Cedar's fraud-agent asks to run a job on the Shared Runner

1What problem?

Autonomous agents from different customers share the same infrastructure. For every request we must prove who the agent is, decide what it may touch and by which route, reserve finite shared capacity without ever overspending, and let the destination check that the request really was granted that route.

The whole page uses one small cast. Remember these five names and the three numbers on the picture.

2What model?

Three separate questions, answered by three separate parts. Keeping them apart gives clean failure reasons, and it stops one part from quietly doing another part's job.

Authorized is necessary, not sufficient. A request can be allowed and still not fit. And nothing adaptive may ever turn "not allowed" into "allowed": feedback may only change preference among routes that policy already permits.

System map: what runs, and what is only designed

3How does a request work? BUILT · RUNNING

Follow the state card on the right. It shows what the system knows about the request after each step, and highlights what just changed.

Release, and what can go wrong

To give capacity back, the agent signs a fresh assertion and authn forwards release. Only the owner may release: another agent gets NOT_OWNER and the lease stays live. If nobody releases, the lease expires on its own; idle partitions sweep expired leases every second. Retrying the same request id returns the original answer instead of a second lease.

The eight recorded scenarios

A driver runs these against the deployed services. They use the read request from the cast: Cedar fraud-agent reads transaction-features through Cedar's gateway (two links, capacity 6 each). The runner request above takes the same code path but is not one of the eight.

All eight pass on a local three-node Minikube cluster and on a one-node managed cluster, including with the agent workload deployed (§9).

4Why graphs?

Each "may this identity do this action on this resource" rule decides whether an edge exists. Routes are then simply paths through the graph, and cross-tenant edges are absent by construction. That is the running part. Policy is compiled into a bitmap index, so one decision is a few AND operations. An independent "scan" evaluator that shares no code with the index must agree with it on every test case. Every allow must carry the owner-scope condition, and a deny that can never match is a compile error rather than a silent gap.

Max-flow: how much can run at once? BUILT · IN-PROCESS

Treat capacities as pipe widths. Max-flow asks: if every agent pushes as hard as it can, how many jobs reach the Shared Runner at the same time?

The policy gap: the real canonical result

Run max-flow twice, once on the physical network and once on only what policy allows, and the difference shows whether hardware or policy is the bottleneck. In the canonical fixture, Cedar's fraud-agent wants to read transaction-features at 4 units at once:

measured on canonical Computed by Dinic's algorithm, confirmed by an independent Edmonds–Karp oracle and by an exact assignment search (also 2). A quota-aware variant shows tenant shares binding: three runner requests of 2 units each (Cedar, Northstar and a third customer) fit 4 without quotas, but only 3 with each customer's declared share of the runner.

Residual graphs: permission to undo BUILT · IN-PROCESS

A residual graph marks how much room is left on every link. It also adds a reverse edge for every unit already sent. A reverse edge is not traffic flowing backwards. It is permission to undo an earlier allocation so that the total can grow.

This rearranging happens inside the max-flow and assignment algorithms. The running system never moves a granted lease. Moving capacity between partitions is a separate, designed-only idea (§6).

5Why a ledger? BUILT · RUNNING

Planners may suggest; only the ledger commits. A request is a demand vector: one entry for every limited thing on its route. The vector either fits completely and is reserved in one step, or nothing is reserved.

The ledger takes the current time as an argument, so every expiry boundary can be tested without sleeping, and it returns a named reason for every outcome. In the deployed system, S3 fills a route until it is full: the 7th admit is FULL and no link is ever above capacity. S4 shows that a release frees capacity immediately. S8 shows that only the owner can release.

The grant never outlives its lease. Its expiry is capped below the lease's (§3, step 5). One gap is disclosed: there is no revoke. A released lease frees capacity immediately, but a grant issued for it stays valid until it expires, at most 60 seconds later.

6Why partitions? BUILT · RUNNING

One ledger is one point of contention. So capacity is split across two partitions, and each partition owns a disjoint share of every limit outright. No two processes can ever both believe they own the same unit, so "never overspend" stays a simple, local property of each partition.

• Choice: "power of two choices". With two partitions the manager compares both partitions' last published spare room. Ties are broken by a hash of the request key, so the same request always lands in the same place.
• One fallback: on a definite FULL the manager tries the other partition once. That is why S3 admits all 6 units on a capacity-6 route (3 + 3) before the first FULL.
• Retries: re-sending the same request key returns the original answer from the partition that holds it, never a second lease.

The price of ownership is fragmentation. A route can fit globally yet fit in neither partition, for example if one link has room only in partition 0 and the next only in partition 1. The system does not measure how often this happens yet. Scaling is also unmeasured: the partition claim is functional only PARTIAL. Moving capacity between partitions is designed, not built. Fencing a restarted or duplicate manager is built (§10).

7Why fairness? DESIGNED · DRAFT

Policy decides who may compete. Capacity decides what fits. Neither decides who goes next when Cedar and Northstar both have work waiting. First-come-first-served lets a flood from one tenant crowd out the other. Deficit round robin (DRR) gives each tenant a fixed allowance of credit per round, and an expensive job spends more of it.

Allowance 2 per round.  Cedar's job costs 3, Northstar's costs 1.
Round 1:  Cedar   0 + 2 = 2   (3 > 2, waits)      Northstar 0 + 2 = 2   (1 ≤ 2, runs; 1 left)
Round 2:  Cedar   2 + 2 = 4   (3 ≤ 4, runs; 1 left)

teaching numbers The running system has no queue: every lease is 1 unit per link, and a request either fits now or gets FULL.

Credit is a scheduling budget, not capacity. A tenant can have plenty of credit and still nowhere safe to run; capacity always belongs to the ledger. The design (per-tenant queues and batched admission inside a partition) is a reviewed draft with open findings. Nothing is built.

8Why PCA? BUILT · IN-PROCESS DESIGNED · DRAFT

Suppose the Fast Relay starts to behave differently: latency and errors begin to move together in a new way. No single number may cross a threshold, yet the shape of the measurements has changed. PCA finds the direction in which several measurements vary together. Drift shows up as that direction rotating.

Built (in-process): the maths. Timestamped samples become fixed windows, are scaled, and give a covariance matrix whose eigenvectors (Jacobi's method) are the directions. Windows are compared by the angles between their main directions. Every degenerate input returns a named status rather than a number: too few samples, a constant feature, near-tied eigenvalues, or a solver that did not converge. Results are checked against hand fixtures, power iteration, planted spectra and an independent distance computation.

Not built: the feedback. There is no telemetry pipeline. Deciding whether a drift is significant, turning it into a bounded penalty, and letting that penalty re-rank the legal routes are all designs. Whether PCA detects real drift usefully is untested. And by design, feedback may only change which allowed route is preferred. It can never grant a permission or create capacity.

9What is live today? BUILT · RUNNING

Seven core processes run in one Kubernetes namespace: authn, policy, the manager, partition-0 and partition-1, the sidecar and the read-only ui (this site). They exchange one-line JSON messages over TCP and share no code. Together they used about 230 MiB of memory. The same stack runs on a local three-node Minikube cluster and on a one-node managed cluster.

The agents and the data plane

A continuous workload adds a dataset-store and seven agents, for 15 pods in total. Each agent loops: sign, get a grant, download one dataset from the store, train a small model, release the lease, log one line. The store asks the sidecar to check every grant before it sends a single byte.

The six allowed agents share one identity and one route with 6 units per link (3 per partition), so they compete for the same capacity. The seventh agent's tenant does allow that read, but an environment-wide deny for dev overrides the allow. Measured results are on the Journey page.

Network policy and the public UI

• Default-deny network policy. Each pod may receive connections only from named peers: only authn and the ui may reach the manager, only the manager may reach the partitions, and only the six allowed agents may reach the store. It is enforced on the managed cluster (11 of 11 probes behaved as expected). The local cluster's network plugin ignores network policies.
• Public UI. This read-only site is also published over TLS only, through one cloud load balancer with a connection throttle. It serves synthetic data only. Every other service is internal.
• Logs. Every process writes one JSON line per event with a trace id, so one request can be followed across processes. Keys and token values are never logged.

Claims hold for clients that use the front door. Traffic between services is plaintext and mutual TLS is deferred. The network policy narrows who can try, on the managed cluster only.

10How is it deployed? BUILT · RUNNING

This chapter follows the running system from the outside in: which processes exist and what each one owns, which Kubernetes objects hold them, who may talk to whom, and how a request and its data move through. Everything here was read from the renderer's output, the code and the live namespace (read-only), not from memory.

The components and what each owns

The calls only go one way: agents → authn → policy, then manager → partitions; and for data, agent → dataset-store → sidecar. The ui reads from the manager, policy and the store's stats port. Nothing calls back into an agent. The members share no code: every hop is one-line JSON over TCP (NDJSON), or HTTP for the store and the ui.

Partition-owned leases, with epoch fencing BUILT · RUNNING

A lease lives only in the partition that granted it. The manager never holds units, so a lost manager loses no capacity. To stop two managers from both writing (for example an old pod stuck while a new one starts), every partition has exactly one owner at a time:

• Owner pair. The owner is the pair (epoch, manager_id). The id is 16 random hex characters chosen at process start; the epoch is a number that only grows.
• Takeover. A starting manager reads every partition, picks an epoch above any it saw, and configures each partition with its own pair. Leases and replay records survive the takeover. Then it re-reads all partitions; only when every one shows its pair does it start serving.
• STALE_EPOCH. Every admit and lookup carries the pair. A partition refuses any other pair with STALE_EPOCH and writes nothing. release is not fenced: it can only free units, and it is already checked against the lease owner.
• Superseded is final. A manager that sees a greater pair than its own stops serving for good and reports not ready. Only a process restart brings it back, as a new manager with a new id.

Spec docs/superpowers/specs/2026-10-02-partition-epoch-fencing.md; code poc/atomic-lease/partition.py (configure, admit, lookup) and manager_server.py (_on_foreign, _poll_round, skip_lookup). The code in the cluster's ConfigMaps was compared with the working tree on 2026-10-02 and matched file for file.

What the renderer creates

One script, scripts/poc/render.py lke, writes every object (40 of them) as JSON files. It refuses to write anything if its own checks fail (see the gates below). Then scripts/poc/deploy.sh applies them.

Services and ports

Pods: requests and limits

Without the Job the set needs 530m CPU and 736Mi memory in requests, and 1888Mi in memory limits (the denied agent's 64Mi init container counts there). Source: the renderer's preflight_ok line.

Secrets

No Secret is ever rendered or kept in the repository. deploy.sh creates four from files outside the repository, through a pipe so no value is printed. The renderer's SECRET_OWNERS table fixes which workloads may mount each one, and the gate refuses any other reference.

• authn-keys: the issuer signing key and the agents' public keys. Only authn.
• agent-fraud-detector-key: the allowed agent's private key. Only the six allowed agents and the loadgen Job.
• agent-route-optimizer-key: the denied agent's private key. Only agent-denied-fleet.
• poc-tls: the public site's certificate and key. Only ui.

All keys except the TLS certificate are synthetic test keys made by scripts/poc/keygen.py.

The same safety settings on every pod

• Runs as a non-root user (uid 10001), with no privilege escalation.
• Read-only root filesystem. Code and dependencies are written once by init containers, then mounted read-only.
• All Linux capabilities dropped. The seccomp profile is RuntimeDefault.
• No service-account token is mounted, and service links are off. No pod holds credentials for the Kubernetes API.
• Only three volume kinds: ConfigMap, Secret and size-limited scratch (emptyDir). No persistent volumes.
• authn, sidecar and the agents need PyJWT. An init container installs it with pip --require-hashes from a hash-locked list, binary wheels only, into a read-only /deps mount.

What the gates refuse

render.py runs these checks before it writes, and deploy.sh runs them again (render.py --check) on the files on disk:

• More than one LoadBalancer Service; any NodePort Service or nodePort field; external IPs; source ranges anywhere but the public ui Service; a NodePort quota that differs from the LoadBalancer count.
• Any volume claim, persistent volume or StorageClass; any object kind outside Namespace, ResourceQuota, ConfigMap, Service, NetworkPolicy, Deployment and Job (so no Secrets, RBAC or Ingress in the set).
• A replica count above 1; a Job with parallelism above 1, retries, no deadline, or a deadline or cleanup time above two hours.
• Any pod that breaks the safety settings above, uses another image, or sets a host port.
• A Secret mounted by a workload not in SECRET_OWNERS; the NetworkPolicy labels on any workload other than their intended ones.
• A NetworkPolicy set that is not exactly the expected one, or that lacks the default deny; a set whose requests exceed the quota; a ConfigMap of 900 KiB or more.

The network

The gate also checks that the two agent labels sit only on the workloads meant to carry them, so a new pod cannot give itself store access by copying a label.

The edge of the cluster

• One load balancer. The ui-public Service makes the cloud create one NodeBalancer. It passes TCP 443 straight to the ui pod's port 8443; TLS ends in the ui process, using the site certificate. A connection throttle of 20 is set by annotation.
• Node firewall. A Cloud Firewall sits in front of the node: inbound traffic is dropped by default. It admits only the cluster's own control traffic from the provider's private ranges, and NodePorts only from the NodeBalancer's backend range, so the load balancer's NodePort is no longer a public side door. No public SSH. Outbound is open. Akamai's Cloud Firewall Controller keeps it attached, and scripts/poc/firewall_reconcile.sh re-checks it after every deploy.

The firewall rules are taken from the runbook (docs/operations/lke-agents.md); no probe of the node's public NodePort is recorded since the firewall was added.

The cluster

No persistent volumes, no autoscaling, no paid add-ons. A local three-node Minikube cluster runs the same manifests without the load balancer; its network plugin does not enforce NetworkPolicy.

How data moves

Grant and download, one iteration

1. Ask. The agent signs a fresh RS256 assertion (30-second life, unique id) and sends it with a new request id, the target and the action.
2. Who. authn checks the signature with that agent's public key, the lifetime (at most 60 seconds, 5 seconds leeway) and the claims. Its anti-replay cache remembers each (agent, assertion id) until the assertion expires; a repeat is REPLAYED. When the cache is full it refuses rather than forgets.
3. May. policy decides from the agent's own catalog row and returns up to 8 routes.
4. Route. The manager sends a key it remembers straight to its partition. A new key is first looked up on both partitions, unless the lookup can be skipped: the manager holds every partition, its key memory has never dropped an entry, and no partition holds records from an earlier owner. Then it tries the partition that power-of-two-choices picks: two candidates from the key's hash, the one whose routes have more spare units first.
5. Fit. The partition checks the owner pair, then reserves 1 unit on each capacity-carrying link of the first route that fits, all or nothing. If it is FULL (or restarted and unconfigured), the manager tries the other partition once. Any other answer goes straight back.
6. Grant. authn signs a path grant naming the lease, route and hops; it expires before the lease does.
7. Download. The agent sends the grant to the dataset-store. The store asks the sidecar over a reused connection, passing its own configured ingress chain as the hops, never the client's claim. Only on OK does it stream the file, and only a file whose size and sha256 matched the manifest. The kernel copies it (sendfile), with a plain read loop as fallback.
8. Release. After training, the agent signs a new assertion and releases the lease through authn and the manager. If nobody releases, the lease expires by itself (5 to 30 seconds).

The ui snapshot RUNNING

When a browser asks for /api/snapshot, the ui refreshes at most once a second: it asks the manager for its snapshot (which includes every partition's), policy for the catalog, and the store's port 8091 for /stats. Each call has a 0.8-second budget. If the store or policy fails, the view still comes back, with that part marked. If the manager fails, the last view is served, marked cached, for up to 5 seconds, then the answer is an error. The view goes to the browser over TLS, through the load balancer.

Seeding the store BUILT

The store downloads nothing. An operator runs scripts/poc/seed_datasets.sh on a laptop: it checks the local files against their manifest, copies each into the store pod with kubectl cp under a temporary name, then renames them all into place in one step, manifest last. The store's catalog re-reads every 5 seconds and serves a file only after its own size and hash check, so it never sees half a file. The script waits until the store reports this exact manifest. The data sits in a scratch volume and is lost when the pod restarts.

Logs and evidence BUILT

Every process writes one JSON line per event to standard output, with a trace id that follows a request across processes. Key and token values are never logged. After a run, the driver saves each pod's log, and scripts/poc/collect_logs.py merges them into one time-sorted logs.jsonl (non-JSON lines go to a side file). Results stay in a local, untracked poc-results/ folder.

11How is the code built? BUILT · RUNNING

Inside each process: which modules import which, the main classes and what each one guarantees, how main() wires them up, the data each process keeps and its limits, and the order in which each request is checked. Every fact cites its file and function. The code in the cluster matched the working tree file for file on 2026-10-02.

Rules every member follows

• Layers. Imports point one way: entry points and adapters → application → domain logic → domain data. Domain logic does no I/O and is given the time as an argument; sockets, clocks and environment variables live in the *_server.py entry points.
• No shared code. No member imports another. Processes talk one-line JSON over TCP (NDJSON) or HTTP, so each member has its own small NDJSON module.
• Standard library only, except where named: poc/authn-sidecar uses PyJWT with cryptography (pinned and hash-locked, a user-approved exception), and the agent pods borrow that same install for signing.

12What remains open?

Living document, checked against the code, the canonical fixture, the specs and the recorded runs. Read the Journey for how it was built and what changed on the way, and use the Simulator to explore the full fixture.