Deployment

How Agentry is deployed in its four shapes, and the operational targets. Production runs on Vultr: the site's umbrella chart on Vultr Kubernetes Engine, applied by terraform/vultr/. The AWS ECS design in terraform/aws/ is kept, validated and never applied, as the reference. Infra-as-code lives in the site repository's terraform/, the site's umbrella chart deploy/helm/secureagentry/, and the core's deploy/helm/agentry/ (self-host HA). For self-hosting, see the self-hosting guide.

What only compose carries

The compose stack and the production umbrella chart carry the same pieces: the directory service (authentik), the site's edge, the hosted downloads and the log stack (agentry-logging's own chart, a release of its own). Only the Cloudflare Tunnel connector is compose's alone. It is a compose deployment's way in, and production is reached through a load balancer instead. The core's chart deploys the registry alone; the umbrella adds the rest.

The four deployment shapes

Shape	What runs	Where it's defined
Dev / small self-host	agentryd all (every role in-process, including the queue worker) + Postgres; Redis/S3 optional	deploy/compose/
Self-host HA	Helm: api ×N, git ×N (shared RWX volume), worker ×N; Postgres + object store operator-supplied; a site in front is its own deployment	deploy/helm/agentry/ (core)
Production (Vultr)	The umbrella chart on VKE: api ×2, git ×1, worker ×1 (the enterprise build), web, docs, developers, the edge behind a PROXY-protocol balancer, the directory; the log stack and Velero as releases of their own; managed Postgres, Object Storage, a VFS claim for git	terraform/vultr/ + deploy/helm/secureagentry/ (site repository)
AWS reference (never applied)	ECS Fargate per role, RDS, EFS, S3, Redis, Bedrock judge	terraform/aws/ (site repository)

Same binary throughout; the shape only changes how the roles are split and what backs Postgres / object storage.

agentryd all runs the queue worker in-process, so webhook deliveries, audit exports and queue-driven scans are claimed on a single-process install. Do not also run a separate agentryd worker against the same database in this shape; it is harmless (River's claim is transactional) but doubles the poll traffic for no gain.

Container images

One image per repository per release, each built and signed by that repository's .github/workflows/release.yaml:

Image	Dockerfile	Contents
agentryd	core deploy/compose/Dockerfile	The server — the api, git and worker roles, and all — on debian-slim with git
web	site deploy/Dockerfile.web	The Next.js app, production deps only

One agentryd image rather than a separate git image: the git role needs a git binary, and so does all, which mounts the git routes in-process. Splitting would have fixed one and left the other broken, and would mean two tags per commit to keep in step. The base is debian-slim rather than distroless because git needs a real userland.

Production on Vultr (as built)

Cloudflare (DNS, WAF, Authenticated Origin Pulls)
  └─ Vultr load balancer (TCP passthrough, PROXY protocol; 80/443 from Cloudflare only, 22 open)
       └─ edge ×2 (the site's nginx; Origin CA certificate) ─┬─ web ×2 · docs · developers
                                                             ├─ agentry api ×2 (HPA to 4) · git ×1 (VFS claim) · worker ×1
                                                             └─ authentik server · worker (auth.)
   VKE, one vpc_only pool of 3 vhp-2c-4gb nodes (autoscaler to 5), egress through the VPC's NAT gateway
   Managed Postgres (Startup, PG 17, verify-full) · Object Storage `app` (ewr1) · backups on `atl2`
   Releases of their own: agentry-logging (the log stack), velero

What each piece is

• Terraform (terraform/vultr/envs/prod) owns the cluster, the network, the database, the object stores, the VFS claims and every Secret. OpenTofu keeps its state encrypted client-side on the ops subscription, and a copy is kept in the second region after each apply (terraform/README.md).
• The umbrella chart (deploy/helm/secureagentry, values values-vultr-prod.yaml) carries the core chart as a dependency, the site's services, the edge, the directory, the downloads sync (a CronJob that puts each app's newest published vX.Y.Z on the download page through the verified import, reading the releases with terraform's agentry-downloads-read) and the backup CronJobs.
• The log stack is agentry-logging's chart, release agentry-logging, with this repository's deploy/helm/agentry-logging/values-vultr-prod.yaml.
• The images are built from each repository's main by the deploy workflow, tagged by commit (secureagentryd:sha-e<enterprise>-c<core>-l<cli>, web:sha-<web>, downloads-import:sha-<web>, and the docs images by their content refs), pushed to ghcr.io/blaktron/agentry-web/production/… and signed keylessly by the workflow that built them. The cluster pulls with terraform's read-only agentry-ghcr-pull.

Deploying (.github/workflows/deploy.yaml)

Production is main, and it is rebuilt only by a push to this repository's main (or by hand: Actions → deploy → Run workflow, on main, which redeploys what main pins). The other five repositories production is built from (agentry, agentry-enterprise, agentry-cli, agentry-docs, agentry-logging) are pinned in deploy/production-versions.yaml, one full commit SHA on each repository's main. The deploy builds exactly those commits and this repository's pushed one; a merge to another repository's main deploys nothing by itself. The commits a revision runs are its description in helm history agentry -n agentry. One run:

1. preflight reads the pinned commits and refuses one that is not a full SHA or not on its repository's main. It stops, green, when the run is not on main or the secrets are not set.
2. infra runs scripts/deploy/tofu-apply.sh with the scoped agentry-deploy key. It refuses to plan unless every operator-held input is set (an empty one would empty its Secret), and applies only a plan of creates and in-place updates: one that deletes, replaces or forgets anything is applied by hand (below). The refusal is by action, so an in-place update that is disruptive in effect (a node pool's size, the database's plan) is applied like any merged change. Then scripts/dr/state-copy.sh copies the state to the second region. The job is skipped while its secrets are missing.
3. site-images, server-image and docs-images build, push and sign the images, the docs and developers sites from agentry-docs at its main tip with the build arguments of its own images.yaml. A tag already pushed and signed is reused; one pushed but never signed is built again.
4. deploy runs scripts/deploy/verify-images.sh over the six images it built and stops if any is not pinned by digest, or not signed by deploy.yaml in a run of this repository on main. The images the charts pin themselves (nginx, authentik, rclone, alpine/git, the log stack's) are their publishers' and are not checked. It then fetches the kubeconfig for the run (kubeconfig.sh) and upgrades agentry, then agentry-logging (helm-upgrade.sh, --rollback-on-failure: a rollout that does not become ready is rolled back and the run fails). The agentry upgrade, and its rollback, run inside the rollout guard (below, "A rollout waiting on a volume attach"). After that it empties the edge's page cache in place and runs scripts/smoke.ts against https://secureagentry.net. The kubeconfig is deleted before the smoke's npm ci (run with --ignore-scripts), whatever happened. The run's summary names the commits and the two revisions.

The workflow needs the secrets AGENTRY_BUILD_TOKEN and VULTR_DEPLOY_API_KEY, and the infrastructure inputs listed at the top of deploy.yaml. The workflow names no other repository's workflow: GitHub resolves a called workflow when it creates a run, so a call into a private repository that does not allow it fails every run before any job starts.

By hand. When the workflow cannot run (its secrets are missing, GitHub is down, or a plan deletes something), the same scripts run from a machine whose address the Vultr API admits, from an export of main:

1. tofu apply from main's terraform/ (terraform/README.md), then scripts/dr/state-copy.sh. A destructive plan is read line by line before it is applied.
2. The images: built and pushed by the workflow if it can run, otherwise from git archive exports of each main with a push token, to ghcr.io/blaktron/agentry-web/web, …/agentry-enterprise/secureagentryd and …/agentry-docs/docs, never under production/. They are not signed, so the next workflow run builds its own and replaces them.
3. KUBECONFIG=… scripts/deploy/kubeconfig.sh, then DRY_RUN=1 scripts/deploy/helm-upgrade.sh (a server-side render against the live cluster that changes nothing; it lists the images each release would run), then the same without DRY_RUN. SERVER_IMAGE, WEB_IMAGE, IMPORT_IMAGE, DOCS_IMAGE, DEVELOPERS_IMAGE and EGRESS_PROXY_IMAGE (the runner image, for the worker's egress proxy) are repository:tag@sha256:…; SITE_DIR and LOGGING_DIR are the exports; DESCRIPTION names the commits.
4. AGENTRY_BASE_URL=https://secureagentry.net node scripts/smoke.ts.

A rollout waiting on a volume attach

Production's two ReadWriteMany volumes, agentry-git-data (git and the worker) and agentry-downloads (web and the edge), are Vultr VFS. Vultr's CSI attaches one to a node only when a pod there first needs it, and that attach often does not finish: the managed CSI's attacher gives up at its 15-second default, and Vultr's VFS API is slow to answer. Without precautions, a rollout's new pod that lands on a node without its volume waits past its Deployment's 10-minute progress deadline, and helm rolls back. The rollback then waits on the same attach.

Two things keep a rollout off that path, and neither pins a pod to a node:

• The VFS anchor (deploy/vultr/vfs-anchor.yaml), a DaemonSet that keeps both volumes mounted, read-only, on every node. A rollout's pods find their volumes already attached wherever they land. It runs a pause container and nothing else. rollout-guard.sh anchor applies it with kubectl before each upgrade, and helm does not manage it: helm's wait would otherwise wait on an anchor that cannot attach. An anchor that is not Ready means its node does not hold the volumes: kubectl -n agentry get pods -l app.kubernetes.io/name=agentry-vfs-anchor -o wide.
• The rollout guard (scripts/deploy/rollout-guard.sh run -- <command>), around the upgrade and the api/worker restart:
• first, it uncordons any node an earlier run left marked as its own (the node annotation secureagentry.net/cordoned-by), which is what a run the runner killed outright leaves behind;
• before the command, it cordons every node whose anchor has not been Ready for two minutes;
• while the command runs, it deletes a Deployment's pod that is still Pending on an attach that has failed for two minutes and failed again recently, and cordons that pod's node so the ReplicaSet's replacement can land where the volume is attached. It never deletes a Job's, a StatefulSet's or the anchor's pod, and a Multi-Attach refusal is not counted as a stuck attach;
• it never leaves fewer than two schedulable nodes, never cordons when it cannot read the nodes, and never touches a node someone else cordoned;
• when the command ends, whatever its status, it uncordons every node marked as its own. Its exit status is the command's.

Each thing it does is a rollout-guard: line in the run's log. A cordon keeps the node's running pods, and the anchor there keeps attaching. A node whose anchor never becomes Ready is one to look at with Vultr, or to replace.

scripts/deploy/rollout-guard-selftest.sh checks each of these promises against a fake kubectl, and CI runs it. A rollback by hand gets the same cover: scripts/deploy/rollout-guard.sh run -- helm rollback agentry <revision> -n agentry --wait --timeout 20m.

The attach timeout is ours, not VKE's. VKE's csi-vultr-controller StatefulSet (kube-system) runs a csi-attacher sidecar whose own default --timeout is 15 seconds. Vultr's attach API takes longer than that under load, and from 2026-09-28 to 30 every VFS attach stalled until the argument was raised to --timeout=90s by hand; with it the stuck attaches published in 21 to 32 seconds. VKE manages that StatefulSet, so any VKE operation that re-applies Vultr's manifest would bring the 15-second default back without a word. Every deploy therefore runs scripts/deploy/csi-attacher-timeout.sh before it rolls anything: it logs already --timeout=90s, or patches the argument back (a JSON patch that first tests the container's name, so a reordered manifest fails rather than patching the wrong container), waits for the rollout, and logs patched from <old>. A StatefulSet or container it cannot find, more than one --timeout, or an update strategy that would not roll the patch stops the deploy on purpose: the fix is to the script, not a skip. csi-attacher-timeout-selftest.sh runs in CI.

Production's Critical alerts

Five conditions email security@secureagentry.net. There is no paging and no acknowledgement: each is an email, sent at most once an hour while its condition holds.

Alert	Rule	Sent by
a fail-open event	any of the six failopen.* events the log contract lists, in the last 10 minutes (a new name alerts once the monitor lists it)	the log stack's monitor, through Postmark
the log collector silent	no line in agentry-logs-* for 10 minutes	the log stack's monitor, through Postmark
a production deploy failed	any job of a deploy run failed; a run cancelled by hand or skipped by its preflight sends nothing	deploy.yaml's last job, alert (scripts/deploy/alert-failed.sh), through Postmark's API with SMTP_PASSWORD
the database or the object store near full	the database's disk at 80% or more (Vultr's usage API: both databases and the WAL), or 800 GB or more across every version in the app object store (80% of the 1000 GB the Standard tier includes; it bills above that, with no hard cap); checked hourly	watch.yaml (scripts/watch/capacity.py), through Postmark's API with SMTP_PASSWORD
production unreachable from outside	any of six URLs (the site, the registry API's /api/v1/meta, sign-in's /-/health/live/, docs, developers, and www's 301) answers anything but its expected status, checked every 3 minutes from the US and Europe	Better Stack's uptime monitors, from outside the cluster and outside Postmark

The two log-stack monitors are agentry-logging's (opensearch/monitors/), applied by the store-init Job of every agentry-logging upgrade with this repository's alerting values. The store reads the relay's sign-in from its keystore, which the chart fills from terraform's agentry-log-alerts-relay Secret (Postmark's server token as user and password) when the store starts; a rotated token therefore needs the store restarted (kubectl -n agentry-logging rollout restart statefulset/agentry-logs-store). Each email names the events and counts, never a line's content; the lines are in Dashboards. To see the monitors and their last runs, port-forward the store and read _plugins/_alerting/monitors/_search as admin.

The deploy-failed email is sent from the workflow itself, not the cluster, so it still goes out when the cluster cannot be reached. It names the failed jobs and links the run, and each failed run sends one. It cannot tell how far the run got: a failure in deploy after the helm upgrades (a restart, the edge purge, the smoke test) leaves the new release live, so read the failed job's log before deciding to roll back. It is not sent when no runner picks the run up at all (the self-hosted runners down): the jobs queue rather than fail.

The near-full check prints both readings in every hourly run of watch, whether or not it alerts, and emails once per reading over its threshold on each run; a reading it cannot take (an API error, an empty listing) is emailed as capacity check failed and fails the run. The thresholds are the two fallbacks in watch.yaml's step env (|| '80', || '800'); prove the path with a dispatch that lowers them (gh workflow run watch.yaml -R blaktron/agentry-web --ref main -f db_threshold=1 -f store_threshold_gb=0), which sends both emails. Its subscription keys come from the Vultr API with the deploy key, masked before use, and are never stored.

The outside watch and the public status page, status.secureagentry.net, are one Better Stack account's (the operator's; free plan: 3-minute checks, ten monitors, one status page). deploy/status/betterstack.json declares the monitors and the page, and scripts/status/betterstack-apply.py makes the account match it (a monitor is adopted by its URL; one the file drops is reported, not deleted), with the team's Uptime API token in BETTERSTACK_TOKEN. No one is on call, so every incident emails the whole Better Stack team: whoever should receive it, security@ first, has to be a member. The page's custom domain is a DNS-only CNAME, status → statuspage.betteruptime.com, made by hand in Cloudflare; the site's footer links it (AGENTRY_STATUS_URL). Nothing is sent to Better Stack but these checks: no logs.

To prove the deploy-failed email without breaking a deploy, dispatch deploy on main with alert_probe set: the preflight fails on purpose before anything is built, and the email follows (gh workflow run deploy.yaml -R blaktron/agentry-web --ref main -f alert_probe=true). Dispatch it only when no deploy is running or waiting (gh run list -R blaktron/agentry-web --workflow deploy.yaml -L 3): the deploy-production group keeps one pending run, so a probe queued behind a running deploy replaces a pushed deploy that was waiting, and that commit is then not deployed until the next push or dispatch.

Rolling back

1. helm history agentry -n agentry lists the revisions. Each description names the commits the revision ran.
2. scripts/deploy/rollout-guard.sh run -- helm rollback agentry <revision> -n agentry --wait --timeout 20m returns to that revision's manifest and images, with the rollout guard around it (above). Use the same with agentry-logging for the log stack.
3. Empty the edge's cache, as the workflow does: kubectl -n agentry exec <each edge pod> -c edge -- sh -c 'rm -rf /var/cache/nginx/edge/*'. Then run the smoke.

A rollback restores what the pods run. It does not undo a migration, because the core's migrations are forward-only, and it does not undo a terraform apply. The next push to this repository's main deploys what the version file pins again, so a rollback that must hold is followed by a PR that pins the component back (or reverts the site's change) before anything else is promoted.

Releasing a component to production

A change in agentry, agentry-enterprise, agentry-cli, agentry-docs or agentry-logging reaches production in two steps after its own promotion to that repository's main:

1. scripts/deploy/bump.sh <component> pins that repository's main tip (or a given commit on it) in deploy/production-versions.yaml and opens a PR to this repository's dev branch. The commit body lists what the bump releases.
2. The PR is gated and merged to dev, then promoted dev → main. The promotion's push is the deploy.

Several components can ride in one promotion, and a site change rides with them. Nothing else starts a production deploy: there is no timer and no dispatch from the other repositories.

Backups and the restore drill are below, in "Production on Vultr: backups and the restore drill".

Limits of the production deployment

• The deploy verifies the signatures of the images it built before helm runs. Nothing in the cluster checks them again (there is no admission policy).
• The deploy's jobs run on the shared self-hosted runners, beside every repository's CI and on the same Docker daemon. A signature proves which workflow built an image, not what else ran on that runner.
• The keyless signatures are recorded in the public Rekor log, which names the repositories, the workflow paths and the commits (not their content).
• The core chart deployed is the one vendored in deploy/helm/secureagentry (charts/agentry-*.tgz), not the core's main: a chart change on the core's main reaches production when this repository bumps the vendored version.

Hosted runs' configuration

Production's runs extension needs two things, and terraform writes both as Secrets in agentry, together or not at all (modules/secrets/runs.tf):

• agentry-runs: the runs CA's master key and bundle. They are kept as the GitHub secrets RUNS_CA_MASTER_KEY and RUNS_CA_BUNDLE on this repository, as the Origin CA key is. The deploy workflow's apply passes them as TF_VAR_runs_ca_*. A by-hand apply passes the same two variables.
• agentry-runs-store: the runs store's endpoint, buckets and key, read from the envs/prod-runs root's outputs through its state (same bucket and passphrase). They are never copied by hand.

secureagentryd stops at start-up with a runs store and no CA, so the store's Secret exists only when the CA's does. The api and worker pods reference both as optional: until the CA exists, the pods run and hosted runs stay off. The api serves hosts' enrolment. The worker runs the dispatcher, and its egress admits the runs host's mTLS endpoint (envs/prod-runs's hosts output).

Setting the two GitHub secrets starts nothing by itself. Set both (one alone fails the apply's validation and holds back every deploy), then run the deploy (Actions → deploy → Run workflow, on main). Its apply creates the Secrets, and the deploy job restarts the api and the worker so they read them. An enrolment token for a host names its control address, the host's mTLS endpoint behind the runs gateway (envs/prod-runs' hosts output). The worker's egress admits that address.

Once the Secrets exist, removing a CA secret plans their deletion, which the deploy's apply refuses. A lost GitHub secret stops the deploy; it never empties production's runs configuration.

Where the sandbox hosts run

Production's residency is one line: the United States, on Vultr, thirty days, encrypted to the user. The sandbox hosts and the runs buckets are in the registry's Vultr region, New Jersey. The hosts' model is Vultr Serverless Inference. A run and its output are deleted thirty days after the run was created, and every output is encrypted to the person who ran it. A region per organization is not offered. The user-facing statement is on hosted runs.

The hosts sit in a network of their own that shares no route with the registry's. A host has no public address. runhost's port is forwarded to it through that network's gateway, from the registry's egress address alone. The design also allows a break-glass SSH forward from the operator's address, which no host has today. Outbound, a firewall of the host's own admits DNS to the host's resolvers and TCP 80 and 443 to public addresses, and refuses private ranges, CGNAT, link-local and loopback addresses. The containers are also refused the cloud's metadata address; the host itself may read it at boot, and may reach NTP. The same shape for a deployment of your own, with the rules a self-hosted sandbox host needs, is in the self-hosting guide's sandbox hosts for hosted runs.

The runs fleet's settings

The registry's fleet makes, drains and destroys the sandbox hosts from the Staff panel. A deployment without the fleet makes its hosts by hand. Only the enterprise build (secureagentryd) reads these settings; the core does not. The fleet's key and image settings belong only to the process that makes hosts, the worker role.

Variable	Default	What
AGENTRY_RUNS_FLEET_PROVIDER	none	vultr; fake, for a deployment with no runs network to make hosts in; or unset, for hosts made by hand
AGENTRY_RUNS_FLEET_CONFIG	none	the runs network, as JSON, as the runs terraform root outputs it
AGENTRY_RUNS_FLEET_ORIGIN	none	Agentry's public https:// origin, where a new host enrols
AGENTRY_RUNS_FLEET_HOST_DOMAIN	none	the DNS suffix of the hosts' names
AGENTRY_RUNS_FLEET_DEADLINE	45m	how long a new host has from its request to its enrolment before it is destroyed and reported; at least 10m
AGENTRY_RUNS_FLEET_VULTR_API_KEY, AGENTRY_RUNS_FLEET_RUNHOST_IMAGE, AGENTRY_RUNS_FLEET_GHCR_USER, AGENTRY_RUNS_FLEET_GHCR_TOKEN	none	on vultr, all or none, and only on the process that makes hosts (the worker role): the fleet's Vultr key, the runhost image a new host runs (by digest), and the credential it pulls that image with, once, at boot

The scanning worker's egress

The worker opens untrusted archives, so its pods are default-deny egress (the core chart's agentry-worker-egress policy). Calico's open-source policies can name addresses, not hosts, so what the worker reaches outside the cluster is split two ways:

• By host, through the egress proxy. The AI Review (api.typesafe.ai, S5) and OSV (api.osv.dev, S7). The worker's HTTPS_PROXY is http://agentry-egress-proxy:3128 (agentry.worker.proxy). That is two replicas of the runner image's CONNECT-allowlist proxy (egressProxy in the umbrella chart; the CLI repository's runnerimage/netproxy), allowing tunnels only to the hosts in egressProxy.allow, by exact name. The worker's pods alone may connect to it, and it may reach DNS and 443 on public IPv4 only. Each decision is one allow host:443 or deny host:443 line in its log (kubectl -n agentry logs deploy/agentry-egress-proxy).
• By address, direct. The database, the object store (ewr1.vultrobjects.com), the sandbox hosts' mTLS endpoint and api.vultr.com (the fleet's key answers only from production's NAT address). Each of these is in the worker's NO_PROXY and on its egress list. A changed address there still cuts the worker off from it until the list moves: check getent ahostsv4 <host> before an upgrade.

A new outside service for the worker is a host on egressProxy.allow, never an address on the egress list, unless its client ignores proxy variables. The CI job helm fails a render that pins the CDN addresses, lets anything but the worker reach the proxy, or changes its allowlist without changing that check.

The AI Review and OSV are not pinned by address because they sit behind shared CDN addresses, which go stale when the provider moves the name. The proxy follows the names. It is not narrower than address pins, though. It does not read TLS, so a tunnel to an allowed host can carry another site's name (SNI) to the same CDN, as a pinned address could. The proxy does not check the SNI.

Rotating the registry's directory token

The registry reaches the directory as the service account agentry-registry, with the one API token the umbrella's blueprint declares (agentry-registry-directory-admin, key AGENTRY_IDP_ADMIN_TOKEN). authentik lets every account mint API tokens for itself, and lets an owner delete and re-create its own tokens; 2026.8 has no setting against either. A holder of a leaked key could therefore mint successors, or replace the blueprint's token with one of its own under the same identifier and read the next key written into it. Rotating the key alone cuts off neither. From a machine with production's kubeconfig:

1. The new key. In an export of main's terraform/vultr/envs/prod: tofu apply -replace='module.secrets.random_password.registry_directory_admin_token'. The plan replaces that one password and updates the two Secrets that carry it, agentry-app and authentik-config. Then run scripts/dr/state-copy.sh. The deploy workflow's apply refuses a replace on purpose, so this step is never unattended.
2. The directory worker reads it. Run kubectl -n agentry rollout restart deployment/agentry-authentik-worker, then kubectl -n agentry rollout status deployment/agentry-authentik-worker.
3. The sweep. scripts/authentik-sweep-tokens.sh does five things:
4. It makes the account inactive. authentik then refuses every token the account has and deletes its sessions.
5. It deletes every token the account owns, the blueprint's included.
6. It applies the blueprint, which re-creates the token with the new key. The account did not create that token, so it has no permission to read its key.
7. It checks that key against EXPECTED_KEY.
8. It makes the account active again.

If any step fails, the script stops and leaves the account inactive. Run it in a subshell, so the admin token and the port-forward end with it:

(
  kubectl -n agentry port-forward svc/agentry-authentik-server 9000:80 >/dev/null & pf=$!; sleep 3
  sec() { kubectl -n agentry get secret "$1" -o "jsonpath={.data.$2}" | base64 -d; }
  AUTHENTIK_URL=http://127.0.0.1:9000 \
  AUTHENTIK_ADMIN_TOKEN=$(sec authentik-config AUTHENTIK_BOOTSTRAP_TOKEN) \
  EXPECTED_KEY=$(sec agentry-app AGENTRY_IDP_ADMIN_TOKEN) \
    scripts/authentik-sweep-tokens.sh
  kill $pf
)

The script's other inputs (SERVICE_USER, KEEP_IDENTIFIER, BLUEPRINT) default to production's values. It prints how many tokens it removed. The registry mints none itself, so any beyond the blueprint's are someone else's: treat that as an incident. 4. The registry reads it. Run kubectl -n agentry rollout restart deployment/agentry-api (the one role of the registry that holds it), and wait with rollout status. From step 3 until this finishes, account erasure and device removal fail against the directory, and the erasure retries try again later.

The sweep also runs on its own, with the current key and without steps 1, 2 and 4: after a suspected leak, before deciding whether to rotate, or as a check. The directory's only OAuth provider allows the authorization-code grant alone, so a token the account mints cannot be exchanged for an API-scoped access token.

Remembered browsers on the directory's sign-in

Production offers "Remember this device for 30 days" on the directory's code page. The pieces are:

• The check secret. Terraform's random_password.registry_trust_check_token, written into both agentry-app (the registry reads AGENTRY_IDP_TRUST_CHECK_TOKEN through the chart's extraEnv, optionally) and authentik-config (the directory's skip policy reads the same name from its environment when it runs).
• The page. The flow-page override, files/authentik/templates/if/flow.html in the chart, rendered into a ConfigMap and mounted at /templates/if in the directory's server and worker.
• The claim. The agentry-remember-device scope mapping of the account flows' blueprint (files/authentik/blueprints/agentry-account-flows.yaml), which the site blueprint puts on the registry's provider.
• The path. AGENTRY_REGISTRY_INTERNAL_URL points at the api Service. Two policies of their own, trust-check-egress and trust-check-ingress, let the directory's server (not its worker) reach the api pods (not git, not the worker) on 8080, and nothing else. The edge never routes /internal/, and the registry refuses a request carrying a proxy's forwarding header.

Without the secret on the directory's side, the skip policy asks for the code before any request. Without it on the registry's side, nothing is minted and the check answers 404. Either way the code is asked every time. With it on the registry's side only, records are still minted at sign-in, but nothing honours them.

The page is authentik's own flow page with one block added, and it replaces authentik's page for every flow. It is byte-matched to the authentik release this chart pins (authentik.global.image in values.yaml; the compose overlay pins the same image, and scripts/acceptance/directory-remember.sh checks the page against it). Move the image pin, the page and that check together, one release line at a time (the directory, "Upgrading authentik"): a chart on a newer authentik with an older page could break every flow page.

To rotate the secret, from a machine with production's kubeconfig:

1. In an export of main's terraform/vultr/envs/prod, run tofu apply -replace='module.secrets.random_password.registry_trust_check_token', then scripts/dr/state-copy.sh. The plan replaces the one password and updates agentry-app and authentik-config. The deploy workflow's apply refuses a replace, so this step is never unattended.
2. Both sides read the value only at start. Run kubectl -n agentry rollout restart deployment/agentry-api deployment/agentry-authentik-server deployment/agentry-authentik-worker, and wait for each rollout to finish.

Until both sides have restarted, the directory's check is refused, so every sign-in is asked for the code. That is the fail-closed direction, and it lasts only as long as the rollout.

The console's sources token

Organization sign-in (the directory, "Organization sign-in") needs AGENTRY_IDP_SOURCES_TOKEN on both sides. Terraform's random_password.registry_sources_token is written into authentik-config, where the organization blueprint makes the console's role, service account and token from it, and into agentry-app, from which the chart's extraEnv gives it to the api (the console). The deploy reads the console's role back on every run (scripts/directory-social-apply.sh --cluster): it fails when the worker has the key and the directory has no role or token, or when the role holds a right its file does not grant.

To rotate it, from a machine with production's kubeconfig:

1. In an export of main's terraform/vultr/envs/prod, run tofu apply -replace='module.secrets.random_password.registry_sources_token', then scripts/dr/state-copy.sh. The deploy workflow's apply refuses a replace, so this step is never unattended.
2. Run kubectl -n agentry rollout restart deployment/agentry-api deployment/agentry-authentik-server deployment/agentry-authentik-worker and wait for each rollout.
3. The blueprint's file has not changed, so the directory does not apply it again by itself: run kubectl -n agentry exec deploy/agentry-authentik-worker -c worker -- ak apply_blueprint mounted/cm-agentry-directory-blueprints/agentry-org-sources.yaml.

Until step 3, the console's calls are refused; sign-ins through existing sources are unaffected.

The AWS reference (Terraform, never applied)

Region: single region, us-east-1 by default (set per env in terraform.tfvars). Accounts: prod, nonprod (dev + staging as separate VPCs), security (org CloudTrail + audit-archive replication target). Humans use IAM Identity Center; no IAM users. GitHub Actions deploys via an OIDC role.

Route53 ─┬─ CloudFront ── WAF(web) ── ALB(app) ─┬─ ECS web
         │        └── OAC ── S3 releases         └─ ECS api
         └─ git.* ─────────── WAF(git) ── ALB(git) ── ECS git ── EFS
                                     VPC 10.0.0.0/16 (3 AZ)
   private-app:  web · api · git · worker-core · worker-sandbox
   private-data: RDS Postgres 16 (Multi-AZ) · ElastiCache Redis · EFS
   VPC endpoints: S3(gw) · ECR · Logs · Secrets · KMS · Bedrock · SES

Module map (site terraform/aws/, the ECS reference; production is terraform/vultr/)

Module	Provisions
modules/network	VPC, 3 subnet tiers × AZ, IGW, NAT, route tables, S3 gateway + interface endpoints, tier/identity SGs
modules/ecs-service	Reusable Fargate service: task def, service, autoscaling, SG + egress profile, log group, optional EFS mount, optional ALB target group, ADOT sidecar
modules/rds	Postgres 16 db.r6g.large Multi-AZ, gp3, PI, PITR, encrypted; publishes the DSN to Secrets Manager
modules/elasticache	Redis 7 (2× cache.t4g.small); publishes rediss:// URL to Secrets Manager
modules/efs	EFS elastic, uid-scoped access points, mount targets, AWS Backup daily/35d
modules/s3-suite	releases (object-lock 2y, deny-delete, OAC, EventBridge), evidence (IA 90d), audit-archive (compliance 400d, cross-account replication)
modules/cloudfront	Distribution: S3 releases via OAC + ALB origin (cache off), security headers, TLS 1.2+
modules/waf	web/API ACL (core + known-bad + rate) and git ACL (core minus packfile body rules + rate)
modules/kms	Data key (rotating) + ECDSA P-256 signing key restricted to worker-core
modules/observability	ADOT config param, dashboards (API/git/pipeline/security), alarms, SNS page/ticket
global/{accounts,dns,ecr}	Org + OIDC deploy role · Route53 + ACM · ECR repos (scan-on-push, lifecycle)

The five services (all from modules/ecs-service)

Service	Command	LB	Egress profile	Extra
web	(Next.js)	app ALB	full (NAT)	none
api	agentryd api	app ALB	full (NAT) + S3/RDS/Redis	task role: read releases, r/w evidence
git	agentryd git	git ALB	no NAT; endpoints + RDS + EFS	EFS mount /data/git; no S3-releases write
worker-core	agentryd worker --queues=core	none	full (NAT) + S3/RDS/Redis/EFS	only role with kms:Sign + SES
worker-sandbox	agentryd worker --queues=sandbox	none	no NAT; endpoints (incl. Bedrock) + S3 + RDS	read-only rootfs; read releases, findings via DB only; bedrock:InvokeModel for S5, which runs inside the scan job

--queues names the queues a worker drains, a subset of the two that exist (core, sandbox; both when omitted, the self-host shape). There is no judge queue: S5 runs inside the scan pipeline's DAG, on the sandbox worker, so the judge runs under the sandbox worker's profile: no NAT, and the Bedrock interface endpoint is its only model egress. A name that is not a queue is refused at start.

modules/ecs-service expresses the sandbox jail through allow_all_egress = false: with no 0.0.0.0/0 route, the only egress rules that exist are the scoped egress_* allowances. Sandbox reaches S3 (archives) + RDS (findings) + the interface endpoints (ECR/Logs/KMS to start & log, Bedrock for S5, all through the shared endpoints SG). Data-tier ingress (RDS/Redis/EFS) is restricted to the app-tier / git-tier identity SGs, so "only app SGs" holds without SG cycles.

App configuration wiring

agentryd reads AGENTRY_* (see internal/config/config.go). The env sets:

• Non-secret (task env): AGENTRY_ENV, AGENTRY_BASE_URL, AGENTRY_STORAGE_KIND=s3, AGENTRY_STORAGE_S3_BUCKET/REGION, AGENTRY_SIGNING_KIND=kms + AGENTRY_SIGNING_KMS_KEY_ARN, AGENTRY_JUDGE_PROVIDER=bedrock + model/region, AGENTRY_OTEL_ENDPOINT=http://localhost:4318 (the sidecar), and AGENTRY_TRUSTED_PROXIES (2 for api/web behind CloudFront→ALB, 1 for git).
• Secret (Secrets Manager → container secrets): AGENTRY_DATABASE_URL (RDS DSN), AGENTRY_REDIS_URL (api/worker-core only), AGENTRY_AUTH_SEALING_KEY (random 32B). Add AGENTRY_SMTP_PASS here if SMTP notifications are turned on.

The terraform envs set AGENTRY_SIGNING_KIND=kms on every agentryd service so config validation passes. Only worker-core's task role holds kms:Sign in IAM, and the KMS key policy further restricts Sign to that exact role ARN by condition. The KMS signer fetches its public key once at startup, so serving signing-keys.json never turns into kms:Sign volume and cannot muddy the kms:Sign anomaly alarm.

agentryd validates its config at boot and refuses to start on a contradiction rather than degrading: storage.kind=s3 with no bucket, signing.kind=kms with no key ARN, notify.kind=smtp with no host or sender, an SMTP username without STARTTLS outside dev, an inverted signing validity window, or a session/PAT TTL above the maxima (24 h session, 1 y PAT, 90 d org-bound PAT).

Self-host HA (Helm)

helm install agentry deploy/helm/agentry. Deploys api, git (shared RWX PVC at /data/git) and worker, with Services, Ingress (registry + git hosts), HPAs, a ConfigMap (agentry.yaml), and a Secret. The site in front is not in the chart.

Operator-supplied (NOT bundled; see values.yaml): PostgreSQL 16 (secret.databaseUrl), object storage (config.storage.* + IRSA/creds), and optional Redis (secret.redisUrl). Provide a ReadWriteMany StorageClass for the git volume (EFS CSI, Filestore, NFS, Vultr VFS). An RWX class whose driver leaves the volume root-owned needs git.persistence.initOwnership, as Vultr VFS does in production. The self-hosting guide lists the values that make the chart installable off the AWS reference.

Deploy pipeline (site .github/workflows/terraform.yaml)

1. Every PR / push: OpenTofu tofu fmt -check, init -backend=false + validate for each module and root of both trees (terraform/aws/, terraform/vultr/), and tflint. (helm lint and scripts/acceptance/helm.sh run in the core's own CI.)
2. Plan on PR (AWS tree only): tofu plan per env, via OIDC role assumption (no long-lived keys). Requires AWS_DEPLOY_ROLE_ARN repo var, which is unset: the jobs no-op.
3. Apply on merge to main (AWS tree only, same gate): dev + staging apply automatically; prod apply is gated by the prod GitHub Environment.

The AWS tree's state would be S3 + DynamoDB lock, one key per env (backend.tf, placeholders; never initialised). The Vultr tree is applied by the deploy workflow (deploy.yaml, above) once its secrets are set, and from an operator's machine for anything that deletes; its state is on Vultr object storage, locked with the backend's lockfile and encrypted client-side (terraform/README.md).

First-time bring-up order

1. Apply global/ (org, OIDC role, Route53 zones + ACM certs, ECR repos).
2. Delegate the registrar to the Route53 name servers; wait for ACM validation.
3. Populate each env terraform.tfvars from global outputs (cert ARNs, image URIs, zone id, security-account bucket ARN).
4. Apply envs/dev, then envs/staging, then envs/prod.
5. Push images to ECR; ECS rolls out.

Backup / DR targets on the AWS reference

Target	Value	Mechanism
RPO	15 min	RDS PITR (backup_retention_period); EFS/S3 near-real-time durable
RTO	4 h	Runbook-driven in-region restore; infra is terraform apply-reproducible

• RDS: automated backups + PITR (prod 35d), Multi-AZ standby; failover raises a page alarm (EventBridge → SNS).
• EFS: AWS Backup daily, 35d retention.
• S3: releases object-lock governance 2y + deny-delete; audit-archive object-lock compliance 400d + replication to the security account.
• Quarterly restore drill: the RDS restore and EFS restore runbooks are the procedure. Multi-region is out of scope.

Production on Vultr: backups and the restore drill

Production on Vultr uses these targets in place of the AWS ones above. Vultr offers no object replication, no etcd backup and no stated snapshot of the file system volume, so the copies are made by jobs, in a second region.

Data	Copy	How often	RPO	Kept
Postgres (registry and directory)	the managed service's point-in-time recovery, in ewr	continuous	minutes (the newest restorable point trails the present by a few)	2 days on Startup, 14 on Business
The store (release archives, run artefacts)	rclone sync to the backup bucket's store/	hourly (backup-objects)	1 hour	noncurrent versions 30 days
Git repositories (the VFS claim)	git bundle --all with each repository's config, HEAD and shim, plus the SSH host key, to git/<date>/	nightly (backup-git)	1 day	30 nights
Postgres, outside ewr	pg_dump -Fc of the agentry and authentik databases, with SHA256SUMS, to db/<date>/	nightly (backup-db)	1 day	30 nights
Cluster objects	Velero, Secrets excluded (terraform's, or helm's release records)	daily	1 day	30 days
Terraform state (already encrypted)	scripts/dr/state-copy.sh to the backup bucket's tfstate/	after every apply	the last apply	versions 30 days

The backup bucket, agentry-prod-backup, is in the backup subscription on atl2 (Atlanta). The cluster, the database and the store are in ewr.

The database's copy outside ewr is the nightly dump. Its point-in-time history is the managed service's own, in the same region, so for a lost ewr the dump is what is left: RPO a day. Each dump holds every row, the directory's password hashes and the registry's sealed secrets among them. It sits in the bucket readable with the backup key alone, held as below. backup-db reaches the database through a policy that admits it alone, and ends with backup: db done: …. scripts/dr/db-dump-restore.sh fetches the newest dump, checks it and restores it into a scratch fork, which is the procedure a lost ewr follows against a new instance.

The backup key is held by:

• the three CronJobs (backup-objects, backup-git, backup-db) and Velero, in the cluster;
• the drill's scratch registry, for the few minutes it runs;
• state-copy.sh and db-dump-restore.sh on the operator's machine, while they run.

backup-db also holds the two databases' credentials, which is what makes its dump complete. Its policies let it reach the database, DNS and public HTTPS addresses (the object stores among them), and nothing else in the cluster.

No pod of the production registry mounts it. Vultr issues one full-rights key per subscription, so there is no read-only key, and the bucket has no object lock: whoever holds the key can delete every copy.

Each CronJob ends with a backup: … done: … line. Its other output lines go to the log stack as well. The Overview dashboard's "Backups (last runs)" search shows the newest.

The database's fork. The managed service's point-in-time history is restored to a new cluster by its fork call. Give it the source database, a label, the plan, the region, type: pitr, and a date and time. The result is a new cluster with production's roles and passwords. Allowlist it before anything connects. The history lives in the same region as the database, so it does not survive the loss of that region.

The repositories' copy. The repositories are on a Vultr File System claim, of which Vultr states no snapshot. The nightly backup-git job, in the site's Helm chart, copies each repository to the backup bucket in a second region as a git bundle --all, with the repository's config, HEAD and pre-receive shim beside it, and the git role's SSH host key. The copy is nightly, so the RPO is a day: pushes since the last bundle come back only from a clone.

RTO: four hours. The restore itself, as the drill below runs it, takes under eight minutes from the first step to a verified version, most of it the database fork. The rest of the four hours is for deciding to restore, and for the steps the drill does not repeat: a new cluster by tofu apply, and the installs.

Proving the database's point-in-time recovery

scripts/dr/pitr-fork.sh makes two audited requests that change nothing, four minutes apart, with the fork point between them. It then forks the managed Postgres to that second, and queries the fork. The fork must hold the first request's audit row, matched by its id from the log stack's audit mirror, and must not hold the second's. The fork is deleted afterwards. The fork takes under six minutes.

The restore drill (quarterly)

scripts/dr/drill.sh restores production into a scratch namespace beside it, then deletes everything it made. The data comes from the copies: the store and the repositories from the backup bucket, the cluster's objects from Velero, the database from its point-in-time history. What a real restore gets from tofu apply and the installs from main, the drill reads from the live cluster:

• the signing Secret;
• from agentry-app, the DSN (pointed at the fork) and the sealing key only. Every other key is emptied, so the scratch registry sends no mail and reaches no directory;
• the pull token;
• the backup key, as the scratch store's credential;
• production's computed helm values and running image.

The steps:

1. A fork of the database, five minutes back, admitting the cluster's NAT address only.
2. The Secrets above.
3. The newest night's git bundles, restored into a scratch claim by scripts/dr/restore-git.yaml (git init --bare, then the saved config, HEAD and shim, then the bundle fetched and fscked).
4. The core chart, from the chart the umbrella vendors, with production's computed values and image. Its store is the backup bucket's store/ (s3Prefix), and its database is the fork.
5. agentry verify --strict of a version through a port-forward. The scratch registry's signing key ids must be production's.
6. Every restored repository's refs compared with production's.
7. Velero's newest backup restored into a second scratch namespace.

KUBECONFIG=… VULTR_API_KEY=… AGENTRY_TOKEN=… \
VERSION=<ns/name@version> AGENTRY_BIN=<the CLI> \
CHART=deploy/helm/secureagentry/charts/agentry-<version>.tgz \
  scripts/dr/drill.sh

Every credential is read from a credential file or the cluster into the environment, and none is printed.

A run serves and verifies the version in about five minutes, of which the fork takes four to six. The digest passes, and the attestation verifies against production's own signing key. Repositories with no content have nothing to bundle and are restored empty.

Restoring for real

What the drill does, pointed at production instead of a scratch namespace:

• The database. Fork the point-in-time history and point agentry-app's DSN at the fork (terraform's database module), or restore in place with the service's restore call (RestoreFromBackup) once the fork has shown that the chosen time is right.
• The store. rclone sync from store/ back into the store bucket, in the opposite direction to backup-objects.
• The git claim. The drill's restore Job (restore-git), pointed at agentry-git-data. Stop the git role first. The Job makes each repository as the server's InitBare would (git init --bare, then the saved config, HEAD and shim), fetches the bundle and checks connectivity.
• A lost cluster. tofu apply, which makes a new cluster and every Secret from the encrypted state. Then the log stack's and the umbrella's installs from main. Velero's backups hold what those do not remake.
• A lost ewr. The state's copy in tfstate/ is the starting point: fetch it and point the backend at a new bucket. The store, the repositories and the cluster's objects are rebuilt from it and the copies above. The database is restored from the newest db/<date>/ into a new managed Postgres in the new region, with pg_restore --no-owner as its admin user (scripts/dr/db-dump-restore.sh shows the steps against a fork). Every restored object is then owned by that admin. Handing them to the users terraform makes is a further step that is not scripted, and terraform's database module says what the admin may and may not change on this managed service.

Observability

agentryd emits metrics two ways from one instrument set (internal/metrics):

• OTLP/HTTP when AGENTRY_OTEL_ENDPOINT is set. The terraform envs set it to http://localhost:4318, the ADOT sidecar every agentryd service runs. The collector ships those metrics to CloudWatch as EMF under the Agentry namespace. Metrics are the only signal it receives: agentryd imports no tracing SDK and starts no tracer, so the collector's traces pipeline and its X-Ray exporter sit configured and idle (the ADOT config says so beside them). Unset (the default) disables OTLP entirely: no exporter, no connection attempts.
• Prometheus at GET /metrics, always on, for self-host. See the bind-address note below.

Metrics actually emitted

These are the exact instrument names, and they are the names the dashboards and alarms in modules/observability reference. The Prometheus exposition uses the same names, so a CloudWatch alarm and a self-hosted scrape describe the same series.

Metric	Type	Attributes	Used by
HttpRequests	counter	route, method, status_class	RED
HttpRequestDurationSeconds	histogram	route, method	RED latency
QueueDepth	gauge	queue	pipeline dashboard, worker autoscaling
QueueAgeSeconds	gauge	queue	PAGE: queue age > 15 min
QueueJobs	counter	queue, kind, outcome	job outcome mix
ScanStageRuns	counter	stage	PAGE: scan error rate > 5% (denominator)
ScanStageErrors	counter	stage	PAGE: scan error rate > 5% (numerator)
ScanStageDurationSeconds	histogram	stage	pipeline dashboard
ScanStageHits	counter	stage	security dashboard (an S3 spike is an attack-wave signal)
PublishSubmitted	counter	none	publish funnel
PublishVerdictClean / …Review / …Blocked	counter	none	publish funnel, verdict mix
JudgeCostCents	counter	none	budget anomaly (ticket); not yet emitted, see below
GitOps	counter	op, outcome	git dashboard
GitOpDurationSeconds	histogram	op	git dashboard
GitOpConcurrency	gauge	op	git dashboard, git autoscaling signal
GitCloneFirstByteSeconds	histogram	none	SLI: p95 < 2 s
FailOpenEvents	counter	path (audit_stream_drop, history_not_scanned, osv_not_checked, osv_error)	TICKET: any fail-open event in an hour
WebhookDeliveries	counter	outcome (ok/error/throttled)	delivery health
McpSweepRuns	counter	outcome (ok/error)	the daily MCP server vulnerability sweep ran
McpDependencyAlertsOpened	counter	none	dependency alerts opened on published versions; a spike is an ecosystem advisory wave

route is the matched route pattern (/agents/{ns}/{name}), never the concrete path; a per-path metric would let any caller mint unbounded series. status_class is 2xx/4xx/5xx for the same reason.

An instrument with no observations yet is absent from a scrape; the alarms are configured treat_missing_data = "notBreaching" to match.

JudgeCostCents carries no data today. The instrument exists and the pipeline dashboard plots it, but no judge provider reports a cost: the openai, anthropic, bedrock and classifier providers all run real adjudication, yet scan.JudgeVerdict has no cost field to carry one (a provider that cannot be built still resolves to UnconfiguredProvider, which routes to review). The recording seam is metrics.RecordJudgeCost, one call from the S5 stage away; until a provider populates it, that dashboard panel is empty rather than zero. Do not read an empty panel as "no spend".

Dimension rollup. The alarms address QueueAgeSeconds, ScanStageErrors and ScanStageRuns with no dimensions, while the app emits them with queue / stage attributes for the dashboards. The terraform therefore sets the collector to ZeroAndSingleDimensionRollup, which publishes the zero-dimension aggregate alongside the per-attribute series. Under NoDimensionRollup the zero-dimension series would not exist and those three page alarms would never fire, while looking correctly configured.

/metrics is unauthenticated, so bind it accordingly

/metrics is mounted outside /api/v1 next to /healthz and /readyz, and like them it has no auth: a scraper is infrastructure, not a principal, and the RBAC chokepoint has no way to express one.

It exposes no user content (request rates, queue depth, stage timings) but it does describe traffic shape and catalogue activity. On SaaS the ALB listener rules only forward /api/*, /.well-known/* and /scim/* to the api service, so /metrics is not reachable from the internet. Self-hosting: either bind AGENTRY_HTTP_ADDR to a private interface, or block /metrics at your ingress. The chart has no ingress.apiPaths value: its ingress sends every path on the registry host to the api service, so it does not filter that route for you.

Alarms and routing

Page: availability burn, queue age > 15 min, scan error rate > 5%, RDS failover, KMS Sign anomaly (from CloudTrail data events), WAF block spike. Wire page_subscriptions / ticket_subscriptions (PagerDuty, email) per env.

Notifications / email

AGENTRY_NOTIFY_KIND selects the transport:

Kind	Behaviour
log (default)	Writes what it would have sent to the log. A fresh install never silently fails to deliver; the line says the mail was not sent.
smtp	Real delivery via AGENTRY_SMTP_{HOST,PORT,USER,PASS,FROM} + AGENTRY_SMTP_STARTTLS.

Sent on: email verification, review verdict decided, quarantine after rescan, advisory published, a dependency alert on a declared MCP server, and org membership granted — and, the verification mail apart, only to an address that has been verified. Bodies are plain text and say "scanned, N findings, signed", never "safe".

Org membership is offered and accepted: an admin creates an invitation (by handle or email) that the person accepts to take a seat, and someone whose verified email is at the organization's domain is folded in as a pending member for an admin to approve. Either way, when the membership becomes active the member.added webhook fires and the "added to an organization" mail rides that same transition; an invitation still outstanding fires no event and counts toward the seat allowance until it is accepted or expires.

Dev. The compose stack runs mailpit; deploy/compose/docker-compose.yaml points the server service at mailpit:1025 with STARTTLS off, and every message lands in the web UI on :8025. Plaintext SMTP is correct for mailpit and wrong for anything else; config validation refuses smtp.username without STARTTLS outside dev mode.

AWS. Point SMTP at SES's SMTP endpoint (email-smtp.<region>.amazonaws.com:587, STARTTLS on) using SES SMTP credentials, which are IAM credentials converted to an SMTP username/password. The terraform already grants ses:SendEmail to worker-core and provisions the SES VPC endpoint. There is deliberately no SES-SDK integration and no SES terraform: domain identity, DKIM keys, and the SPF/DMARC DNS records are operator setup that depends on the region and domain you choose.

Put the SMTP password in Secrets Manager (SaaS) or secret.smtpPassword / existingSecret (Helm), never in the ConfigMap.

Signing keys and rotation

GET /.well-known/agentry/signing-keys.json publishes keyId, algorithm, publicKeyPem, and an optional notBefore / notAfter validity window per key. The window is what makes rotation by overlapping validity windows expressible: the document can list more than one key at a time.

To rotate:

1. Create the new key; publish it as signing.additionalKeys[] with a notBefore, while the current key keeps signing.
2. Switch signing.kmsKeyARN to the new key and give the outgoing key a notAfter, moving it into additionalKeys.
3. Drop the outgoing key from additionalKeys once nothing needs to verify attestations it signed. Attestations are signed once and never re-issued, so old envelopes stay verifiable only while their key is published.

Windows are optional; a key with neither bound is unbounded, which is the correct description of a deployment that has never rotated.

Quotas

internal/entitlements resolves the limits, and its OSS implementation reads them from config, so an operator raises a limit by setting a value, not by patching a constant. Chart: config.limits.*. Env: AGENTRY_LIMITS_* (see internal/config/config.go).

agentryd enforces them with in-process token buckets, so they are approximate across replicas: an N-replica fleet allows roughly N× the budget. The request limits (anonymous per IP, authenticated per user, login per IP) sit in the auth middleware; two more are:

• webhook deliveries, 5/s per org (over budget, the delivery is deferred back onto the queue, not dropped)
• git pushes, 60/h per repo (over budget, 429 before git is even started)

Redis is the answer for exact multi-node limits; that swap goes behind the same internal/ratelimit interface.

Client IP behind a proxy

Per-IP rate limits and audit records key on AGENTRY_TRUSTED_PROXIES, the number of reverse-proxy hops in front of agentryd:

Deployment	Value
Direct (default, 0)	X-Forwarded-For ignored entirely
Behind one ingress/ALB	1
CloudFront → ALB → task	2 (what the terraform envs set for api/web; git is 1; no CDN in front of git.*)

X-Forwarded-For is client-supplied. Setting this higher than the real hop count lets a caller choose which rate-limit bucket to spend, including someone else's, and forge the address the audit log records. It must match the deployment exactly.

Inputs the AWS reference leaves as tfvars

• Region / data residency: aws_region (default us-east-1) in each env and in global; backend.tf region comment.
• Judge model: judge_model (default Sonnet-class Bedrock id).
• Cross-account audit replication + T-09 alarm source: audit_replication_destination_bucket_arn / ..._account_id, cloudtrail_log_group_name (empty ⇒ feature off).
• Domains + certs: apex_domain, git_domain, acm_*_certificate_arn, route53_zone_id (all placeholder; empty zone id skips DNS records).
• Image URIs, alert subscriptions, and the state bucket/table names.