The log contract¶
What a line the server writes contains, what is never in one, and the names
a dashboard or an alert may key on. This page is the reference a
collector, a log store, a dashboard or an alert is built against; the code
that keeps it is internal/logx (the mechanism)
and internal/logevent (the names), and a test fails when the code and this
page disagree.
A line¶
agentryd (and the enterprise build, secureagentryd, which is the same
runtime) writes one JSON object per line to stderr and nowhere else. Every
line carries:
| Field | Meaning |
|---|---|
time, level, msg |
when, at what level (DEBUG, INFO, WARN, ERROR), and the line's own words |
service |
the binary's name as invoked: agentryd or secureagentryd |
role |
the role this process runs: all, api, git or worker |
version |
the build |
request_id |
on every line written inside a request (see below); absent outside one |
event |
a name from the catalog below, when the line is a signal rather than narration; absent otherwise |
The level is the line's own level. When the lines are shipped through a
syslog transport, the syslog severity says which stream the line came from
(everything is stderr) and nothing about its level.
The minimum level is log.level in agentry.yaml or AGENTRY_LOG_LEVEL
(debug, info — the default — warn, error).
The request id¶
Every request the server handles has one id, returned to the caller as
X-Request-Id and carried on every line the request writes — the access
line, a handler's warning, the panic line — and on the audit row of any
mutation the request made (detail.request_id). A row and its lines find
each other by it.
The server mints the id (16 hex characters). It honours an inbound
X-Request-Id instead — so that one id names a request on the proxy in
front and on the server — only when the operator has said that the hop
in front sets the header on every request it forwards
(http.proxyRequestID: true / AGENTRY_PROXY_REQUEST_ID=true, which
itself needs http.trustedProxies ≥ 1, and with http.trustedProxyCIDRs
set, a request whose peer is in one of its ranges) and the value matches
^[A-Za-z0-9-]{8,64}$. The hop count alone is not enough: anyone can send
the header, and a proxy that merely forwards it would let a caller stamp
its own id onto the server's lines and audit rows. The compose stack's
headless edge sets it (proxy_set_header X-Request-Id $request_id); a
site's edge config in front of the server must do the same before the flag
is turned on there.
The access line¶
One line per request, event=http.request, msg=http, at INFO:
| Field | Meaning |
|---|---|
method, path |
as requested |
route |
the matched route pattern (/api/v1/agents/{ns}/{name}), never the concrete path — the low-cardinality key dashboards group by |
status, dur_ms, bytes |
the response |
client_ip |
the caller's address as the http.trustedProxies and http.trustedProxyCIDRs rule resolves it |
user_agent |
as sent |
actor, actor_via |
the principal's opaque id and how it authenticated (session, pat, ssh); actor=anonymous when unauthenticated. A PAT's id may appear (actor_pat); the PAT itself never does |
The probes — /healthz, /readyz, /metrics — write no access line. Their
metrics are still recorded.
The event catalog¶
A line that is a signal carries event=<name>. The names are constants in
internal/logevent; a string literal reaching a line is a test failure, as
is a name missing from this table.
| Event | Level | When | Fields beside the common ones |
|---|---|---|---|
http.request |
INFO | the access line | see above |
http.rate_limited |
INFO | a request refused by a rate class | class (login, anon, authed, abuse_reports, scim, an organization's SCIM data plane over its budget, and model-input, a model-input caller or an address presenting job credentials over its budget), route; for the git transports, class push with agent (the repository's push budget, over HTTP or SSH), class ssh-key-offer with source (the address's public-key offers to the SSH listener), and class git-concurrent with source (the client address, as resolved through the declared proxies, at its budget of concurrent git operations, over HTTP or SSH) |
auth.social_rejected |
WARN | a sign-in through the OpenID provider was rejected at the token exchange | provider, err |
saml.state_unavailable |
ERROR | a SAML login refused because the shared state behind it could not be read or written; the fail-closed opposite of failopen.* |
set (assertions, requests), err |
ssrf.refused |
WARN | the private-range guard refused a webhook target | org, delivery, endpoint, host |
gate.verdict |
INFO | the gate decided a version | outcome (published, rejected, needs_review), version, agent, run, worst_severity, findings, reason; and ruleset when a new ruleset's redo of the review queue published a held version |
gate.error |
WARN | the platform's own failure sent a version to review: stage errors, an unusable policy, the watchdog | version, run, and stage_errors or kind + err |
scan.stage |
INFO | a scanner stage finished | stage, outcome (ok, error), dur_ms, findings, err |
failopen.audit_stream_drop |
ERROR | an audit archive record was dropped on a full buffer | action, id |
failopen.history_not_scanned |
WARN | a git history the S2 stage did not scan, or scanned only to its size cap; a history scan that failed (S2/history-error) holds the version for review and is not counted here |
stage, scanner, title |
failopen.osv_not_checked |
WARN | a dependency set S7 did not check against OSV | stage, scanner, title |
failopen.version_not_checked |
WARN | declared MCP server versions S7 did not check against their package registry: the check is off (AGENTRY_SCAN_NPM_REGISTRY=none), or npm or PyPI gave no answer, and the version publishes (scanner is S7/version-not-checked) |
stage, scanner, title |
failopen.osv_error |
WARN | an OSV lookup errored and left a declared server with no data (scanner is S7/osv-no-data; a server S7 does not cover makes the same finding and is not counted here) |
stage, scanner, title |
failopen.directory_erasure |
ERROR | an account erasure's directory half did not happen: the attempt failed, the directory could not tell the account apart, or a directory that erases on its own answered no outcome the response defines; the sign-in account is left for the operator to delete | user, err |
git.push_refused |
WARN | the git door refused a submission after a tag push | agent, tag, err |
mail.suppressed |
INFO | a notification was withheld from an unverified address (WARN when the user row could not be read) | user, kind |
webhook.delivered |
INFO | a webhook delivery attempt finished | outcome (ok, error), org, delivery, attempt, status or err |
model_input.call |
INFO | a call to a model-input route finished: AI Review of text a model is about to read, or a brokered web search | op (scan, search), caller (staff, job), job (a hosted job's id, when a job called), verdict (pass, withheld, not_scanned; a search is withheld when any result was), source, scanned, withheld, requests, input_tokens. Never the text, the query or a result's address |
audit.<action> |
INFO | the audit mirror: one line per audit event, named by the audit action (audit.user.login, audit.pat.create, …) |
see below |
Each failopen.* name is also a FailOpenEvents{path} metric increment;
the line is the where and the why, the metric is the count.
The audit mirror¶
With audit.streamKind: log (AGENTRY_AUDIT_STREAM_KIND=log) every audit
event is also written as one line, msg=audit, event=audit.<action>, with
audit_id, occurred_at, actor, actor_kind, org, target_kind,
target_id, client_ip, user_agent, request_id and the record's
detail as an object. It is a mirror, not a second record: the row in
audit_events is the record and stays append-only; the line is what makes
the event searchable wherever the server's lines are shipped. The detail is
carried as the record has it — what a handler puts on an audit row is bound
by the audit rules, not by this page — and passes through the redaction
below like everything else. The mirror writes synchronously, so it has no
buffer and no drop path of its own; the file kind keeps its buffer and its
drop accounting.
What is never in a line¶
- A token or a secret. Before a line is written, a value under a key
ending in
token,secret,password,passwd,authorization,cookie,api_key,private_key,signing_key,secret_key,access_key,client_secretorcredential(s)becomes[redacted], and any token the registry minted — 52 base32 characters, with the kind's prefix (agy_PATs,agys_sessions,agyv_verification tokens,agyi_invitations,agyd_remembered browsers,agyscim_SCIM tokens,agd_desktop grants,whsec_webhook secrets,csrf_CSRF tokens) or bare (OIDC state and nonce, domain verification) — is masked wherever it appears (in a message, a value, an error, a nested object), keeping its 12-character display prefix so an incident still traces to the credential. Masked, not dropped: the line about the incident is worth more than the value it would have leaked. The collector applies the same token pattern again as a second net. - A person's name or email address. Lines carry opaque principal ids.
The log notifier (
notify.kind: log, the default when no sender is configured) says a mail would have gone and to which mailbox with the address masked (a…@example.test), and never carries a verification token; an address cannot be verified through it. Use a sender. - A value above 4 KB. Docker splits a line above 16 KB into pieces that
are no longer JSON and cannot be reassembled, so every string value is cut
at 4,096 bytes (on a character boundary) with a
…[truncated]marker; a panic'sstackkeeps 8 KB, the one value worth most of a line. - The probes. No access line for
/healthz,/readyzor/metrics.
The client address is in a line (client_ip) for as long as the lines are
kept, which is the retention of whatever store they are shipped to, and on
the audit row for the audit retention; the privacy page says so.
Shipping is the deployment's¶
Where the lines go after stderr is the deployment's choice (the "Logs"
section of self-hosting). Ship them best-effort: a
transport that cannot block, such as a Docker logging driver in
mode: non-blocking, never stalls the server or stops a container from
starting, at the price of the lines sent while the collector was down. The
audit record does not depend on any of it.