Section 2 · Application. The request loop, the routing table and the response cache.

Fig. 1 · one request, end to end

Your apps hold no provider keyThey send one base URL and one virtual key.
Guardrail screen6 classes, off until you give one an action. A block costs nothing.
Response cacheExact, then semantic. A hit ends here, at $0.00, using no call slot.
Budget reserveDurable, survives a restart, downroutes at 90% instead of cutting off.
RouterTier to model, with a failover chain, a breaker per endpoint and a 3.0× price guard.
Cost meterThe provider's own rate, not an estimate.
The provideropenai, anthropic, openrouter, or your own key. Real credentials live only here.
Back to your appA byte-faithful OpenAI response, with the real cost recorded per call.

The loop, in order

Every surface runs the same eight steps.

#	Step	What happens	If it fails
1	Authenticate	The bearer virtual key resolves the project. Identity is never taken from a client header.	401
2	Screen	6 classes, redact or block, set per project. Off until you give a class an action.	403 guardrail_blocked
3	Look up the cache	Exact first, then semantic. A hit returns here.	Treated as a miss. Both layers fail open
4	Reserve the budget	Durable, and it survives a restart.	Downroute at 90%, refuse at 100%
5	Route	Sender override, then the project default, then straight through.	400 no models available for route
6	Prove the endpoint	The breaker must be closed and the price inside the 3.0× guard.	The next entry in the chain
7	Inject the credential	Server-side only. The real key never reaches a client.	502
8	Record the cost	Read from the provider’s answer and priced per model.	—

Why step 2 runs before steps 3, 4 and 7

Redaction happens before the cache lookup, so a card number never reaches an embedding stored in our own database. It happens before the reservation, so a blocked request consumes no call slot. It happens before the body is recorded, so one rewrite covers the upstream request, the cache key and the stored body alike.

Routing

Three steps, always in this order.

Order	Rule	Source	Example
1	Sender override	axon_route_overrides	`x-axon-source: batch` asking for `standard` goes somewhere else
2	Project default	axon_tier_defaults	Everything else asking for `standard`
3	Passthrough	the request itself	`openrouter/qwen-3-235b` is used as sent

You name the tiers. Most people configure trivial, standard and heavy, but nothing in llmyard requires those words — a tier is whatever you put in the routing table, and a key may point at any tier this gateway actually serves. Two names are reserved: auto and classifier mean “decide per request” rather than “use this row”. A new key starts on trial until you move it.

A project with neither its own vendor key nor a grant on the shared pool can route nowhere: the key authenticates, the tier resolves, and every candidate is then skipped. The refusal says which of the two you are missing instead of failing silently.

A change reaches every replica within 30 seconds, so a revoke is not instant fleet-wide. The response that applies it reports the window.

The response cache

Off unless CACHE_ENABLED=true. Both layers fail open.

Layer	Store	Key or match	Cost of a hit
Exact	Dragonfly	sha256(tenant + model + endpoint + body)	$0.00
Semantic	pgvector	cosine ≥ 0.95, inside 3600s, same tenant and model	$0.00

The lookup runs before the budget reservation, so a hit is free in both senses: no upstream call, and no call slot. A semantic hit is promoted into the exact layer, so the second repeat costs nothing to find. The query embedding is reused for the write-back, so a miss embeds exactly once.

Caution · a mismatch here fails silently

CACHE_EMBED_DIM must equal the vector(N) column width. The shipped column is vector(1024). If they disagree, every embedding is dropped: the semantic layer stops working, the exact layer keeps hitting, and the deployment looks healthy.