How it works

FitGuard sits in the request path between your application and your provider. This page covers what it does with each request and why the steps are ordered the way they are.

The whole picture

Your app (OpenAI SDK)
      │  POST /v1/chat/completions
      │  Authorization: Bearer sk-guard-...
      ▼
┌──────────────────────────────────────────────────────────┐
│ fitguard                                                 │
│                                                          │
│  1. authenticate   → resolve virtual key to a user_id    │
│  2. cache lookup   → serve from cache if hit ($0)        │
│  3. budget reserve → estimate worst case, reserve it     │
│  4. call upstream  → try model, then configured fallbacks│
│  5. finish_reason  → warn on truncated responses         │
│  6. log + release  → persist cost, release the hold      │
│                                                          │
└──────────────┬───────────────────────┬───────────────────┘
               │                       │
       provider APIs           SQLite (fitguard.db)
   (OpenAI / Anthropic /               │
    Gemini / Groq / Together)   /dashboard (live UI)

One Go binary. Your application only ever talks to FitGuard, and never sees your real provider keys — those live in FitGuard's config and are attached only on the upstream leg.

The request lifecycle

Every call to /v1/chat/completions goes through these steps in order.

1. Authenticate

The bearer token resolves to a user_id. A missing or unrecognized token is a 401. In single-tenant mode (empty keys:) everything authenticates as "default".

Budgets are tied to the key rather than to anything the caller reports about itself. A client-supplied X-User-Id header or the OpenAI user field can't be trusted for spend enforcement, because a single code path that forgets to set it lets that traffic escape its budget entirely.

2. Validate

model must be present. Malformed JSON or a missing model is a 400.

3. Cache lookup, before the budget check

The request is hashed and checked against the cache. A hit is served immediately, logged at $0, and skips the budget check entirely. A cache hit costs nothing, so there is nothing to gate. Checking the budget first would refuse a user at their limit an answer that was free to serve.

4. Budget reservation

On a miss, FitGuard estimates the worst case this request could cost and reserves that amount. If already-spent plus other in-flight reservations plus this estimate would exceed the daily limit, you get a 429 and nothing is sent upstream. Details in budget enforcement below.

5. Upstream call, with fallback

The primary model is tried first. On an error, 5xx, or 429 from the provider, each model in fallback: is tried in order.

6. finish_reason guard

If the response came back with finish_reason: "length", the completion was cut off. FitGuard sets X-AI-Guard-Warning and logs it. This is a common signal of a truncation or runaway-loop bug in the calling application, and it usually goes unnoticed because the response is still a 200.

7. Log and release

Real cost, from real token usage, is written to SQLite. The cache is populated. The reservation from step 4 is released, so the next request sees actual spend rather than an estimate.

Budget enforcement

Budgets are enforced by reserving cost before the upstream call rather than by checking already-logged spend.

Logged spend only reflects requests that have finished, so a "check spend so far" gate has a race window. Fire twenty concurrent requests at a user with $1 left and all twenty read "under budget" before any of them finish and get logged, so all twenty go through. Reserving up front closes that window, because an in-flight request's estimated cost is visible to concurrent requests immediately.

How the estimate is computed

The estimate is a ceiling rather than a prediction, so it usually reserves more than the request ends up costing. The question it answers is whether this request could exceed the budget in the worst case, which is what has to be decided before the call goes out. The reservation is released as soon as the request finishes, and the real cost is what gets logged.

local vs. redis

Redis EVAL runs single-threaded, so the Lua script acts as a mutex shared by every instance rather than by one process's memory. Two instances pointed at the same Redis can never jointly admit more than the configured limit. This is covered by TestRedisEnforcer_TwoInstancesShareOneBudget, which runs two independent enforcers concurrently and checks that the admitted total never exceeds what the shared limit allows.

If you run more than one instance, set budget.backend: redis. Otherwise each instance enforces its own copy of the budget and real spend can reach roughly (instance count) × the limit. See deployment.

Caching

The cache key is a SHA-256 hash of the model name plus the full request payload, minus user and stream (neither changes the answer). Identical requests inside the TTL are served at zero cost and zero upstream latency.

Model routing

The provider is chosen from the model name:

Routing is prefix-based and not yet configurable per-model. If a model lands on the wrong provider, prefix it explicitly: groq/my-custom-model. If the resolved provider isn't configured, that attempt is skipped and fallback continues.

Fallback routing

fallback: is a flat list of model names tried in order after the requested model, and only when a request errors or the upstream returns 5xx/429. Each fallback is routed through the same rules above, so falling back to claude-haiku-4-5 after a gpt-4o failure hits your Anthropic provider.

X-AI-Guard-Model-Used tells you which model actually served the response. If every attempt fails, you get one 502, not a cascade of confusing errors.

The pricing table

Cost is computed from a hardcoded USD-per-1K-token table covering OpenAI, Anthropic, Gemini, Groq, and Together models. Lookup falls back to a prefix match, so claude-sonnet-5-20241022 matches a claude-sonnet-5 entry, and finally to a conservative default for anything unrecognized — an unknown model degrades to "probably overestimated" rather than silently free.