Frontier llm-router

Frontier llm-router is a Kubernetes service for OpenAI-compatible routing: local-first escalation, role-based access, budgets, cost caps and spend logging.

The Frontier llm-router is its own Kubernetes service. It runs as a pod in its own namespace on the Federal Frontier FMC (Fleet Management Cluster) or another conformant Kubernetes cluster, such as Amazon EKS in AWS or RKE2 on-prem via CAPO, and sends model calls to llm-d and vLLM inference pods. Its callers, including Compass chat and Frontier SRE dispatch, get one OpenAI-compatible endpoint. Compass also hosts the FinOps and Admin screens that read and configure it. It sends each request to the cheapest model that can do the job, from self-hosted open-weight models up to commercial frontier models from leading AI providers. It also enforces who may use which model and keeps spend inside budget.

Request flow: a caller such as Compass chat or SRE dispatch passes the OIDC identity check (multiple issuers), then role rules (deny by default, 403 off-list), then the budget and cap check (monthly budget, RPM/TPM, per-turn cap), then the escalation ladder (self-hosted via llm-d, then mid-tier and top-tier commercial frontier models for high-risk or critical work). Every pick is written to a spend ledger that feeds the AI FinOps & SLA dashboard.

Cheapest-first escalation

Tier Model Used for
1 Self-hosted, served through llm-d First choice for every request
2 Mid-tier commercial frontier model Escalation when tier 1 is not enough
3 Top-tier commercial frontier model from a leading AI provider High-risk or critical work

Self-hosted is the low-cost first tier. The router escalates to frontier models when the task needs it, under the same budgets and caps.

Per-role model access

Access is deny-by-default and mapped to identity-provider roles. A request for a model outside the role’s allowed list returns HTTP 403. Every pick is logged.

Identity

OIDC token validation supports multiple issuers.

Budgets and caps

Control Scope
Monthly AI budget Per user and per role
Rate limits (RPM/TPM) Per user and per role
Per-turn cost cap Each model call
Per-session dollar cap Automatically stops a runaway agent’s job

Admins manage all of it in the Compass Admin AI Spend panel.

To track spend, budgets and forecasts, see Compass FinOps.

AI FinOps & SLA dashboard

The Grafana AI FinOps & SLA dashboard shows spend by model, app, role and user, plus escalations.

Frontier SRE agents

Frontier SRE agents (OpenCode and Claude Code runners) send every model call through the router under their own service identity and budget.

SRE remediation flow: an alert goes to sre-dispatch, then the SRE agent (OpenCode or Claude Code runner), then the llm-router with its per-session cap, then Remuda MCP tools, then the fix is applied. A guardrail branch shows the session cap stopping the Job if it is exceeded.

In a live ImagePull remediation, the agent finished in about 2.5 minutes for about $0.07.