LLM Inference Overview

Inference routing starts on the user’s own workstation, adds an optional department or enterprise model that the customer self-hosts, and escalates to Amazon Bedrock frontier models when local is not enough. Air-gapped sites run vLLM inside the cluster.

The Federal Frontier Platform uses large language models (LLMs) for agentic tool calling — CompassAI selects and invokes MCP tools from natural language. Inference is not one fixed server. Inference routing selects the backend from where the work runs and whether the request must leave a boundary-approved AI-enabled workstation. The Frontier llm-router, a Kubernetes service that CompassAI and other clients call, sends each request first to a local model on the user’s workstation, then to an optional department or enterprise model the customer self-hosts, and escalates to Amazon Bedrock frontier models when the task needs it, under the same budgets and caps.

Which Guide Do I Need?

Why Two Paths?

Ollama excels at zero-configuration LLM serving on Apple Silicon. Metal GPU acceleration is automatic, models download from the Ollama registry with a single command, and the native /api/chat endpoint provides reliable tool calling. It is the fastest path from zero to a working AI-powered infrastructure chat on an operator workstation. However, Ollama is not designed for multi-tenant Kubernetes deployments, does not ship as a Helm chart, and depends on the Ollama model registry for model distribution.

vLLM is a production-grade inference server built for Kubernetes. It deploys via Helm chart, exposes an OpenAI-compatible API, supports tensor parallelism across GPU nodes, and serves models from local storage or container images — making it suitable for FIPS-aligned air-gapped environments where no external registry access is available. vLLM is the inference backend for production F3Iai deployments.

Connected deployments that need managed Claude (or similar) use Amazon Bedrock over VPC PrivateLink (or GovCloud at IL5). That path is covered under Sovereign Inference; this page focuses on how CompassAI selects among local, Bedrock, and vLLM.

Inference routing

Federal Frontier inference routing selects which LLM backend serves CompassAI and related agents. Inference starts on a boundary-approved AI-enabled workstation with a local model. A customer can add an optional shared department or enterprise model on customer hardware, behind llm-router and CompassAI, where Keycloak roles control which models each user can reach. When local is not enough, requests escalate to Amazon Bedrock; without a shared tier, they go from the workstation straight to Bedrock. See Choosing Where Inference Runs.

CUI and higher IL data that never leave a boundary-approved workstation remain under that workstation’s controls. Any time tokens, tool payloads, or conversation context leave the customer’s controlled boundary, exposure goes up. Managed cloud and remote cluster endpoints are real options — they are also egress paths.

How selection works today

Routing is configured at deploy time, not by changing application code for each classification level. CompassAI API is the LLM client in the stack (CompassAI UI → CompassAI API chat / tool loop → MCP tools). It calls the endpoint and model set in the environment (LLM_ENDPOINT, LLM_MODEL; for vLLM also LLM_MAX_TOKENS and LLM_TEMPERATURE). The harness and tools stay the same; only the backend behind the API changes.

In air-gapped Kubernetes, CompassAI API is typically the sole consumer of cluster vLLM (LLM_ENDPOINT → the in-cluster vLLM OpenAI-compatible URL). On an operator workstation, the same pattern points at local Ollama when that is the documented path.

Where traffic goes

Inference routing starts on the user’s own workstation, adds an optional department or enterprise model that the customer self-hosts, and escalates to Amazon Bedrock frontier models when local is not enough. Air-gapped sites run vLLM inside the cluster.

1. First → a local model on the user’s workstation. If the data must not leave the device → local only.
2. If the workstation model is not enough and the customer self-hosts a department or enterprise model → that shared tier behind llm-router and CompassAI.
3. If local cannot meet capability, policy, or production needs → Bedrock (connected) or vLLM (air-gapped cluster).

Classification-level detail and the full IL table live on Sovereign Inference. Workstation install and API guidance live on Ollama on Apple Silicon. Cluster serving lives on vLLM on Kubernetes.

Local inference on AI workstations

An AI-enabled workstation is any boundary-approved operator machine running a local inference server for CompassAI and related tooling. Hardware can vary; the routing rule does not: the workstation is the first tier; requests leave it only when the local model is not enough, and data that must stay on the device stays local.

Apple Silicon is the preferred class for many FFP developer and operator setups because of Unified Memory Architecture: CPU and GPU share one physical memory pool, so large quantized models load without a discrete-VRAM ceiling, and Metal accelerates inference without a CUDA stack. That is why the documented local CompassAI path today is Ollama on Apple Silicon (:11434).

Under the hood, Metal GGUF-style serving on Macs rests on llama.cpp (ggml-org). See Related clients on the Ollama overview for how llama.cpp, Ollama, and Llama.app relate — and which path is validated for CompassAI tool calling.

Model Selection

Local (Ollama) and air-gapped (vLLM) paths use open-weight instruct models. Tags and HF ids churn — pick the strongest model that fits your hardware and tool-calling needs. Illustrative options:

On Bedrock, these docs cover families such as Claude, Meta Llama, Amazon Nova, Mistral, AI21, Cohere, Amazon Titan, and Gemma where available. Connected agent examples often use Claude via VPC PrivateLink; that is common, not exclusive. See Sovereign Inference and the Model Selection Guide.

Architecture

CompassAI API selects the configured backend via environment (LLM_ENDPOINT, LLM_MODEL). No application code changes are required when moving between compatible chat endpoints.

Next Steps

• Inference routing and IL backends? Start with Sovereign Inference
• Choosing between workstations, a shared Mac Studio, or GPU nodes? See Choosing Where Inference Runs
• Setting up an AI workstation? Start with Ollama on Apple Silicon
• Deploying to production K8s? Start with vLLM Production Inference
• Choosing a local model? See the Model Selection Guide