Choosing Where Inference Runs: Workstations, Shared Mac Studio, or GPU Nodes
Placement guidance for self-hosted LLM inference on the Federal Frontier Platform — default to shared inference behind llm-router and Compass rather than a GPU in every workstation.
This page explains where self-hosted (“local”) LLM inference should run for a Federal Frontier team, and why the default is a shared inference service on the network rather than a GPU in each workstation. It applies wherever the platform runs: on Amazon EKS in AWS, and on RKE2 clusters built with Cluster API Provider OpenStack (CAPO).
Every claim on this page carries one of three labels:
| Label | Meaning |
|---|---|
| Tested | Built and exercised on Federal Frontier infrastructure; date and environment given. |
| Design guidance | Our recommended approach and reasoning. Not a measured result. |
| Not yet measured | A question we have not benchmarked. No numbers are given. |
Summary
Design guidance. Default to shared inference on the network. Start small teams on a shared Mac Studio behind llm-router, then move to an NVIDIA GPU node on Kubernetes (RKE2 via CAPO, or Amazon EKS) when demand grows. Clients use the same OpenAI-compatible API in both stages, so the move does not change how users or agents connect. Put GPUs in individual workstations only for users with a specific need: disconnected operation, strict latency, or data that must stay on the device.
Why shared inference is the default
Utilization
Design guidance. A GPU in a workstation is used only while its owner is actively prompting, which is a small share of the working day. Pooling the same budget into one shared system lets a team run a larger, more capable model than any single desktop could hold, and keeps the hardware busy across many users.
Governance and audit
Shared inference sits behind llm-router and Compass, which gives one place to enforce and record access:
- Identity and per-role model rules — model access is tied to Keycloak (CNCF Incubating) roles and is deny-by-default: a role reaches only the models it is explicitly allowed. Tested: the llm-router and Compass role rules are built and enforced.
- Logging and audit — requests pass through a single controlled path that can be logged and reviewed. Design guidance.
Design guidance. For Controlled Unclassified Information (CUI), demonstrating access control and auditability across one governed endpoint is far simpler than doing so across many independently managed desktop models.
The placement tiers
Shared Mac Studio — small-team starting tier
Design guidance. A Mac Studio configured with up to 512 GB of unified memory can hold large models that would otherwise need multiple data-center GPUs. It is a low-friction first shared endpoint for a small team. Its weakness is concurrency: it is less efficient at batching many simultaneous requests than vLLM on NVIDIA GPUs, so it suits a small group of interactive users rather than many parallel agent workloads.
Not yet measured. We have not benchmarked how many concurrent users a Mac Studio supports before latency becomes unacceptable. This page gives no user-count threshold. Treat rising queue times or user complaints as the signal to plan the next tier.
NVIDIA GPU node on Kubernetes — department scale and agent workloads
Design guidance. For department-scale use, and for agent workloads such as SRE runners that issue many requests in parallel, run inference on an NVIDIA GPU node in a Kubernetes cluster (RKE2 built with CAPO, or Amazon EKS), served by vLLM. llm-d (CNCF Sandbox) adds Kubernetes-native distributed inference and request routing on top of vLLM for larger deployments. Deployments are managed through GitOps with Argo CD (CNCF Graduated), consistent with the rest of the platform.
Tested (2026-10-21). Production-configuration validation on a single GPU node (one T4, AWS bare metal), running TinyLlama 1.1B on vLLM v0.9.2 on RKE2 via CAPO: a single request ran at about 71–96 tokens/s, and aggregate throughput reached about 620 tokens/s with 8 concurrent requests and about 1,160 tokens/s with 16 (about 1,170 tokens/s over a 48-request run at 16 concurrent).
These figures show the batching behavior that makes a GPU node the right tier for many parallel requests: aggregate throughput rises roughly in line with concurrency up to 16 while each request continues to be served. They come from a small model on a single T4; results with larger models, on larger GPUs, or across more nodes are Not yet measured.
Workstation GPUs and Apple-silicon laptops — exceptions
Design guidance. Reserve dedicated workstation GPUs for users who:
- must work disconnected from the network,
- need latency the shared service cannot provide, or
- are required to keep data on the device.
Apple-silicon laptops already run small models locally and cover lightweight, on-device needs without extra hardware.
Decision diagram
Design guidance.
need on-device latency,
or keep data on device?} B -- Yes --> C{Small model sufficient?} C -- Yes --> D[Apple-silicon laptop
local small model] C -- No --> E[Dedicated workstation GPU
exception, approved per user] B -- No --> F[Shared inference
behind llm-router + Compass] F --> G{Small team,
interactive use only?} G -- Yes --> H[Shared Mac Studio] G -- No: department scale
or parallel agent workloads --> I[NVIDIA GPU node
RKE2 via CAPO or Amazon EKS
vLLM / llm-d] H -- Demand grows --> I
Recommended path
Design guidance.
- Start with a shared Mac Studio behind llm-router, with Keycloak identity and deny-by-default role rules in Compass.
- Graduate to an NVIDIA GPU node on RKE2 (via CAPO) or Amazon EKS running vLLM, adding llm-d as scale requires.
- Keep the API constant. Both tiers expose the same OpenAI-compatible API through llm-router, so users, Compass, and agents do not change their configuration when the backend moves.