Secrets Management (Vault + ESO)

How the Federal Frontier Platform manages secrets — HashiCorp Vault as the backend, External Secrets Operator materializing Kubernetes Secrets, and ArgoCD GitOps holding only pointers, never values.

Secrets management

The Federal Frontier Platform keeps secret values out of Git entirely. HashiCorp Vault is the system of record for secrets; the External Secrets Operator (ESO) pulls them into Kubernetes Secrets on demand; and Git (Gitea, reconciled by ArgoCD) holds only pointers — which secret to fetch and where to put it — never the secret itself.

The rule: a plaintext secret must never appear in a Git repository or a Helm values file. If a workload needs a secret, it gets an ExternalSecret that references a path in Vault.

Why this architecture

• GitOps without secrets in Git. Everything else on the platform is declarative in Gitea. Secrets are the one thing that cannot be — so ESO bridges the gap: the declaration (an ExternalSecret) is in Git, the value lives in Vault.
• Central rotation and revocation. Rotate a secret in Vault and ESO re-materializes it on its refresh interval; no commits, no redeploys.
• Auditability and least privilege. Vault policies scope exactly which paths a consumer can read; Kubernetes auth ties that to a specific ServiceAccount.

Components

All three are deployed via ArgoCD — Vault and ESO as Helm-chart Applications, the ESO configuration (store + external secrets) as a kustomize Application.

How a secret flows

The diagram below traces the Grafana OIDC client secret end to end — the concrete example wired up on the platform. Git holds the ExternalSecret pointer; the value lives in Vault; ESO bridges them; Grafana consumes an ordinary Kubernetes Secret and never knows Vault exists.

The workload is unaware of Vault — it consumes an ordinary Kubernetes Secret. Only ESO talks to Vault. The secret value never appears in Git: Gitea holds only the ExternalSecret (which Vault path to read) and the ClusterSecretStore (how to reach Vault).

Vault

• Storage: single-node Raft on a ceph-rbd PVC. Scales to a 3-node HA quorum later by raising replicas.
• Secrets engine: KV v2 mounted at secret/.
• Agent injector: disabled — secret delivery is ESO’s job, not Vault sidecars.
• Sealing: initialized with Shamir key shares (5 shares, threshold 3). Unseal keys and the root token are held out-of-band — never in Git or a cluster Secret. After a Vault pod restart it must be unsealed with 3 of the 5 keys.

Operational note: with single-node manual unseal, a Vault restart leaves Vault sealed until an operator provides the unseal keys. Auto-unseal (cloud KMS or Vault transit) is a tracked follow-up; in an air-gapped federal context the transit option is preferred over a cloud KMS.

Kubernetes authentication

ESO authenticates to Vault with its own ServiceAccount token using Vault’s Kubernetes auth method:

• Auth method kubernetes/ is configured with the in-cluster API (https://kubernetes.default.svc:443). Vault validates incoming ServiceAccount tokens via the TokenReview API (its ServiceAccount holds system:auth-delegator).
• A Vault policy eso-read grants read on secret/data/* and secret/metadata/*.
• A Vault role eso binds that policy to the external-secrets ServiceAccount in the external-secrets namespace.

External Secrets Operator

The ClusterSecretStore vault-backend points ESO at Vault:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: http://vault.vault.svc:8200
      path: secret          # KV v2 mount
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

An ExternalSecret then materializes a Kubernetes Secret from a Vault path. Example — the Grafana OIDC client secret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: grafana-oidc
  namespace: monitoring
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-backend
  target:
    name: grafana-oidc        # the K8s Secret ESO creates
    creationPolicy: Owner
  data:
    - secretKey: client_secret
      remoteRef:
        key: monitoring/grafana-oidc   # path under the KV v2 mount
        property: client_secret

Network policy dependency

ESO lives in external-secrets and must reach Vault in vault. Because every namespace is locked by a default-deny-ingress policy (see Network Policy Architecture), an explicit allow is required or ESO login fails with context deadline exceeded:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-eso-to-vault
  namespace: vault
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: external-secrets
      ports:
        - protocol: TCP
          port: 8200

Onboarding a new secret

1. Write the value into Vault (out-of-band, not in Git): vault kv put secret/<area>/<name> key1=value1 key2=value2
2. Add an ExternalSecret in the consuming namespace referencing <area>/<name> and the properties you need, with target.name set to the Kubernetes Secret name the workload expects.
3. Commit the ExternalSecret to Gitea (the ESO-config kustomize overlay). ArgoCD applies it; ESO materializes the Secret within the refresh interval.
4. Confirm: kubectl get externalsecret -n <ns> shows SecretSynced=True and kubectl get secret -n <ns> <name> exists.

Verifying health

# Vault sealed/unsealed + HA status
kubectl exec -n vault vault-0 -- vault status

# ClusterSecretStore reachable & authenticated
kubectl get clustersecretstore vault-backend \
  -o jsonpath='{.status.conditions[0].type}={.status.conditions[0].status}'

# All external secrets in a namespace
kubectl get externalsecret -n <ns>

Current scope & follow-ups

Materialized today: the Grafana OIDC client secret and the Grafana break-glass admin credentials (both in monitoring). The pattern extends to every platform secret — migrating the remaining imperatively-created Secrets into Vault/ESO is ongoing.

Tracked follow-ups: Vault auto-unseal (transit/KMS), TLS on the Vault listener, and HA (3-node Raft quorum).