Synced from Hive. This page is pulled from hivecommons/hive@v5 during the docs build. Edit the canonical source in the Hive repository.

Provisioning a Hosted Hive

This is a Kubernetes operator runbook, including a separate standalone path. If you only want a hosted dashboard, use Hosted Hive Hub onboarding: request a hive and wait for an administrator to provision it. You do not need cluster credentials.

Choose one provisioning task

Use only the prerequisites for your chosen task. A heartbeat is an outbound status report from a hive to its hub; a heartbeat-only cluster can send those reports even though the hub cannot run kubectl against it.

The manifests and gotchas below record a ten-placeholder deployment across hub-reachable and heartbeat-only clusters. Its example storage classes, inference placement, and OpenShift routing are not requirements for every deployment.

Reading route: complete your chosen path, verify it, then return to Getting started. Config precedence applies to all Kubernetes paths. Placeholder pools, identity labels, and the failure-mode reference are advanced administration/recovery, not first-install steps.

Background: how the two clusters differ

The provisioning path is dictated entirely by whether the hub can kubectl to the target cluster.

Why methods map to clusters. Public-method hives run subscription CLIs and need no in-cluster inference, so they live on the hub’s own (hub-reachable) cluster. Private-method hives point at a self-hosted inference endpoint that lives on the heartbeat-only cluster, so the hive runs there too — and that cluster is heartbeat-only, which is exactly why it needs the manual path.

Prerequisites (both paths)

Hosted Paths A and B only. Standalone operators skip this section. You need the context for each cluster you actually administer; two contexts are needed only when target and hub are on different clusters.

• kubectl access to the hub cluster for Path A; for Path B, access to the heartbeat-only target as well as the hub cluster.
• The hub is running on the hub-reachable cluster in namespace hive-hub, backed by a RWX PVC named hive-hub-data-rwx mounted at /data. The hub’s SaaS store lives at /data/saas/hives/<id>/meta.json; its fleet registry at /data/hub-registry.json.
• For manual (heartbeat-only cluster) provisioning: the system:openshift:scc:anyuid ClusterRole must exist (it does by default on OpenShift).

Standalone / self-hosted (no hub)

This section is the complete path for the hub-less scenario: an operator (“Joe”) running their own hive on their own firewalled cluster, against their own repos, with self-hosted inference (litellm / vllm / llm-d) and no attachment to hub.hivecommons.dev. Everything after this section (Path A, Path B, meta.json, placeholder pools, claiming, upgrade-via-hub) is hub-only and does not apply — skip it.

Required order: check the swap list, NET_ADMIN decision, firewall, and agent backend; then edit, render, and apply the overlay. Verify with Standalone completion checks. The private-repository examples are optional configuration-management reference, not a second installation you must perform.

What “standalone” means

A standalone hive has no hub. In src/pkg/config/config.go the hub: block is a HubConfig whose zero value has Enabled: false, so a config that simply omits hub: is already hub-less. Concretely, a standalone hive needs none of the hub plumbing the rest of this guide describes:

• No HIVE_HUB_SECRET, no HIVE_HUB_URL, no derived HIVE_HEARTBEAT_KEY / HIVE_SESSION_KEY / HIVE_SSO_KEY — those authenticate a spoke to a hub; a hive with nothing to heartbeat to needs none of them.
• No meta.json / My-Hives / hub SaaS record — there is no hub SaaS store.
• No claim / placeholder-pool / Upgrade steps — placeholders and hub-driven auto-upgrade are hub features. You upgrade the image yourself (below).

Because there is no hub auth proxy in front of it, a standalone hive enforces its own dashboard login: dashboard.hub_proxied: false (the default) plus a dashboard.authorized_users allowlist and a github.oauth_client_id for the device-flow login. All three are covered in the swap checklist.

Install method: Kustomize

Use the Kustomize overlays — a peer already built a base and a standalone overlay, so this is the least-effort, git-trackable path (no operator, no Helm values to reverse-engineer). The overlay layers two in-repo bases — the core workload (src/deploy/k8s) and an in-cluster OpenAI-compatible inference backend (src/deploy/inference, llama.cpp serving Qwen2.5-0.5B) — so you get a working model endpoint out of the box. See the overlay and its README here:

• Standalone overlay — github.com/hivecommons/hive/tree/v4/src/deploy/kustomize/overlays/standalone (annotated patch-configmap.yaml, patch-pvc-storageclass.yaml, patch-advisory-mode.yaml, and a README.md with the full swap list).
• Filled-in example (“Joe on Spyre”) — github.com/hivecommons/hive/tree/v4/src/deploy/kustomize/overlays/standalone/example-joe-spyre. Every value there is a placeholder — copy the shape, don’t apply it verbatim.

Both flows need the Gateway API Inference Extension CRDs on the cluster first. The inference base ships an InferencePool (inference.networking.k8s.io/v1), and without the CRDs kubectl apply -k creates everything else and then fails with no matches for kind "InferencePool".

There are two flows. Pick based on whether you just want to see it run or you’re doing a real deployment.

1. Remote quickstart (see the shape). Kustomize can build straight from GitHub — no clone:

kubectl apply -k "https://github.com/hivecommons/hive//src/deploy/kustomize/overlays/standalone?ref=v5"

(Note the // between the repo and the sub-path — kustomize needs it to find the directory inside the repo; a single slash does not resolve.)

This applies the overlay with its PLACEHOLDER values (YOUR_ORG, YOUR_OAUTH_CLIENT_ID, YOUR_RWX_STORAGE_CLASS, …). It is only for seeing the manifest shape come up — it will not be a working hive, because the repo, storage class, OAuth app, and inference endpoint are all placeholders you must swap first.

2. Real path (clone → edit → apply). This is the path you actually deploy, because you must swap values before it works:

git clone -b v5 https://github.com/hivecommons/hive.git
cd hive/src/deploy/kustomize/overlays/standalone
# Edit the placeholders — see "What you must swap" below:
#   patch-configmap.yaml       (org/repos, owner login, OAuth client id, litellm endpoint)
#   patch-pvc-storageclass.yaml (your storage class — RWO is fine, but network-attached, not local-path)
# Review the rendered manifests, then apply:
kubectl kustomize .
kubectl apply -k .
kubectl -n hive rollout status deploy/hive
kubectl -n hive-inference rollout status deploy/vllm

Upgrading the image. The base tracks ghcr.io/hivecommons/hive:stable. To pin (so upgrades are deliberate — there is no hub to auto-upgrade you), resolve the channel you reviewed to a digest and write it in from the overlay directory, then re-apply:

TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:hivecommons/hive:pull" | jq -r .token)
DIGEST=$(curl -sI -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json" \
  https://ghcr.io/v2/hivecommons/hive/manifests/stable \
  | awk 'tolower($1)=="docker-content-digest:" {print $2}' | tr -d '\r')
kustomize edit set image ghcr.io/hivecommons/hive=ghcr.io/hivecommons/hive@"$DIGEST"
kubectl apply -k .

Which image tags exist. ghcr.io/hivecommons/hive carries the channel tags (stable, candidate, edge, v4-latest) and a short-SHA tag per merge. A vX.Y.Z image tag exists only when the automated tagged-release workflow (.github/workflows/tagged-release.yml, see Tagged releases) has cut that version — it retags the merge’s short-SHA images. That workflow landed after the v4.0.0 git tag, so there is no :v4.0.0 image; newTag: v4.0.0 is an ImagePullBackOff. Pin a version tag only after confirming it on the Releases page or with curl -sI -H "Authorization: Bearer $TOKEN" https://ghcr.io/v2/hivecommons/hive/manifests/vX.Y.Z (200 = exists). Digests always work.

Storage. The base hive-data PVC is ReadWriteOnce. That is correct for the single-replica standalone hive — any network-attached block storage class works; you do not need an RWX class. (RWX is only relevant to the hub-provisioned path further down, whose template runs a surge rollout.) What you must avoid on a multi-node cluster is a node-local provisioner — see the next section.

Multi-node clusters: do not put /data on node-local storage

/data is the hive’s whole state (agent homes, config overlay, audit log, knowledge). Where it lives decides which nodes the pod can run on:

Check before applying:

kubectl get storageclass
# A production spoke wants a network-attached class, ideally with allowVolumeExpansion: true.
# If the only class is local-path / openebs-hostpath, install a CSI driver for your cloud
# first — or accept that the hive is pinned to one node and size that node's disk for it.

If a spoke is already on local-path and must move: scale the deployment to 0, create a new PVC on the network class, copy with a one-off pod that mounts both (kubectl run … --overrides or a small Job running cp -a /old/. /new/), point the Deployment’s hive-data volume at the new claim, scale back up, and delete the old PVC once the dashboard shows the agents and config intact. The hive does not yet split its stateless API from its stateful agent workers, so one pod still carries both; keeping that one pod off node-local storage is what lets the scheduler place it where there is room.

On OpenShift. The standalone overlay does not create a Route or grant any SCC. Two more pieces supply the OpenShift-only deltas:

• The overlays/openshift overlay — a Route exposing the dashboard port (3002) and the anyuid SCC RoleBinding (the pod starts as root to set up the ACMM iptables and chown the PVC).
• The overlays/openshift-netadmin overlay — the dedicated hive-netadmin SCC + RoleBinding, applied once by a cluster-admin. This one is not optional on OpenShift: the base deployment adds capabilities (NET_ADMIN + the su-exec set), and no stock SCC — anyuid included — permits adding them, so without this grant the pod is rejected at admission (unable to validate against any security context constraint). The SCC is restricted-v2 loosened by exactly what the base deployment declares (see the manifest comments) — not privileged. See Does your cluster grant NET_ADMIN? to check your cluster first.

A standalone deployment on OpenShift composes standalone + these platform deltas (the SCC grant is applied separately, by cluster-admin: oc apply -k src/deploy/kustomize/overlays/openshift-netadmin). For the app-level overlays, two equivalent ways:

• Combined overlay (recommended): in your own copy of the standalone kustomization.yaml, add route.yaml and anyuid-scc-rolebinding.yaml as additional resources — this is exactly what the example-joe-spyre overlay does (it adds its own route.yaml; add the SCC binding the same way). Set the Route spec.host to your cluster’s apps domain (or drop host to let OpenShift auto-generate one).
• Apply both: kubectl apply -k <standalone> then kubectl apply -k <openshift> — but note the openshift overlay re-includes the src/deploy/k8s base, so prefer the combined form to avoid re-applying the base twice.

Keeping your overlay private

my-hive-config/                     ← your private repo
├── overlays/
│   └── spyre/                      ← one subdirectory per cluster/environment
│       ├── kustomization.yaml      ← references the public base by URL
│       ├── patch-configmap.yaml    ← your real org/repo/OAuth values
│       ├── patch-pvc-storageclass.yaml
│       └── route.yaml              ← OpenShift only; your cluster's apps domain
└── README.md                       ← internal runbook: how to install/upgrade

overlays/
├── spyre-staging/
│   ├── kustomization.yaml
│   └── patch-configmap.yaml
└── spyre-prod/
    ├── kustomization.yaml
    └── patch-configmap.yaml

# overlays/spyre/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  # ── Public hive standalone overlay, pinned to a reviewed git ref ──────────
  # kubectl/kustomize fetches this from GitHub at apply time — no local clone.
  # Bump the ref= value in a git commit when you want to upgrade; review the
  # diff of what changed upstream before you do. A git TAG (v5.186.4) or a full
  # commit SHA is immutable; a branch (v5) moves under you.
  - https://github.com/hivecommons/hive//src/deploy/kustomize/overlays/standalone?ref=v5.186.4
  #
  # OpenShift only: add your Route here alongside the remote base.
  # On plain Kubernetes, use an Ingress resource instead.
  - route.yaml
 
# Pin the image for extra safety. Without this the base tracks
# `ghcr.io/hivecommons/hive:stable`; pinning means upgrades are deliberate —
# you bump this in git, review, and apply. Pin by DIGEST: a git tag does not
# imply an image tag (there is no `:v4.0.0` image — see "Upgrading the image"
# above for how to resolve `stable` to a digest and how to check whether a
# `vX.Y.Z` image tag exists before using one).
images:
  - name: ghcr.io/hivecommons/hive
    newName: ghcr.io/hivecommons/hive
    digest: sha256:<digest-you-resolved-from-stable>
 
patches:
  - path: patch-configmap.yaml
  - path: patch-pvc-storageclass.yaml
  # Uncomment ONLY if your cluster denies NET_ADMIN — see "The NET_ADMIN
  # decision" section below. Leaving this commented keeps the fail-closed gate.
  # - path: patch-advisory-mode.yaml

# overlays/spyre/patch-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hive-config
  namespace: hive
data:
  hive.yaml: |
    project:
      org: your-github-org             # the GitHub org (or user) that owns your repos
      repos:
        - your-repo                    # repo(s) hive manages; list all of them
      ai_author: your-bot-login        # GitHub login agents use when authoring PRs
      primary_repo: your-repo          # must appear in repos: above
 
    acmm_level: 2                      # L2 = Advisory (Instructed); no new issues or PRs
 
    hub:
      enabled: false                   # standalone; no hub connection
 
    agents:
      supervisor:
        enabled: true
        backend: litellm               # routes to governor.litellm.endpoint below
        beads_dir: /data/beads/supervisor
        clear_on_kick: true
      scanner:
        enabled: true
        backend: litellm
        beads_dir: /data/beads/scanner
        clear_on_kick: true
      architect:
        enabled: true
        backend: litellm
        beads_dir: /data/beads/architect
        clear_on_kick: false
 
    governor:
      eval_interval_s: 300
      litellm:
        endpoint: http://litellm-svc.llm-gateway.svc.cluster.local:4000
        api_key_env: HIVE_LITELLM_API_KEY
        default_model: your-model-name
 
    dashboard:
      port: 3002
      hub_proxied: false
      authorized_users:
        - your-github-login:owner      # first entry is the owner; add more as :member
 
    github:
      oauth_client_id: YOUR_OAUTH_CLIENT_ID   # from your GitHub OAuth App settings

# overlays/spyre/patch-pvc-storageclass.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hive-data
  namespace: hive
spec:
  storageClassName: ocs-storagecluster-cephfs   # ← a storage class on your cluster

kubectl get storageclass
# Look for one marked (default) or annotated storageclass.kubernetes.io/is-default-class=true
# The base PVC is ReadWriteOnce, so any NETWORK-ATTACHED block class works (AKS managed-csi, EBS, RBD, ...).
# Avoid node-local classes (local-path, openebs-hostpath): they pin the pod to one node and cannot
# be expanded — see "Multi-node clusters: do not put /data on node-local storage" above.
# On Spyre/ODF clusters, ocs-storagecluster-cephfs (RWX) also works — RWX is allowed, not required.

# overlays/spyre/route.yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: hive
  namespace: hive
spec:
  host: hive.apps.your-cluster.example.com    # ← your cluster's apps domain
  port:
    targetPort: dashboard
  to:
    kind: Service
    name: hive
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

# GitHub App PEM + OAuth client secret
kubectl create secret generic hive-secrets -n hive \
  --from-file=github_app_private_key=./your-app.pem \
  --from-literal=HIVE_GITHUB_OAUTH_CLIENT_SECRET=your-oauth-secret
 
# If your LiteLLM endpoint requires an API key:
kubectl patch secret hive-secrets -n hive \
  --type=merge \
  -p '{"stringData":{"HIVE_LITELLM_API_KEY":"your-litellm-key"}}'

# Install controller once (cluster-admin):
kubectl apply -f https://github.com/bitnami/sealed-secrets/releases/latest/download/controller.yaml
 
# Seal your secret:
kubectl create secret generic hive-secrets -n hive \
  --from-file=github_app_private_key=./your-app.pem \
  --from-literal=HIVE_GITHUB_OAUTH_CLIENT_SECRET=your-oauth-secret \
  --dry-run=client -o yaml \
  | kubeseal --controller-namespace kube-system -o yaml \
  > overlays/spyre/sealed-hive-secrets.yaml
 
# Add to kustomization.yaml:
#   resources:
#     - sealed-hive-secrets.yaml
git add overlays/spyre/sealed-hive-secrets.yaml
git commit -m "Seal hive-secrets for spyre cluster"

# Clone your private repo (not the hive repo):
git clone git@github.example.com:your-org/my-hive-config.git
cd my-hive-config
 
# Preview — see every manifest kustomize will send to the cluster:
kubectl kustomize overlays/spyre | less
 
# Create the secret first (Option 1 above), then apply:
kubectl apply -k overlays/spyre/
kubectl -n hive rollout status deploy/hive     # wait for Ready
kubectl -n hive-inference rollout status deploy/vllm     # inference backend

# Edit overlays/spyre/patch-configmap.yaml, then:
git add overlays/spyre/patch-configmap.yaml
git commit -m "Add jane to authorized_users"
git push
kubectl apply -k overlays/spyre/
# This updates the ConfigMap seed, not an existing hive's effective config.
# A rollout restart alone still boots from /data/hive.yaml.runtime.

# 1. Read the release notes for the new tag:
#    https://github.com/hivecommons/hive/releases
 
# 2. Bump the ref and the image pin in kustomization.yaml:
#    resources:
#      - https://github.com/hivecommons/hive//src/deploy/kustomize/overlays/standalone?ref=<new git tag or sha>
#    images:
#      - name: ghcr.io/hivecommons/hive
#        digest: sha256:<digest resolved from stable, or from the release's image tag>
#    (use `newTag: vX.Y.Z` only after confirming that IMAGE tag exists — see
#    "Which image tags exist" above)
 
# 3. Preview the diff — see exactly what changes upstream:
kubectl kustomize overlays/spyre | diff - <(kubectl kustomize overlays/spyre 2>/dev/null) || true
# Or simpler: use `kustomize build` and diff the YAML files manually.
 
# 4. Commit and apply:
git add overlays/spyre/kustomization.yaml
git commit -m "Upgrade hive to v4.3.0"
git push
kubectl apply -k overlays/spyre/
kubectl -n hive rollout status deploy/hive

git revert HEAD   # or git checkout <old-sha> -- overlays/spyre/kustomization.yaml
kubectl apply -k overlays/spyre/

# One-time: clone the public hive repo at the pinned tag and copy just the
# kustomize overlay you need into a vendor/ directory:
git clone -b v4.2.0 https://github.com/hivecommons/hive.git hive-upstream
cp -r hive-upstream/src/deploy/kustomize overlays/spyre/vendor/
rm -rf hive-upstream
 
# Update kustomization.yaml to use the local path instead of the URL:
#   resources:
#     - vendor/kustomize/overlays/standalone   ← local, no network needed
 
git add overlays/spyre/vendor/
git commit -m "Vendor hive kustomize base at v4.2.0"

What you must swap

Every value below is a placeholder in the overlay. Cited location is where the key is defined (config struct) and where you edit it (overlay file).

Also populate the base hive-secrets Secret with a GitHub App PEM or HIVE_GITHUB_TOKEN, and — if your inference endpoint needs auth — the key named by governor.litellm.api_key_env (default HIVE_LITELLM_API_KEY).

The NET_ADMIN decision (stated precisely)

The base deployment already requests the NET_ADMIN capability (src/deploy/k8s/deployment.yaml) — the entrypoint uses it to install the forced-proxy-egress iptables REDIRECT that enforces the ACMM MITM egress proxy.

• Cluster GRANTS NET_ADMIN: retain that grant and verify the full, fail-closed gate starts. The node also needs required netfilter modules and working iptables tools; the capability alone is not sufficient.
• Cluster DENIES NET_ADMIN: the entrypoint FATALs (refusing to start with an advisory-only capability model — src/deploy/entrypoint.sh) and the pod crash-loops, with exit code 77 when the missing capability is the cause (see the checks section below for what 77 means and how to read it; since #6003 the same code also fires when the node’s kernel lacks a netfilter module the gate needs, which neither remedy here fixes). Two remedies:
  1. Grant the capability (preferred — full gate). On OpenShift, a cluster-admin applies the overlays/openshift-netadmin overlay (dedicated hive-netadmin SCC + RoleBinding); on plain Kubernetes, ensure the namespace’s Pod Security admission allows the capability (see below).
  2. Deliberately run with advisory-only egress. Set HIVE_PROXY_ADVISORY_OK=true via the commented-out patch-advisory-mode.yaml — uncomment its - path: patch-advisory-mode.yaml line in your kustomization.yaml.

Security trade-off (one honest sentence): in advisory-only mode forced proxy egress becomes advisory — the MITM egress gate is best-effort rather than enforced, so an agent holding a raw token could bypass the proxy policy — an explicit risk decision even on a single-tenant hive, and not a control you should silently disable on a shared one. This is different from degradation of the proxy’s own connection exemption; see the capability decision table.

Does your cluster grant NET_ADMIN?

The question every firewalled-cluster adopter hits first. Three checks, from authoritative to empirical — run them before deploying and you’ll know which of the two remedies above (if either) you need.

1. Ask OpenShift directly (scc-subject-review dry-run). This is the authoritative answer: it evaluates the actual pod spec against every SCC the hive ServiceAccount can use, without creating anything:

# From your (placeholder-swapped) overlay directory — build the real manifests
# and let OpenShift judge them as the hive SA:
kubectl kustomize . > /tmp/hive-rendered.yaml
oc -n hive policy scc-subject-review -z hive -f /tmp/hive-rendered.yaml

Read the ALLOWED BY column for the hive Deployment: an SCC name (e.g. hive-netadmin) means admission will pass under that SCC; <none> means no SCC admits the pod — deploying now would leave the ReplicaSet stuck with unable to validate against any security context constraint events and no pod at all.

2. Which SCCs allow the capability, and who can use them?

# SCCs that would permit adding NET_ADMIN (allowedCapabilities contains it or "*"),
# plus privileged SCCs (which imply all capabilities):
oc get scc -o json | jq -r '.items[]
  | select(((.allowedCapabilities // []) | index("NET_ADMIN") or index("*"))
           or .allowPrivilegedContainer == true)
  | .metadata.name'
 
# For each candidate, is the hive SA allowed to use it?
oc adm policy who-can use scc/hive-netadmin
# Look for system:serviceaccount:hive:hive in the output.

On a stock cluster the first query returns only privileged / node-exporter style SCCs the hive SA cannot (and should not) use — that is the “cluster denies” case until hive-netadmin is applied. On plain Kubernetes (no SCC API) the equivalent gate is Pod Security admission: check kubectl get ns hive -o jsonpath='{.metadata.labels}' — an pod-security.kubernetes.io/enforce: baseline (or restricted) label denies NET_ADMIN; the capability needs privileged enforcement (or no enforcement) on the namespace.

3. The 30-second probe pod (empirical). Runs as the same ServiceAccount, requests only NET_ADMIN, prints its own capability bounding set, and exits:

oc -n hive run netadmin-probe --restart=Never --image=busybox:1.36 \
  --overrides='{"spec":{"serviceAccountName":"hive","containers":[{"name":"netadmin-probe","image":"busybox:1.36","command":["sh","-c","grep CapBnd /proc/self/status"],"securityContext":{"capabilities":{"drop":["ALL"],"add":["NET_ADMIN"]}}}]}}'
oc -n hive logs netadmin-probe && oc -n hive delete pod netadmin-probe

Two outcomes: the run is rejected at admission (unable to validate against any security context constraint) — the cluster denies the capability; or the pod runs and prints CapBnd: 0000000000001000 — bit 12 (0x1000, CAP_NET_ADMIN) is set and the cluster grants it. (The probe requests only NET_ADMIN; the real deployment also adds the su-exec capability set, so treat check 1 as the authoritative pre-flight and this probe as the quick smoke test.)

Reading the failure after the fact — the exit-77 signature. If the pod was admitted but the container’s bounding set still lacks CAP_NET_ADMIN (e.g. an admission webhook stripped the capability, or a hand-edited manifest dropped the request), the entrypoint refuses to start and exits with a distinct code, 77 (sysexits.h EX_NOPERM), after logging:

[entrypoint] FATAL: refusing to start. Grant NET_ADMIN + install iptables, or set HIVE_PROXY_ADVISORY_OK=true to deliberately run in advisory mode.
[entrypoint] FATAL: CAP_NET_ADMIN is not in the container's capability bounding set — exiting 77 (EX_NOPERM) rather than 1.

kubectl -n hive get pod -l app.kubernetes.io/name=hive \
  -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.exitCode}'

77 has exactly two causes, and only one of them is the capability. Since #6003 the entrypoint probes the kernel’s netfilter extension modules in a throwaway chain before building the real ruleset, and a node whose kernel is missing xt_mark (the packet-mark exemption) or xt_REDIRECT (the :443 redirect) also exits 77, after a different set of lines:

[entrypoint] ERROR: netfilter REDIRECT target unavailable on this node (kernel module xt_REDIRECT): ...
[entrypoint] FATAL: this node's kernel is missing netfilter module(s) required by the forced-egress gate: xt_REDIRECT.
[entrypoint] FATAL: load the module(s) on this node - the durable fix is a MachineConfig writing an /etc/modules-load.d/ drop-in ...
[entrypoint] FATAL: exiting 77 (EX_NOPERM) rather than 1 - see EXIT_NET_ADMIN_REQUIRED near the top of entrypoint.sh.

Tell them apart by the log, not the code: the module case names the module and never prints Grant NET_ADMIN; the capability case prints Grant NET_ADMIN and then the bounding-set line. Granting NET_ADMIN does not fix the module case (the capability is already there), and the SCC overlay above is the wrong remedy for it: load the modules on the node (on OpenShift/RHCOS, a MachineConfig writing an /etc/modules-load.d/ drop-in so they survive a node rebuild) and taint or label the node until then. xt_owner is probed too but only warns. HIVE_PROXY_ADVISORY_OK=true is the opt-out for both causes. Any other cause of the same FATAL (no iptables binary, netfilter lock contention) still exits 1, since neither fix would help those. Full rationale and the check order: net-admin-requirement.md.

Firewall / egress (locked-down cluster)

A standalone hive needs no hub egress. The minimal outbound allowlist is:

• Your repo’s GitHub API host — api.github.com, or your GHE instance’s <host>/api/v3.
• ghcr.io (and its blob backend) — for the image pull.
• Your inference endpoint — if it’s in-cluster (the shipped src/deploy/inference backend, or your own litellm/vllm Service) this is cluster-internal traffic, not external egress at all.

Hub hosts (hub.hivecommons.dev) are not required. For the full egress matrix (ports, hosts, when each is needed) see network-requirements.md; for the capability rationale see net-admin-requirement.md.

Agent backend

Standalone uses a self-hosted agent backend, not the subscription-CLI backends (copilot / claude) the internal fleet’s base ConfigMap uses. The overlay sets each agent’s backend: litellm (valid enum values: litellm | vllm | llm-d | watsonx). The litellm connection config lives under governor: (LiteLLMConfig on Config.Governor):

governor:
  litellm:
    endpoint: http://vllm-svc.hive-inference.svc.cluster.local:8000  # your OpenAI-compatible proxy
    api_key_env: HIVE_LITELLM_API_KEY   # env var NAME holding the key (default)
    default_model: qwen2.5-0.5b-instruct

Swap backend: vllm to hit HIVE_VLLM_ENDPOINT directly. Hive does not default this to an in-cluster Service; set it only when the target cluster can resolve and reach that endpoint (hosted provisioning injects it from the cluster inference_endpoint setting). Verify any key against src/pkg/config/config.go before adding it.

Standalone completion checks

• Both kubectl -n hive rollout status deploy/hive and the selected inference backend’s rollout succeed. Inspect logs for authentication or egress errors; a running pod is not proof the backend works.
• Open the dashboard using your configured Route/Ingress, sign in as an allowed owner, and verify the intended repository and model endpoint.
• Confirm L2 — Advisory in the dashboard. L2 workers report findings without creating issues or PRs; reviewer can comment on existing PRs. See the current policy matrix.
• Save one intended setting through the dashboard, restart only in a planned window, and verify it persists. Do not edit the ConfigMap expecting a running hive to reload it.
• No hub heartbeat or My Hives entry is expected. Continue with Getting started. If blocked, use the failure-mode reference or troubleshooting.

Path A — Automated provisioning (hub-reachable cluster)

The hub exposes POST /api/saas/hives (handleCreateHive). It generates the hive ID, creates the namespace + RBAC + PVC + Service + Ingress + Deployment, and writes the SaaS meta.json — all automatically.

A.1 Call the API as the hub admin

Who runs this: the provisioning operator with access to the hub pod. The API requires an authenticated, unblocked SaaS account with hosted-hive quota; localhost does not bypass authentication. A hive_hub_user cookie must be a valid signed session cookie, not a bare username. The example uses the supported GitHub bearer-token authentication instead. Supply a token for the intended authorized account without putting it in the command history; do not copy session cookies or tokens into manifests, tickets, or logs.

read -r -s -p "GitHub token for the authorized hub account: " HUB_API_TOKEN
printf '\n'
HUB_POD=$(kubectl --context <hub-reachable-cluster> -n hive-hub get pods -l app=hive-hub \
  --no-headers | grep Running | awk '{print $1}' | head -1)
 
kubectl --context <hub-reachable-cluster> -n hive-hub exec "$HUB_POD" -- curl -s -X POST \
  -H "Authorization: Bearer ${HUB_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
        "org": "myorg",
        "repos": "myrepo",
        "primary_repo": "myrepo",
        "project_name": "My Hive",
        "acmm_level": 2,
        "cluster_id": "<hub-reachable-cluster>",
        "auth_method": "app",
        "app_id": "999999999",
        "installation_id": "",
        "app_private_key": "",
        "is_public": false
      }' \
  http://localhost:80/api/saas/hives
unset HUB_API_TOKEN

Response:

{"id":"hosted-myorg-myrepo-ab12","status":"provisioning",
 "subdomain":"hosted-myorg-myrepo-ab12.hive.hivecommons.dev"}

The generated ID is hosted-<org>-<primary_repo>-<4char>.

app_id: 999999999 above is the placeholder sentinel, not an arbitrary number. It is config.PlaceholderAppID, and the spoke recognises it as “this hive has no GitHub App yet”. Use exactly this value — any other non-zero number is read as a real App ID and the hive will try (and fail) to authenticate as it. See Placeholder app_id below.

CreateHiveRequest fields that matter:

Gotcha — auth_method: app with no key provisions but reports error. The “app later” path leaves the hive with no valid credentials, so its meta.json status becomes "error" with "provisioning failed — check hub logs for details", and it 401-loops until the App is installed. This is expected for a not-yet-claimed hive. If you want it to read as available instead of error, patch the meta.json status (see Placeholder pools).

Path A completion checks

1. Record the returned hive ID and verify the workload reaches Ready on its registered target cluster.
2. Verify its SaaS record exists automatically and the intended owner sees the hive in My Hives.
3. Open that dashboard link; complete App/backend setup and confirm a recent heartbeat, correct dashboard_url, and reported version on the hub. An unclaimed placeholder may report the documented App-auth error until installation.
4. If the link is wrong or health is partial, use the RBAC reference below and the failure-mode table. Do not broaden tenant permissions to make a health badge green.

A.2 What the automated path creates that the manual path does not

The automated provisioner also provisions an OCI file-system export for the PVC and records it in meta.json (oci_file_system_id, oci_export_id). The manual path uses an in-cluster RWX PVC directly and has no OCI export. This is the main structural difference between the two meta.json shapes.

A.3 RBAC the template emits

The provisioning template creates the namespace RBAC for you, including the namespace-scoped read-only hive-route-reader Role and RoleBinding (get/list on networking.k8s.io/ingresses and route.openshift.io/routes).

It is not unused. The spoke reads its own Route/Ingress through it to learn the hostname it actually serves, and reports that to the hub as dashboard_url. Remove it and the hub falls back to synthesising <hiveID>.<hub host>, which 503s for any spoke not fronted by the hub’s wildcard domain — see hive-route-reader under the manual path for the full rationale, the YAML, and the ServiceAccount-derivation caveat.

The template also emits a read-only cluster-scoped hive-node-health-reader-* ClusterRole/ClusterRoleBinding so push-reported spokes can send node health in their outbound heartbeat. It grants only nodes get,list and metrics.k8s.io/nodes list. It must not grant nodes/proxy (cross-tenant kubelet exec) or cluster-wide pods list (other tenants’ pod specs). Live disk usage, pod/hive counts and request-based remaining hive capacity are unavailable with this safe tenant role; a pods-API 403 is expected partial health, not a reason to broaden access. If metrics-server is absent or that last rule is denied, the spoke still reports node count, vCPU, memory and disk capacity from the core Node API and carries the precise node_health_error reason to the hub.

The template binds hive-sa on RequiresSCC (OpenShift) clusters and default elsewhere. Namespaces provisioned before either reader was added do not have it and need it applied retroactively. Existing overbroad node-health roles are narrowed by the hub’s startup/15-minute reconciliation on kubectl-reachable clusters. On push-reported clusters, a cluster administrator must replace the role’s entire rules list with the safe rules below (do not merely append them), and inspect any orphaned hive-node-health-reader-* roles as well. A tenant pod upgrade does not revoke old ClusterRole permissions. For each affected namespace, an administrator with the cluster’s kubeconfig can revoke the legacy grants immediately (repeat with the appropriate NS on every hosting cluster):

kubectl --request-timeout=15s patch clusterrole "hive-node-health-reader-${NS}" \
  --type=merge -p '{"rules":[{"apiGroups":[""],"resources":["nodes"],"verbs":["get","list"]},{"apiGroups":["metrics.k8s.io"],"resources":["nodes"],"verbs":["list"]}]}'
kubectl get clusterrole "hive-node-health-reader-${NS}" -o yaml

Verify the resulting role has only the two node rules; it must have no nodes/proxy, pods, wildcard resources, or aggregation rule. Also check other bindings to the tenant ServiceAccount: narrowing this role cannot undo grants from independently installed roles.

Existing push-reported spokes also need the cluster identity that enables the heartbeat cluster_health block. If a hive was provisioned before hub.cluster_id/HIVE_CLUSTER_ID were emitted, patch the Deployment and then restart the pod (repeat for each namespace):

CLUSTER_ID=vllm-d
kubectl --request-timeout=15s -n "${NS}" set env deploy/hive HIVE_CLUSTER_ID="${CLUSTER_ID}"
kubectl --request-timeout=15s -n "${NS}" rollout restart deploy/hive

This is in addition to the read-only ClusterRole/ClusterRoleBinding above; RBAC alone cannot help if the spoke omits cluster_health because its cluster ID is blank.

Path B — Manual provisioning (heartbeat-only cluster)

Who runs this: the target-cluster operator, with a cluster administrator for ClusterRole/SCC grants and the hub operator for the final SaaS record. Run target commands from your workstation using $CTX; hub commands name their separate context. Do not attempt cluster-admin actions as the hive’s tenant ServiceAccount.

The hub cannot reach a heartbeat-only cluster, so every object is applied by hand with kubectl --context <heartbeat-only-cluster>. The full set, in order:

1. Namespace
2. ServiceAccount (hive-sa)
3. RBAC — three Roles (hive-secrets-writer, hive-self-upgrade, hive-route-reader), one ClusterRole (hive-node-health-reader-${NS}), four RoleBindings (the three above plus hive-anyuid), and one ClusterRoleBinding (hive-node-health-reader-${NS})
4. PVC (hive-data, RWX cephfs, 50Gi)
5. ConfigMap (hive-config) — the first-boot config seed
6. Secret (hive-secrets) — dashboard token, GitHub App key, LiteLLM key
7. Service (hive, ports 3002 dashboard / 3001 terminal)
8. Routes (hive-dashboard on /, hive-terminal on /terminal)
9. Deployment (hive)
10. Hub SaaS record — meta.json on the hub PVC

Set these shell variables for the target hive:

CTX=<heartbeat-only-cluster>    # kubectl context for the target cluster
ID=hosted-myorg-myrepo          # the hive ID
NS=hive-hosted-$ID              # namespace is always hive-hosted-<id>
ROUTE_HOST=$ID.apps.<your-cluster-domain>
IMAGE=ghcr.io/hivecommons/hive:v4-latest   # what the hub provisioner stamps by default (saas_provision.go); use `stable` for production
SC=ocs-storagecluster-cephfs
 
# The hub heartbeat secret — the SAME for every spoke on a given hub. Copy it
# from any working hive on the same cluster (see the Deployment gotcha). The
# spoke's heartbeats 401 without it.
HUB_SECRET=$(kubectl --context "$CTX" -n hive-hosted-<any-working-hive> \
  get deploy hive -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep '^HIVE_HUB_SECRET=' | cut -d= -f2-)

B.1 Namespace + ServiceAccount

kubectl --context "$CTX" create ns "$NS"
kubectl --context "$CTX" label ns "$NS" app=hive
kubectl --context "$CTX" -n "$NS" create sa hive-sa

B.2 RBAC — Roles and RoleBindings

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hive-secrets-writer, namespace: ${NS} }
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get","list","watch","create","update","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hive-self-upgrade, namespace: ${NS} }
rules:
- apiGroups: ["apps"]
  resources: ["deployments","deployments/scale"]
  verbs: ["get","list","watch","update","patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","list","watch","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hive-route-reader, namespace: ${NS} }
rules:
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-secrets-writer, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hive-secrets-writer }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-self-upgrade, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hive-self-upgrade }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-route-reader, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hive-route-reader }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-anyuid, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: system:openshift:scc:anyuid }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: hive-node-health-reader-${NS} }
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get","list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: hive-node-health-reader-${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: hive-node-health-reader-${NS} }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
YAML

Gotcha — the RoleBinding subject namespace must be the hive’s own namespace. If you build these by copying another hive’s manifests, the subjects[].namespace fields will still point at the source namespace. The bindings then bind a ServiceAccount that does not exist here, the pod does not get the anyuid SCC, it falls to restricted-v2, and it crash-loops with exit 255. Always template subjects[].namespace to $NS. Verify after applying:

kubectl --context "$CTX" -n "$NS" get rolebinding hive-anyuid \
  -o jsonpath='{.subjects[0].namespace}'   # must equal $NS

hive-route-reader — why the dashboard link 503s without it

The spoke discovers the external hostname its own Route/Ingress actually serves and reports it to the hub as dashboard_url (SpokeServedHost, called from the heartbeat path). hive-route-reader is what lets it read them.

Without the Role, that lookup returns “no answer” and the spoke falls back to synthesising <hiveID>.<hub host>. That is correct only for spokes fronted by the hub’s own wildcard domain. Anywhere else it is a guaranteed 503: the wildcard resolves, so DNS looks healthy, but it sends the name to the hub’s router, which has no backend for a hive living on another cluster. This was a live user-visible outage on the heartbeat-only pool — the hub minted links at <id>.hive.hivecommons.dev while the spoke’s Route served <id>.apps.<your-cluster-domain>.

The spoke is the only party that can answer this on a heartbeat-only cluster, since the hub has no kubectl path there by design.

Notes on the shape of the Role:

• Read-only and namespace-scoped. get/list, no write verbs. A compromised spoke learns only its own hostname — which it already advertises — and can neither create nor retarget routing.
• Routes are listed unconditionally, not gated on OpenShift. A cluster can serve Routes without requiring an SCC, and a Role naming a CRD-backed resource is inert where that API is absent.
• Ingress is consulted before Route, matching the traffic path, so a cluster carrying both resolves to the same object.

Gotcha — derive the ServiceAccount, do not assume it. The subject is not uniform across the fleet. OpenShift/SCC spokes bind hive-sa; others bind default. Measured live: the hub-reachable cluster is 22× default; one heartbeat-only cluster is 2× default and 41× hive-sa; another spoke cluster is 5× hive-sa. Binding the wrong one fails silently — the pod runs, the lookup is denied, and you get the 503 fallback with no error anywhere obvious. Read it off the Deployment:

kubectl --context "$CTX" -n "$NS" get deploy hive \
  -o jsonpath='{.spec.template.spec.serviceAccountName}'

An empty result means the Deployment does not set one, which is Kubernetes for default — bind default, not the empty string. The manifests in B.2 above use hive-sa because this manual path creates it in B.1; substitute whatever the command returns if you are patching an existing namespace.

Retrofit — namespaces provisioned before this change do not have it. The Role/RoleBinding is emitted by the provisioning template, so new hosted hives get it automatically. Every namespace created before it was added needs it applied retroactively, or its dashboard link keeps 503ing. Check the fleet:

kubectl --context "$CTX" get rolebinding -A \
  --field-selector metadata.name=hive-route-reader

The spoke also has to be running a build new enough to perform the lookup, so a namespace that already has the RBAC may still need a rollout restart to start reporting.

B.3 PVC

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: hive-data, namespace: ${NS}, labels: { app: hive } }
spec:
  accessModes: ["ReadWriteMany"]
  resources: { requests: { storage: 50Gi } }
  storageClassName: ${SC}
YAML

Use RWX (ReadWriteMany). Both the old and new pod mount the same volume during a rolling upgrade (maxUnavailable: 0), so the volume must support multi-attach — hence cephfs, not RWO block storage.

B.4 ConfigMap (the config seed)

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: ConfigMap
metadata: { name: hive-config, namespace: ${NS}, labels: { app: hive } }
data:
  hive.yaml: |
    project:
      org: myorg                      # a POOL PLACEHOLDER can seed any bootstrap org here; the
      repos: [myrepo]                 # AUTHORITATIVE identity is the meta.json org
      primary_repo: myrepo            # (available-<cluster>-<date>-<suffix>), which is what the hub
                                      # renders — see Placeholder pools below
    agents:
      guide:   { backend: copilot, model: claude-sonnet-4-6, enabled: true }
      scanner: { backend: copilot, model: claude-sonnet-4-6, enabled: true }
    governor:
      eval_interval_s: 300
      modes:
        idle: { threshold: 0,  guide: 4h, scanner: 4h }
        busy: { threshold: 10, guide: 2h, scanner: 2h }
    github:
      app_id: 999999999               # the placeholder sentinel (config.PlaceholderAppID)
      installation_id:
      key_file: /secrets/gh-app-key.pem
      oauth_client_id: Ov23ligE2p0gjXg6xAUf   # public device-flow client (no secret)
    dashboard:
      port: 3002
      authorized_users:
        - owner-github-login:owner    # a POOL PLACEHOLDER lists the pool owner: <hub-admin-login>:owner
      hub_proxied: false              # heartbeat-only cluster has no hub auth proxy → direct device-flow
    hub:
      enabled: true
      url: https://hive.hivecommons.dev
      cluster_id: vllm-d
      dashboard_url: https://${ROUTE_HOST}
      hive_type: hosted
      is_public: false
    acmm_level: 2
YAML

Gotcha — github.app_id cannot be empty. Config validation requires either github.token or github.app_id to be set (pkg/config/config.go: github.token or github.app_id is required). A hive awaiting its real App must carry the placeholder sentinel app_id: 999999999 or it will refuse to boot.

Use 999999999 exactly — it is a recognised sentinel, not a spare number. It is declared as config.PlaceholderAppID, and GitHubConfig.HasApp() reports a hive carrying it as having no GitHub App. Such a hive boots into dashboard-only mode: the dashboard serves, the heartbeat runs, and the “GitHub App required” banner shows the install link. Agents stay idle until a real App is linked.

Any other non-zero placeholder is read as a real App ID. The hive will then try to authenticate as an App that does not exist — and once installation_id also becomes non-zero (which happens the moment the owner installs an App), it attempts to load a private key that was never provisioned. Before this was fixed, that combination exited the process before the HTTP listener bound: permanent CrashLoopBackOff, no heartbeat, the hive shown offline on the hub, and any in-flight rollout stuck forever because the new pod never went Ready. The spoke now degrades instead of exiting, but the hive still cannot do any GitHub work until the app_id is corrected.

Replacing the placeholder with the real value is normally automatic — the hub delivers app_id, installation_id and the private key over the heartbeat when the App is installed. To do it by hand, patch the PVC overlay (/data/hive.yaml.dashboard, not the ConfigMap) and restart the pod.

Gotcha — hub_proxied must be false on a heartbeat-only cluster. A heartbeat-only cluster has no hub nginx auth proxy. If hub_proxied is true, dashboard sign-in enters an OAuth redirect loop. On the hub-reachable cluster it is the opposite — the hub proxy handles auth.

B.5 Secret

DASH_TOKEN=$(openssl rand -hex 32)
cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: Secret
metadata: { name: hive-secrets, namespace: ${NS}, labels: { app: hive } }
type: Opaque
stringData:
  dashboard-token: "${DASH_TOKEN}"
  gh-app-key.pem: "PLACEHOLDER-awaiting-github-app-key"
  litellm_api_key: "sk-PLACEHOLDER"
YAML

B.6 Service

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: Service
metadata: { name: hive, namespace: ${NS}, labels: { app: hive } }
spec:
  selector: { app: hive }
  ports:
  - { name: dashboard, port: 3002, targetPort: 3002 }
  - { name: terminal,  port: 3001, targetPort: 3001 }
YAML

B.7 Routes (OpenShift)

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: route.openshift.io/v1
kind: Route
metadata: { name: hive-dashboard, namespace: ${NS}, labels: { app: hive } }
spec:
  host: ${ROUTE_HOST}
  path: /
  port: { targetPort: dashboard }
  tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
  to: { kind: Service, name: hive, weight: 100 }
  wildcardPolicy: None
---
apiVersion: route.openshift.io/v1
kind: Route
metadata: { name: hive-terminal, namespace: ${NS}, labels: { app: hive } }
spec:
  host: ${ROUTE_HOST}
  path: /terminal
  port: { targetPort: terminal }
  tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
  to: { kind: Service, name: hive, weight: 100 }
  wildcardPolicy: None
YAML

B.8 Deployment

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: apps/v1
kind: Deployment
metadata: { name: hive, namespace: ${NS}, labels: { app: hive } }
spec:
  replicas: 1
  strategy: { type: RollingUpdate, rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  selector: { matchLabels: { app: hive } }
  template:
    metadata: { labels: { app: hive } }
    spec:
      serviceAccountName: hive-sa
      # ── init containers — REQUIRED. The ConfigMap is mounted read-only at
      # /etc/hive-seed and copied into a WRITABLE emptyDir at /etc/hive. The hive
      # process must WRITE /etc/hive/hive.yaml at runtime (on first boot the entrypoint
      # seeds it, then merges the PVC overlay over it; later boots restore it from
      # /data/hive.yaml.runtime; the dashboard's Save writes it). Mounting the ConfigMap DIRECTLY at /etc/hive makes it read-only and
      # every config save fails with "open /etc/hive/hive.yaml: read-only file
      # system" — and dashboard-installed GitHub App auth / ACMM changes are lost.
      initContainers:
      - name: copy-config
        image: ${IMAGE}
        command: ["sh","-c","if [ -s /data/hive.yaml.runtime ]; then echo runtime-config-exists-for-recovery; elif [ -s /data/hive.yaml.bak ]; then echo legacy-runtime-config-exists-for-recovery; else cp /etc/hive-seed/hive.yaml /etc/hive/hive.yaml && echo configmap-copied-first-boot; fi"]
        volumeMounts:
        - { name: config,          mountPath: /etc/hive-seed, readOnly: true }
        - { name: config-writable, mountPath: /etc/hive }
        - { name: data,            mountPath: /data }
      - name: init-permissions
        image: ${IMAGE}
        command: ["sh","-c","chown -R 1001:1000 /data 2>/dev/null || true; echo permissions-set"]
        volumeMounts:
        - { name: data, mountPath: /data }
      containers:
      - name: hive
        image: ${IMAGE}
        imagePullPolicy: Always
        ports:
        - { name: dashboard, containerPort: 3002 }
        - { name: terminal,  containerPort: 3001 }
        env:
        - { name: HIVE_ID, value: "${ID}" }
        # HIVE_HUB_SECRET authenticates every heartbeat to the hub. WITHOUT it,
        # the hub rejects the spoke's heartbeats with 401 and the hive shows
        # OFFLINE forever (see the gotcha below). HIVE_HUB_URL points at the hub.
        #
        # SECURITY (C2 domain separation): the hub no longer signs everything with
        # this one key. It derives a distinct sub-key per trust domain
        # (heartbeat / session cookie / SSO), so a spoke never needs the master.
        # Automated hub-hosted provisioning injects ONLY the derived keys
        # (HIVE_HEARTBEAT_KEY, HIVE_SESSION_KEY, HIVE_SSO_KEY) and NEVER
        # HIVE_HUB_SECRET. Manual provisioning MAY still set HIVE_HUB_SECRET as
        # shown below: the spoke derives the same sub-keys from it locally, so
        # heartbeats/SSO keep working. If you prefer least privilege, set the three
        # HIVE_*_KEY vars instead (the hub derives them per-hive; the exact
        # derivation lives in src/pkg/hub/hub_keys.go — heartbeat is HMAC-based
        # while session/SSO are now Ed25519 asymmetric seeds, so do NOT assume a
        # single HMAC(master, "hive-<domain>-v1") formula) and omit
        # HIVE_HUB_SECRET entirely.
        #
        # HUB-ONLY: this whole HIVE_HUB_SECRET / derived-key block is irrelevant
        # to a standalone (hub-less) hive — see "Standalone / self-hosted (no
        # hub)" at the top of this guide.
        - { name: HIVE_HUB_SECRET, value: "${HUB_SECRET}" }
        - { name: HIVE_HUB_URL,    value: "https://hive.hivecommons.dev" }
        # startupProbe keeps the liveness clock from starting until boot
        # finishes, so a slow start can't race liveness into a restart loop.
        # Match spoke.HostedStartupBudget: 10s + 118 x 5s = 600s; keep
        # these timings in sync with its hosted startup probe constants.
        startupProbe:
          httpGet: { path: /api/health, port: 3002 }
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 118
        livenessProbe:
          # /api/livez (NOT /api/health) — it also fails when the heartbeat
          # loop stops *attempting* sends, so a wedged/dead heartbeat
          # goroutine gets the pod auto-restarted. It does NOT fail merely
          # because the hub is unreachable (see the gotcha below).
          httpGet: { path: /api/livez, port: 3002 }
          periodSeconds: 30
          failureThreshold: 3
          timeoutSeconds: 2
        readinessProbe:
          httpGet: { path: /api/health, port: 3002 }
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3
          timeoutSeconds: 2
        volumeMounts:
        - { name: data,            mountPath: /data }
        - { name: config-writable, mountPath: /etc/hive }         # WRITABLE (emptyDir), not the ConfigMap
        - { name: secrets,         mountPath: /secrets, readOnly: true }
      volumes:
      - { name: data,            persistentVolumeClaim: { claimName: hive-data } }
      - { name: config,          configMap: { name: hive-config } }   # seed only → /etc/hive-seed
      - { name: config-writable, emptyDir: {} }                       # runtime /etc/hive
      - { name: secrets, secret: { secretName: hive-secrets } }
YAML

maxUnavailable: 0 keeps the current pod serving until the new one is ready — uninterrupted upgrades. It only works because the PVC is RWX.

Gotcha — HIVE_HUB_SECRET is REQUIRED or the hive is permanently offline. The hub authenticates every heartbeat against a shared secret. A spoke without HIVE_HUB_SECRET in its deployment env sends unauthenticated heartbeats, the hub replies 401 unauthorized, and the hive shows offline in the fleet even though the pod is 1/1 Running. Copy the value from any working hive on the same hub — it is the same for all spokes:

HUB_SECRET=$(kubectl --context "$CTX" -n hive-hosted-<any-working-hive> \
  get deploy hive -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep '^HIVE_HUB_SECRET=' | cut -d= -f2-)

Symptom to look for in the spoke logs: hub heartbeat rejected status=401.

Gotcha — point the liveness probe at /api/livez, not /api/health. /api/health only checks the HTTP server is up, so a pod whose heartbeat goroutine has silently died still passes it and is never restarted — the hive shows a persistent “offline” dot while 1/1 Running. /api/livez additionally fails (503) when the heartbeat loop stops attempting to send, so the kubelet restarts the pod and the heartbeat revives. It returns 200 unauthenticated once healthy. Existing deployments provisioned before this note still point at /api/health — migrate them: kubectl -n <ns> patch deploy hive --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path","value":"/api/livez"}]'

Note — /api/livez does not fail on an unreachable hub. It keys off heartbeat attempts, not successes. A hub that is down, firewalled, or rejecting beats leaves the process perfectly healthy, and restarting it cannot fix a network partition — an earlier version gated liveness on heartbeat freshness and crash-looped healthy spokes on firewalled (heartbeat-only) clusters. Heartbeat freshness is reported by /api/health/deep under the hub_heartbeat check, and the hub greys the hive’s dot on its own. Spokes deployed before this fix need a redeploy (or a probe patch) to stop restarting on hub outages.

Gotcha — some clusters enforce an owner label. A ValidatingAdmissionPolicy on the heartbeat-only cluster requires an owner label on PersistentVolumeClaim, ConfigMap, Secret, Service, and Deployment. Objects still apply without it, but each emits a warning. Add owner: <github-login> to every metadata.labels block to silence them.

B.9 Verify the pod comes up on anyuid

kubectl --context "$CTX" -n "$NS" rollout status deploy/hive --timeout=120s
POD=$(kubectl --context "$CTX" -n "$NS" get pods -l app=hive \
  -o jsonpath='{.items[0].metadata.name}')
# MUST be "anyuid" — if it says "restricted-v2", the anyuid binding is wrong
kubectl --context "$CTX" -n "$NS" get pod "$POD" \
  -o jsonpath='{.metadata.annotations.openshift\.io/scc}'

Path B completion checks

1. Confirm rollout readiness and the expected SCC. An anyuid grant alone does not grant NET_ADMIN; follow the capability decision and diagnose exit 77 rather than silently bypassing egress enforcement.
2. Create the hub SaaS record below. Verify the owner sees the hive in My Hives.
3. Open the reported Route URL, sign in as the allowed owner, and complete App/backend setup. Confirm recent outbound heartbeats and a reported version on the hub; the hub still cannot kubectl into this cluster.
4. If heartbeats work but management says hive not found, inspect meta.json; if the link 503s, inspect route-reader RBAC and dashboard_url. Use the failure-mode reference for recovery.

The hub SaaS record (meta.json) — required for both paths

Automated provisioning writes this for you. Manual provisioning does not — you must create it by hand, or the hive is invisible in My Hives and every management action (upgrade, claim) returns hive not found.

The read is loadSaaSHive(id) → /data/saas/hives/<id>/meta.json, read live from disk on every call, so a new or edited file takes effect immediately with no hub restart.

ID=hosted-myorg-myrepo
HUB_POD=$(kubectl --context <hub-reachable-cluster> -n hive-hub get pods -l app=hive-hub \
  --no-headers | grep Running | awk '{print $1}' | head -1)
 
cat > /tmp/meta.json <<JSON
{
  "id": "${ID}",
  "owner": "owner-github-login",
  "project_name": "My Hive",
  "org": "myorg",
  "repos": ["myrepo"],
  "primary_repo": "myrepo",
  "acmm_level": 2,
  "status": "",
  "created_at": "",
  "subdomain": "",
  "auto_upgrade": true,
  "is_public": false,
  "cluster_id": "<heartbeat-only-cluster>"
}
JSON
 
kubectl --context <hub-reachable-cluster> -n hive-hub exec "$HUB_POD" -- \
  sh -c "mkdir -p /data/saas/hives/${ID}"
kubectl --context <hub-reachable-cluster> -n hive-hub cp /tmp/meta.json \
  hive-hub/"$HUB_POD":/data/saas/hives/${ID}/meta.json

Visibility rule. handleMyHives shows a hive to a user when the user is its owner, has a role on it, or is the hub admin — even if the hive has no fleet-registry entry (never heartbeated). So a hive with owner: <someone> in its meta.json appears in that person’s My Hives (and the admin’s) as soon as the file exists.

Hub access roles. Manage Access grants four ordered roles (separate from the ClankeR trust tiers shown on /contribute): read < read-write < merger < owner. merger inherits read-write dashboard access and can approve/queue other people’s PRs for the hive auto-merge-on-green flow. The spoke enforces the self-merge ban against the GitHub login bound to the session; owners are exempt because they already have repository-level merge authority.

Queueing a PR approves it as the hive App and applies a label. That label is configurable, because it is applied in a repository the hive does not own and whose review conventions already exist:

governor:
  labels:
    automerge: lgtm

It defaults to lgtm, the long-standing Prow/Kubernetes label for “a second person signed off”, which is the decision the merger tier records. Set it to whatever the managed repositories already use — the label is created on demand if it does not exist.

meta.json requirement is why heartbeats can be accepted but upgrades fail. The heartbeat path is separate from loadSaaSHive. A hive can heartbeat and appear “online” while a missing meta.json makes the Upgrade button (and claim, toggle-auto-upgrade) return hive not found. If you see that, the fix is almost always a missing meta.json.

Config precedence — the PVC overlay is authoritative

At runtime the effective config is not the ConfigMap. On the first boot (no non-empty /data/hive.yaml.runtime or legacy /data/hive.yaml.bak) the entrypoint:

1. seeds /etc/hive/hive.yaml from the hive-config ConfigMap, then
2. merges the PVC dashboard overlay /data/hive.yaml.dashboard over it, and
3. writes /data/hive.yaml.runtime.

On every later boot it copies /data/hive.yaml.runtime over /etc/hive/hive.yaml and boots from that; the seed is not used and the overlay merge does not run. With neither file and no seed, it exits 1.

The dashboard’s Config.Save() writes the effective source, the dashboard overlay, and /data/hive.yaml.runtime. The runtime file is the later-boot input, not merely a backup. Therefore:

• Editing the ConfigMap after first boot does nothing to the running config — the PVC overlay wins.
• To change a running hive, prefer dashboard/settings API saves. Editing only .dashboard is not a later-boot repair: .runtime wins before that merge.
• For offline repair, stop the hive first and back up the runtime, dashboard, and legacy files securely. Reconcile the intended change into the runtime boot input and its corresponding overlay, preserving credentials and file permissions. Otherwise an old layer or a live process save can undo it.
• A deliberate reseed requires no non-empty .runtime or legacy .bak; a remaining .dashboard may still override seed fields. Back up all three before clearing any of them, review the new seed, and plan to restore any dashboard-only state. A rollout restart alone never performs a reseed.
• Verify effective config by grep-ing /etc/hive/hive.yaml in the running pod, never the ConfigMap.

Access-grant verification. Current device-flow login checks Config.Dashboard.AuthorizedRole(username) at login, not a permanently cached startup list. Verify the actual GitHub login and the effective dashboard.authorized_users if a new grant is rejected. Editing the ConfigMap or only the PVC overlay is not the same as a live dashboard save.

This bites hardest right after provisioning or a cross-cluster migration onto a fresh PVC, where the first grants are written after the pod already booted. Finish provisioning with the verification below. On an older build or after an offline repair, a planned rollout can be needed to load the repaired runtime state; it is not a way to make a changed ConfigMap authoritative.

# 1. Confirm the grant reached the effective config:
kubectl -n hive-hosted-<id> exec deploy/hive -c hive -- \
  grep -A6 authorized_users /etc/hive/hive.yaml
# 2. Only if an offline runtime repair or an older build needs a reload:
kubectl -n hive-hosted-<id> rollout restart deploy/hive
# 3. Confirm no post-roll rejections:
kubectl -n hive-hosted-<id> logs deploy/hive -c hive | grep 'not authorized'

Do not hand-create /data/hive.yaml to “fix” a missing config — the effective config is ConfigMap-seed + .dashboard overlay; a stray /data/hive.yaml is never read and only adds confusion.

Placeholder pools

A placeholder is a fully-provisioned but idle hive, waiting to be claimed — so a request for access is satisfied in seconds instead of the full 5–10 minute provision. Two pools, one per method type:

• Heartbeat-only-cluster placeholders serve private methods — provisioned manually (Path B) and left running at replicas: 1.
• Hub-reachable-cluster placeholders serve public methods — provisioned via the API (Path A) and left running at replicas: 1.

Gotcha — placeholder IDs are date + random suffix, NOT sequential integers. The live pool uses hosted-available-<cluster>-<YYMMDD>-<suffix>, where <suffix> is a short unique token (4 lowercase base36 chars) — e.g. hosted-available-<cluster>-260731-fikn, hosted-available-<cluster>-260806-1bo3. Do not use sequential numbers like hosted-available-<cluster>-01: they collide with existing pool slots and don’t match the ID convention the fleet uses, so renumbering is needed every time the pool grows. Unique suffixes let you add a batch on any date without touching existing slots. Generate a batch of N suffixes:

tr -dc 'a-z0-9' < /dev/urandom | fold -w4 | head -N | sort -u

Then collision-check each suffix before creating — skip any that already exist — against both the hub meta dirs and the target cluster’s namespaces:

kubectl --context <hub-reachable-cluster> -n hive-hub exec "$HUB_POD" -- ls /data/saas/hives/   # existing hive IDs
kubectl --context <heartbeat-only-cluster> get ns | grep hive-hosted-hosted-available-<cluster>  # existing namespaces

Provision each placeholder as above at replicas: 1. The pod boots immediately, lands on the anyuid SCC, and heartbeats to the hub as an available slot ready to be claimed. Both pools stay at replicas: 1:

# manual (heartbeat-only cluster): apply the Deployment at replicas: 1 (it boots and heartbeats).
# If you applied it at 0, scale it up:
kubectl --context <heartbeat-only-cluster> -n "$NS" scale deploy/hive --replicas=1
 
# automated (hub-reachable cluster): the API already provisions at replicas=1 — leave it there.

Gotcha — do NOT scale placeholders to 0. A replicas: 0 slot is not assignable: the claim/assign path delivers the claimant’s config to a running, heartbeating spoke, so a slot that isn’t booted can never be claimed. Every assignable slot in the live pool runs at replicas: 1, ready 1/1. (Earlier guidance to “scale to 0” was incorrect.)

Give every placeholder a meta.json with owner: <hub-admin-login> (admin-only visibility), status: "available", and acmm_level: 2. Set org to a unique value per slot — available-<cluster>-<YYMMDD>-<suffix> mirroring the id’s date-suffix — with empty repos: [] and primary_repo: "" (they get their real values when the slot is claimed). The meta.json also carries github_host when the placeholders target GitHub Enterprise (github.your-company.com), not github.com:

{
  "id": "hosted-available-<cluster>-260806-1bo3",
  "owner": "<hub-admin-login>",
  "project_name": "Available slot (private) 260806-1bo3",
  "org": "available-<cluster>-260806-1bo3",
  "repos": [],
  "primary_repo": "",
  "acmm_level": 2,
  "status": "available",
  "created_at": "",
  "subdomain": "hosted-available-<cluster>-260806-1bo3.apps.<your-cluster-domain>",
  "claim_delivered": false,
  "auto_upgrade": true,
  "is_public": false,
  "cluster_id": "<heartbeat-only-cluster>",
  "github_host": "github.your-company.com"
}

The org must be unique per slot (mirror the id’s date-suffix) so each renders as a distinct “Available slot (private) <date>-<suffix>” row in the hub. A literal shared org: "placeholder" (with repos: ["placeholder"]) makes every slot render identically as “placeholder / placeholder” — indistinguishable — and is wrong.

Verify a provisioned placeholder

A correct heartbeat-only-cluster placeholder runs at replicas: 1 with its pod up (Running, ready, restartCount: 0) on the anyuid SCC, its hive-anyuid binding points at its own namespace, its PVC is Bound, its livez returns 200, and its hub meta.json exists. This set verified all ten placeholders in the 2026-08-06 batch:

ID=hosted-available-<cluster>-<date>-<suffix>; NS=hive-hosted-$ID
kubectl --context <heartbeat-only-cluster> -n $NS get deploy hive -o jsonpath='{.spec.replicas}'          # must be 1
POD=$(kubectl --context <heartbeat-only-cluster> -n $NS get pod -l app=hive -o jsonpath='{.items[0].metadata.name}')
kubectl --context <heartbeat-only-cluster> -n $NS get pod $POD -o jsonpath='{.status.phase}'              # must be Running
kubectl --context <heartbeat-only-cluster> -n $NS get pod $POD -o jsonpath='{.metadata.annotations.openshift\.io/scc}'  # must be anyuid
kubectl --context <heartbeat-only-cluster> -n $NS get pod $POD -o jsonpath='{.status.containerStatuses[0].ready}'       # must be true
kubectl --context <heartbeat-only-cluster> -n $NS get pod $POD -o jsonpath='{.status.containerStatuses[0].restartCount}' # must be 0
kubectl --context <heartbeat-only-cluster> -n $NS exec $POD -c hive -- curl -s -o /dev/null -w '%{http_code}' localhost:3002/api/livez  # must be 200
kubectl --context <heartbeat-only-cluster> -n $NS get rolebinding hive-anyuid -o jsonpath='{.subjects[0].namespace}'  # must equal $NS
kubectl --context <heartbeat-only-cluster> -n $NS get pvc hive-data -o jsonpath='{.status.phase}'         # must be Bound
kubectl --context <hub-reachable-cluster> -n hive-hub exec $HUB_POD -- test -f /data/saas/hives/$ID/meta.json  # must exist

The hive-anyuid subject-namespace check is still load-bearing at apply time (it is the same restricted-v2 crash-loop gotcha from B.2): a wrong subject namespace means the pod can’t get the anyuid SCC. But because a placeholder now boots at replicas: 1, you confirm the SCC directly — verify the pod reaches phase=Running, scc=anyuid, ready=true, restartCount=0, and livez == 200.

Note — placeholders carry a fleet-registry entry. Because both pools run at replicas: 1, every placeholder heartbeats and holds a registry entry that pins its version and last-known ACMM and shows it “online”. This is expected and required: the claim/assign path delivers config to the running, heartbeating spoke, so the entry is what makes the slot assignable — do not strip it.

Claiming a placeholder

Until the dashboard “assign” flow lands, claiming is manual:

1. Edit the placeholder’s meta.json: set the real owner, org, repos, primary_repo, acmm_level, and is_public.
2. Update the running config on the PVC overlay (/data/hive.yaml.dashboard) — real project.org / repos, and the claimant’s GitHub App app_id / installation_id.
3. Install the GitHub App key (owner does this from the dashboard’s Install GitHub App flow — the hive is already awake at replicas: 1).
4. Roll the pod so it picks up the new config (kubectl rollout restart deploy/hive) — it is already running, so there is nothing to scale up.

Mapping a namespace back to its hive (identity labels)

A claimed placeholder’s namespace keeps its original placeholder name forever — e.g. hive-hosted-hosted-available-<cluster>-01-placeholder-bb95 — even after the hub API’s claim/assign flow gives the hive a real name and org. Kubernetes has no atomic namespace-rename primitive, and renaming would mean recreating every object inside it (including the PVC), so the namespace name is never changed. Historically this meant there was no way to look at a namespace on the cluster and tell which hive/org it belonged to without cross-referencing the hub registry and reverse-engineering the placeholder slug.

Instead, the hub stamps identity labels and a display-name annotation onto the namespace itself, keeping it self-describing even though its name never changes:

Sanitization: label values are lower-cased, restricted to [a-z0-9.-] (everything else — spaces, underscores, unicode, punctuation — is dropped outright rather than transliterated), trimmed of leading/trailing separators, and capped at 63 characters (the RFC-1123 label-value limit). So a hive named "TradingAsBuddies" gets the label value tradingasbuddies, while the hive.hivecommons.dev/display-name annotation preserves the original TradingAsBuddies exactly. A field that sanitizes to empty (or wasn’t set yet — e.g. a fresh placeholder with no real name/org) is omitted from the labels rather than written as an empty/invalid label.

Given a hive name, find its namespace:

kubectl --context <hub-reachable-cluster> get ns \
  -l hive.hivecommons.dev/hive-name=tradingasbuddies

Given a namespace, find the hive that owns it:

kubectl --context <hub-reachable-cluster> get ns hive-hosted-hosted-available-<cluster>-01-placeholder-bb95 \
  -o jsonpath='{.metadata.annotations.hive\.hivecommons\.dev/display-name}'
# -> TradingAsBuddies

When the labels/annotation are (re)written — the hub calls the same stampHostedNamespaceIdentity helper at all three points a hive’s identity is known or changes, so the namespace never goes stale:

1. Automatic provisioning (provisionHive, Path A above) — stamps whatever is known at namespace-creation time. For a freshly-created pool placeholder this is often just hive.hivecommons.dev/hive-id, since org/name are still placeholder values at this point.
2. Claim (handleApproveProvision) — the real name/org become known here for the first time, so the labels/annotation are (re)written.
3. Assign / reassign (handleAssignHive) — same, so a placeholder that gets reassigned to a different owner has its namespace identity refreshed too.

The stamp is applied via idempotent kubectl label/kubectl annotate --overwrite (merged into whatever the namespace already carries, not replaced) and is best-effort: a failure to label (no kubectl on PATH, a transient API error) is logged and does not fail the claim/assign request — this is a cosmetic/operability nicety, not a required step for the hive to work.

Name-bearing vanity Route

The same problem — a URL permanently encoding the placeholder slug — applies to the dashboard’s public URL. The assign path (handleAssignHive) also creates an additional OpenShift Route whose host is derived from the hive’s own display name, rather than the org/repo-derived host used previously:

<sanitized-hive-name>-<4-char-suffix>.<cluster-domain>

For example, a hive named TradingAsBuddies on a cluster with domain apps.example.com gets a Route host like tradingasbuddies-a1b2.apps.example.com, adopted as https://tradingasbuddies-a1b2.apps.example.com. The random suffix keeps two hives with similar/identical sanitized names from colliding on the same host.

This is an additional Route, not a rename: the original Route (host derived from org/repo or the placeholder ID) is left completely alone, so any in-flight bookmark or callback against it keeps working. The namespace itself is still never renamed or recreated — only a second Route pointing at the same Service is added. If the hive has no display name set, the vanity host falls back to the previous org/repo-derived scheme.

Namespace shown in the dashboards

The namespace is now surfaced in two read-only UI spots, so an operator or hive owner doesn’t need kubectl at all to find it:

• Spoke dashboard → Hub tab. The hive’s own dashboard shows its Kubernetes namespace next to the hub URL/name info. The spoke learns its own namespace from the POD_NAMESPACE env var, set via the Kubernetes downward API (fieldRef: metadata.namespace) on the hive Deployment — falling back to a NAMESPACE env var, then to the in-cluster service-account namespace file, if POD_NAMESPACE isn’t set (e.g. a spoke provisioned before this env var existed). The row is omitted entirely when none of the three resolve.
• Hub dashboard → per-spoke status hover. Hovering a hive in the hub’s fleet view shows its namespace, computed client-side as "hive-hosted-" + <hive-id> — the same convention the hub uses server-side — so it can never disagree with the namespace the identity labels above are stamped on.

Deprovisioning

# delete the workload namespace
kubectl --context "$CTX" delete ns "$NS"
# remove the hub SaaS record
kubectl --context <hub-reachable-cluster> -n hive-hub exec "$HUB_POD" -- \
  rm -rf /data/saas/hives/"$ID"
# if it ever heartbeated, also drop its registry entry (see the placeholder gotcha)

Quick failure-mode reference