Self-hosted edge hardening rollout

This runbook applies to self-hosted edges attached to a platform: the edge owner operates MistServer, Helmsman, Caddy, and the edge host; the platform operator runs Foghorn, federation, Livepeer, Chandler, object storage, DNS/TLS, analytics, billing, and the control plane. On the hosted platform that operator is FrameWorks. If you self-host the entire stack, you are the platform operator and follow the same release order for your own platform.

Rollout order and gates

Prepare signed source authority before Foghorn. Generate/import the media-authority signing key tuple, then deploy and health-check Commodore before any Foghorn replacement. A non-development Commodore refuses to start without MEDIA_AUTHORITY_SIGNING_KEY_ID and MEDIA_AUTHORITY_SIGNING_PRIVATE_KEY_PEM_B64. Confirm signed tenant authority is compiling and delivery lag is healthy. During rolling skew, new Foghorn accepts an old Commodore’s legacy origin/peer envelope only when the new serve_policy_resolved field is absent; an upgraded Commodore’s explicit false remains fail-closed.
Contain source disclosure. Deploy Foghorn source authorization only after step 1. Verify foghorn_source_authorization_rejected_total remains quiet for valid edges and that a two-tenant negative request increments cluster_not_authorized without federation, replication, or source-success effects. Deploy Gateway owner/admin action checks and Purser’s ApplyTenantBillingEntitlements client before Quartermaster rejects legacy direct writes.
Apply the DNS-safe expand set. The Quartermaster expand migration adds the entitlement columns with an unobserved epoch fence; its backstop ignores those rows, so it cannot tear down existing aliases or custom domains before authoritative billing state arrives. The Purser expand migration inserts both entitlement keys for every built-in tier. The Quartermaster data migration does not infer entitlement from legacy tier names and runs only after Purser’s complete tenant sweep proves that every existing Quartermaster tenant has an observed entitlement stamp. Subscription-less lifecycle rows are explicitly stamped fail-closed instead of blocking the handoff. Never deploy the new workers against only one side of this expand set.
Schedule identity interruption. There is no report-only identity mode: the new Quartermaster binary rejects every keyless or malformed stored node identity, so schedule recovery credentials and a per-node reconnect window before replacing it. The identity-key column is added by the v0.3.0 Quartermaster expand migration and starts empty; it is filled only when a node re-enrolls with a token and requests key rotation. Every active edge enrolled before v0.3.0 is therefore keyless after expand. frameworks cluster release apply lists them in its post-expand report, printed after expand and before the new Quartermaster is installed: operator-managed nodes (which block postdeploy) and customer-enrolled nodes (which are refused at their next connect) are listed separately with cluster, tenant, and last seen. To see the list before starting the service upgrades, apply expand on its own with frameworks cluster migrate --phase expand --to-version v0.3.0 and run frameworks cluster release apply --version v0.3.0 --dry-run. A dry-run before expand prints post-expand report: not run (expand not applied), which is not an empty census.
Deploy strict Quartermaster and recover identity keys. After the new binary is installed, list the affected operator-managed nodes with:
Terminal window
```
frameworks cluster data-migrate verify quartermaster.quartermaster_node_identity_keys_v0_3_0
```
The verification fails with the count and the first 100 node IDs until every operator-managed active node has a valid identity key. Customer-enrolled edges that reconnect without a key are rejected and counted in quartermaster_node_identity_rejections_total{reason="keyless"} (alert QuartermasterKeylessNodeReconnectRejected). For each affected node, issue a fresh tenant/cluster-scoped edge_node token and follow Legacy Keyless Node recovery. The token alone is insufficient: the reconnect must explicitly request rotation. Re-run the verification until it passes; postdeploy remains blocked while a keyless operator-managed active node exists. Never bind a presented key from fingerprint knowledge alone.
Preflight subscriptionless DNS teardown, then verify convergence. Every tenant without a Purser subscription will be stamped free with both DNS grants disabled, and its live alias/custom-domain desired state will be removed. Quartermaster generates a requested subdomain for ordinary tenants, so this can affect most subscriptionless tenants even when only a smaller subset currently has an enabled grant. The post-expand report of frameworks cluster release apply lists every tenant whose entitlements are still unobserved and that has a custom subdomain or custom domain enabled, with the subdomain and domain, and states whether Purser’s purser_dns_entitlements_v0_3_0 handoff receipt exists yet. These are the grants quartermaster_tenant_dns_entitlements_v0_3_0 clears if the Purser sweep does not observe the tenant first. Before starting upgraded Purser, check each listed tenant that should keep its subdomain or custom domain, and create any missing subscription through the supported operator billing workflow. Do not insert subscription rows manually.

Start upgraded Purser and wait at least one minute for its first tier-access sweep before running quartermaster_tenant_dns_entitlements_v0_3_0; later repair attempts run hourly. A fully successful sweep records purser_dns_entitlements_v0_3_0 in quartermaster.billing_entitlement_handoffs; a partial page, failed apply, or Purser subscription row with no matching Quartermaster tenant records nothing. A fence rejection means Quartermaster atomically accepted a newer entitlement observation and does not block the sweep. Serialization failures are retried three times, with 100 ms and 200 ms delays before the second and third attempts. Subscription-less Quartermaster tenants are legitimate during signup, deletion, and system bootstrap, so the sweep stamps them free with both DNS grants disabled. The receipt’s historically named subscription_count is Quartermaster’s serializable tenant census at first completion; subsequent completion calls return from the immutable receipt without repeating that census. A tenant created above the sweep’s descending cursor starts observed and fail-closed, so it neither invalidates nor delays the receipt. An in-band deactivation after a tenant’s page was visited can remain visible until the next hourly sweep, which then closes both DNS grants. The migration requires the receipt but does not compare it to a changing live count. Never insert or rewrite the receipt manually. The sweep stamps even value-matching subscriptions and resolves explicit per-subscription overrides. New tenant creation and bootstrap both write an observed, fail-closed DNS state and do not create new epoch rows. The first sweep emits bounded per-tenant Navigator and media-authority work; size both drains for the tenant census before rollout. A demo bootstrap tenant has no implied commercial subscription and is closed unless its billing state is explicitly provisioned. Verify upgrades preserve existing entitled names and downgrades/suspensions remove served state while retaining requested names; reactivation immediately reapplies the authoritative grants. Subscription/entitlement changes retain a durable retry obligation until both Quartermaster entitlement reconciliation and Commodore authority refresh succeed; balance/usage-only rows go directly to Commodore and do not depend on Quartermaster availability.
Verify delegated API-token enforcement. Exercise an owner token with infrastructure:write and billing:write, then repeat with a member role, billing:read, and each scope removed. Gateway, Quartermaster, and Purser must independently deny the disallowed calls without a write or outbox row. Replay one captured delegated RPC assertion and confirm the second use is rejected.
Census and replace coarse or unsupported API tokens. The historical read and write values never authorized namespaced Gateway operations and are rejected for new tokens; they are not migrated or widened automatically. A tenant owner/admin lists the tenant’s tokens and their permissions with frameworks admin tokens list --output json. Any active token whose permissions include a value outside account:read, analytics:read, billing:read, billing:write, consultant:use, developer:read, developer:write, infrastructure:read, infrastructure:write, mcp:high-risk, security:read, security:write, settings:write, streams:read, streams:write, and support:read needs replacing. A platform operator lists those tokens across every tenant with frameworks admin tokens list --all-tenants --unsupported-scopes (add --tenant-id <id> for one tenant); each token shows its tenant, status, scopes, the unsupported scopes, created, last used, and expiry, never its value. This needs a platform-operator login (frameworks login).

For each such token, have the creating user or a tenant owner/admin create a replacement with frameworks admin tokens create --name <name> --perms <exact,scopes>, update the consuming secret, and verify the replacement. These commands use the authenticated operator session from the active CLI context. A tenant owner/admin can enumerate the tenant’s tokens with frameworks admin tokens list and revoke any one by ID with frameworks admin tokens revoke <token-id> --yes, including a token whose creating user has departed. Ordinary members list and revoke only their own tokens. There is no in-place re-scope operation because token grants are immutable. Re-run the listing until no unsupported token remains and retain the redacted count—not token identifiers or secret values—with the upgrade evidence. The database default is changed to streams:read for omitted permissions, but existing rows remain unchanged. Retention mutations require billing:write; set_preferred_cluster and update_tenant_settings require settings:write.
Tighten inventory. Deploy owner-scoped full cluster/node reads after Gateway and Quartermaster policy are both live. Confirm expired, suspended, unknown-provenance, and inactive-cluster grants reveal no full inventory.

Required evidence

Before postdeploy, retain the output of:

make proto-check
make sqlc-check
make validate-migrations
make verify-schema-migrations
make test-foghorn
make test-quartermaster
make test-purser
make test-navigator
make test-bridge
make test-helmsman
make test-privateer
make test-pkg
make test-accesspolicy

Also run the two-tenant scenario: enroll distinct keyed edges, deny tenant A’s request for tenant B’s origin with no side effect, allow B’s request, deny member/foreign/under-scoped lifecycle calls at both Gateway and direct gRPC, require token-authorized key recovery, and exercise DNS entitlement plus final-cluster loss/recovery.

Operational signals

Signal	Interpretation	Owner/action
`foghorn_source_authorization_rejected_total{reason="authority_unavailable"}`	Stream ownership could not be proven	Foghorn/control-plane owner restores authority; lookup remains fail-closed
`quartermaster_grpc_requests_total{status="PermissionDenied"}`	Caller or resource scope rejected	Identity/API owner corrects claims, role, or service credential
`quartermaster_node_identity_rejections_total{reason=~"keyless\|invalid_stored_key\|key_mismatch"}`	Strict enforcement rejected a missing, malformed, or impersonated node key	Edge operator verifies the node and rotates it with a fresh tenant/cluster-scoped token
`quartermaster_billing_entitlement_stale_total`	Older Purser observation was fenced	Billing owner inspects webhook/bootstrap/sweep ordering
`purser_tier_access_reconciliation_failures_total{operation}`	Access or DNS entitlement reconciliation failed	Billing owner inspects the bounded reconcile/revoke/sweep path and Quartermaster availability
`purser_media_authority_refresh_worker_ready`	`0` means the durable refresh worker lacks a required dependency	Billing owner restores Purser database or Commodore refresh connectivity
`purser_media_authority_refresh_pending` and `_oldest_pending_seconds`	Billing-owned authority refresh work is accumulating or aging	Billing/control owners inspect the failing delivery stage before authority becomes stale
`purser_media_authority_refresh_failures_total{stage}`	Claim, reconcile, delivery, completion, or observation failed	Follow the stage label; the durable row remains retryable
`purser_media_authority_refresh_completions_total{outcome="superseded"}`	A newer revision replaced an in-flight refresh	Expected during churn; investigate only if pending age also rises
`quartermaster_dns_backstop_repairs_total`	Desired/applied DNS drift was found	Quartermaster/Navigator owner verifies the repair lands
`quartermaster_navigator_outbox_pending` and `_failures_total`	DNS hand-off is delayed or failing	Navigator/DNS owner restores delivery; do not delete the obligation
`helmsman_credential_cleanup_pending`, `_pending_seconds`, and `_last_success_timestamp_seconds`	A readable cleanup request is not draining	Edge operator checks the Helmsman error log for the preserved direct-scrub cause, repairs the credential scrubber, and never reuses the consumed token
`helmsman_credential_cleanup_observation_error`	Helmsman cannot safely read or trust cleanup state	Edge operator checks the state mount, ownership, permissions, unexpected symlinks, and the Helmsman error log; a disappeared mount retains known pending age

Tenant, stream, cluster, node, hostname, fingerprint, and token values belong only in access-controlled logs/traces, never metric labels.

Forward-fix boundaries

Foghorn: repair tenant/cluster authority resolution; never bypass the signed caller check.
Quartermaster: repair caller propagation or canonical ownership data; never weaken owner/admin or service-only fields.
Enrollment: reissue and consume a scoped token; never mutate fingerprint keys directly.
Purser: replay the authoritative entitlement reconciliation with a fresh observation; never infer current access from a tier-name allowlist.
Navigator: leave failed outbox rows pending and restore idempotent delivery; never manually publish DNS that contradicts Quartermaster desired state.