Running Cluster Upgrades

Do not use provision as the normal upgrade path

frameworks cluster provision converges the full manifest and may run install, configure, service, validation, and initialization tasks. Use it for initial deployment or intentional reconciliation, not as the routine way to roll a running cluster forward.

Release notes can require provisioning

Some releases change manifest-rendered runtime config, service environment, DNS/TLS provider settings, or cluster metadata that a binary-only upgrade cannot apply. When a release note explicitly includes frameworks cluster provision in the upgrade sequence, run it at that point; otherwise keep using cluster upgrade for normal service rollouts.

Use frameworks cluster upgrade for service binaries and images, and frameworks cluster migrate for PostgreSQL/YugabyteDB schema migrations. The upgrade command downloads or pulls the target release artifact, lets the role restart the service when the artifact or config changes, validates health, and rolls back on health-check failure unless --no-rollback is set.

Migrations are not magic data migrations

cluster migrate applies PostgreSQL/YugabyteDB schema migrations. It does not prove that existing rows have been transformed into a new semantic model, and it does not make destructive migrations safe for rollback. Release notes must call out any required data migration, read/write flip, or contract step.

Recommended flow

1. Check the current state:

   frameworks cluster status

2. Preview database migrations:

   frameworks cluster upgrade plan --version stable
   frameworks cluster migrate --phase expand --dry-run

3. Preview the service or cluster upgrade:

   frameworks cluster upgrade quartermaster --version stable --dry-run
   frameworks cluster upgrade --all --dry-run

4. Apply PostgreSQL/YugabyteDB migrations before upgrading services that depend on new schema. This normal path assumes the pending migrations are compatible with both the currently running service version and the target version:

   frameworks cluster migrate --phase expand

5. Upgrade one service, or all enabled services in dependency order:

   frameworks cluster upgrade quartermaster --version stable --yes
   frameworks cluster upgrade --all --yes

6. Run any catalog-declared data migration after compatible binaries are deployed. For example, a billing model migration may need to populate new normalized tables from old JSONB columns before reads can be flipped:

   frameworks cluster data-migrate list --to-version v0.0.0
   frameworks cluster data-migrate run <service>.<id> --dry-run
   frameworks cluster data-migrate run <service>.<id>

7. Verify health and migration-specific checks after the rollout:

   frameworks cluster status
   frameworks cluster doctor

Control plane first, then edges

Deploy and migrate the control plane first; a new Foghorn comes up without requiring any edge to be ready. Foghorn enforces a minimum control-protocol version at sidecar registration: a sidecar declaring a version below the deployed minimum is rejected (FailedPrecondition, logged with the observed and required versions) and cannot connect until it is upgraded. Because rejection happens before the control stream is established, that sidecar cannot receive an in-band updater command. When a release advances the minimum with no mixed-version window, operators must redeploy affected edges out of band; the specific minimum and interruption window belong in that release’s notes.

0.3.0 makes a one-time protocol-v3 hard cut

Foghorn 0.3.0 rejects protocol-v2 Helmsmans. The provider control plane is upgraded before the edge release target is synchronized, so existing v2 edges temporarily disappear from platform routing and cannot auto-update over their rejected control stream. After the control-plane rollout, manually redeploy every self-hosted edge to the 0.3.0 edge image. An offline v2 edge returning later must also be manually updated before it can reconnect.

Because a connected sidecar is always on the current protocol, its artifact inventory is always a versioned whole-node snapshot (a monotonic per-connection sequence), which Foghorn treats as authoritative — it can add and remove copies immediately. There is no unversioned/legacy inventory handling.

Foghorn still applies an artifact-only readiness cordon: on every (re)connect an edge node is excluded from stored-artifact routing until its active connection delivers its first complete versioned snapshot. Live ingest and stream routing are unaffected — only stored-clip/VOD/DVR playback from that edge is withheld while it has not yet reported.

Therefore:

• Upgrade/migrate the control plane first; it does not wait on the edge.
• Redeploy every edge afterward. A sidecar below the minimum is absent from platform routing until it reconnects on a supported protocol. Existing local media processes may continue running, but that does not make the rejected node available for new platform assignments. A supported sidecar is cordoned only for stored artifacts until it sends its first versioned snapshot.
• A supported N−1 / third-party edge protocol range applies only where a release declares one; absent that, treat the deployed minimum as a hard floor.

Thumbnails: deterministic serving, install-time ordering

Chandler is a dumb static-object server: a request path maps DETERMINISTICALLY to an S3 key (thumbnails/{asset}/{file}), which it streams and caches. It makes no per-request call to Foghorn and needs no FOGHORN_INTERNAL_URL. SERVICE_TOKEN gates only its internal cache-invalidation endpoint; serving does not need it.

has_thumbnails (and the API URL) is exposed only after Foghorn has projected a thumbnail to that deterministic served key, so the API never advertises a thumbnail Chandler cannot serve. The publish/projection internals (how Foghorn picks and projects the winning attempt, and how it recovers a published-but-unprojected one) are in the thumbnails architecture doc.

At serving time there is no per-request resolver call and no publish-time handshake, so Chandler has no runtime dependency on Foghorn’s version. Installation carries one ordering edge: Chandler’s readiness check reads a sentinel the in-cell Foghorn establishes at boot, so the planner deploys Chandler after that Foghorn. The topology rule also stands: a Foghorn is rejected if its cluster has no Chandler (a publisher with nothing to serve its thumbnails).

The served URL is {chandlerBase}/assets/{asset}/{file} — a stable, deterministic shape that is Chandler’s only public route — so a URL producer that upgrades before the media-cell Chandler never emits a path an already-running Chandler cannot serve.

The projection to the deterministic key is eventually consistent, not strictly serial (a database lock cannot fence a non-atomic S3 copy): a straggler overwrite that lands within the max-copy window is corrected by the winner’s automatic re-copy, and a straggler that resurrects a deleted object within the window is reclaimed by a delayed second cleanup sweep. Deletion tombstones the asset (the API stops returning its URL immediately) and sweeps the thumbnails/{asset}/ prefix; the tombstone stops any new projection, and the delayed sweep reclaims a copy that lands within the window. A copy whose provider tail exceeds the assumed window is not reclaimed — the accepted, rare residual risk (a re-exposed cosmetic thumbnail), since thumbnails are regenerable derivatives, not primary media.

Live-stream thumbnails are minted on the ingest cell (its own per-cell Foghorn database). The owning cell is recorded durably before any thumbnail is minted, and deletion fans the cleanup out to every recorded owning cell — so per-cell databases converge without a shared DB. An empty owner set means “no thumbnails”: there is no cross-cell guess.

The release command: reconciliations run automatically

Cross-service reconciliations are not a manual intermission. frameworks cluster release apply runs the whole release as one ordered, resumable plan and interleaves reconciliation transitions at their declared points:

expand migrations
→ service upgrades in dependency order, with reconciliation transitions
  interleaved at their declared points
→ postdeploy migrations

frameworks cluster release apply --dry-run   # show the plan + every gate/Check, mutate nothing
frameworks cluster release apply --yes        # run it

A reconciliation transition is a constrained Check → Apply → Verify node registered by a compiled handler id. It converges a declared invariant against natural authoritative state, so the release is resumable: if a later step fails, rerun release apply — each transition re-Checks reality and skips whatever is already Complete. A Blocked transition (e.g. authoritative replicas that disagree, or an authority that is unreachable) halts the release with a clear reason instead of guessing, and a transition whose precondition does not apply reports Not applicable rather than mutating.

cluster migrate, cluster upgrade, and the low-level reconciliation subcommands remain as recovery/debugging tools; release apply is the normal path for a cluster that is already installed — it upgrades existing services and runs the reconciliation transitions, and it refuses to “upgrade” a service that is not yet installed. A from-scratch cluster is brought up with cluster provision (see When to use provision), which is where install ordering and initial descriptor establishment happen. After the initial provision, use release apply for subsequent rollouts. At serving time Chandler has no runtime dependency on Foghorn’s version (it serves a deterministic key with no per-request Foghorn call); the Foghorn-before-Chandler ordering is an install-time property of the planner, shared by both provision and release apply. A direct cluster upgrade foghorn still enforces the storage-descriptor prerequisite. Upgrade the frameworks CLI to at least the release’s min_cli_version before applying — the CLI validates that from the fetched release metadata, but a CLI predating that check cannot be stopped by it, so treat the CLI upgrade as the first step.

Cold-storage freeze: the control-protocol contract

Cold-storage freeze (uploading a stored clip/VOD to durable S3) uses a staged, server-minted protocol: Foghorn mints the attempt id, presigns an attempt-scoped staging PUT, and publishes the verified staging object itself — copying it to a fresh immutable attempt-versioned key and atomically flipping the object pointer. The sidecar must upload to the staging key it is handed and echo the server-minted attempt id at completion.

Foghorn gates freeze dispatch on the sidecar’s observed control-protocol version (never a per-request self-asserted flag): a sidecar below the staged-freeze minimum is refused freeze assignments — both the interactive permission request (denied with sidecar_protocol_unsupported) and proactive reconciler dispatch fail closed, and the artifact simply stays local and retryable.

The registration minimum (see “Control plane first, then edges” above) is at or above every staged-operation minimum — so any sidecar that connects at all already satisfies the freeze gate, and a below-minimum sidecar is rejected at registration rather than run in a degraded mode. There is no “upgrade edges first” or “run old sidecars against a new Foghorn” step: deploy the control plane, then redeploy every edge.

Version selection

--version accepts:

Value	Meaning
stable	Latest stable release manifest
rc	Latest release-candidate manifest
v0.0.0-rc1	Specific release manifest
omitted	The cluster’s configured channel, or stable

Set the default channel with:

frameworks cluster set-channel stable
frameworks cluster set-channel rc

Schema and data migration phases

PostgreSQL and YugabyteDB migrations are managed by frameworks cluster migrate. Run them before deploying service versions that require new tables, columns, or indexes.

Normal rolling upgrades must follow an expand/data/postdeploy/contract model:

Phase	What changes	Rollback expectation
Expand	Add nullable/defaulted columns, tables, indexes, or broader constraints	Old and new binaries both run against the expanded schema
Deploy	Roll out binaries that can read/write both shapes	Roll back by redeploying the old binary
Data	Populate new rows/columns from old data in batches	Background data migration is idempotent and safe to pause/resume
Postdeploy	Prefer or require the new shape once verified	Roll back only while fallback/dual-write remains
Contract	Drop old columns/tables/values and fallback code	Not a normal rollback point; requires a new forward fix or restore

Do not put destructive contract work in the same normal upgrade step as the expand work. Column drops, table drops, enum/check narrowing, required fields without a complete background data migration, and semantic rewrites that old binaries cannot handle require a later contract release or an explicit downtime runbook.

For billing and other audit-sensitive domains:

• finalized invoices should remain immutable unless the release explicitly documents a correction,
• pricing and rating changes should be effective-dated,
• draft/open records may be recalculated only by a documented background data migration or rating command,
• release verification should include counts and totals, not just service health.

cluster migrate applies ClickHouse schema migrations (expand/postdeploy/contract phases) for the configured ClickHouse databases alongside Postgres, and refuses to proceed if the cluster is below the required baseline floor. Follow the ClickHouse Migrations runbook for phase ordering and replicated-cluster specifics, and apply the expand phase before upgrading the analytics services that depend on it.

Rollback behavior

If health validation fails, cluster upgrade attempts to roll the service back to the previously detected version. Keep rollback enabled for normal production rollouts.

This rollback only swaps the service artifact back. It does not undo schema or data migrations. That is intentional for expand-compatible changes: the old binary should keep working while extra tables/columns remain in place. If a release includes a migration that makes the old binary invalid, treat it as a downtime upgrade with a tested database restore or forward-fix plan.

Use --no-rollback only when you want to inspect a failed upgraded service in place:

frameworks cluster upgrade bridge --version rc --no-rollback

CLI support status

Today, frameworks cluster migrate --phase expand applies pending embedded PostgreSQL/YugabyteDB migrations from the database/version/phase directory layout. The command records phase and checksum in _migrations, rejects edited applied migrations, and does not run service-specific data migrations.

frameworks cluster doctor also checks the migration ledger: embedded PostgreSQL/YugabyteDB migrations are compared with _migrations in each configured database so pending required migrations and checksum drift show up during normal diagnostics.

frameworks cluster upgrade plan shows the target rollout order and embedded SQL migrations by phase. The CLI embeds a release catalog for service database ownership, data-migration requirements, and the per-release min_cli_version tooling floor. cluster upgrade uses that catalog together with live _migrations and _data_migrations state to refuse unsafe rollouts before changing binaries. Engine schema checks (expand + prior postdeploy) apply to a service that owns a PostgreSQL/YugabyteDB or ClickHouse schema; the required data-migration check applies to every service, because a data migration can gate a binary that only reads the migrated data without owning its schema. Ordering is enforced against the actual migration ledger, not a version label. A non-concrete (dev/unversioned) CLI is refused against a release’s min_cli_version unless you pass --unsafe-ignore-cli-version-floor. cluster provision runs the same required-data-migration gate before it deploys applications, so a clean-compute redeploy cannot start newer binaries ahead of a required migration on a preserved database. The gate runs after database initialization and reads durable _schema_baseline provenance: a data migration introduced strictly below the floor a database was born at is folded into that baseline, so a fresh (baseline-born) database is exempt without special-casing database existence.

frameworks cluster data-migrate is the first-class surface for resumable service-owned data migrations. list shows catalog-declared work and adoption state; run --dry-run is read-only; run, status, verify, pause, and resume operate through the service binary on the target host. If a release declares a required data migration but the service binary cannot report state, upgrade and postdeploy/contract gates fail closed instead of treating the unknown as safe.

Contract migrations are intentionally separate. cluster migrate --phase contract runs only after the matching data-migration gate succeeds, so destructive cleanup cannot be mixed into the routine expand/deploy step.

Local compose exception

The root docker-compose.yml is a seeded development stack, not an operator-managed cluster. For local development, incompatible schema changes can be handled by recreating volumes and reseeding:

docker compose down -v
docker compose up -d

That shortcut is acceptable for local compose because demo data is disposable. It is not an operator upgrade path.

When to use provision

Use frameworks cluster provision for:

• first deployment of a cluster,
• adding a new service or host to the manifest,
• repairing drift where you intentionally want a full converge,
• re-rendering infrastructure after changing role inputs.

For routine release rollouts, use cluster upgrade plus cluster migrate.

Why some services keep their older version label

A FrameWorks release manifest pins every service by content identity (Docker image digest, native tarball SHA-256), not by version label. When a service’s source code did not change since an earlier release, the new manifest carries that service’s previous service_version label forward verbatim — for example, a v0.2.40 release can legitimately list helmsman with service_version: v0.2.37 because that is the release that actually produced the bytes you are running.

This is the artefact provenance model: platform_version says “what release this manifest belongs to,” and service_version per component says “which release actually produced this component’s bytes.”

Practical consequences:

• cluster upgrade plan shows each service’s target image digest and native tarball checksum, not just version labels. If your installed identity equals the target identity, nothing is re-pulled regardless of the umbrella version.
• For edge nodes, Foghorn’s release reconciler compares per-component versions against foghorn.node_components. Carried-forward components keep their old label, so a release that doesn’t touch helmsman/mist/caddy does not roll any edge node.
• A “carried” service is not a downgrade. It is the same artefact that shipped earlier, content-pinned forward into the new release.

Edge nodes auto-update in both modes

Foghorn auto-updates Helmsman, MistServer, and Caddy components over the Helmsman control stream for native nodes and single-image container nodes alike; MistServer binary updates use an in-place controller reload in both. Container nodes keep binaries on persistent volumes, so in-place updates survive container restarts. Only the retired multi-container docker mode was excluded from reconciliation.

Release notes call out carried components when relevant.