Running Cluster Upgrades

Use frameworks cluster release apply to move a running cluster to a new platform release. It runs the schema phases, platform service upgrades, and gates of a release as one ordered, resumable operation. cluster migrate and cluster upgrade remain available as lower-level tools for recovery and diagnosis.

Do not use provision as the normal upgrade path

frameworks cluster provision converges the full manifest and may run install, configure, service, validation, and initialization tasks. Use it for initial deployment or intentional reconciliation, not as the routine way to roll a running cluster forward. See When to use provision.

This page describes the procedure every release shares. Each release’s GitHub Release note has an Upgrade section with what is specific to that release: the supported source version, the minimum CLI version, operator-managed inputs to add or change, required data migrations, and any extra steps. Read it before you start. See Release Lifecycle for channels and supported upgrade sources.

Before you start

• Upgrade the CLI first. Run frameworks update and check frameworks version against the release’s minimum CLI version. release plan and release apply refuse a CLI below that floor. A development build without a release version is refused unless you pass --unsafe-ignore-cli-version-floor.
• Set the operator inputs the release note names in GitOps, using the repository’s SOPS helper for secrets. release apply renders every service it will install or upgrade before changing anything and refuses, listing each key, when a required input is missing.
• Declare services the release introduces. release apply installs a new platform service only when the manifest names it: add the service under services: and, when it owns a database, an entry under infrastructure.postgres.databases. release apply then creates and migrates that database before the service starts. A service another service requires, such as Bosun (added in v0.3.11, required by Bridge), fails release plan until it is declared. Bosun also needs BOSUN_FIELD_ENCRYPTION_KEY, which cluster secrets generate-shared does not produce.
• Pick the channel. Production manifests use channel: stable. Staging manifests use channel: candidate, which follows the newest tag, GA or RC. See Version selection.

Recommended flow

1. Print the static plan. It needs the manifest and the release metadata, but no SSH session, service connection, or control-plane login. An age key is needed only when the manifest’s host inventory is SOPS-encrypted.

   frameworks cluster release plan --version stable

2. Check the active context, then run the live dry-run. The dry-run connects to the cluster and runs every preflight gate and host precheck without changing anything:

   frameworks context check
   frameworks cluster release apply --version stable --dry-run

3. Apply the release:

   frameworks cluster release apply --version stable --yes

   If a step fails, fix the cause and rerun the same command. Work that already completed is skipped.

4. If the release declares required data migrations, release apply stops at the postdeploy phase until they are complete. Run and verify each one, then rerun release apply:

   frameworks cluster data-migrate list --to-version vX.Y.Z
   frameworks cluster data-migrate run <service>.<id>
   frameworks cluster data-migrate verify <service>.<id>
   frameworks cluster release apply --version vX.Y.Z --yes

   See Data migrations.

5. If the release note says the rendered ingress or observability configuration changed, apply the interface layer. release apply does not own it:

   frameworks cluster provision --version vX.Y.Z --only interfaces --dry-run
   frameworks cluster provision --version vX.Y.Z --only interfaces

   --version pins the provision to the release you applied, whatever the manifest channel currently resolves to.

6. Reconcile service-owned control-plane state only when the desired bootstrap model changed (tenant metadata, billing catalog, managed accounts, assignments, and similar platform state):

   frameworks cluster control-plane plan
   frameworks cluster control-plane reconcile

   The frameworks system-tenant alias is a stable identity. Reconciliation updates its mutable metadata but never replaces its UUID.

7. Verify health and state:

   frameworks cluster status
   frameworks cluster doctor --deep
   frameworks cluster diff

8. After the rollback window closes, apply any deferred contract migrations. release apply prints the command but never runs it. Contract migrations need a backup that started less than an hour earlier and still matches the migration ledgers:

   frameworks cluster backup create --to /var/backups/frameworks --all
   frameworks cluster migrate --phase contract --to-version vX.Y.Z --dry-run --backup <printed path>
   frameworks cluster migrate --phase contract --to-version vX.Y.Z --backup <printed path> --yes

   See Backup and Restore.

What release apply does

Preflight

Before the first change, and under --dry-run too, release apply refuses to start when:

• the CLI is below the release’s minimum CLI version;
• a platform service replica runs a version below a declared minimum source version, or a version that is not a release version;
• an earlier release is unfinished: its postdeploy migrations are missing from a PostgreSQL, YugabyteDB, or ClickHouse ledger, or its required data migrations are incomplete. The error names the release to apply first. Databases this release creates are exempt;
• a planned artifact does not resolve for its host’s deploy mode and architecture;
• a required operator input is missing from a service’s rendered environment.

Stages

release apply prints four numbered stages:

1. Pre-upgrade host convergence. In order: the node baseline and OS tuning on every host, Privateer on every mesh host, every Redis server and Sentinel, the data services and managed dependencies described below, the declared Kafka topics, and MirrorMaker2 workers. Each host runs the same role as cluster provision in check mode first, and a host that already matches is skipped without a restart. cluster release plan and release apply --dry-run list every step and, per host, the role tasks that would change. The node baseline (operator tools, journald and logrotate retention, key-only sshd, a fail2ban sshd jail; see Key-Only SSH) and OS tuning converge in waves that start with a single canary host followed by batches of four; after each wave a new SSH login must succeed on every host of that wave before the next wave starts. They restart no platform service: sshd reloads and journald restarts only on a host whose configuration changed. Privateer restarts in waves that start with a single canary host, and mesh health is verified after each wave before the next starts. Privateer’s DNS socket stays open across its restart, so .internal lookups are delayed rather than failed. A Sentinel-mode Redis instance converges one host at a time: the servers that are not the live primary, each of which must then report a synced replication link, then each Sentinel, which must then pass SENTINEL CKQUORUM, then the live primary. When the primary’s configuration changed, Sentinel first fails it over to a synced replica, and the demoted server must sync from the new primary. The live primary is read from the servers, not the manifest, so a primary Sentinel moved is handled. The stage refuses to continue when two servers both report the primary role. A single-mode instance restarts in place, and only when its configuration changed. Redis and Sentinel log to journald (journalctl -u frameworks-redis-<instance>); convergence removes the old unrotated files under /var/log/frameworks/redis. The Kafka topic step creates declared topics and applies declared topic configuration. MirrorMaker2 workers converge their configuration first, then every changed worker restarts together.
2. Expand migrations. Service databases the manifest declares but the cluster does not have yet are created first, with owner and runtime roles and the current baseline. Then expand migrations run on PostgreSQL/YugabyteDB and ClickHouse. The post-expand report follows.
3. Service upgrades and reconciliations. Platform services upgrade in dependency order, with any reconciliation transitions the release declares run at their declared points. A platform service the manifest adds or moves to a new host is installed there. Once the services are upgraded, the Quartermaster bootstrap runs with this release’s rendered desired state, as cluster provision runs it: the service registry, including each instance’s health endpoint, the service catalog, and clusters. The desired state renders before the first mutation, so a render failure refuses the release up front. Replicas and MirrorMaker2 workers the manifest no longer places are removed only after every desired replica is healthy, and then the service-cluster assignments are reconciled. A dry run reports these steps without running them.
4. Postdeploy migrations. These refuse to run until the release’s required data migrations are complete. The Purser tier catalog reconcile follows.

After postdeploy, release apply checks that every replica of the auth services and the web app runs the release’s version, since a readiness check cannot detect an older web app bundle. It then sets the edge release target, from which Foghorn rolls edge nodes (see Edge nodes). Contract migrations are printed, never run.

A reconciliation transition is a Check, Apply, Verify step that converges a declared invariant against the authoritative state. On a rerun it is checked again and skipped when complete. A blocked transition, for example when authoritative replicas disagree, stops the release with its reason. release plan lists the transitions a release declares, or none.

Data services and managed dependencies

Stage 1 applies the roles of the data services and managed dependencies after Redis, in this order. Only a host whose role check reports a change is touched, one host at a time, and the first gate that does not pass stops the release with the resume hint. A rerun skips converged hosts and gates every remaining change on a healthy service first. release plan and release apply --dry-run list the hosts; the dry-run shows each host’s pending role changes.

Service	Order and gates
YugabyteDB	The engine first, when a node runs another engine than the release pins: every master, then every tserver, then finalize (below). Then masters: a changed master restarts only after the masters confirm it can go down, and every master must be alive with one leader before the next node. Then every tserver whose applied configuration differs restarts, each only after the masters confirm the node can go down, and the next only once the node serves YSQL, its tserver is alive, and no tablet is under-replicated or leaderless.
ClickHouse	Node by node. Before and after each change, every node must reach Keeper and have no replica that is read-only, lost its Keeper session, is more than 30 s behind, or has parts to fetch.
Kafka	Per cluster, controllers then brokers. Before and after each change, read through another broker: a quorum leader, every controller within 100 records of it, and no under-replicated or unavailable partition.
vmagent, vmalert, vmauth	A changed scrape config, rule file, or auth config is validated with -dryRun and reloaded; a new binary or unit restarts the process.
Control-plane nginx and Caddy	A changed config is tested (nginx -t, caddy validate) before the reload; a failed test keeps the running config.
Chatwoot, Listmonk, Metabase, Grafana	The compose project is rendered and brought up, which recreates only the containers whose configuration changed.

When the release pins a new YugabyteDB engine, release apply upgrades it before converging any Yugabyte configuration, the same way frameworks cluster upgrade yugabyte does. Each node still on the previous engine installs the new one and restarts only its master, after the masters confirm that master can go down and before every master is alive with one leader again. Then every master still on the previous binary restarts, then every tserver through the whole-node gate, and once every node serves YSQL and no tablet is under-replicated or leaderless the upgrade is finalized (AutoFlags promoted, YSQL catalog upgraded). Nodes already on the pinned engine skip this step. If a gate stops the release, rerun it: each node’s engine and running binaries are read again, and a marker the upgrade leaves on every node until it is finalized makes the rerun finish it. A release that pins an older engine than a node runs is refused. release apply --dry-run prints would upgrade engine <from> -> <to> (masters, tservers, finalize) without changing anything, and release plan names the pinned engine; each node’s engine is checked at apply.

On the first release that carries a new Yugabyte flag or unit, every node’s master and tserver restart once through these gates. Kafka brokers keep at most log.retention.bytes (2 GiB) per partition in addition to each topic’s retention.ms.

Rollout order within a service

One service finishes on every replica before the next service starts. Replicas roll by the service’s tier:

Tier	Services	Replica order
Control	Quartermaster, Navigator, Commodore, Purser, Bridge, and unclassified ones	First replica alone as canary, then one at a time
Media	Foghorn, Chandler	One replica per cell at a time, cells in parallel
Other	Periscope, Decklog, Signalman, Bosun, Lookout, Skipper, Steward, and similar	Together, up to the service’s max_unavailable

max_unavailable caps how many replicas of a media or other service are down at once. The built-in default is 1 for Foghorn, Chandler, and the control services. Stateless workers such as Signalman, Decklog, the Periscope services, Lookout, Bosun, Skipper, Deckhand, and Steward default to all replicas at once. Override it per service in the manifest:

services:
  decklog:
    hosts: [regional-eu-1, regional-eu-2, regional-eu-3]
    update_strategy:
      max_unavailable: 1

A failed replica stops the service’s rollout before any further replica starts, and the release stops there.

Post-expand report

Between expand and the service upgrades, release apply reads the Quartermaster database in a read-only transaction and prints:

• operator-managed nodes without an identity key, which block postdeploy until they re-enroll;
• customer-enrolled nodes without an identity key, which the new Quartermaster refuses at their next connect until they re-enroll;
• tenants with a custom subdomain or domain whose DNS entitlements are still unobserved, and whether Purser’s entitlement handoff receipt exists.

Each list shows up to 50 rows with the total, and keyless nodes come with the re-enrollment instruction. The report never fails the release. post-expand report: nothing pending is the all-clear. post-expand report: not run (expand not applied) appears on a dry-run before expand has been applied and is not an all-clear. To see the full report before any service is replaced, apply expand on its own and then run the dry-run:

frameworks cluster migrate --phase expand --to-version vX.Y.Z
frameworks cluster release apply --version vX.Y.Z --dry-run

What release apply does not change

• Other managed dependencies with their own manifest pins: VictoriaMetrics, Alertmanager, Prometheus, MistServer, and the Livepeer gateway and signer. Use cluster diff to see desired and observed state, and cluster provision when their pins change.
• A standalone PostgreSQL server, which cluster provision converges.

A direct cluster upgrade <service> refuses a host where the service is not installed; only release apply installs a newly placed platform service.

Edge nodes

Deploy and migrate the control plane before the edges. A new Foghorn does not wait for any edge.

After the release sets the edge release target, Foghorn updates Helmsman, MistServer, and Caddy over the Helmsman control stream on native and single-image container nodes. A component whose version did not change is not rolled.

Foghorn rejects a Helmsman whose control-protocol version is below its minimum at registration. Such an edge is absent from platform routing and cannot receive an update over the rejected stream, so it has to be redeployed out of band:

frameworks edge provision --manifest <edge-manifest>

A release that raises the minimum says so in its release note, with the interruption window. Always redeploy an edge through edge provision rather than pulling a new image against an old rendered deployment: the render carries the persistent Helmsman state volume, which holds node identity and undelivered control acknowledgements.

Edge configuration

Foghorn rolls component versions; it does not change an edge’s rendered configuration (env files, systemd units, the compose file). A new CLI carries the current edge configuration, and re-running edge provision with it is how that configuration reaches the edges:

1. Install the new CLI.

2. See which edges drifted and what would change:

   frameworks edge provision --manifest <edge-manifest> --dry-run

   Every node is checked in Ansible check mode. The plan per node is skip (no drift), apply (changes that keep media running), or drain+apply, with the diff and the changed tasks.

3. Apply it:

   frameworks edge provision --manifest <edge-manifest>

Re-running provision reuses each node’s enrolled identity. Enrolled nodes are prechecked first and only nodes that drifted are applied, one at a time; --parallel only applies to fresh installs. A MistServer binary change goes through MistServer’s rolling reload and keeps its streams. When the apply must restart MistServer or Caddy, or recreate the edge container (a new compose file, image, or env file), the CLI first sets the node to draining through Foghorn, waits until its sessions end (up to 4 hours, as cluster nodes remove does), applies, and then restores the mode the node had. A node an operator already set to draining or maintenance keeps that mode. If the sessions do not end in time the node is left unchanged and back in normal; if the apply fails the node stays draining until you resume it with frameworks edge mode normal --ssh <target>.

Self-hosted single edges use the same path: frameworks edge provision --ssh <user@host> --dry-run, then without --dry-run, with the flags the node was provisioned with.

Each provision writes a provisioned-config marker on the node (the CLI version and a digest of the rendered configuration). Helmsman reports it to Foghorn, which returns it in node health, and frameworks cluster doctor lists every edge whose configuration was rendered by an older CLI (or before markers existed). frameworks edge doctor shows the same for the node it runs on.

Version selection

--version accepts:

Value	Meaning
stable	Latest stable release
candidate	Newest release, including release candidates
latest	Alias of candidate
rc	Newest release candidate only
vX.Y.Z[-rcN]	That release
omitted	The manifest’s channel, or stable when it has none

cluster release plan, cluster release apply, cluster upgrade, and cluster provision all accept it. Set the manifest channel with set-channel, which accepts stable, candidate, or rc:

frameworks cluster set-channel stable
frameworks cluster set-channel candidate

Schema and data migration phases

Phase	What changes	Rollback expectation
Expand	Add nullable/defaulted columns, tables, indexes, or broader constraints	Old and new binaries both run against the expanded schema
Deploy	Roll out binaries that can read and write both shapes	Roll back by redeploying the old binary
Data	Populate new rows or columns from old data in batches	Resumable and idempotent
Postdeploy	Prefer or require the new shape once verified	Roll back only while fallback or dual-write remains
Contract	Drop old columns, tables, values, and fallback code	Not a normal rollback point; requires a forward fix or restore

cluster migrate applies the PostgreSQL/YugabyteDB and ClickHouse phases, records each migration with its checksum, rejects edited applied migrations, and refuses a database below the release’s baseline floor. It does not run data migrations. cluster doctor compares the embedded migrations with each database’s ledger and reports pending migrations and checksum drift. See PostgreSQL Migrations and ClickHouse Migrations.

Data migrations

A release can require data migrations to complete before a later phase. release apply, cluster upgrade, cluster migrate --phase postdeploy|contract, and cluster provision check them and refuse to continue; none of them runs one. Run them with cluster data-migrate:

Command	Purpose
list --to-version vX.Y.Z	Data migrations the release catalog requires
run <service>.<id> [--dry-run]	Run one to completion; --dry-run is read-only
status <service>.<id>	Persisted state on the owning service host
verify <service>.<id>	Read-only verification
pause / resume <service>.<id>	Pause or resume a running migration

A service binary that cannot report a required migration’s state counts as incomplete. Running an irreversible data migration needs --backup with a backup that started less than an hour earlier and whose ledgers match; --dry-run does not. A migration is irreversible when the release lists its service under automatic-rollback exclusions, or when the service binary marks it irreversible or does not say.

Foghorn also runs cell-local data migrations against each cell’s own database in the background. They are not in the release catalog, so no gate checks them. When a release note names one, check it in every cell, using each cell’s Foghorn service name from the manifest:

frameworks cluster data-migrate status foghorn-eu.<id>

Rollback behavior

When a service fails its readiness check after an upgrade, cluster upgrade and release apply redeploy the previous version and check it again. Rollback is skipped when:

• the release lists the service under automatic-rollback exclusions, because the previous binary could not pass the new health contract (release plan lists these);

• the service was newly installed or its previous version could not be detected;

• you pass --no-rollback, to inspect a failed service in place:

  frameworks cluster upgrade bridge --version rc --no-rollback

Rollback swaps the service artifact only. It does not undo schema or data migrations. Expand migrations stay in place for the older binary, which is why a release keeps destructive changes in the contract phase.

Health and readiness

Every FrameWorks Go service answers two endpoints on its HTTP port:

Endpoint	Meaning
/health	Live: the process is running. Container health checks and ingress probes use it.
/ready	Able to serve: 200 only after the service’s own database, ClickHouse, Kafka, or object-store checks pass and its gRPC listeners serve. It returns 503 while a check fails and from the moment the service starts draining.

Services serve /ready from v0.3.11 on; Chandler has always served it. Earlier releases answer 404 on /ready.

cluster upgrade and release apply wait up to 150 seconds for each deployed service to report ready before the upgrade counts as failed. A v0.3.11 or later binary is checked on /ready, and a rollback to an earlier binary on /health. A service that cannot reach its own database within the wait fails its rollout: fix the dependency and rerun the command.

cluster apply, cluster doctor, Quartermaster, and Navigator cannot tell which release an instance runs, so they check /health for every Go service except Chandler. They move to /ready in the release that requires v0.3.11 as the minimum source.

Post-rollout staging smoke

After a requested staging rollout, run the stack scenario suite against the deployed staging services from an operator checkout on the private network:

STACK_TARGET=staging \
STACK_ENDPOINTS_FILE=/private/path/staging-stack-endpoints.sh \
STAGING_SERVICE_MAP_FILE=/private/path/staging-service-map.txt \
make verify-stack

The same scenario scripts and per-scenario log checks run on remotedev before the rollout. The private endpoint file supplies the test tenant, credentials, mailbox, webhook receiver, and deployed CLI context; the service map connects scenario aliases to native hosts and systemd units. A failed or blocked scenario is not a passing smoke. STACK_SCENARIOS selects an individual scenario for diagnosis, and STACK_REPEAT repeats a race check. The staging target neither provisions nor tears down the cluster. See docs/development/remote-development.md in the monorepo for the endpoint and service-map contract.

When to use provision

Use frameworks cluster provision for:

• first deployment of a cluster;
• adding a host, an infrastructure component, or a managed dependency (a new platform service is installed by release apply);
• applying changed managed-dependency pins or the interface layer;
• repairing drift where you intentionally want a full converge.

cluster provision runs the same required-data-migration gate before it deploys applications, so redeploying compute against a preserved database cannot start newer binaries ahead of a required migration. A database created from a baseline is exempt from migrations that baseline already includes.

Why some services keep their older version label

A release manifest pins every service by content identity (Docker image digest, native tarball SHA-256), not by version label. When a service’s code did not change, the new manifest carries its previous service_version forward. A v1.2.3 release can list helmsman at v1.2.1 because that release produced the bytes you run. platform_version names the release a manifest belongs to; service_version names the release that built each component.

• cluster upgrade plan shows each service’s target image digest and tarball checksum. When the installed identity matches the target, nothing is pulled again.
• Foghorn compares edge component versions with the ones each node reports, so a release that carries Helmsman, MistServer, and Caddy forward does not roll any edge.
• A carried service is the same artifact that shipped earlier, not a downgrade.

Local compose

The root docker-compose.yml is a seeded development stack, not an operator-managed cluster. Incompatible schema changes are handled there by recreating volumes and reseeding (docker compose down -v, then docker compose up -d). That is not an operator upgrade path.