Operator CLI Reference

The FrameWorks CLI provisions, upgrades, and inspects clusters and edge nodes. Provisioning uses Ansible for some infrastructure tasks, while most commands run direct SSH or API calls.

Global Flags

--config string     Path to config.yaml (default: $XDG_CONFIG_HOME/frameworks/config.yaml,
                    which resolves to ~/.config/frameworks/config.yaml when XDG is unset)
--context string    Context name for this invocation; overrides the saved current context
--output string     Output format: json|text (default: text)
--verbose, -v       Enable verbose logging

Run frameworks config path --kind config to print the canonical path the CLI is using right now, honoring any --config override.

Note: the manifest-source flags (--manifest, --gitops-dir, --github-repo, --github-ref, --cluster, --age-key, --github-app-id, --github-installation-id, --github-private-key) are persistent on the frameworks cluster command group — set on any cluster * subcommand. They are not global.

Command Catalog

frameworks commands prints the Cobra command tree for tools that need to discover available CLI paths and flags without hardcoding them.

frameworks commands
frameworks commands --output json
frameworks commands --include-hidden --output json

Personas and Access Modes

frameworks setup creates a context with a persona and an access mode. The persona captures the operator’s intent; the access mode controls how operator-originated CLI calls reach control-plane gRPC services.

Persona	Default shape	Typical commands
`platform`	Public Bridge plus GitOps-backed access to core services. Setup defaults to `access_mode=ssh`.	`admin`, `services`, `dns doctor`, `mesh status`, `context check`, `cluster *`
`selfhosted`	Public Bridge for a tenant-owned BYO edge footprint. The operator does not run Quartermaster or Foghorn.	`login`, `edge deploy`, local edge diagnostics
`user`	Public Bridge/account API only; this is the persona for someone who is not running their own edge node.	`login`, insights, Skipper, account-facing workflows

Access modes:

Mode	Behavior
`local`	Use the endpoint fields saved in the context. Existing contexts default to this when `access_mode` is absent.
`ssh`	Platform contexts load the saved GitOps manifest, merge `hosts_file`, locate the host running the service, and open an SSH local-forward for the gRPC call.
`mesh`	Platform contexts dial the manifest WireGuard address directly. Use this only from a machine already joined to the mesh.

frameworks context set-access-mode ssh
frameworks context check

For platform contexts, per-service non-local URLs saved with frameworks context set-url still override manifest-derived targets, but they must declare transport explicitly: use --tls for TLS endpoints or --allow-insecure for plaintext gRPC. Localhost defaults, manifest SSH tunnels, and mesh-derived addresses keep their established transport behavior. Localhost defaults are ignored in ssh and mesh modes so a newly created platform context does not accidentally dial the operator laptop for core services. Self-hosted/BYO edge contexts use Bridge and do not collect core gRPC endpoints.

Older configs that still say persona: edge are normalized to user; edge is not a user-facing persona.

Cluster Manifest-Source Flags

All frameworks cluster * subcommands share a persistent flag set for selecting the cluster manifest. The resolver picks the highest-priority input the operator provided (flag > env > the active context’s saved default > cwd heuristic).

Flag	Env var	Meaning
`--manifest`	`FRAMEWORKS_MANIFEST`	Path to a single `cluster.yaml`
`--gitops-dir`	`FRAMEWORKS_GITOPS_DIR`	Local gitops repo root (uses `<dir>/clusters/<cluster>/cluster.yaml`)
`--github-repo`	`FRAMEWORKS_GITHUB_REPO`	`owner/repo` to fetch from via GitHub App
`--github-ref`	`FRAMEWORKS_GITHUB_REF`	Branch/tag for `--github-repo` (default `main`)
`--cluster`	`FRAMEWORKS_CLUSTER`	Cluster name within the gitops repo
`--age-key`	`SOPS_AGE_KEY_FILE`	Age private key for SOPS-encrypted files
`--ssh-key`	—	SSH private key path, overriding ssh-agent/ssh config defaults
`--github-app-id`	`FRAMEWORKS_GITHUB_APP_ID`	GitHub App ID (for `--github-repo`)
`--github-installation-id`	`FRAMEWORKS_GITHUB_INSTALLATION_ID`	GitHub Installation ID
`--github-private-key`	`FRAMEWORKS_GITHUB_PRIVATE_KEY`	Path to GitHub App private-key PEM

Save defaults once with frameworks setup (interactive) or the non-interactive equivalents (frameworks context create <name> + frameworks context set-gitops-*). Explicit flags on a specific invocation always win over saved defaults.

Detection & Health Commands

frameworks cluster detect

Detect the current state of all services in the cluster using multi-method detection.

Usage:

frameworks cluster detect [flags]

Detection Methods:

Check inventory file (/etc/frameworks/inventory.json)
Check Docker containers
Check systemd services
Check listening ports
Try health endpoints
Try direct connections (DB, Kafka, etc.)

The text report groups discovered service state by host and detection method. Use --output json when a pipeline needs machine-readable state.

Flags: inherits the Cluster Manifest-Source Flags.

Examples:

# Detect all services (uses the active context's saved manifest source)
frameworks cluster detect

# Detect against a specific manifest
frameworks cluster detect --manifest production.yaml

# Detect against a local gitops repo
frameworks cluster detect --gitops-dir ~/work/gitops --cluster production

frameworks cluster doctor

Run read-only health checks across the cluster.

Usage:

frameworks cluster doctor [flags]

Checks:

Infrastructure: Postgres/Yugabyte, ClickHouse, and Kafka broker connectivity
Applications: Health endpoints for enabled services
Interfaces: Health endpoints for enabled interfaces
Observability: Health endpoints for enabled observability services
Database migrations: Embedded PostgreSQL/YugabyteDB migrations compared against each configured database’s _migrations ledger
Yugabyte layout: Declared and observed placement for every table and secondary index of each platform database, with distinct tablets and tablet peers. A database created before its colocated layout is reported as a warning; see YugabyteDB Physical Layout
Edge config version: Every registered edge whose configuration was rendered by an older CLI than the one running doctor, or that has no provisioned-config marker yet, from the marker each node reports through Foghorn node health. Needs lifecycle access (Quartermaster and Foghorn) from the active context; without it the check is reported as not checked. Re-run edge provision (with --dry-run first) to bring those edges current

With --deep, doctor additionally executes every enabled database-backed binary’s read-only schema capability pack against the live PostgreSQL/YugabyteDB and ClickHouse engines. PostgreSQL/YugabyteDB probes assume the prepared runtime role, so the check proves required objects and least-privilege grants together.

This is targeted binary-contract introspection, not a full physical-schema diff. Deep mode decrypts the shared env for authenticated service checks and ClickHouse credentials; Yugabyte administrative probes still use the role’s localhost-only trust path and then SET ROLE to validate application access.

The report groups infrastructure, application, interface, observability, and migration-ledger checks. Use --output json when a pipeline needs the current result shape.

The migration-ledger check treats contract migrations that have not run yet as healthy: after a release they wait for the rollback window, and doctor says so and prints the cluster migrate --phase contract command to run once it closes.

Flags: inherits the Cluster Manifest-Source Flags.

--deep - Also run authenticated control-plane checks and the read-only schema capability packs

Examples:

# Full health check against the active context
frameworks cluster doctor

# Against a GitHub-sourced manifest
frameworks cluster doctor --github-repo org/gitops --cluster production

# With verbose output
frameworks cluster doctor --verbose

# Verify live schema capabilities and prepared runtime-role grants
frameworks cluster doctor --deep

frameworks cluster status

Compare deployed service versions against the available versions for the cluster’s release channel.

frameworks cluster status
frameworks cluster status --manifest /etc/frameworks/cluster.yaml
frameworks cluster status --json

The report includes enabled services, interfaces, and observability workloads. It also prints control-plane readiness information when the active context has a system tenant ID from a previous provision run.

frameworks cluster preflight

Check host readiness before provisioning.

frameworks cluster preflight
frameworks cluster preflight --domain example.com

Checks Docker/Compose, service manager availability, disk space, standard edge ports, ulimits, Linux sysctls, /dev/shm, optional DNS resolution, and infrastructure connectivity when a manifest source is configured.

frameworks cluster drift

Survey observed service state and fail when it diverges from the manifest.

frameworks cluster drift
frameworks cluster drift --manifest /etc/frameworks/cluster.yaml
frameworks cluster drift --output json

Drift states include missing, stopped, wrong_mode, and wrong_version. This is the CI-friendly observed-state check; it exits non-zero when any divergence is found.

frameworks cluster diff

Typed change-kind survey: compare desired (manifest + release artifacts) against observed (sha256 of files on each host) for every service that has registered a fingerprinter.

frameworks cluster diff
frameworks cluster diff --only-services foghorn,bridge
frameworks cluster diff --only-hosts regional-eu-1
frameworks cluster diff --output json

Diff kinds: binary, env, unit, cert, infra, unknown. Services without a registered fingerprinter report unknown; cluster apply --confirm falls through to the heavy cluster provision path for them. Exits non-zero when any kind diff is detected.

frameworks cluster apply

Rolling-update planner that composes typed diffs (cluster diff) with per-service rolling-update strategy (max_unavailable, region_stagger, canary, primary_last). Per-service update_strategy blocks in the manifest override those defaults when a service needs a different rollout budget. Without --confirm it prints the plan and exits (dry-run). With --confirm it executes each host through the same role provisioner used by cluster provision, so env files, systemd units, handlers, and validation stay owned by the Ansible roles. It handles only non-version rollouts (env, unit, cert): a binary diff is a service version change and is refused, routed to the release-gated cluster upgrade (which runs the compatibility, schema / data-migration, storage-backend, and rollback gates); infra and unknown diffs are routed to cluster provision. The action label is reload when the diff is env-only on a SIGHUP-capable service; otherwise it is restart. After each wave, cluster apply waits for a per-service readiness gate (HTTP health for HTTP services, systemctl is-active otherwise) before advancing.

frameworks cluster apply
frameworks cluster apply --only-services foghorn,bridge
frameworks cluster apply --output json
frameworks cluster apply --confirm

--output json is a dry-run format only; --confirm --output json is rejected so execution logs cannot be mixed into structured plan JSON.

Hard rule: if any host has a diff classified as unknown or infra, the command refuses and lists those entries under Skipped — those go through cluster provision instead. This is the safety guarantee that keeps the fast path from attempting structural changes.

--confirm halts on the first host that fails its action or its readiness gate. Recovery is to fix the failing host and re-run cluster apply — the diff classifier only re-emits hosts that are still out of sync, so a second run resumes from where the first stopped.

Provisioning Commands

frameworks cluster provision

Deploy services to hosts based on the cluster manifest.

Usage:

frameworks cluster provision [flags]

Deployment Modes:

Docker: Uses docker-compose files (default for most services)
Native: Uses systemd units (for services with native binaries)

Process:

Load the cluster manifest and validate required GitOps mesh identity
Build an execution plan
Generate configuration files (docker-compose.yml or systemd units)
Transfer files to target hosts via SSH and provision (Ansible for some infra)
Start services
Validate health (unless --ignore-validation)
For application/all phases, run static SQL seeds, service-owned bootstrap reconciliation, and edge release target sync

Database initialization (baseline schema + expand migrations, behind the baseline-floor guard) runs before the services that depend on it: inline after each database’s infrastructure batch for --only infrastructure / --only all, and — because --only applications targets already-live infrastructure whose plan carries no infrastructure tasks — before the application plan executes for that phase. No application starts against an uninitialized schema.

Flags:

Inherits the Cluster Manifest-Source Flags.
--only string - Phase to provision (infrastructure, applications, interfaces, all)
--version string - Target release (stable, candidate, legacy rc, or vX.Y.Z), resolved the same way as cluster release apply --version; defaults to the manifest channel
--dry-run - Show what would be deployed without actually deploying
--force - Force re-provision even if services exist
--ignore-validation - Continue even if health validation fails (dangerous)
--bootstrap-admin-* / --strict-control-plane - Optional bootstrap controls; see frameworks cluster provision --help for the full list.

Examples:

# Provision + init + static seeds + admin bootstrap in one shot
frameworks cluster provision \
  --bootstrap-admin-email you@co --bootstrap-admin-password-env DEPLOY_PW

# Provision entire cluster
frameworks cluster provision --gitops-dir gitops --cluster production

# Provision only infrastructure
frameworks cluster provision --only infrastructure

# Provision only applications
frameworks cluster provision --only applications

# Preview changes
frameworks cluster provision --dry-run

On success, the CLI remembers the manifest path and system tenant ID in the active context, so follow-up commands (cluster doctor, cluster status, admin users create) don’t need to re-specify them. When a saved default is used, the command announces it with Using manifest from context: … — explicit flags always win.

cluster provision is read-only against GitOps. For managed native clusters, run frameworks mesh wg generate and frameworks mesh wg check before provisioning; missing mesh identity is a hard preflight error.

Bootstrap Flow:

During provisioning, the CLI automatically bootstraps the system tenant and clusters:

Service Token: SERVICE_TOKEN is loaded from your manifest env_files (gitops, SOPS-encrypted) — the CLI decrypts with --age-key or SOPS_AGE_KEY_FILE and uses the token to authenticate to Quartermaster. No separate login or env var is required; the token is platform configuration, not operator identity.
System Tenant Creation: If the “FrameWorks” system tenant doesn’t exist, it’s created. Provisioning does not set a deployment tier; the tier is derived from billing.
Cluster Registration:
- If the manifest has a clusters section, one cluster record is created per entry. Cluster IDs come from the manifest keys (e.g., prod-control, prod-media).
- If there is no clusters section, a single cluster is created with ID {type}-{profile} (e.g., cluster-production).
- Each cluster is registered with its name, type, and base URL derived from Bridge (if present) or Quartermaster.
Node Registration: Each host is registered under the cluster(s) its services belong to. The CLI resolves cluster assignment per service using role-based matching (see Cluster Manifest: clusters).

Provisioning does not mint infrastructure bootstrap tokens. To enroll additional infrastructure nodes, use frameworks cluster nodes add or create a token with frameworks admin bootstrap-tokens create --kind infrastructure_node.

Important: When using clusters, IDs are explicit from the manifest. When not using clusters, the auto-generated format {type}-{profile} applies. These IDs are needed when using:

admin bootstrap-tokens create --cluster-id
services discover --cluster-id

To discover existing cluster IDs after provisioning:

frameworks admin clusters list

frameworks cluster control-plane

Plan or reconcile service-owned platform state without provisioning hosts, deploying services, or running database migrations.

Usage:

frameworks cluster control-plane plan [flags]
frameworks cluster control-plane reconcile [flags]

The domains are:

quartermaster — tenants, infrastructure clusters, ingress, and the service catalog
billing — Purser tiers, pricing, and billing invariants
accounts — Commodore managed operator accounts and bootstrap state
assignments — service-instance placement and stale-assignment cleanup
validation — cross-service control-plane checks
all — every domain in dependency order

plan resolves the platform release, renders and validates the bootstrap desired-state document, and prints the ordered steps. It does not connect to the cluster or apply changes. reconcile runs those idempotent service-owned steps against the cluster.

The system tenant is keyed by the stable frameworks alias. Reconciliation may update its name and branding, but it does not rotate the alias-to-UUID mapping. Replacing that UUID requires a purpose-built data migration and is never an implicit effect of cluster provision or this command.

Examples:

# Validate desired state and inspect the full ordered plan
frameworks cluster control-plane plan --gitops-dir gitops --cluster production

# Reconcile all service-owned platform state
frameworks cluster control-plane reconcile --gitops-dir gitops --cluster production

# Reconcile only pricing and billing invariants
frameworks cluster control-plane reconcile --domain billing

# Reconcile only service-cluster assignments
frameworks cluster control-plane reconcile --domain assignments

Both subcommands inherit the cluster manifest-source flags. Use --domain to select one domain. --skip-validation omits the validation steps for all or billing; reconcile also accepts --ignore-validation. The --bootstrap-admin-*, --bootstrap-reset-credentials, and --strict-control-plane flags are available where their selected steps use them.

frameworks cluster finalize (deprecated)

Resume the post-provision control-plane finalization steps without provisioning or restarting services. This compatibility command keeps its existing --only behavior; use cluster control-plane reconcile --domain ... for new workflows.

Usage:

frameworks cluster finalize [flags]

Process:

Load the cluster manifest and SOPS-backed env_files
Render the same bootstrap desired-state document used by provisioning
Run quartermaster bootstrap
Run purser bootstrap and purser bootstrap validate
Run commodore bootstrap
Reconcile service-cluster assignments
Run control-plane validation

Flags:

Inherits the Cluster Manifest-Source Flags.
--only string - Finalization slice to run (all, quartermaster, purser, commodore, assignments, validation)
--skip-validation - Skip final control-plane validation when --only=all
--ignore-validation - Continue even if control-plane validation has warnings
--bootstrap-admin-* / --bootstrap-reset-credentials / --strict-control-plane - Same bootstrap controls as cluster provision

Examples:

# Resume every post-provision finalization step
frameworks cluster finalize --gitops-dir gitops --cluster production

# Bootstrap only Quartermaster (the tenant/entitlement control plane)
frameworks cluster finalize --gitops-dir gitops --cluster production --only quartermaster

# Retry only Commodore bootstrap after fixing stale stream state
frameworks cluster finalize --gitops-dir gitops --cluster production --only commodore

# Reconcile only the service-cluster assignments
frameworks cluster finalize --gitops-dir gitops --cluster production --only assignments

# Run only the final control-plane validation
frameworks cluster finalize --gitops-dir gitops --cluster production --only validation

frameworks cluster init

Initialize databases and create required schemas/tables.

Usage:

frameworks cluster init [service] [flags]

Initialization:

Creates PostgreSQL databases and schemas
Creates ClickHouse databases and tables
Creates Kafka topics
Applies embedded schemas and static seeds (per service)

Flags:

--manifest string - Cluster manifest file

Examples:

# Initialize everything
frameworks cluster init all

# Initialize just Postgres
frameworks cluster init postgres --manifest production.yaml

Generate a dedicated Quartermaster review signer for an existing deployment:

frameworks cluster secrets generate-capacity-consent --out /secure/path/capacity-consent.env

The command writes a new mode-0600 fragment and refuses to overwrite an existing file. Import CAPACITY_CONSENT_REVIEW_KEY_ID and CAPACITY_CONSENT_REVIEW_PRIVATE_KEY_PEM_B64 into your operator-owned secret store. Do not distribute this private key to other services. The production renderer passes the tuple only to Quartermaster; generate-shared includes a separate signer for fresh installations. Importing or generating keys does not deploy services.

Without the tuple, consent reads and committed-change recovery remain available, but new reviews and writes are unavailable. Configure both values together; partial or malformed configuration prevents Quartermaster startup. Rotating the signer requires reviewing unapplied changes again; an already committed command remains recoverable by its currently authorized owner.

Consent management is not yet activated end to end. Its gRPC API records requested owner permissions and reports changes as pending, not enforced. Public web/API integration and policy-bearing edge admission remain release gates. Configuring this signer does not activate viewer or ingest placement enforcement.

Additional Cluster Provisioning Commands

Command	Purpose
`frameworks cluster control-plane plan/reconcile`	Validate or reconcile service-owned platform state by the `quartermaster`, `billing`, `accounts`, `assignments`, or `validation` domain without provisioning or deploying services.
`frameworks cluster finalize`	Deprecated compatibility alias for control-plane reconciliation. Its existing `--only quartermaster`, `--only purser`, `--only commodore`, `--only assignments`, and `--only validation` behavior remains available.
`frameworks cluster seed`	Apply idempotent static seed data; add `--demo` for sample tenant/user/stream data and `--force` to skip the prompt.
`frameworks cluster migrate`	Apply pending PostgreSQL/YugabyteDB schema migrations from `pkg/database/sql/migrations/` and ClickHouse migrations from `pkg/database/sql/clickhouse/migrations/`; contract-phase runs require `--yes`; use `--phase expand --dry-run` for Ansible check/diff mode. Filtered to `--to-version vX.Y.Z` (defaults to cluster’s resolved platform version). Populated databases without a baseline marker or migration history require the explicit `--complete-interrupted-baselines` recovery flag. A migration with a precheck is refused before its first statement when the precheck returns rows; the failure prints the total and up to 50 rows.
`frameworks cluster migrate validate`	Validate migration path layout, checksums-at-source assumptions, conservative expand-phase SQL safety rules, and that every precheck is a single `SELECT` keyed to an offered migration.
`frameworks cluster secrets generate-shared --out <file>`	Generate the complete production shared-secret set for a fresh deployment into a new mode-0600 dotenv fragment. The output passes the same production validation used by lifecycle commands; import it into the operator-owned SOPS store. It never overwrites a file or mutates the manifest/secret store.
`frameworks cluster secrets generate-internal-ca --out <file>`	Generate a fresh managed internal root certificate, Navigator signing intermediate, and intermediate private key in a new mode-0600 dotenv fragment. Import all three values together into the SOPS store before provisioning a non-development cluster. The root signing key is discarded after the intermediate is created, so rotating this material establishes a new trust domain.
`frameworks cluster secrets generate-media-authority --out <file>`	Generate only the five media-authority values needed when upgrading an existing deployment: the Ed25519 signer/trust tuple, seal root, and Foghorn state-encryption root. Merge the new mode-0600 fragment into the existing shared secret set. The command never overwrites a file and never rotates unrelated platform credentials.
`frameworks cluster data-migrate list`	List required data migrations declared in the embedded release catalog up to `--to-version`. Empty catalog reports “no required data migrations declared” — never silent OK.
`frameworks cluster data-migrate status <service>.<id>`	Show persisted state for one data migration on its owning service host (queries the service binary’s `data-migrations status --format json`).
`frameworks cluster data-migrate run <service>.<id>`	Run one data migration to completion or until error; honors `--batch-size`, `--dry-run`, `--scope-kind/--scope-value`. `--dry-run` runs inside a read-only transaction, needs no backup, and prints the migration’s summary lines and findings (`findings: showing N of TOTAL`) when it reports them.
`frameworks cluster data-migrate verify <service>.<id>`	Run the migration’s read-only verification on the service host, printing the migration’s report first when it has one (for example the Commodore field-encryption quarantine).
`frameworks cluster data-migrate pause <service>.<id>`	Mark a data migration paused.
`frameworks cluster data-migrate resume <service>.<id>`	Mark a paused data migration runnable again.
`frameworks cluster skipper knowledge reset`	Clear Skipper knowledge embeddings and crawl cache so configured sources are fetched and embedded again.
`frameworks cluster skipper posts`	Query recent Skipper social post drafts from the skipper database over the manifest-selected SSH/database path.
`frameworks cluster metabase sync`	Sync repo-managed Metabase Periscope cards and attach missing cards to the dashboard without overwriting unmanaged manual edits.
`frameworks cluster sync-geoip`	Upload GeoLite2-City MMDB data to Foghorn/Quartermaster hosts; supports `--source maxmind` or `--source file`.
`frameworks cluster set-channel <channel>`	Set the manifest release channel. Use `stable` for production, `candidate` for the newest GA or RC, or legacy `rc` for the newest RC only; `cluster release plan/apply` uses this when no explicit version is given.
`frameworks cluster clickhouse migrate backfill`	Bulk-copy periscope data into the new Replicated node via `remote()` over the WireGuard mesh, using the same idempotent staging + `REPLACE PARTITION`/`EXCHANGE` mechanics as `sync` (re-runnable after a partial failure); stops refreshable MVs for the duration. Requires `--from <source host key>` (and `--from-port` if source/dest run on swapped ports).
`frameworks cluster clickhouse migrate sync`	Idempotent per-partition re-copy of the growing tail (staging table + atomic `REPLACE PARTITION`; additive-safe — re-runs replace, never double). Unpartitioned tables use an atomic `EXCHANGE`.
`frameworks cluster clickhouse migrate verify`	Parity fingerprint — FINAL row count plus a `cityHash64` content hash over the shared (source-and-destination) columns, each side normalized to the destination type (`AggregateFunction` columns finalized; a type-evolved source column cast to the destination type). Destination-only/new tables absent on the source are asserted EMPTY instead of hashed. Plus a catalog-coverage assertion that fails if the live schema has a table the migration catalog doesn’t classify.
`frameworks cluster clickhouse migrate cutover`	Refuses unless `--ingest-stopped` is passed (periscope-ingest must be stopped first; Kafka buffers). Stops the refreshable MVs on BOTH the source (quiescing the old node so a timer refresh can’t race the copy) and the destination, runs the final idempotent sync, RE-VERIFIES parity (aborting before any MV starts on a mismatch), then starts the destination MVs; any post-quiesce failure restores the source MVs so the old node stays usable. It then prints the operator repoint steps — either flip BOTH endpoints (downtime, ingest stays stopped) or write-first/drain/read (online) — plus the rollback path. The endpoint flips are operator gitops steps, not done by the command.

cluster migrate --phase contract (with or without --dry-run) and cluster data-migrate run of an irreversible migration take --backup <dir|s3-url>: a backup from cluster backup create that started less than an hour earlier and whose migration ledger digests still match the databases the command changes. Without one they refuse. A migration is irreversible when the service marks it so, when the service binary does not report the field, or when the release that declares it disables rollback for its service. See Backup and Restore.

Examples:

# Static seeds only
frameworks cluster seed

# Include demo tenant/user/stream records
frameworks cluster seed --demo --force

# Preview migrations without applying them
frameworks cluster migrate --phase expand --dry-run

# Validate embedded migration files
frameworks cluster migrate validate

# Apply expand-compatible schema migrations
frameworks cluster migrate --phase expand

# Inspect recent Skipper social drafts
frameworks cluster skipper posts --status draft --limit 25

# Clear platform knowledge vectors and crawl cache before an embedding-model recrawl
frameworks cluster skipper knowledge reset --yes

# Adopt existing matching Metabase cards, then keep them managed by hash
frameworks cluster metabase sync \
  --url https://metabase.example.com \
  --session-id "$METABASE_SESSION_ID" \
  --adopt

# Distribute a local GeoIP database and restart target services
frameworks cluster sync-geoip \
  --source file \
  --file /path/to/GeoLite2-City.mmdb \
  --services foghorn,quartermaster \
  --restart

# Track the newest release, including release candidates
# (the legacy `rc` channel tracks release candidates only)
frameworks cluster set-channel candidate

Operations Commands

frameworks cluster skipper

Inspect Skipper operational state.

frameworks cluster skipper knowledge reset

Clear Skipper knowledge embeddings and page-cache rows so the next crawl cycle fetches and embeds sources from scratch. The command does not delete conversations, usage rows, social posts, reports, or diagnostic baselines.

By default it resets the active context’s system tenant. Pass --tenant-id for a specific tenant, --all-tenants to reset every tenant, and --source to limit the reset to one source URL or source root.

Usage:

frameworks cluster skipper knowledge reset [flags]

Flags:

--tenant-id string - Tenant UUID to reset (defaults to active context system tenant)
--all-tenants - Reset knowledge for every tenant
--source string - Limit reset to one source URL or source root
--dry-run - Show matching row counts without deleting data
--yes / -y - Skip confirmation prompt

Examples:

frameworks cluster skipper knowledge reset --dry-run
frameworks cluster skipper knowledge reset --yes
frameworks cluster skipper knowledge reset --tenant-id 00000000-0000-0000-0000-000000000001 --yes
frameworks cluster skipper knowledge reset --all-tenants --yes

frameworks cluster skipper posts

List recent Skipper social posts from skipper.skipper_posts.

Usage:

frameworks cluster skipper posts [flags]

Flags:

--limit int - Maximum rows to show (default: 50, max: 500)
--status string - Filter by post status, for example draft or sent

Examples:

frameworks cluster skipper posts
frameworks cluster skipper posts --status draft --limit 25

frameworks cluster metabase

Manage repo-owned Metabase content for a cluster.

frameworks cluster metabase sync

Sync the FrameWorks-managed Metabase card specs. Without --file, every spec embedded in the CLI (pkg/metabase/specs/) is synced to the dashboard it declares; missing collections and dashboards are created automatically. The Metabase URL is derived from the cluster manifest’s root_domain (https://metabase.<root_domain>) and the API key is read from the manifest env_files (METABASE_API_KEY); flags and environment variables override both. The only manual prerequisite is the Metabase database connection itself (it carries the ClickHouse credentials).

The command creates missing cards and updates cards that still carry the Frameworks managed hash. Existing cards without that marker are left alone unless --adopt confirms the SQL already matches, or --force is used to replace them.

Usage:

frameworks cluster metabase sync [flags]

Flags:

--url string - Metabase base URL (defaults to METABASE_URL, else derived from the manifest)
--session-id string - Metabase session ID (defaults to METABASE_SESSION_ID)
--api-key string - Metabase API key (defaults to METABASE_API_KEY, else the manifest env_files)
--file string - Sync a single card spec file instead of the embedded specs
--database string / --database-id int - Target Metabase database
--collection string / --collection-id int - Target collection
--dashboard string / --dashboard-id int - Dashboard override (requires --file)
--dry-run - Print intended changes without writing
--adopt - Mark matching unmanaged cards as Frameworks-managed
--force - Replace existing unmanaged cards

Examples:

frameworks cluster metabase sync --url http://localhost:3001 --session-id "$METABASE_SESSION_ID" --dry-run
frameworks cluster metabase sync --url https://metabase.example.com --api-key "$METABASE_API_KEY" --adopt

frameworks cluster grafana

Manage repo-owned Grafana content for a cluster.

frameworks cluster grafana sync

Sync the repo-owned Grafana dashboard (the consolidated FrameWorks Ops, embedded in the CLI from pkg/grafana/dashboards/) into the FrameWorks folder, creating the folder and the pinned datasources it references when missing: uid victoriametrics (the VictoriaMetrics datasource plugin, addressed by mesh IP from the manifest) and uid clickhouse (the dedicated frameworks_analytics ClickHouse user, password from CLICKHOUSE_ANALYTICS_PASSWORD in the manifest env_files — the same identity the Metabase datasource uses). The Grafana URL is derived from the manifest’s root_domain and credentials come from GF_SECURITY_ADMIN_USER / GF_SECURITY_ADMIN_PASSWORD; flags and environment variables override both.

Dashboards are pushed with overwrite keyed on their stable uid, so the sync is idempotent and UI edits to the managed dashboard are overwritten on the next sync — the repo is the source of truth. Hand-created datasources keep their own uids and are never touched.

Usage:

frameworks cluster grafana sync [flags]

Flags:

--url string - Grafana base URL (defaults to GRAFANA_URL, else derived from the manifest)
--api-key string - Grafana service-account token (defaults to GRAFANA_API_KEY, else admin basic auth from the manifest env_files)
--file string - Sync a single dashboard JSON file instead of the embedded dashboards
--dry-run - Print intended changes without writing

Examples:

frameworks cluster grafana sync --dry-run
frameworks cluster grafana sync

frameworks cluster logs

View or stream service logs.

Usage:

frameworks cluster logs <service> [flags]

Supports:

Docker Compose logs (docker compose logs)
Systemd journal logs (journalctl -u)

Flags:

--manifest string - Cluster manifest file
--follow, -f - Follow log output (stream)
--tail, -n int - Number of lines to show (default: 50)

Examples:

# View last 50 lines (default)
frameworks cluster logs commodore

# Stream logs
frameworks cluster logs commodore --follow

# Show last 500 lines
frameworks cluster logs bridge --tail 500

# Show last 100 lines and follow
frameworks cluster logs decklog -n 100 -f

frameworks cluster logs snapshot

Collect a bounded debugging snapshot from every cluster host. Each host gets a local log file with host metadata, failed units, frameworks-* service status, journal excerpts, suspicious log summaries, and internal PKI cert/key checks.

Usage:

frameworks cluster logs snapshot [flags]

Flags:

--manifest string - Cluster manifest file
--since string - Journal window when --boot is not set (default: 4 hours ago)
--boot - Collect logs from the current boot instead of --since
--tail int - Maximum journal lines per unit; set 0 for all lines in scope (default: 500)
--output, -o string - Local output directory; defaults to a temp directory
--parallel int - Maximum hosts to collect in parallel (default: 6)
--edge-manifest string - Optional edge manifest to include edge nodes

Examples:

frameworks cluster logs snapshot --since "2 hours ago"
frameworks cluster logs snapshot --boot --tail 800 --edge-manifest ./clusters/production/edge.yaml
frameworks cluster logs snapshot --output /tmp/fw-logs

frameworks cluster snapshot

Collect a bounded incident bundle from the cluster manifest. The snapshot includes HA-aware host logs and read-only database metadata for PostgreSQL/YugabyteDB and ClickHouse: table sizes, row estimates, migration ledgers, ClickHouse parts, and pending mutations.

Usage:

frameworks cluster snapshot [flags]

Flags:

--manifest string - Cluster manifest file
--since string - Journal window when --boot is not set (default: 4 hours ago)
--boot - Collect logs from the current boot instead of --since
--tail int - Maximum journal lines per unit; set 0 for all lines in scope (default: 500)
--output, -o string - Local output directory; defaults to a temp directory
--parallel int - Maximum hosts to collect in parallel (default: 6)
--edge-manifest string - Optional edge manifest to include edge nodes
--skip-logs - Skip host and service logs
--skip-db - Skip database metadata

Examples:

frameworks cluster snapshot --gitops-dir ../gitops --cluster production --since "2 hours ago"
frameworks cluster snapshot --boot --tail 800 --output /tmp/fw-snapshot
frameworks cluster snapshot --skip-logs --output /tmp/fw-db-state

frameworks cluster os

Run OS-level fleet maintenance operations.

frameworks cluster os update

Inspect or apply pending OS updates across manifest hosts. Check mode is the default and exits with code 2 when updates, stale services, or a reboot are pending. Apply mode is mutating and should run only during a maintenance window.

Usage:

frameworks cluster os update [flags]

Flags:

--check - Inspect pending OS updates (default when --apply is absent)
--refresh - Refresh apt indexes during check mode
--apply - Apply package upgrades, restart stale services, and reboot when needed
--hosts string - Comma-separated manifest host names to target
--mode string - Apt upgrade mode for apply: safe or full (default: safe)
--serial int - Number of hosts to process in parallel during apply (default: 1). Yugabyte nodes are always updated one at a time after the other hosts, through the same gate as cluster provision.
--no-reboot - Skip reboot even if /var/run/reboot-required exists
--continue-on-error - Continue after per-host failures during apply
--json - Emit one JSON object per host in check mode

Examples:

# Inspect pending updates without refreshing apt indexes
frameworks cluster os update

# Refresh apt indexes first, then return exit code 2 if anything is pending
frameworks cluster os update --refresh

# Apply updates to one host without rebooting
frameworks cluster os update --apply --hosts central-eu-1 --no-reboot

frameworks cluster restart

Restart services with optional health validation.

Usage:

frameworks cluster restart <service> [flags]

Process:

Stop service gracefully
Wait for shutdown
Start service
Validate health (if enabled)

frameworks cluster restart yugabyte restarts every Yugabyte node, one at a time: nodes that are down first, each serving node only once the masters confirm the universe survives losing it, and the next node only after the restarted one has recovered. The first failure stops the restart.

Flags:

--manifest string - Cluster manifest file
--validate - Validate service health after restart

Examples:

# Restart without a post-restart health check
frameworks cluster restart commodore

# Restart with health validation
frameworks cluster restart bridge --validate

frameworks cluster upgrade

Upgrade services to new versions from GitOps manifests.

For the production rollout sequence, including migrations and rollback behavior, see Running Cluster Upgrades.

Usage:

frameworks cluster upgrade <service> [flags]

Version Resolution:

stable - Latest stable release
candidate - Newest release, including release candidates (latest is an alias)
rc - Legacy track: newest release candidate only
v0.0.0-rc1 - Specific version tag

Process:

Fetch GitOps manifest for target version (and fail closed if that fetched metadata says this CLI is too old or is missing a required reconciliation transition)
Run the pre-deploy gate once (schema + data migrations; cluster-level)
For every host the service runs on: detect state, deploy the new version (pull image or download binary), and validate health. A host that already runs the target version runs the role’s check-mode precheck instead: it redeploys only when its rendered configuration (env, secrets, role files) changed, without automatic rollback, and is otherwise skipped.

Each replica renders exactly as cluster provision renders it: the planner’s task for that host (its cluster, instance identity, and phase, so infrastructure credentials are included) and the same runtime identities (system tenant, cluster owner tenants, Quartermaster address).

upgrade redis converges every Redis server and Sentinel of every instance through the same stage as release apply host convergence: servers that are not the live primary, then the Sentinels, then the live primary after a Sentinel failover. Each host changes only when its role precheck reports a difference.

For an HA service, upgrade (and --all) moves every replica, not just the first — a partial failure aborts the sequence. upgrade yugabyte moves every node one at a time through the Yugabyte gate described under cluster restart, in the order YugabyteDB requires: every node’s master on the new engine first, then every tserver. Once every node serves and the universe is healthy, it finalizes the upgrade with yb-admin finalize_upgrade (promotes the new AutoFlags and upgrades the YSQL catalog). It is the only command that may change the engine of a node that already joined the universe; provisioning such a node with another engine fails and points here. After finalizing, nodes cannot go back to the previous Yugabyte binary. If finalizing fails, rerun the same upgrade: nodes already on the version are skipped and finalizing is retried. The CLI does not write the deployed version back to your local manifest (the resolved manifest carries SOPS-decrypted host inventory, so serializing it would leak host IPs/users/key paths). There is no durable “deployed version” record: your GitOps manifest holds the desired release, while the observed version is read from actual state (edge nodes self-report to foghorn.node_components; control-plane services are detected on disk). Record the intended version in your source GitOps manifest.

Flags:

--manifest string - Cluster manifest file
--version string - Target version. Defaults to the manifest release channel, or stable if no channel is set.
--dry-run - Preview upgrade without executing
--skip-validation - Skip health validation after upgrade
--yes - Skip confirmation prompt
--no-rollback - Skip automatic rollback on health check failure
--all - Upgrade all enabled services in dependency order

Examples:

# Upgrade to latest stable
frameworks cluster upgrade commodore --version stable

# Upgrade to specific version
frameworks cluster upgrade bridge --version v0.0.0-rc1

# Show rollout order and embedded migration phases
frameworks cluster upgrade plan --version stable

# Preview upgrade
frameworks cluster upgrade signalman --version stable --dry-run

# Upgrade without validation
frameworks cluster upgrade decklog --version stable --skip-validation

# Upgrade all enabled services in dependency order
frameworks cluster upgrade --all --yes

frameworks cluster release

Run a full cluster release as one ordered, resumable plan: pre-upgrade host convergence (node baseline and OS tuning on every host, Privateer seeds, Redis servers and Sentinels, Yugabyte, ClickHouse, Kafka controllers and brokers, vmagent/vmalert/vmauth, nginx/Caddy, compose dependencies, declared Kafka topics, MirrorMaker2 workers; changed hosts only, behind health gates) → expand migrations → service upgrades in dependency order → reconciliation transitions interleaved at their declared dependency points → stale placement cleanup → postdeploy migrations. Contract migrations are printed as a deferred operator step and run only after the rollback window closes. This is the normal upgrade path; cluster migrate, cluster upgrade, and the low-level reconciliation subcommands remain recovery tools.

Usage:

frameworks cluster release plan [flags]
frameworks cluster release apply [flags]

cluster release plan is the credential-free planning mode. It resolves the release metadata and manifest topology, classifies platform artifacts, independently pinned dependencies, and host infrastructure, and prints schema/reconciliation ordering. It does not decrypt secrets, open SSH, or dial the control plane. cluster release apply --dry-run is the live preflight: it additionally runs database, artifact, required operator input, host convergence, health, and reconciliation checks against the cluster.

A reconciliation transition is a constrained Check → Apply → Verify node registered by a compiled handler id (never a shell command); an unknown handler or an outdated CLI fails the release closed. Each converges a declared invariant against natural authoritative state, so a rerun after a mid-release failure re-Checks reality and skips whatever is already complete — the release is resumable. --dry-run runs every gate and Check and prints the planned transitions without mutating anything.

Between expand migrations and service upgrades, release apply prints a read-only post-expand report from the Quartermaster database: operator-managed and customer-enrolled nodes without an identity key, and tenants whose custom subdomain or domain grant the tenant-DNS entitlement migration will clear, with whether Purser’s handoff receipt exists. Each list shows up to 50 rows and the total. The report never fails the release. Before expand has been applied (including a --dry-run on such a cluster) it prints post-expand report: not run (expand not applied). See Running Cluster Upgrades.

Flags:

--version string - Target version (defaults to the cluster channel)
--dry-run - Show the plan and run every gate/Check without mutating anything
--yes - Skip the confirmation prompt
--skip-validation - Skip health validation after each service upgrade
--no-rollback - Skip automatic rollback on a service health-check failure
--complete-interrupted-baselines - Reapply and verify baselines for inspected populated databases without migration provenance

Examples:

# Inspect the complete topology and release ordering without cluster credentials
frameworks cluster release plan --version stable

# Run every live gate without mutating the cluster
frameworks cluster release apply --dry-run

# Apply it
frameworks cluster release apply --version stable --yes

frameworks cluster storage

Inspect the immutable S3 storage descriptor a media cell is bound to. Each cell is bound to one backend (bucket/endpoint/region/prefix): Quartermaster persists it, Chandler serves from it, and Foghorn establishes its cell identity against it on first boot. See Media Storage & Playback for the full model.

Usage:

frameworks cluster storage descriptor <cluster> --manifest <cluster.yaml>

The descriptor command performs a structural manifest read without decrypting host inventory and prints the non-secret bucket, prefix, and endpoint as JSON. Desired-state bootstrap establishes the authoritative Quartermaster descriptor; deployed components enforce it as immutable.

frameworks cluster yugabyte relayout

Move one existing YugabyteDB database into the colocated layout declared in the repository. The database is copied from its own schema and data into a shadow database created with the layout, verified, and renamed into place. Services keep running; only during cutover can they not reach the database. Each step records its effect in a leased journal in the yugabyte admin database, renews the lease while it runs, and resumes from that record when re-run. The full procedure is in YugabyteDB Physical Layout.

Usage:

frameworks cluster yugabyte relayout plan --database <name>
frameworks cluster yugabyte relayout preflight --database <name>
frameworks cluster yugabyte relayout prepare --database <name>
frameworks cluster yugabyte relayout cutover --database <name> --yes
frameworks cluster yugabyte relayout finish --database <name> --yes

Step	What it does
`plan`	Read-only. Checks every tserver’s local superuser YSQL, the YSQL Connection Manager, client binaries, and dump space, then shows placement, tablets and peers, affected services, journal state, and blockers.
`preflight`	No downtime. Rehearses the copy into a scratch database in the order a relayout runs it, verifies it, reports the time of each phase and the estimated window, then drops the copy and dump.
`prepare`	No downtime. Creates the shadow colocated with the source owner and restores the source’s schema, including indexes, constraints, and triggers, rewritten for the layout; requires an equal schema.
`cutover`	Requires `--yes`. The window: revokes access and ends every session, copies the data, verifies the copy, renames the source aside and the shadow into place, and restores access. `--rollback` undoes it until access is restored.
`finish`	Requires `--yes`. Drops `<database>__pre_relayout` and the staged dump after a completed cutover.

Flags:

Flag	Default	Description
`--database`		Physical database, such as `quartermaster` or `foghorn_eu`
`--dump-dir`	`/var/lib/frameworks/relayout`	Absolute directory on the Yugabyte node for the staged dump
`--dry-run`	`false`	Print the step’s actions without connecting
`--rollback`	`false`	`cutover` only: return the original database instead
`--yes`	`false`	`cutover` and `finish`: confirm the window or the final drop

Also inherits the Cluster Manifest-Source Flags.

Backup & Restore Commands

Backups cover every PostgreSQL/YugabyteDB service database, the databases of named postgres instances, and the ClickHouse billing facts. See Backup and Restore for what is covered, the manifest, and the restore procedure.

frameworks cluster backup create

Dump the selected databases to a local directory or an S3 prefix.

Usage:

frameworks cluster backup create --to <dir|s3-url> (--all | --database <name>... | --clickhouse) [flags]

The dump tools (pg_dump, ysql_dump, clickhouse-client) run on the database hosts over SSH; their gzip-compressed output streams to the CLI and is written under a new frameworks-backup-<UTC time> directory below --to. The manifest is written last. Any failure fails the command, and a backup without a manifest is incomplete.

Flags:

--manifest string - Cluster manifest file
--to string - Destination directory or s3://bucket/prefix (required)
--all - Every PostgreSQL/YugabyteDB database and the ClickHouse billing facts
--database string - Database to back up: physical name, or <instance>/<name> for a named postgres instance (repeatable)
--clickhouse - The ClickHouse billing facts

S3 destinations use the standard AWS configuration: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or a profile), AWS_REGION, and AWS_ENDPOINT_URL_S3 for an S3-compatible store.

Examples:

frameworks cluster backup create --to /var/backups/frameworks --all
frameworks cluster backup create --to s3://ops-backups/frameworks --database purser
frameworks cluster backup create --to /var/backups/frameworks --database support/chatwoot

frameworks cluster backup verify

Read a backup’s manifest and check the size and SHA-256 of every file.

Usage:

frameworks cluster backup verify --from <dir|s3-url>

Flags:

--from string - Backup directory or s3:// URL (required)

frameworks cluster restore

Restore databases from a backup. Each database is restored into <name>__restore, compared with the manifest (ledger digest and per-table row counts), checked against the schema migration floor and, for Foghorn cells, the cell storage identity. Only then are the services that use it stopped, the restored database swapped in, and the services started and validated. The live database is kept as <name>__prerestore. ClickHouse fact rows are moved into the live tables partition by partition, and the previous rows are kept in <table>__prerestore.

Usage:

frameworks cluster restore --from <dir|s3-url> (--all | --database <name>... | --clickhouse) [flags]

Warning: Restore stops affected services and replaces their database contents, preserving the previous copies until finish. Foghorn restores must raise the durable media-authority fence before restart.

For a native PostgreSQL or Yugabyte restore, use the complete cluster manifest containing every Foghorn replica sharing those databases:

frameworks cluster restore-fence prepare --yes
# Perform the database-native restore and apply the target release's migrations.
frameworks cluster restore-fence complete --yes
frameworks cluster doctor

prepare stops and verifies every replica before fencing. complete refuses while any replica is running, fences every restored cell database, and only then starts the replicas. Failure before fencing completes leaves them stopped. Use the same manifest for both commands. Do not restart replicas between these steps or clear a fence to bypass an unreachable core. For database-native restores outside the backup-manifest workflow, use the fenced native workflow above.

Authority recovery requires matching core/cell protocol support. Upgrade Commodore before Foghorn within the coordinated release. An unsupported core logs an upgrade requirement and is retried without requiring a Foghorn restart. Valid local authority stays usable on ordinary restart; a durable restore fence stays raised until recovery succeeds. There is no legacy replay fallback.

Flags:

--manifest string - Cluster manifest file
--from string - Backup directory or s3:// URL (required)
--all - Every database and the ClickHouse facts in the backup
--database string - Database to restore (repeatable)
--clickhouse - The ClickHouse billing facts
--yes - Skip the confirmation prompt

Examples:

frameworks cluster restore --from /var/backups/frameworks/frameworks-backup-20260919T101500Z --database purser
frameworks cluster restore --from s3://ops-backups/frameworks/frameworks-backup-20260919T101500Z --all --yes

frameworks cluster restore finish

Drop the __prerestore copies (and any leftover __restore shadows) once the restored data is confirmed.

frameworks cluster restore finish --database purser

frameworks cluster restore rollback

Stop the services again, put the __prerestore copies back in place, and start the services.

frameworks cluster restore rollback --all

Diagnostic Commands

frameworks cluster diagnose network

Test network connectivity between all hosts in the cluster.

Usage:

frameworks cluster diagnose network [flags]

Tests:

Ping test from each host to every other host
Reports success/failure matrix

The report shows host-to-host connectivity results for the manifest targets.

Examples:

frameworks cluster diagnose network

frameworks cluster diagnose resources

Check CPU, memory, disk, and load average on all hosts.

Usage:

frameworks cluster diagnose resources [flags]

Metrics:

CPU usage (%)
Memory usage (used/total)
Disk usage (/)
Load average (1m, 5m, 15m)

The report shows CPU, memory, disk, and load data for each manifest host.

Examples:

frameworks cluster diagnose resources

frameworks cluster diagnose ports

Check for port conflicts on all hosts.

Usage:

frameworks cluster diagnose ports [flags]

Checks Standard Ports: Uses the CLI’s built-in port list for core services, interfaces, and gRPC endpoints. For the canonical port table, see Architecture Overview.

The report checks the CLI’s built-in port set against each manifest host.

Examples:

frameworks cluster diagnose ports

frameworks cluster diagnose kafka

Check Kafka cluster health, topics, consumer groups, and broker status.

Usage:

frameworks cluster diagnose kafka [flags]

Checks:

List all topics
List consumer groups
Broker API version check
Cluster responsiveness
MirrorMaker2 topic coverage

The report covers topic listing, consumer groups, a committed-offset lag snapshot, broker API reachability, and cluster responsiveness for one broker in every configured Kafka region. For every enabled MirrorMaker2 link it compares each mirrored topic’s end offset on the source cluster with its <source>.<topic> copy on the target cluster and warns when the source has records but the copy has none, which is how a topic without a replication task shows up. The coverage check only reads offsets. Native deployments use the tools under /opt/kafka/bin; an explicitly configured Docker deployment uses its Compose toolchain. Any failed probe makes the command exit non-zero and includes the remote command error.

Examples:

frameworks cluster diagnose kafka

frameworks cluster diagnose media

Capture read-only service state, listeners, TLS certificate details, recent errors, and Quartermaster media placement from every manifest host that runs a media/control-path service.

frameworks cluster diagnose media [flags]
frameworks cluster diagnose media --stream-id <uuid> --tenant-id <uuid>
frameworks cluster diagnose media --output json

--since controls the journal window (default 4 hours ago). JSON output is a single structured report containing every host’s services, stdout, stderr, exit code, and connection/probe error. A missing host, SSH failure, or non-zero probe causes a non-zero command exit instead of a silent partial report.

frameworks cluster diagnose media-authority

Trace the media-authority pipeline end to end: Quartermaster and Purser refresh outboxes, Commodore’s refresh obligations, issued versions, delivery queue and compile fences, and each cell’s apply audit.

frameworks cluster diagnose media-authority [flags]
frameworks cluster diagnose media-authority --window-hours 6 --since "1 hour ago"
frameworks cluster diagnose media-authority --output json

Two kinds of probe run, both read-only:

Journals run on each service’s own host (one host per manifest service entry) and print its authority log lines with timestamps and IDs normalized, so a line repeated a thousand times reads as one line with a count.
SQL runs on the database host through the same connection path as cluster snapshot, in a session forced read-only with a 20-second statement timeout. Service hosts carry no SQL client, and each Foghorn cell has its own database (foghorn_eu, foghorn_us), which only this path enumerates.

--window-hours sets the database lookback (default 24, max 720); --since sets the journal window. JSON output is one report with every probe’s stdout, stderr, exit code, and error. A failed probe makes the command exit non-zero.

The Commodore section answers the questions that matter during an authority incident:

Version churn — versions per authority per hour, how many re-signs carried an identical payload, and how many were issued within 60 seconds of their predecessor (two compiles racing for the same authority).
What is stuck — refresh obligations by lane and status, how far behind each lane is, parked targets with the reason they cannot compile, retrying targets, and current authorities that have no renewal scheduled.
What is in use — authorities by renewal state, with dormant meaning nobody has used the object for thirty days so it is no longer renewed or kept in cells; the authorities still in use whose version ran out (there should be none); recorded use; and what each cell attested.
Delivery health — deliveries waiting per cell, split into fresh and replayed, unacknowledged deliveries against the current version, index validity, and the measured plan of the short-lease claim predicate.

Each cell section shows apply outcomes, rejections per hour, the most recent rejections next to the version the cell actually holds, and how many held authorities are past refresh_after or past their validity, by lifecycle. A cell holding an expired authority is not by itself a fault: an object nobody uses runs out there, is fetched again on its next use, and is forgotten thirty days after it was issued.

frameworks cluster doctor runs the pass/fail form of the same state as its Media authority convergence check: unhealthy when a refresh target is parked, a refresh or a current delivery has been stuck for more than five minutes, a cell refuses the current version, or an authority that is still in use is past its validity. A fan-out backlog older than an hour and a cell that is not forgetting expired authorities are warnings. It probes Commodore and every cell’s Foghorn database; a database or a count it could not read is reported as degraded, never as healthy.

frameworks cluster diagnose dvr

Show one DVR recording as its origin cell’s Foghorn sees it. Needs a platform-operator login.

frameworks cluster diagnose dvr <dvr-hash>
frameworks cluster diagnose dvr <dvr-hash> --service foghorn-eu --output json

Commodore resolves the hash to the recording’s origin cluster, and the command asks the manifest Foghorn entry whose clusters include that cluster; a manifest with a single unassigned Foghorn uses it. The report covers:

Recording - status and error, start/end, storage location, sync status with its last attempt, node, failure count, and error, .dtsh sync state, retention, the chapter policy snapshot, and a pending start dispatch.
Segments - count, first and last media time, recorded duration and bytes, counts per segment status, and the gaps in the ledger (the first 100, plus the total).
Chapters - each chapter’s state, bounds, finalize attempts, latest finalize job ID, assigned node, and last failure reason.
Finalize queue - the chapters in closed or finalizing state, waiting for or running a finalize attempt.

Flags:

--service - Manifest Foghorn entry to ask (e.g. foghorn-eu) instead of the one serving the origin cluster

DNS Commands

frameworks dns doctor

Verify public DNS records match Quartermaster inventory.

Usage:

frameworks dns doctor --domain example.com

Validates Cloudflare-proxied names by resolution and Bunny smart records as a geographically selected subset of Quartermaster’s healthy inventory. It checks both cluster-specific media names and platform-official global entrypoints.

Mesh Commands

frameworks mesh wg generate

Generate missing GitOps WireGuard identity for cluster hosts. This intentionally writes cluster.yaml and the local SOPS hosts inventory.

frameworks mesh wg generate \
  --manifest gitops/clusters/production/cluster.yaml \
  --hosts-file gitops/clusters/production/hosts.enc.yaml

Mutation commands require a local checkout. Remote GitHub sources are rejected; run them against a checked-out gitops repository, review the diff, then commit.

frameworks mesh wg check

Read-only validation for CI and preflight.

frameworks mesh wg check --gitops-dir gitops --cluster production

frameworks mesh wg rotate

Rotate one host’s WireGuard key. The mesh IP is preserved by default; pass --readdress only when the address must change.

frameworks mesh wg rotate core-2 --gitops-dir gitops --cluster production

frameworks mesh status

Show WireGuard mesh status from Quartermaster inventory.

Usage:

frameworks mesh status

Displays node IDs, roles, internal/WireGuard IPs, last heartbeat, and agent status. Useful for verifying Privateer mesh connectivity.

Additional Mesh Commands

Command	Purpose
`frameworks mesh doctor`	Simulate Privateer’s runtime WireGuard apply policy for each manifest host and report validation failures.
`frameworks mesh join <ssh-target>`	Write Privateer’s runtime-enrollment env file to a remote host over SSH. Requires `--token` and `--bootstrap-addr`.
`frameworks mesh reconcile`	Adopt `runtime_enrolled` nodes into GitOps. Use `--write-gitops` to mutate `cluster.yaml` and `hosts.enc.yaml`.
`frameworks mesh wg audit`	Compare GitOps WireGuard identity against Quartermaster node state; exits non-zero on authoritative divergence.
`frameworks mesh wg promote <host>`	Finish `adopted_local` to `gitops_seed` promotion after `mesh wg rotate` and `cluster provision` converge.

Examples:

frameworks mesh doctor --gitops-dir gitops --cluster production
frameworks mesh wg audit --gitops-dir gitops --cluster production
frameworks mesh join root@edge-2 \
  --token <bootstrap-token> \
  --bootstrap-addr https://bridge.example.com \
  --cluster-id production
frameworks mesh reconcile --write-gitops --gitops-dir gitops --cluster production
frameworks mesh wg promote edge-2 --gitops-dir gitops --cluster production

Admin Commands

frameworks admin tokens

Manage developer API tokens.

Subcommands:

# Create a token
frameworks admin tokens create --name "my-app" --expires 720h --perms "streams:read,streams:write"

# List tokens (shows name and ID)
frameworks admin tokens list

# Platform operator: token metadata across every tenant
frameworks admin tokens list --all-tenants

# Platform operator: tokens holding a scope this release no longer accepts
frameworks admin tokens list --all-tenants --unsupported-scopes

# Revoke by name (preferred)
frameworks admin tokens revoke --name "my-app"

# Revoke by ID
frameworks admin tokens revoke <token-id>

Token listings contain only the current user’s tokens. A tenant owner or admin may revoke another user’s token by its tenant-scoped ID; ordinary members may revoke only their own tokens.

--all-tenants lists token metadata (ID, tenant, name, scopes, status, created, last used, expiry) across every tenant and needs a platform-operator login; token values are never shown. With it, --tenant-id narrows the listing to one tenant and --unsupported-scopes keeps only tokens holding a permission that create no longer accepts, each marked with the unsupported scopes.

Flags for create:

--name - Token name/label (required)
--expires - Expiry duration (e.g., 720h, 30d)
--perms - Comma-separated permissions

Permissions are enforced as exact scopes. The coarse historical values read and write are not wildcards and are rejected for new tokens. If --perms is omitted, the token receives only streams:read. Use mcp:high-risk in addition to the domain scope when an API token must call a high-risk MCP tool.

frameworks admin bootstrap-tokens

Manage bootstrap tokens for edge node enrollment.

Subcommands:

# Create bootstrap token
frameworks admin bootstrap-tokens create --name "edge-nyc-01" --kind edge_node --ttl 24h

# Create runtime mesh token with pre-service topology intent
frameworks admin bootstrap-tokens create \
  --name "regional-eu-2 privateer" \
  --kind infrastructure_node \
  --cluster-id regional-eu-1 \
  --desired-service-type foghorn \
  --desired-cluster-id media-eu-1

# List tokens (shows name and ID)
frameworks admin bootstrap-tokens list

# Revoke by name (preferred)
frameworks admin bootstrap-tokens revoke --name "edge-nyc-01"

# Revoke by ID
frameworks admin bootstrap-tokens revoke <token-id>

Flags for create:

--name - Display name (default: Bootstrap Token)
--kind - Token kind: edge_node, service, or infrastructure_node (default: edge_node)
--ttl - Time-to-live (e.g., 24h, 7d) (default: 24h)
--tenant-id - Tenant ID (defaults to the active context’s system tenant; required for edge_node)
--cluster-id - Cluster ID (defaults to the active context’s cluster; required for edge_node)
--expected-ip - Expected client IP for validation (optional)
--usage-limit - Maximum uses; omit or set 0 for single use

frameworks admin tenants

List and manage tenants.

# List all tenants (shows name and ID)
frameworks admin tenants list

# Cross-tenant activity rollup (god view): live streams, ingest/viewer
# hours, egress, unique viewers, and API usage per tenant
frameworks admin tenants activity

# Wider window, top 10 only, machine-readable
frameworks admin tenants activity --since 30d --limit 10 --output json

Sample output:

Tenants (2)
 - FrameWorks (id=abc123-...) tier=global
 - Acme Corp (id=def456-...) tier=standard

Tenant activity — last 168h0m0s (2 tenants with activity)
TENANT      TIER      LIVE  VIEWERS NOW  INGEST H  VIEWER H  EGRESS GB  UNIQUES  API REQS  LAST STREAM
Acme Corp   standard  2     14           40.5      120.2     33.50      77       900       2026-06-10
FrameWorks  global    0     0            1.5       0.0       0.00       0        321       2026-06-09

activity joins tenant names and tiers from Quartermaster with the Periscope cross-tenant rollup. The Periscope RPC only answers service credentials, so the command needs an active context with a resolvable service token.

frameworks admin clusters

List and manage clusters.

Subcommands:

list - List clusters
create - Create a cluster
update - Update a cluster
access - Manage tenant access (frameworks admin clusters access list, frameworks admin clusters access grant)
invites - Manage invites (frameworks admin clusters invites create, frameworks admin clusters invites list, frameworks admin clusters invites revoke, frameworks admin clusters invites list-mine, frameworks admin clusters invites accept)
subscriptions - List cluster subscriptions (frameworks admin clusters subscriptions list)
migrate-artifacts - Migrate artifact metadata from a source cluster for one tenant
reassign-control-cell - Move a tenant-private cluster to another Foghorn control cell (--to, --timeout)
control-cell - Show a tenant-private cluster’s control cell, reassignment state, and edges not yet moved

# List all clusters (shows name and ID)
frameworks admin clusters list

# Create an internal cluster
frameworks admin clusters create \
  --cluster-id central-production \
  --cluster-name "FrameWorks Central Production" \
  --cluster-type central \
  --base-url http://10.0.1.10:18002

# Create a customer-facing media cluster with Foghorn and platform flags
frameworks admin clusters create \
  --cluster-id edge-eu \
  --cluster-name "EU Edge Cluster" \
  --cluster-type edge \
  --base-url eu.frameworks.network \
  --foghorn-count 2 \
  --is-platform-official \
  --is-default-cluster

# Update cluster flags
frameworks admin clusters update --cluster-id edge-eu --is-platform-official
frameworks admin clusters update --cluster-id edge-eu --is-default-cluster

# Emergency operator override (not private invites or marketplace onboarding)
frameworks admin clusters access grant --cluster-id edge-eu --tenant-id <tenant-id>

# List cluster access rows for a tenant
frameworks admin clusters access list --tenant-id <tenant-id>

# Create and accept an invite
frameworks admin clusters invites create \
  --cluster-id edge-eu \
  --owner-tenant-id <owner-tenant-id> \
  --invited-tenant-id <invited-tenant-id>
frameworks admin clusters invites list --cluster-id edge-eu --owner-tenant-id <owner-tenant-id>
frameworks admin clusters invites list-mine --tenant-id <tenant-id>
frameworks admin clusters invites accept --invite-token <invite-token>
frameworks admin clusters invites revoke --invite-id <invite-id> --owner-tenant-id <owner-tenant-id>

# List cluster subscriptions for a tenant
frameworks admin clusters subscriptions list --tenant-id <tenant-id>

# Migrate artifact metadata after moving a tenant between clusters
frameworks admin clusters migrate-artifacts \
  --tenant-id <tenant-id> \
  --from-cluster <source-cluster-id>

# Move a tenant-private cluster to another Foghorn control cell, then follow
# its edges moving; reassigning back to the previous cell is the rollback
frameworks admin clusters reassign-control-cell <cluster-id> --to <control-cell-id> --timeout 30m
frameworks admin clusters control-cell <cluster-id>

Sample output:

Clusters (2)
 - FrameWorks Central Production (id=central-production) type=central url=http://10.0.1.10:18002
 - FrameWorks Regional EU (id=regional-eu) type=regional url=http://10.1.1.10:18002

Note: Cluster IDs are either explicit (from the clusters section of your manifest) or auto-generated as {type}-{profile} when no clusters section is defined.

frameworks admin service-pool

Manage pool-assigned media services across logical clusters.

Subcommands:

# Show pool status (total, assigned, unassigned per cluster)
frameworks admin service-pool status --service-type foghorn

# Release instances back to the pool
frameworks admin service-pool add --service-type foghorn --instance-id <uuid>
frameworks admin service-pool add --service-type livepeer-gateway --count 2 --from-cluster <cluster-id>

# Drain an instance from its cluster
frameworks admin service-pool drain --service-type chandler --instance-id <uuid>

# Assign service instance(s) to a cluster (many-to-many)
frameworks admin service-pool assign --service-type foghorn --cluster-id <id> --count 1
frameworks admin service-pool assign --service-type livepeer-gateway --cluster-id <id> --instance-id <uuid>

# Remove service instance(s) from a cluster
frameworks admin service-pool unassign --service-type chandler --cluster-id <id> --instance-id <uuid>

Flags for add:

--instance-id - Instance UUID(s) to release (repeatable)
--count - Number of instances to release from cluster
--from-cluster - Cluster to release instances from

Requires either --instance-id or both --count and --from-cluster.

Flags for assign:

--cluster-id - Cluster to assign Foghorn(s) to (required)
--instance-id - Specific Foghorn instance UUID(s) (repeatable)
--count - Pick N least-loaded Foghorn instances

Flags for unassign:

--cluster-id - Cluster to unassign from (required)
--instance-id - Foghorn instance UUID(s) to remove (required)

Flags for drain:

--instance-id - Instance UUID to drain (required)

frameworks admin clusters create-edge

Create an edge cluster with automatic Foghorn assignment and enrollment token.

frameworks admin clusters create-edge --cluster-name "My Edge" --tenant-id <uuid>

Flags:

--cluster-name - Display name for the cluster (required)
--tenant-id - Tenant ID (defaults to JWT tenant)
--description - Short description

frameworks admin clusters enrollment-token

Create an enrollment token for an existing cluster. Requires cluster owner/operator or provider lifecycle authority; ordinary subscriber access cannot mint node enrollment tokens.

frameworks admin clusters enrollment-token --cluster-id <id> --ttl 7d

Flags:

--cluster-id - Cluster ID (defaults to the active context’s cluster; required if the context has none)
--tenant-id - Tenant ID (defaults to JWT tenant)
--name - Token display name
--ttl - Time-to-live (e.g. 24h, 7d; default 30d)

frameworks admin clusters cert-status

Show cluster readiness including health, Foghorn instances, and certificate status.

frameworks admin clusters cert-status --cluster-id <cluster-id>

Flags:

--cluster-id - Cluster ID (required)

Output includes: cluster name, type, base URL, health status, registered Foghorn instances with status, and certificate readiness.

frameworks admin billing

Billing tier and subscription management.

Subcommands:

tiers - List billing tiers (shows default prepaid/postpaid flags)
init-postpaid - Initialize postpaid billing for a tenant (resolves default postpaid tier, provisions cluster access)
promote - Promote tenant from prepaid to postpaid billing (re-evaluates cluster subscriptions)
set-cluster-pricing - Set or update pricing config for a cluster
credit - Add (or with --debit, remove) EUR prepaid balance for a tenant
set-tier - Put a tenant on a billing tier as an operator decision
grant set / grant show / grant revoke - Operator billing arrangements: base fee, usage waiver, manual invoicing, expiry
record-payment - Record a bank transfer that paid an invoice
prepaid-double-charges - List invoices that charged prepaid tenants twice for the same usage (read-only)

# List billing tiers
frameworks admin billing tiers

# Initialize postpaid account for a tenant
frameworks admin billing init-postpaid --tenant-id <UUID>

# Promote prepaid tenant to postpaid
frameworks admin billing promote --tenant-id <UUID>

# Set cluster pricing (for marketplace or custom cluster billing)
frameworks admin billing set-cluster-pricing \
  --cluster-id <id> \
  --pricing-model tier_inherit \
  --required-tier-level 0 \
  --allow-free-tier

frameworks admin billing credit

Add EUR prepaid balance to a tenant, typically to fund a staging or test tenant or a comped account. The tenant’s tier and billing model do not change: prepaid tenants spend the balance on usage, and postpaid tenants have it applied as credit when their invoice closes. Purser creates the balance if the tenant has none, records operator credit: <reason> (or operator debit: <reason>) as the transaction’s reason, and records your user as its actor when you are logged in. The command prints the transaction ID and the resulting balance.

Flags:

--tenant-id - Tenant UUID (this or --email)
--email - Email of a user in the tenant; resolving it needs a platform-operator session (frameworks login)
--amount - EUR amount with at most two decimals, for example 25.00 (required)
--reason - Why the balance changes; recorded on the transaction (required)
--debit - Confirms a negative --amount; a negative amount without it is refused, and --debit with a positive amount is refused

frameworks admin billing credit --email [email protected] --amount 25.00 --reason "staging storage test"
frameworks admin billing credit --tenant-id <UUID> --amount=-5.00 --debit --reason "reverse test credit"

Write negative amounts as --amount=-5.00 so the shell argument is not read as a flag.

frameworks admin billing set-tier

Assign a billing tier to a tenant as an operator decision. The change applies immediately in either direction, and unlike the self-serve tier change it does not require a Stripe or Mollie subscription or a complete billing profile. Purser refuses a tier that does not exist or is inactive, a billing model the tier does not run (level-0 tiers such as payg run prepaid, all other tiers run postpaid), and a tenant whose account was closed. A tenant without a subscription gets one on the assigned tier. Moving to prepaid creates the prepaid balance and returns any credit an open draft invoice holds to it; moving from prepaid to postpaid keeps the balance, which each postpaid invoice takes as credit when it is finalized, and reactivates a subscription suspended for low balance. Moving from prepaid to postpaid also closes the prepaid part of the period with a prepaid statement, emailed to the tenant, and starts the postpaid period at the change, so the first postpaid invoice charges only usage from the change on, and the base fee for that part of the period. Moving from postpaid to prepaid ends the postpaid plan at the change: its invoice for the period up to the change is finalized then, with the usage reported until the change at the postpaid tier’s prices and the base fee and monthly cluster fees for that part of the period, and collected like any invoice (off-session charge through the tenant’s provider, or record-payment for a bank transfer). A base fee charged in advance for the period has its unused part, from the change on, returned to the prepaid balance. The prepaid period starts at the change and the prepaid balance pays usage reported after it, including late reports for earlier usage. Each plan pays the base fee only for its own part of the period, also across several changes. Drafts hold no credit, so the balance stays visible while the postpaid period runs; an invoice takes only the credit it uses, up to its total, when it is finalized. A scheduled downgrade is cleared.

After the change Purser reconciles cluster access for the new tier and refreshes cached tenant limits, the same as a self-serve tier change, and records billing.subscription_updated with --reason and the calling identity. Assigning the tier and model the tenant already has changes nothing and records no event; cluster access is reconciled again.

A postpaid paid tier invoices its base price and usage when the period closes, and rated work is admitted only once those charges can be collected: the tenant’s own Stripe or Mollie setup, or an operator arrangement from grant set. A tenant put on a paid postpaid tier with set-tier alone, without either, is refused rated work with PAYMENT_SETUP_REQUIRED. Use grant set --tier to assign the tier together with the arrangement.

Flags:

--tenant-id - Tenant UUID (this or --email)
--email - Email of a user in the tenant; resolving it needs a platform-operator session
--tier - Tier name from frameworks admin billing tiers, for example payg or production (required)
--billing-model - prepaid or postpaid; defaults to the model the tier runs
--reason - Why the tier changes; recorded on the audit event (required)

frameworks admin billing set-tier --email [email protected] --tier production --reason "comped partner"
frameworks admin billing set-tier --tenant-id <UUID> --tier payg --billing-model prepaid --reason "back to pay as you go"

Both commands call Purser RPCs that accept only the manifest service token or a platform-operator session; tenant users and API tokens are refused.

frameworks admin billing grant

An operator billing arrangement for one tenant. It stands in for the self-serve rule that a paid postpaid tier needs Stripe or Mollie collection, for accounts you decide to bill differently: a supporter without a monthly fee, a house account you invoice by hand, a staging tenant that pays nothing until a date. Metering and rating keep running under an arrangement, so invoices still show what the tenant used.

An arrangement has four parts:

Base fee (--base-fee): an EUR amount in place of the tier’s base fee, for example 0. Refused while the tenant has a Stripe or Mollie subscription, because that subscription bills the tier’s price on the provider’s side. While it applies, a tenant adding a card through Checkout saves the card without starting a Stripe subscription, and a Mollie subscription is refused.
Usage (--waive-usage): rate usage at zero. Without it usage is billed at the tier’s rates.
Collection (--collection): provider (default) leaves collection to Stripe or Mollie, so a tenant that owes usage needs a card; until it adds one, rated work is refused with PAYMENT_SETUP_REQUIRED. invoice means you collect invoices by hand: invoices are issued and emailed as usual, stay pending until you record the payment with record-payment, and get the usual overdue reminders.
Expiry (--until): a date (the arrangement runs through that day, UTC) or an RFC 3339 time. After it the tier and the tenant’s own payment setup apply again. Without it the arrangement runs until revoked.

A tenant that owes nothing (base fee 0 and usage waived) needs no collection setup. An invoice uses the arrangement in force when its period is invoiced.

frameworks admin billing grant set

Record an arrangement, replacing any previous one. --tier assigns the tier first, as set-tier does. Records billing.subscription_updated with --reason, and prints the arrangement and whether the tenant’s charges can now be collected.

Flags:

--tenant-id / --email - The tenant, as for set-tier
--tier - Also assign this tier; empty keeps the tenant’s tier
--base-fee - EUR base fee in place of the tier’s, for example 0
--waive-usage - Rate usage at zero
--collection - provider (default) or invoice
--until - End of the arrangement
--reason - Why; recorded on the arrangement and the audit event (required)

# Supporter without a monthly fee; the tenant adds a card for usage
frameworks admin billing grant set --email [email protected] --tier supporter --base-fee 0 --reason "beta supporter"

# House account: no monthly fee, usage invoiced and collected by hand
frameworks admin billing grant set --tenant-id <UUID> --tier supporter --base-fee 0 --collection invoice --reason "house account"

# Staging tenant that pays nothing until the end of the year
frameworks admin billing grant set --tenant-id <UUID> --tier production --base-fee 0 --waive-usage --until 2026-12-31 --reason "staging test"

frameworks admin billing grant show

Print the tenant’s arrangement, including an expired one, the tier and billing model it applies to, and whether its charges can be collected and by whom (stripe, mollie or operator). With --output json, tier is the tenant’s tier after the command, and assignment is set only when grant set --tier assigned one.

frameworks admin billing grant show --email [email protected]

frameworks admin billing grant revoke

Remove the arrangement; the tier and the tenant’s own payment setup apply from then on. --reason is required and recorded on the audit event.

frameworks admin billing grant revoke --tenant-id <UUID> --reason "supporter period over"

frameworks admin billing record-payment

Record a payment you received outside Stripe and Mollie, such as a bank transfer from a tenant on --collection invoice. The payment covers the full open balance of a pending or overdue invoice, in the currency the invoice was presented in, and marks the invoice paid.

Flags:

--tenant-id / --email - The tenant
--invoice-id - Invoice UUID (required)
--reference - The reference the payment arrived with (required)
--reason - Why; recorded on the audit event (required)

frameworks admin billing record-payment --tenant-id <UUID> --invoice-id <UUID> --reference "transfer 2026-10-03" --reason "bank transfer received"

The grant and record-payment commands call Purser RPCs that accept only the manifest service token or a platform-operator session.

frameworks admin billing prepaid-double-charges

List finalized usage invoices that charged a prepaid tenant again for usage its prepaid balance had already paid. A prepaid tenant pays usage from its balance as it is reported, and its period now closes with a prepaid statement that has nothing due. Before that, the month end invoiced prepaid tenants and took the invoice total from the same balance as invoice credit, or charged it to the card on file; a tenant moved from prepaid to postpaid mid-period was also invoiced for the prepaid part of the period.

For each invoice the report prints the tenant, the invoice number and ID, its status and period, the usage on the invoice, what the prepaid balance paid for the same period, the invoice credit, what the invoice charged, and the amount charged twice: the invoice’s usage up to what the balance had already paid. That is the amount to refund. The command only reads; refund with frameworks admin billing credit or through the payment provider.

Flags:

--tenant-id - Only this tenant
--limit - At most this many invoices (default 500); the report says when it stopped at the limit

frameworks admin billing prepaid-double-charges
frameworks admin billing prepaid-double-charges --tenant-id <UUID> --output json

The command calls a Purser RPC that accepts only the manifest service token or a platform-operator session.

frameworks admin users

Manage users in existing tenants.

Subcommands:

create - Create a user in an existing tenant (requires service token auth)

# Right after cluster provision — tenant-id comes from context, password from stdin
echo "$DEPLOY_PW" | frameworks admin users create --email [email protected] --password-stdin --role owner

# Or with an explicit tenant (overrides context default)
frameworks admin users create \
  --tenant-id <UUID> \
  --email [email protected] \
  --password <password> \
  --role owner

Flags:

--tenant-id - Existing tenant UUID. Defaults to the active context’s SystemTenantID (populated by a successful cluster provision); override to target a different tenant.
--email (required) - User email address
--password - User password (plaintext; leaks into shell history — prefer --password-stdin).
--password-stdin - Read password from stdin (recommended for CI).
--first-name - First name
--last-name - Last name
--role - owner or member (default: owner)

Without --password or --password-stdin, the CLI prompts at a TTY with hidden echo.

frameworks admin nodes

List registered infrastructure nodes.

Subcommands:

list - List nodes
create - Create/register a node record
hardware - Update hardware metadata (cpu/mem/disk)

# List all nodes
frameworks admin nodes list

# Filter by cluster
frameworks admin nodes list --cluster-id cluster-production

# Filter by type
frameworks admin nodes list --type edge

# Filter by region
frameworks admin nodes list --region us-east-1

# Create a node
frameworks admin nodes create \
  --node-name edge-nyc-01 \
  --node-type edge \
  --cluster-id central-production

# Update hardware metadata
frameworks admin nodes hardware --node-id edge-nyc-01 --cpu-cores 16 --memory-gb 64 --disk-gb 2000

Flags:

--cluster-id - Filter by cluster ID
--type - Filter by node type (common: edge, core; some deployments use regional/central)
--region - Filter by region

frameworks admin nodes fingerprints

Inspect and remove node fingerprint bindings across every tenant. Both subcommands need a platform-operator login and Quartermaster v0.3.11 or later; the CLI cannot call these RPCs on an older running service.

frameworks admin nodes fingerprints list

frameworks admin nodes fingerprints list
frameworks admin nodes fingerprints list --duplicates
frameworks admin nodes fingerprints list --duplicates --cluster-id media-eu-1 --output json

Each binding shows its node, fingerprint ID, cluster, tenant, the start of its machine-ID and MAC hashes, whether it holds an identity key, and when it was last seen. --duplicates keeps only bindings whose machine-ID or MAC hash another binding shares, which are the rows Quartermaster’s unique fingerprint indexes reject. A hash shared by N bindings is marked (xN). A binding whose node row no longer exists shows cluster=(node row missing).

Flags:

--cluster-id - Only bindings of nodes in this cluster
--duplicates - Only bindings that share a machine-ID or MAC hash

frameworks admin nodes fingerprints unbind

frameworks admin nodes fingerprints unbind <node-id> <fingerprint-id> --reason "host replaced by edge-nyc-02"

Deletes the node’s fingerprint binding, including its stored hashes and identity key; the node must enroll again to rebind. Quartermaster records a node.fingerprint_unbound service event with your user ID and the reason. The command asks for confirmation unless --yes is given.

Flags:

--reason - Why the binding is removed (required)
--yes - Skip the confirmation prompt

Cluster Node Lifecycle Commands

Use frameworks cluster nodes ... for provider/operator edge-node lifecycle operations in platform-managed clusters. These commands require a platform context because they call Quartermaster and Foghorn directly with GitOps/service-token backed control-plane access. BYO edge operators should use frameworks edge deploy, which goes through Bridge instead of direct core gRPC. The commands default to the active context’s cluster_id; when a TTY has no active cluster, the CLI prompts for one. Pass --cluster-id when scripting against another cluster.

frameworks cluster nodes add --ssh [email protected]
frameworks cluster nodes list
frameworks cluster nodes drain --node edge-1
frameworks cluster nodes resume --node edge-1
frameworks cluster nodes remove --node edge-1 --wait 4h
frameworks cluster nodes evict --node edge-1

add expands an existing platform-managed cluster. It uses the active context’s cluster_id or prompts for a cluster on TTYs, mints a short-lived enrollment token internally through Quartermaster, passes it into the edge deployment pipeline, and does not print the token. Quartermaster requires provider authority for this path; ordinary tenant subscriptions cannot mint node enrollment tokens. Lifecycle-managed adds default to native mode and, unless --version is supplied, install from the cluster’s release target. Clusters without a release target use the stable manifest. --mode container installs the single edge image; container nodes converge through the same agent-pull release reconciler as native ones. Before provisioning, it probes the target for an existing FrameWorks edge stack and classifies clean, complete same-cluster, foreign, partial, and unmarked installs. Foreign clusters are refused. Same-cluster targets are treated as already added only when the stack and node marker are present, Quartermaster still has an active node registration, and Foghorn reports live node health. Partial installs, stale markers, missing registrations, non-active registry status, and unmarked installs require --force-reapply after operator confirmation. User contexts are for hosted account, insights, and Skipper workflows, not edge-node lifecycle mutation.

Node-targeted commands accept --node <name-or-id> and prompt with an interactive cluster picker, then an interactive node picker, on TTYs when no context cluster or selector is provided. --node-id remains as a deprecated scripting alias.

drain and resume map to Foghorn operational modes. remove drains by default, waits up to 4 hours for active streams to reach zero unless --wait changes the deadline, switches the node to maintenance, and marks the Quartermaster registry row retired. remove --wait 0 is rejected; use evict when the intent is immediate fencing. evict immediately switches to maintenance and marks the registry row evicted.

list --health includes Foghorn mode/stream health and reported component versions. Helmsman reports component versions in its lifecycle heartbeat; Foghorn stores them in foghorn.node_components and returns them through node health.

frameworks cluster releases

Inspect and repair the edge release catalog and active cluster rollout target. Provider/platform contexts use Quartermaster through the configured control-plane resolver.

frameworks cluster releases list
frameworks cluster releases publish --version stable
frameworks cluster releases target get
frameworks cluster releases target set --channel stable --version v1.2.3
frameworks cluster releases target sync

frameworks cluster releases list

List known edge releases. Use --channel and --version to narrow the catalog.

frameworks cluster releases publish

Fetch a GitOps release manifest and publish its edge component catalog into Quartermaster. Use --version for the GitOps release version or channel. By default the catalog records every platform artifact in the manifest; --os and --arch narrow a repair/backfill publish to one platform. This is the provider repair/backfill path. Normal provider cluster provision and cluster upgrade --all runs publish the selected GitOps manifest and sync cluster targets automatically without clearing an operator pause or rollout override. Release rows must include at least one updateable edge component (helmsman, mist, or caddy); config-schema-only rows are rejected because runtime configuration still flows through ConfigSeed. Operators do not label releases to change rollout behavior; disruptive Mist behavior requires a CI-generated update contract.

frameworks cluster releases target

Inspect or override a cluster’s desired edge release target.

frameworks cluster releases target get

Show the active cluster’s release target. Pass --cluster-id to inspect a different cluster.

frameworks cluster releases target set

Set the active cluster’s target release track, optional pinned version, and rollout plan JSON. Pass --cluster-id when scripting. This is an override/repair path; provider cluster provision follows the manifest channel, and cluster upgrade --all syncs the target after the platform upgrade succeeds. Foghorn reconciles targets against reported node component versions and runtime platform metadata. Native and container (single edge image) nodes both converge; only the retired multi-container docker mode was excluded. Helmsman and Caddy changes apply over the existing Helmsman control stream. For Mist, Helmsman atomically replaces each release-owned file inside the stable install directories, installs the controller last, and sends USR1 to the controller’s main/angel process. Mist then performs its built-in rolling handoff: active session, ingest, and processing workers continue, protocol listeners are re-adopted and roll in place, and newly launched workers use the replacement binaries. On Linux, the update is not reported successful until the running controller executable matches the installed replacement. The node remains available throughout. Cordon, drain, update, warmup version confirmation, endpoint probing, and return-to-normal are reserved for Mist releases whose CI-generated update contract explicitly requires a disruptive apply strategy. Release reconciliation only restores drains owned by update-orchestrator; it never clears an operator-owned drain or maintenance fence. Config-schema versions are reported for visibility; actual runtime config still flows through ConfigSeed.

frameworks cluster releases target sync

Publish the selected GitOps release manifest and sync every manifest cluster’s edge release target. Use --version to override the manifest channel with a specific GitOps version or release channel. This is the targeted repair command for the same edge release target sync that cluster provision and cluster upgrade --all run automatically. Supported rollout-plan controls are limited to implemented scheduler behavior such as batch_size, canary, canary_count, error_abort, and max_failed; capacity-floor fields are rejected until disruptive drain rollouts need them.

Edge Commands

Edge node lifecycle operations. Most commands support --ssh user@host (and --ssh-key) for remote execution.

frameworks edge deploy

Deploy an edge node in one command. Handles edge cluster setup, enrollment token generation, pre-registration, and provisioning.

Mode A — Logged-in tenant (automatic cluster setup):

# Deploys to your edge cluster (creates one if needed)
frameworks edge deploy --ssh ubuntu@edge-1 --email [email protected]

# Deploy to a specific cluster
frameworks edge deploy --cluster-id my-edge-prod --ssh ubuntu@edge-1

Mode B — Pre-existing token:

frameworks edge deploy --enrollment-token <token> --ssh ubuntu@edge-1

The token is the only credential you need. Bridge resolves the cluster’s Foghorn via the public bootstrapEdge mutation and proxies the pre-registration. No Foghorn address required.

Flags:

--cluster-id - Cluster to deploy to (auto-detected or auto-created if omitted)
--cluster-name - Name for new edge cluster if one needs to be created
--enrollment-token - Pre-existing enrollment token (skips login and cluster setup)
--foghorn-addr - Explicit Foghorn gRPC override; debug only. Bridge resolves the addr by default; use this when you need to talk to a specific Foghorn directly.
--ssh - SSH target (user@host) for remote deployment
--ssh-key - SSH private key path
--mode - Deployment mode: container (default; single edge image) or native. docker is a deprecated alias for container. Local deploys without --ssh default to native (no Docker dependency) unless --mode is passed explicitly.
--email - ACME email for TLS certificates
--tune - Apply sysctl/limits tuning (default: true)
--skip-preflight - Skip preflight checks
--version - Platform version for binary resolution
--timeout - HTTPS verification timeout (default: 3m)
--token-ttl - Enrollment token TTL when --cluster-id mints a token

frameworks edge preflight

Check host readiness (DNS, ports, sysctl, ulimits).

# Check all requirements
frameworks edge preflight

# Include DNS validation for a specific domain
frameworks edge preflight --domain edge1.example.com

Flags:

--domain - Edge domain to validate DNS for

frameworks edge tune

Apply recommended sysctl and ulimit settings for streaming workloads.

# Preview changes (writes local preview files)
frameworks edge tune

# Apply changes system-wide (requires sudo)
frameworks edge tune --write

Flags:

--write - Write to system paths (requires sudo)
--sysctl-path - Target sysctl path (default: /etc/sysctl.d/frameworks-edge.conf)
--limits-path - Target limits path (default: /etc/security/limits.d/frameworks-edge.conf)

Tuning Applied:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 8192
net.ipv4.ip_local_port_range = 16384 65535
* soft nofile = 1048576
* hard nofile = 1048576

frameworks edge init

Generate edge node configuration files (edge env file, edge compose file, Caddy config file).

# With explicit domain
frameworks edge init --domain edge1.example.com --email [email protected]

# With enrollment token (auto-assigns domain via PreRegisterEdge)
frameworks edge init --enrollment-token enroll_abc123 --email [email protected]

When --enrollment-token is provided without --domain, the CLI calls PreRegisterEdge to get an auto-assigned domain, pool domain, and cluster configuration. The token carries tenant and cluster scope.

Flags:

--dir - Target directory for templates (default: .)
--domain - Edge domain to configure
--enrollment-token - Enrollment token issued by FrameWorks for node bootstrap
--email - ACME email for Let’s Encrypt certificate issuance
--overwrite - Overwrite existing files

frameworks edge enroll

Start the edge stack and verify HTTPS readiness. The bootstrap token must be set as EDGE_ENROLLMENT_TOKEN in the enrollment environment file (.edge-enroll.env).

# Enroll edge node (token read from the enrollment environment file)
frameworks edge enroll

# From a specific directory
frameworks edge enroll --dir /opt/frameworks-edge

# Run remotely via SSH
frameworks edge enroll --dir /opt/frameworks-edge --ssh [email protected]

Flags:

--dir - Directory containing the edge compose and environment files
--timeout - Maximum time to wait for HTTPS readiness (default: 2m)
--ssh - Run remotely on user@host via SSH
--ssh-key - SSH private key path

Note: This command starts Caddy, MistServer, and Helmsman, then waits for HTTPS to become available.

frameworks edge provision

Orchestrate full edge node provisioning (preflight, tune, register, init, enroll).

Usage:

# Single node provisioning
frameworks edge provision --ssh user@host [flags]

# Multi-node provisioning from manifest
frameworks edge provision --manifest edges.yaml

Process:

Run preflight checks
Apply system tuning (optional)
Register node in Quartermaster (optional)
Deliver enrolled TLS/CA/site config through Foghorn ConfigSeed
Generate and upload templates
Start edge stack
Verify HTTPS readiness

Flags:

--ssh string - SSH target (user@host)
--ssh-key - SSH private key path
--pool-domain string - Shared edge pool domain (e.g., edge.media-eu.example.com)
--node-domain string - Individual node domain
--node-name string - Node name for registration
--cluster-id string - Cluster ID for registration
--region string - Region identifier
--email string - ACME email for certificates
--timeout - HTTPS readiness timeout (default: 3m)
--enrollment-token string - Enrollment token for auto-provisioning (skips explicit domain flags)
--manifest string - Path to edge manifest file
--cluster-manifest string - Cluster manifest used to load platform SERVICE_TOKEN for edge manifest register_qm
--parallel int - Number of fresh nodes installed concurrently (default: 1); enrolled nodes are always applied one at a time
--dry-run - Check every node in Ansible check mode and print its plan (skip, apply, drain+apply, or install) and diff; change nothing
--tune - Apply sysctl/limits tuning
--register - Register node in Quartermaster (manual/admin path only — see note below)
--fetch-cert - Fetch TLS certificate from Navigator
--skip-preflight - Skip preflight checks
--capability string - Enabled capability for single-node provisioning; repeat for ingest, edge, storage, processing
--bandwidth-mbps int - Advertised edge bandwidth limit in Mbps; Helmsman writes it to MistServer bwlimit and reports it to Foghorn
--max-transcodes int - Maximum local transcodes reported by Helmsman
--storage-capacity-bytes uint - Local storage capacity limit reported by Helmsman

Re-running edge provision on an enrolled node reuses its identity and re-renders its configuration from this CLI. Enrolled nodes are checked in Ansible check mode first; nodes without drift are skipped. An apply that would restart MistServer or Caddy, or recreate the edge container, is preceded by a drain through Foghorn (the node’s sessions end first) and followed by restoring the node’s prior mode. See Running upgrades.

--register is for the platform manual provisioning path, where an admin supplies --pool-domain/--node-domain directly without an enrollment token and wants to create the Quartermaster node row in the same step. It is rejected unless the active context uses the platform persona, and it is rejected when combined with --enrollment-token because the token path already registers the node through Foghorn’s PreRegisterEdge. Manifest entries that set register_qm: true follow the same platform-only rule and cannot be combined with an enrollment token.

Examples:

# Provision single node with explicit domains (manual/admin path; requires
# a platform context with Quartermaster reachability)
frameworks edge provision --ssh [email protected] \
  --pool-domain edge.media-eu.example.com \
  --node-domain edge-1.media-eu.example.com \
  --register --fetch-cert --email [email protected]

# Provision with enrollment token (auto-assigns domain via PreRegisterEdge,
# registers the node row as a side effect — do NOT pass --register)
frameworks edge provision --ssh [email protected] \
  --enrollment-token enroll_abc123 \
  --email [email protected]

# Provision from manifest
frameworks edge provision --manifest edges.yaml --parallel 4

When --enrollment-token is provided without --pool-domain and --node-domain, the CLI calls PreRegisterEdge to validate the token and get auto-assigned domains, the initial internal CA bundle, and cluster routing information.

frameworks edge status

Check edge node container status and HTTPS health.

frameworks edge status
frameworks edge status --domain edge1.example.com --ssh [email protected]

Flags:

--dir - Directory with compose/env files
--domain - Override EDGE_DOMAIN from env file
--ssh - Run remotely via SSH
--ssh-key - SSH private key path

frameworks edge update

Refresh the edge services. On a container stack rendered by edge init, the compose file and .edge.env are first re-rendered from this CLI’s templates (node identity, pinned image, endpoints, telemetry, and edited restream settings are kept; the write-once enrollment and secrets files are untouched), then the images are pulled and the stack is brought up. A container stack provisioned by edge provision is re-rendered by re-running edge provision; edge update refuses it. Native stacks reload MistServer (rolling), restart Helmsman, and reload Caddy.

frameworks edge update
frameworks edge update --ssh [email protected]

Flags:

--dir - Directory
--ssh - Run remotely via SSH
--ssh-key - SSH private key path

frameworks edge cert

Check and manage TLS certificates.

frameworks edge cert --domain edge1.example.com
frameworks edge cert --reload  # Reload Caddy after check

Flags:

--dir - Directory
--domain - Domain to check
--reload - Reload Caddy after certificate check
--ssh - Run remotely via SSH
--ssh-key - SSH private key path

frameworks edge logs

View edge service logs.

frameworks edge logs
frameworks edge logs --follow --tail 500

Flags:

--dir - Directory
--follow - Follow/tail logs
--tail - Number of lines to show
--ssh - Run remotely via SSH
--ssh-key - SSH private key path

frameworks edge doctor

Run comprehensive health checks on the edge node. The Config Version section reads the node’s provisioned-config marker and reports whether its configuration was rendered by this CLI (current), an older one (stale), or before markers existed (unmarked); the last two come with an edge provision --dry-run next step.

frameworks edge doctor --domain edge1.example.com

Flags:

--dir - Directory
--domain - Domain to check

Additional Edge Commands

Command	Purpose
`frameworks edge drift`	Survey observed edge service state, `.edge.env` keys, and HTTPS health; exits non-zero on divergence.
`frameworks edge diagnose media`	Run MistServer media diagnostics, including HLS validation and analyzer checks for active or specified streams.
`frameworks edge mist`	MistServer-specific edge diagnostics.
`frameworks edge mist cores`	Collect and analyze MistController core dumps.
`frameworks edge mode [mode]`	Read or change Helmsman node mode. Valid modes are `normal`, `draining`, and `maintenance`.

Examples:

frameworks edge drift --dir /opt/frameworks-edge --ssh [email protected]
frameworks edge diagnose media --ssh [email protected]
frameworks edge diagnose media \
  --analyzer HLS \
  --target http://localhost:8080/hls/live/index.m3u8
frameworks edge mode draining --reason deploy --ssh [email protected]

frameworks edge mist cores collect

Collect the latest MistController core dump, MistServer artifact manifest, config snapshot, and service logs from a native edge over SSH.

frameworks edge mist cores collect [email protected]
frameworks edge mist cores collect [email protected] --out ./edge1-mist-core.tar.gz

Flags:

--out - Local output tarball path
--service - systemd unit to include in logs
--since - journalctl time window to include
--ssh-key - SSH private key path

frameworks edge mist cores analyze

Analyze a collected MistController core bundle. If the bundle contains a MistServer artifact manifest with a debug-symbol artifact URL, the CLI downloads the matching symbols into the local cache before invoking gdb or lldb.

frameworks edge mist cores analyze ./edge1-mist-core.tar.gz
frameworks edge mist cores analyze ./edge1-mist-core.tar.gz --cache-dir ~/.cache/frameworks/mistserver-debug

Flags:

--cache-dir - Debug symbol cache directory

Livepeer Commands

Livepeer wallet reads come from Arbitrum, using the wallet addresses that running gateways advertise in Quartermaster discovery and the cluster’s configured RPC. Discovery queries every media cluster the manifest’s livepeer-gateway services are assigned to. The gateway CLI listener remains loopback-only. Routine TicketBroker funding is Purser-owned; exceptional reserve/unlock/withdraw operations SSH to the selected manifest host and require a temporary maintenance-window flag.

frameworks livepeer status

Show each gateway wallet’s ETH balance and TicketBroker deposit, reserve, and withdraw round.

frameworks livepeer status
frameworks livepeer status --cluster staging-media-us
frameworks livepeer status --address 0x... --rpc https://arb1.example

Flags:

--address - Wallet address; overrides Quartermaster discovery
--rpc - Arbitrum JSON-RPC URL; overrides cluster shared env
--cluster - Gitops cluster that selects the manifest, or one livepeer-gateway cluster to narrow discovery to

frameworks livepeer wallet address

Show the gateway wallets’ ETH addresses.

frameworks livepeer wallet address

frameworks livepeer wallet balance

Show the gateway wallets’ ETH balances.

frameworks livepeer wallet balance

frameworks livepeer deposit reserve

Exceptionally add TicketBroker reserve over SSH to the gateway’s loopback CLI. The amount is wei. Routine funding remains Purser-owned.

frameworks livepeer deposit reserve --host regional-eu-1 --amount 200000000000000000

The command fails with a runbook message while tx routes are disabled. Temporarily set enable_cli_tx_routes: "true" on the selected gateway service, provision, run the command, and remove the setting immediately.

frameworks livepeer unlock / withdraw

Start the TicketBroker unlock period or withdraw after it has elapsed:

frameworks livepeer unlock --host regional-eu-1
frameworks livepeer withdraw --host regional-eu-1

Services Commands

Central-tier service operations (for single-host or development deployments).

frameworks services plan

Generate central-tier compose fragments and a .central.env file.

# Generate with default profile
frameworks services plan

# Use a specific profile
frameworks services plan --profile control

# Interactive service selection
frameworks services plan --interactive

# Include/exclude specific services
frameworks services plan --include commodore,quartermaster --exclude purser

The command writes service fragments and a merged central env file into the selected target directory. Exact filenames are CLI-owned; inspect frameworks services plan --help before scripting against them.

Flags:

--profile - Preset: central-all | platform | control | data | media | interfaces | observability | support (default: central-all)
--include - Comma-separated services to include
--exclude - Comma-separated services to exclude
--dir - Target directory for generated files (default: .)
--overwrite - Overwrite existing files
--interactive - Interactive checkbox selection
--config-base - Path to base env file (default: config/env/base.env)
--config-secrets - Path to secrets env file (default: config/env/secrets.env)
--env-output - Path for generated env file (default: <dir>/.central.env)
--env-context - Value for ENV_CONTEXT (default: central)

frameworks services up

Start central services.

frameworks services up
frameworks services up --only commodore,quartermaster
frameworks services up --ssh admin@central-01

Flags:

--dir - Directory containing generated compose files
--only - Comma-separated services to start
--ssh - Run remotely on user@host via SSH
--ssh-key - SSH private key path

frameworks services down

Stop central services.

frameworks services down
frameworks services down --only commodore

Flags:

--dir - Directory containing generated compose files
--only - Comma-separated services to stop
--ssh - Run remotely via SSH
--ssh-key - SSH private key path

frameworks services status

Show container status for running services.

frameworks services status
frameworks services status --only commodore,bridge

Flags:

--dir - Directory containing generated compose files
--only - Comma-separated services to filter
--ssh - Run remotely via SSH
--ssh-key - SSH private key path

frameworks services logs

View service logs.

frameworks services logs
frameworks services logs --only commodore --follow --tail 500

Flags:

--dir - Directory containing generated compose files
--only - Comma-separated services to filter
--follow - Follow/tail logs
--tail - Number of lines to show (default: 200)
--ssh - Run remotely via SSH
--ssh-key - SSH private key path

frameworks services health

Show aggregated service health from Quartermaster.

# All services
frameworks services health

# Filter by service type
frameworks services health --type commodore

# Filter by specific service ID
frameworks services health --service-id commodore-prod-01

Flags:

--service-id - Filter by service ID
--type - Filter by service type (catalog name)

frameworks services discover

Query service discovery endpoints.

frameworks services discover --type commodore
frameworks services discover --type foghorn --cluster-id central-production

Flags:

--type - Service type/catalog name (required)
--cluster-id - Filter by cluster ID

Context Commands

Manage CLI contexts (endpoints, persona, gitops defaults). A context bundles everything the CLI needs to know for one environment: which control-plane endpoints to call, what kind of operator you are (platform / self-hosted / user), and where your cluster manifests live.

Recommended onboarding: run frameworks setup for an interactive wizard that captures all of the above at once. For CI or scripted provisioning, use context create + the context set-* commands below.

frameworks context create

Create a new (empty) context with default localhost endpoints. Does not switch to it — pair with context use <name>.

frameworks context create platform-prod

This is the non-interactive entry point the setup wizard is built on. Use it when you want to bootstrap a context from a script without going through the TTY prompts.

frameworks context set-*

Non-interactive setters for the fields the setup wizard captures interactively. Each targets the active context, or --context <name> for a specific one. They error if the target context doesn’t exist — use context create first.

frameworks context set-persona platform --context platform-prod
frameworks context set-gitops-source local --context platform-prod
frameworks context set-gitops-path /path/to/gitops --context platform-prod
frameworks context set-gitops-repo Livepeer-FrameWorks/gitops --context platform-prod
frameworks context set-gitops-ref main --context platform-prod
frameworks context set-gitops-cluster production --context platform-prod
frameworks context set-gitops-manifest clusters/production/cluster.yaml --context platform-prod
frameworks context set-age-key ~/.config/sops/age/keys.txt --context platform-prod

Available setters: set-persona, set-access-mode, set-gitops-source, set-gitops-path, set-gitops-repo, set-gitops-ref, set-gitops-cluster, set-gitops-manifest, set-age-key. Plus the existing set-url and set-cluster for endpoints.

frameworks context list

List available contexts.

frameworks context list

Sample output:

* local
  production
  staging

frameworks context use

Switch to a different context.

frameworks context use production

frameworks context show

Show details of a context.

# Show current context
frameworks context show

# Show specific context
frameworks context show production

frameworks context set-cluster

Set the active cluster ID for the current context.

frameworks context set-cluster <cluster-id>

This determines which cluster subsequent commands operate against.

frameworks context set-url

Update a service endpoint URL.

frameworks context set-url bridge http://bridge.example.com:18000
frameworks context set-url quartermaster-grpc quartermaster.internal:19002 --allow-insecure
frameworks context set-url foghorn-grpc 10.88.0.10:18019 --allow-insecure

Valid service keys: bridge, signalman-ws, commodore-grpc, quartermaster-grpc, purser-grpc, periscope-grpc, signalman-grpc, foghorn-grpc, decklog-grpc, navigator-grpc. Saved non-local gRPC endpoints do not infer transport from hostname or IP shape. Pass --tls for TLS endpoints and --allow-insecure for plaintext endpoints.

frameworks context check

Check connectivity to all endpoints in the current context.

frameworks context check

Authentication Commands

Authenticate with the FrameWorks platform using your user session token (the same credential the macOS tray uses).

frameworks login

Token storage. The session is written to the platform-native credential store and never appears in shell history:

macOS: Keychain service com.livepeer.frameworks, account user_session.
Other platforms: $XDG_DATA_HOME/frameworks/credentials with mode 0600.

For CI/automation, set FW_USER_TOKEN in the environment — it overrides whatever the store holds. Run frameworks config path --kind credentials to print the active backend (e.g. keychain:com.livepeer.frameworks on macOS).

The platform SERVICE_TOKEN is not a user credential. It is loaded from your manifest env_files (gitops) during cluster provision and admin commands. There is no login --service-account.

frameworks logout

Clear the stored user session.

frameworks logout

Deletes both user_session and refresh_token so a running tray can’t silently repopulate them. The gitops-sourced SERVICE_TOKEN is unaffected — logout does not touch platform configuration.

Utility Commands

frameworks setup

Run the interactive onboarding wizard. It captures the operator persona, Bridge endpoint, and platform GitOps manifest defaults into a CLI context, then makes that context current. Self-hosted/BYO edge setup uses Bridge and does not ask for Quartermaster or Foghorn gRPC addresses.

frameworks setup

For non-interactive setup, use frameworks context create <name> followed by the relevant frameworks context set-* commands.

frameworks update

Update the CLI to the latest version.

# Update to latest
frameworks update

# Check for updates without installing
frameworks update --check

Flags:

--check - Only check for updates, don’t install

Downloads the latest release from GitHub, verifies the checksum, and atomically replaces the running binary. If the binary is installed in a directory requiring elevated permissions, use sudo frameworks update.

frameworks version

Print CLI and platform version info.

frameworks version

Launch interactive menu for common operations.

frameworks menu

Provides a guided interface with these sections; which ones appear depends on the active context’s persona:

Account & Hosted
Edge Operations
Services & Health
Control Plane (Admin)
Cluster Operations
DNS & Mesh
Settings & Contexts

The platform persona sees every section; selfhosted sees Edge Operations, Account & Hosted, and Settings & Contexts; user sees Account & Hosted and Settings & Contexts. frameworks menu --output json prints the same catalog as JSON.

frameworks config env

Generate a merged .env file from layered configuration.

# Generate .env in current directory
frameworks config env

# Custom base and secrets files
frameworks config env --base config/env/base.env --secrets config/env/secrets.env

# Output to stdout
frameworks config env --output -

# Specify context
frameworks config env --context production

Flags:

--base - Path to base env file (default: config/env/base.env)
--secrets - Path to secrets env file (default: config/env/secrets.env)
--output - Output path (default: .env, use - for stdout)
--context - Value for ENV_CONTEXT (default: dev)

Additional Config Commands

Command	Purpose
`frameworks config set <key> <value>`	Store GitHub App settings used for GitHub-sourced manifests.
`frameworks config get <key>`	Print a stored GitHub App setting.
`frameworks config path`	Print the canonical config path or credential-store location. Supports `--kind config\|credentials`.

Supported config set/get keys are github.app-id, github.installation-id, github.private-key, github.repo, and github.ref.

frameworks config set github.repo org/gitops
frameworks config get github.repo
frameworks config path --kind credentials

Crypto Custody Commands

frameworks crypto contains the manual custody workflow. frameworks crypto sweep is deliberately split into online and offline stages; it never sends the HD deposit xprv to Purser.

frameworks crypto smoke

Checks the deployed public wallet-onboarding and agent-payment contract. It validates the published wallet/x402 metadata and issues an expiring challenge for a fixed non-customer address. It never signs or consumes the challenge, creates an account, authenticates, or moves funds.

frameworks crypto smoke --base-url https://bridge.frameworks.network --output json

Run this after deployment alongside the authenticated operator readiness report.

frameworks crypto readiness

Returns the operator-only structured readiness report for direct deposits, x402, finalized RPC heads, scanner health/lag, treasury and relayer configuration, embedded facilitator readiness, tax configuration, and unresolved accounting anomalies. production_eligible is false until every live dependency is present.

frameworks crypto readiness

frameworks crypto wallet rotate

Activates a new watch-only xpub for future deposit and x402 addresses without changing historical address derivation. The xpub is read from a file; the xprv must remain offline.

frameworks crypto wallet rotate --xpub-file new-deposit.xpub --network mainnet

Retain the previous offline xprv until all sweep manifests for that key have been reconciled and its addresses are empty.

frameworks crypto mutation resolve

Inspects an ambiguous paid mutation in dry-run mode, or attaches a known owner-service result / keeps it blocked for operator review. It never releases the idempotency claim for blind execution.

frameworks crypto mutation resolve --tenant-id <uuid> --idempotency-key <key> --reason "owner lookup pending" --mark-review
frameworks crypto mutation resolve --tenant-id <uuid> --idempotency-key <key> --reason "owner result verified" --result-file result.json --status-code 201 --execute

Flags:

--tenant-id - Tenant UUID (required).
--idempotency-key - Original paid mutation key (required).
--reason - Operator evidence/reason (required).
--result-file - Known result body to attach.
--content-type - Result content type; defaults to application/json.
--status-code - Result status code; required with --result-file.
--mark-review - Keep the mutation blocked without attaching a result.
--execute - Apply the resolution; default is dry-run.

frameworks crypto sweep plan

Reads confirmed, unswept direct-deposit and x402 sources at a canonical block, checks current chain balances, and writes a deterministic checksummed manifest. It is a dry run unless --persist is supplied; only a persisted plan can be signed and broadcast.

frameworks crypto sweep plan --network base --out sweep-manifest.json
frameworks crypto sweep plan --network base --out sweep-manifest.json --persist

Flags:

--network - EVM network name.
--out - New manifest file; existing files are never overwritten.
--persist - Claim the source rows and persist the batch.

frameworks crypto sweep sign

Runs without network access on the offline signer. The input descriptor must contain the external-chain xprv corresponding to the manifest xpub. Secret material is never accepted through arguments or environment variables. The separate allowlist file binds each network to its approved treasury.

exec 3< <(password-manager read frameworks-sweep-xprv)
frameworks crypto sweep sign --manifest sweep-manifest.json \
  --treasury-allowlist treasury-allowlist.json --secret-fd 3 \
  --max-fee-gwei 5 --max-priority-fee-gwei 2 \
  --out sweep-bundle.json
exec 3<&-

--secret-fd must be descriptor 3 or higher. Required flags are --manifest, --treasury-allowlist, --secret-fd, --max-fee-gwei, --max-priority-fee-gwei, and --out. The two fee caps are chosen on the offline host and are never trusted from the online manifest.

frameworks crypto sweep broadcast

Revalidates every signed transaction/authorization, the persisted batch, current treasury configuration, and signature source. It is a dry run unless --execute is supplied. Execution additionally requires --ack to exactly equal the signed bundle checksum.

frameworks crypto sweep broadcast --bundle sweep-bundle.json
frameworks crypto sweep broadcast --bundle sweep-bundle.json --execute --ack <bundle-checksum>

Flags:

--bundle - Signed bundle path (required).
--execute - Persist transaction hashes and submit transactions.
--ack - Exact bundle checksum required with --execute.

frameworks crypto sweep reconcile

Checks transaction receipts against the consensus-labelled finalized head and verifies canonical block hashes. It is read-only unless --apply is supplied.

frameworks crypto sweep reconcile --batch-id <uuid>
frameworks crypto sweep reconcile --batch-id <uuid> --apply

Flags:

--batch-id - Persisted sweep batch UUID (required).
--apply - Persist confirmed/failed item and batch states.

frameworks crypto sweep release

Rechecks chain state for an expired or failed sweep batch. Eligible claims are released for replanning; ambiguous signed or broadcast claims are quarantined. The command is read-only unless --execute is supplied. Run the dry-run first to obtain the manifest checksum, then bind execution to that checksum.

frameworks crypto sweep release --batch-id <uuid> --reason "expired offline ceremony"
frameworks crypto sweep release --batch-id <uuid> --reason "expired offline ceremony" --execute --ack <manifest-checksum>

Flags:

--batch-id - Persisted sweep batch UUID (required).
--reason - Operator reason recorded in the immutable event log (required).
--execute - Apply eligible releases and quarantines.
--ack - Exact manifest checksum returned by dry-run; required with --execute.

Exit Codes

0 - Success
1 - Any error, including invalid arguments and flags. cluster diff, cluster drift, and edge drift also exit 1 when they find changes or divergence, and cluster apply exits 1 when the diff contains entries it cannot apply.
2 - Command-specific status: cluster os update exits 2 when OS updates are pending, and cluster apply exits 2 when the rollout halts on a failing host.

Operator CLI Reference

Global Flags

Command Catalog

Personas and Access Modes

Cluster Manifest-Source Flags

Detection & Health Commands

frameworks cluster detect

frameworks cluster doctor

frameworks cluster status

frameworks cluster preflight

frameworks cluster drift

frameworks cluster diff

frameworks cluster apply

Provisioning Commands

frameworks cluster provision

frameworks cluster control-plane

frameworks cluster finalize (deprecated)

frameworks cluster init

Capacity-owner consent review keys

Additional Cluster Provisioning Commands

Operations Commands

frameworks cluster skipper

frameworks cluster skipper knowledge reset

frameworks cluster skipper posts

frameworks cluster metabase

frameworks cluster metabase sync

frameworks cluster grafana

frameworks cluster grafana sync

frameworks cluster logs

frameworks cluster logs snapshot

frameworks cluster snapshot

frameworks cluster os

frameworks cluster os update

frameworks cluster restart

frameworks cluster upgrade

frameworks cluster release

frameworks cluster storage

frameworks cluster yugabyte relayout

Backup & Restore Commands

frameworks cluster backup create

frameworks cluster backup verify

frameworks cluster restore

frameworks cluster restore finish

frameworks cluster restore rollback

Diagnostic Commands

frameworks cluster diagnose network

frameworks cluster diagnose resources

frameworks cluster diagnose ports

frameworks cluster diagnose kafka

frameworks cluster diagnose media

frameworks cluster diagnose media-authority

frameworks cluster diagnose dvr

DNS Commands

frameworks dns doctor

Mesh Commands

frameworks mesh wg generate

frameworks mesh wg check

frameworks mesh wg rotate

frameworks mesh status

Additional Mesh Commands

Admin Commands

frameworks admin tokens

frameworks admin bootstrap-tokens

frameworks admin tenants

frameworks admin clusters

frameworks admin service-pool

frameworks admin clusters create-edge

frameworks admin clusters enrollment-token

frameworks admin clusters cert-status

frameworks admin billing

frameworks admin billing credit

frameworks admin billing set-tier

frameworks admin billing grant

frameworks admin billing grant set

frameworks admin billing grant show

frameworks admin billing grant revoke

frameworks admin billing record-payment

frameworks admin billing prepaid-double-charges

frameworks admin users

frameworks admin nodes