Skip to content
FrameWorks Docs
Blog

Deployment Overview

This guide covers a full operator-owned deployment. If you only want to add customer-operated media capacity to the managed FrameWorks control plane, use the hybrid edge guide.

Media authority and control-plane outages

Section titled “Media authority and control-plane outages”

Provisioning generates separate signing and per-cell sealing material for the media-authority pipeline. The production CLI renderer—not the development compose file—places the signing private key and recipient map only on Commodore, and the trust set plus one distinct decryption key on each Foghorn cell. The derivation root is not rendered into any workload. Do not copy these variables between cells or hand-edit Mist trigger configuration. The rendered key ID is bound to the control-cell ID and X25519 public key; a cell/key mismatch now fails during service startup rather than during the first sealed-source request.

The derived workload variables are intentionally service-specific:

  • Commodore receives MEDIA_AUTHORITY_SEAL_RECIPIENTS.
  • Each Foghorn receives MEDIA_AUTHORITY_CELL_ID, MEDIA_AUTHORITY_SEAL_KEY_ID, and MEDIA_AUTHORITY_SEAL_PRIVATE_KEY_PEM_B64 for its own control cell.

These four values are renderer output, not shared-secret inputs. Operators configure the seal root and cell topology; provisioning derives and places the per-cell material.

For a fresh production deployment, create the complete shared input once with frameworks cluster secrets generate-shared --out /secure/path/shared.env, import that fragment through the GitOps repository’s supported SOPS editing command, and securely remove the plaintext file. The complete input contains DATABASE_RUNTIME_PASSWORD, SERVICE_TOKEN, CLUSTER_ACCESS_MATERIALIZATION_SECRET, FOGHORN_BALANCER_CAPABILITY_SECRET, FOGHORN_STATE_ENCRYPTION_KEY, MEDIA_AUTHORITY_SEAL_ROOT_SECRET, JWT_SECRET, PASSWORD_RESET_SECRET, FIELD_ENCRYPTION_KEY, USAGE_HASH_SECRET, TELEMETRY_TOKEN_SECRET, MEDIA_AUTHORITY_SIGNING_KEY_ID, MEDIA_AUTHORITY_SIGNING_PRIVATE_KEY_PEM_B64, and MEDIA_AUTHORITY_TRUST_SET. The renderer derives the per-cell variables; operators do not add derived recipients or private cell keys to the shared input.

During the v0.3.0 application-field migration, preserve an outgoing JWT_SECRET in the JSON array FIELD_ENCRYPTION_LEGACY_SECRETS before rotating it. This read-only compatibility channel accepts historical short JWT secrets; do not place JWT secrets in FIELD_ENCRYPTION_PREVIOUS_KEYS, which is reserved for versioned field keys and enforces the current field-key length floor.

After authority has converged, an existing media cell can continue playback, ingest under its signed outage owner, source resolution, outputs, and already dispatched work while Commodore, Quartermaster, or Purser is unavailable. This is bounded by signed hard expiry and certificate validity; never-seen, revoked/tombstoned, corrupt, or hard-expired authority remains unavailable. Foghorn and Helmsman retain last-good runtime configuration across restart.

This is bounded autonomy, not cross-cell consensus. During an asymmetric partition, the deterministic outage owner cannot observe a still-live claim in an isolated peer cell, so dual ingest can exist until connectivity or leases converge. The cross-cell placement claim is leased; tenant capacity is checked from signed policy against durable per-cell ingest sessions. If the complete Mist-liveness signal path is isolated, another cell may admit a publisher after the placement claim expires. Events whose tenant identity cannot be resolved are never allowed to mutate a guessed tenant’s state. See Media-cluster authority and autonomy for the exact boundaries.

During a v0.3 upgrade, an existing node fingerprint without an Ed25519 key is not yet usable as outage admission. Keep Quartermaster available for each existing media node’s first post-upgrade reconnect: after a stable machine-ID or MAC match, Quartermaster atomically binds the node’s proved key. It never seeds that binding from peer IP or replaces a different key. Once Foghorn has stored the resulting bounded admission, later reconnects can use the cell-local copy during a control-plane outage.

Keep Helmsman’s durable state separate from reclaimable media. Container edges mount /data/state independently from /data/storage; native installs render the equivalent state directory through the Helmsman role. If that state is lost, obtain a fresh enrollment token and run frameworks edge provision --ssh <user>@<host> --force-reenroll --enrollment-token <fresh-token> (or use --local). The signed request authorizes Quartermaster to rotate the pinned node key only when the existing tenant/node and stable machine or MAC fingerprint still match. Helmsman binds the one-shot rotation to that token, reuses the replacement key across retries, and records completion after Foghorn accepts it; a persisted provisioning flag cannot rotate it again on restart. Normal redeploys do not rotate it.

The shared production secret named FOGHORN_STATE_ENCRYPTION_KEY is a deployment root, not a cross-cell runtime key. The CLI derives a different key for every Foghorn control cell while keeping replicas in one cell compatible. Development commands persist generated authority roots beside the local manifest in .frameworks/dev-generated-secrets.env; production never relies on process-local generated secrets.

The shipped rules alert on authority delivery backlog/version lag, apply or verification rejection, hard-expired reads, and durable node-admission failures/saturation. During a planned outage canary, query foghorn_media_request_central_rpcs_total{path,service,method} before and after the test: a request covered by ready local authority must not increment it, except for ingest endpoint discovery’s single bounded Commodore live-claim check. A static alert on every central RPC would be wrong during normal connected operation. New management mutations and new x402 settlement remain online-only.

Quick Start

Section titled “Quick Start”

Production Deployment (Recommended):

flowchart LR
    M[Cluster manifest] --> P[frameworks cluster provision]
    S[SOPS-encrypted secrets] --> P
    P --> I[Infrastructure]
    I --> A[Application services]
    A --> X[Ingress and interfaces]
    X --> D[frameworks cluster doctor]
Terminal window
# 1. Install CLI
curl -sSfL https://github.com/Livepeer-FrameWorks/monorepo/releases/latest/download/install.sh | sh


# 2. Review prerequisites
# See: /operators/prerequisites


# 3. Create cluster manifest + secrets file
# See: /operators/cluster-manifest
# Copy config/env/secrets.env.example → gitops/secrets/production.env


# 4. Detect hosts (verify SSH connectivity)
frameworks cluster detect --manifest cluster.yaml


# 5. Provision + init + static seeds (one command, usable platform)
frameworks cluster provision --manifest cluster.yaml \
  --bootstrap-admin-email you@co --bootstrap-admin-password-env DEPLOY_PW


# 6. Verify health — subsequent commands pick up the saved manifest path
#    from the active context, so no need to re-pass --manifest.
frameworks cluster doctor

GitOps Deployment (from a GitHub repo):

If your manifests are in a gitops repo managed by a GitHub App:

Terminal window
# Configure GitHub App credentials (once)
frameworks config set github.app-id 12345
frameworks config set github.installation-id 67890
frameworks config set github.private-key /path/to/key.pem
frameworks config set github.repo org/gitops


# Provision from the repo (fetches cluster.yaml, hosts, .env, etc.)
frameworks cluster provision --github-repo org/gitops --cluster production


# If secrets are SOPS-encrypted (recommended), specify an age key:
frameworks cluster provision --github-repo org/gitops --cluster production --age-key ~/.config/sops/age/keys.txt

This fetches the cluster manifest and referenced files (env files, host inventory) from the repo, then runs the standard provision pipeline. SOPS-encrypted files — both env files and host inventory YAML — are decrypted automatically if an age key is available (see External Services). Operators manage config changes via pull requests.

Local Development:

Terminal window
# Quick local stack with Docker Compose
cd monorepo
cp config/env/secrets.env.example config/env/secrets.env  # First time only
make env                  # Generate .env from layered config
docker-compose up -d

See root README.md for local endpoints and ports.

Deployment Paths

Section titled “Deployment Paths”
Section titled “1. CLI Deployment (Recommended)”

The FrameWorks CLI automates the complex orchestration of multi-tier infrastructure, handling everything from SSH keys to database migrations (via internal provisioners; Ansible is used for some infrastructure components).

Features:

  • Multi-host SSH deployment
  • Docker and native (systemd) support
  • GitOps version management
  • Automated health validation
  • Backup/restore operations
  • Health and configuration diagnostics
CLI Reference Grouped operator CLI overview.
Cluster Manifest Configure your deployment.
Prerequisites System requirements and setup.
External Services Third-party integrations.

2. Manual Deployment Reference

Section titled “2. Manual Deployment Reference”

For operators who need to understand or debug what the CLI configures:

Architecture

Section titled “Architecture”

For detailed architecture documentation including service tables, ports, deployment tiers, and data flow diagrams, see Architecture Overview.

Security

Section titled “Security”

Network Security

Section titled “Network Security”
  • Backend nodes (central/regional) communicate over a private mesh (WireGuard or equivalent). Edge nodes do not join the mesh.
  • When running more than one backend node, a private mesh is required for databases, Kafka, and service-to-service calls.
  • TLS termination varies by tier:
    • Development: Nginx reverse proxy (optional TLS)
    • Edge nodes: Caddy (automatic HTTPS via HTTP-01)
    • Central/Regional: Navigator-issued certificates (DNS-01) or operator-managed proxies
  • Firewall rules restrict access to required ports

Data Security

Section titled “Data Security”
  • TLS at ingress (Caddy/Navigator/proxies) should be enabled; internal gRPC TLS is not enabled yet (RFC draft exists).
  • Kafka inter-broker encryption is not enabled yet; the mesh provides the current security boundary.
  • Service-to-service authentication uses SERVICE_TOKEN (RFC draft exists for stronger service identity).
  • Secrets are currently managed via flat env files pushed during provisioning. This is an interim solution — a HashiCorp Vault integration is planned (RFC draft exists) to replace plaintext env files with dynamic secret injection.

Access Control

Section titled “Access Control”
  • Multi-tenant isolation at application layer
  • User roles and exact API-token scopes are enforced at the Gateway and service boundaries. Coarse read and write values are not wildcards; issue least-privilege namespaced scopes and add mcp:high-risk only for unattended high-risk MCP actions.
  • The Gateway has tenant-aware rate-limit middleware. Free-tier product limits are enforced at Foghorn admission with a three-tier cluster-load policy: under 50% load all free traffic admitted; 50–95% load rejects new ingest from over-allowance free streams; at the 95% redline all new free ingest is rejected. Viewer admission follows the same shape with a higher first threshold (80% / 95%). Free tenants are also capped at 3 concurrent live streams and 200 concurrent viewers. Active sessions are never killed mid-stream. Recordings, VOD uploads, and clip exports are rejected at the per-tenant 10 GB storage cap.
  • SSH key-based authentication for deployments

For monitoring, operations, and CLI commands, see CLI Reference and Operations.