Cluster Manifest Reference

The cluster manifest is the blueprint for your FrameWorks deployment. It’s a YAML file that tells the CLI how to provision infrastructure, services, and interfaces.

Tip

FrameWorks uses a GitOps-friendly approach. We recommend committing your cluster manifest private repository to track changes over time.

File Structure

version: v1
type: cluster
profile: production
channel: stable # optional: stable (default) or rc
root_domain: example.com # optional: used for Caddy routing and cookie domains
env_files:
  - config/production.env
  - secrets/production.env
hosts_file: clusters/production/hosts.enc.yaml # optional: SOPS-encrypted host inventory
bootstrap_overlay: gitops/clusters/production/bootstrap.yaml # optional desired-state overlay
hosts: {}
clusters: {}
wireguard: {}
infrastructure: {}
services: {}
interfaces: {}
observability: {}
geoip: {}
tls_bundles: {}
ingress_sites: {}

Top-Level Fields

version

Manifest schema version (required).

Example:

version: v1

---

type

Cluster manifest type (required).

Values:

• cluster - Full multi-service deployment.
• edge - Single edge node deployment (use the dedicated edge manifest format for multi-node edge fleets; see below).

Example:

type: cluster

---

profile

Environment/profile name. This is used to form the default cluster ID during provisioning ({type}-{profile}), so choose a stable slug (e.g., production, eu-1).

Common values: development, staging, production.

Example:

profile: production

---

channel

Release channel to track (optional). Defaults to stable.

Values: stable (production-ready), rc (release candidate for staging/testing).

Example:

channel: rc

---

env_files

Paths to shared environment files for all services (optional). Relative to the manifest directory. Files are loaded in order. Shared env files override auto-generated environment variables, but are overridden by per-service env_file and inline config entries.

Example:

env_files:
  - config/production.env
  - secrets/production.env

---

root_domain

Base domain used by Caddy routing, public interface metadata, and production cookie-domain derivation.

Example:

root_domain: example.com

---

hosts_file

Optional path to a SOPS-encrypted host inventory. When set, frameworks cluster provision merges sensitive host fields such as external_ip, user, and WireGuard private-key material from this file before validation.

Example:

hosts_file: clusters/production/hosts.enc.yaml

---

bootstrap_overlay

Optional path to a bootstrap desired-state overlay. The CLI renders manifest-derived bootstrap state and then merges this overlay for operator-authored tenants, billing, cluster pricing, and service registry overrides that do not belong directly in the manifest.

Example:

bootstrap_overlay: gitops/clusters/production/bootstrap.yaml

---

hosts

Map of host definitions for SSH access and mesh identity. The CLI connects to each host via SSH using external_ip; internal service traffic uses wireguard_ip or .internal names after Privateer starts.

Note

Managed backend clusters must generate WireGuard identity before provisioning: frameworks mesh wg generate --manifest clusters/production/cluster.yaml. cluster provision validates this state and never writes it.

Format:

hosts:
  <hostname>:
    external_ip: <public-ip>
    user: <ssh-user>
    cluster: <cluster-id>
    wireguard_ip: <mesh-ip>
    wireguard_public_key: <base64-public-key>
    wireguard_port: 51820
    roles: [infrastructure, services] # optional
    labels: # optional
      environment: prod

Fields:

• external_ip (required unless provided by hosts_file) - Public IP address used for SSH/bootstrap and WireGuard endpoint dialing
• user (required unless provided by hosts_file) - SSH username
• cluster (optional) - Explicit cluster membership for this host. When one cluster is declared, hosts implicitly belong to it
• wireguard_ip (required for Privateer-covered hosts) - GitOps-owned internal mesh address
• wireguard_public_key (required for Privateer-covered hosts) - Public half of the host mesh identity
• wireguard_port (required for Privateer-covered hosts) - Host WireGuard UDP listen port
• wireguard_private_key_file (optional) - On-disk private-key path for adopted-local nodes
• wireguard_private_key_managed (optional) - Set to false for adopted-local nodes whose key is preserved on disk
• roles (optional) - Arbitrary tags used for human grouping
• labels (optional) - Free-form metadata

Note

SSH authentication is resolved from the operator’s machine — ssh-agent, ~/.ssh/config, and default identities (~/.ssh/id_ed25519, ~/.ssh/id_rsa). If ssh <external_ip> or ssh frameworks-<hostname> works for you locally, the CLI works. There is no ssh_key field in the manifest.

---

clusters

Map of cluster definitions for multi-cluster deployments. Each key is the cluster ID that will be registered in Quartermaster.

Format:

clusters:
  prod-platform:
    name: Production Platform
    type: central
    owner_tenant: frameworks
    roles: [control, data, analytics, support, infra, mesh, interface, observability]
  prod-media:
    name: Production Media Plane
    type: edge
    default: true
    platform_official: true
    allow_private_pull_sources: false
    owner_tenant: frameworks
    region: us-east
    roles: [media]
    pricing:
      model: tier_inherit
      required_tier_level: 0
      allow_free_tier: true

Fields:

• name (required) - Display name for the cluster
• type (required) - Cluster type: central, edge, etc.
• region (optional) - Region identifier
• roles (optional) - Service roles that belong to this cluster. Used for automatic cluster resolution when services don’t have an explicit cluster field.
• default (optional) - Mark as the default cluster. New tenants auto-subscribe to this cluster on signup. At most one cluster may have default: true.
• platform_official (optional) - Mark as a platform-operated cluster. Used by billing tier access and unauthenticated service discovery.
• allow_private_pull_sources (optional) - Capability flag allowing pull-input streams on this media cluster to use private literal source addresses such as RFC1918, ULA, or non-link-local multicast. Keep this false for platform-managed clusters and enable it only on self-hosted edges that can reach those sources. This does not place a pull stream by itself; private sources must also set allowed_cluster_ids / allowedClusters.clusterIds on the pull source.
• owner_tenant (optional) - Owning tenant. Use frameworks for the system tenant, or a tenant UUID.
• pricing (optional) - Purser cluster pricing config. When present, provision reconciles it authoritatively. When absent, existing pricing is left untouched.
  • model (required) - free_unmetered, metered, monthly, tier_inherit, or custom
  • required_tier_level (optional) - Minimum billing tier level (0-5)
  • allow_free_tier (optional) - Allow free-tier tenants
  • default_quotas (optional) - Cluster pricing metadata for marketplace/self-host offers. Platform Free-tier stream/viewer caps are billing-tier entitlements, not static cluster capacity.

Role-based resolution: When a service doesn’t specify cluster, the CLI matches the service’s role (from the built-in service registry) against each cluster’s roles list. For example, foghorn has role media, so it resolves to whichever cluster lists media in its roles. Available roles: control, data, analytics, media, support, infra, mesh, interface, observability.

Resolution priority:

1. Explicit cluster field on the service
2. Role match against cluster roles
3. If only one cluster is defined, use it
4. Auto-generated {type}-{profile} when no clusters section is declared

Tip

If you omit the clusters section entirely, the CLI creates a single cluster with ID {type}- {profile} (e.g., cluster-production). Use this for single-cluster deployments; declare clusters explicitly for anything more granular.

---

wireguard

WireGuard mesh configuration for backend nodes managed by Privateer (api_mesh). Production/staging managed clusters keep this block in GitOps and generate per-host identity with frameworks mesh wg generate.

Tip

Local compose/dev manifests may leave this disabled only when Privateer is disabled. Managed native clusters require wireguard.enabled: true and complete per-host identity before provisioning.

Format:

wireguard:
  enabled: true
  mesh_cidr: 10.88.0.0/16
  listen_port: 51820

Fields:

• enabled (required)
• mesh_cidr (required when enabled) - CIDR used by generated host wireguard_ip values
• listen_port (required when enabled) - Default UDP listen port used by mesh wg generate

Edge nodes do not join the mesh.

---

Infrastructure Section

Database and messaging infrastructure.

Version Reference: See config in the monorepo for tested/recommended versions.

postgres

PostgreSQL or YugabyteDB configuration.

Format:

infrastructure:
  postgres:
    enabled: true
    engine: postgres
    mode: native
    version: "15.14"
    host: central-01
    port: 5432
    databases:
      - name: quartermaster
        owner: quartermaster_user

Fields:

• enabled (required)
• engine (optional) - postgres by default, or yugabyte
• mode (required) - native
• version (required)
• host (required for single-node Postgres) - Host name from hosts
• nodes (required for multi-node YugabyteDB) - List of {host, id, rpc_port} entries
• port (required)
• replication_factor (optional) - Defaults to the number of nodes
• databases (optional)
• tuning (optional) - key/value overrides for server tuning
• sql_access (optional) - direct by default, or ssh
• password (optional) - Database password override

---

clickhouse

ClickHouse analytics database.

Format:

infrastructure:
  clickhouse:
    enabled: true
    mode: native
    version: "25.9.2.1"
    host: central-01
    port: 9000
    databases: [periscope]

Fields:

• enabled (required)
• mode (required) - native
• version (required)
• host (required)
• port (required)
• databases (optional)
• sql_access (optional) - direct by default, or ssh

---

redis

Redis or Valkey cache instances. Use named instances when services need separate logical caches.

Format:

infrastructure:
  redis:
    enabled: true
    engine: valkey
    mode: native
    version: "8.1"
    instances:
      - name: foghorn
        host: central-01
        port: 6379
      - name: chatwoot
        engine: redis
        host: central-01
        port: 6380

Fields:

• enabled (required)
• engine (optional) - Default engine for instances, valkey or redis
• mode (required) - docker or native
• version (optional)
• instances (required)
  • name (required)
  • engine (optional) - Instance-level engine override
  • host (required) - Host name from hosts
  • port (required)
  • password (optional)
  • config (optional) - key/value overrides such as maxmemory or appendonly

---

kafka

Apache Kafka message broker. Kafka native mode is KRaft-only.

Two modes are supported:

• Dedicated controllers (recommended for production): Add a controllers list. Controllers run the metadata quorum as separate processes; brokers handle client traffic only. Controllers are provisioned first, then brokers join via dynamic discovery.
• Combined mode (suitable for dev): Omit controllers. Each broker runs both broker and controller roles in a single process.

Production format (dedicated controllers):

infrastructure:
  kafka:
    enabled: true
    mode: native
    version: "4.2.0"
    cluster_id: "iwNix2BLQny7Z9tlyna7Qw"
    min_insync_replicas: 2
    controllers:
      - host: central-01
        id: 100
        port: 9093
        dir_id: "2uBL_MXZSAe-9agF6p5aMQ"
      - host: media-01
        id: 101
        port: 9093
        dir_id: "2AyDvMsASN26Tk7joXgJ6Q"
      - host: media-02
        id: 102
        port: 9093
        dir_id: "ZTJH65OoRGqcxUYwht1RSw"
    brokers:
      - host: central-01
        id: 1
        port: 9092
      - host: media-01
        id: 2
        port: 9092
      - host: media-02
        id: 3
        port: 9092
    topics:
      - name: analytics_events
        partitions: 10
        replication_factor: 3

Fields:

• enabled (required)
• mode (required) - native
• version (required) - Apache Kafka version (4.x+)
• cluster_id (required) - KRaft cluster UUID. Generate with kafka-storage.sh random-uuid
• controllers (optional) - dedicated controller nodes. If present, requires at least 3. Controller IDs must not overlap with broker IDs
  • host (required) - host name from hosts map
  • id (required) - controller node ID
  • port (optional) - controller listener port, default 9093
  • dir_id (required) - directory UUID for dynamic quorum bootstrap. Generate with kafka-storage.sh random-uuid
• controller_port (optional) - combined mode only, default 9093
• brokers (required) - at least 1; use 3+ for production
• topics (optional)
• min_insync_replicas (optional)
• delete_topic_enable (optional)
• offsets_topic_replication_factor (optional)
• transaction_state_log_replication_factor (optional)
• transaction_state_log_min_isr (optional)

Services / Interfaces / Observability

All three sections share the same shape. Use them to define application services, public interfaces (reverse proxy + web surfaces), and observability stack (VictoriaMetrics, vmagent, Grafana, etc.).

Service Definition

Format:

services:
  commodore:
    enabled: true
    mode: docker
    version: v0.0.1
    image: livepeerframeworks/frameworks-commodore:latest
    host: central-01
    port: 18001
    grpc_port: 19001
    env_file: /etc/frameworks/commodore.env
    depends_on: [postgres, kafka]
    config:
      EXAMPLE_KEY: value

Fields:

• enabled (required)
• mode (required) - docker or native
• version (optional) - Version or channel to track
• image (optional) - Docker image (docker mode)
• binary_url (optional) - Binary download URL (native mode)
• deploy (optional) - Override deploy slug (container/binary name)
• cluster (optional) - Explicit cluster assignment. Must reference a key in clusters. When omitted, the service’s role determines its cluster automatically
• host (optional) - Single host
• hosts (optional) - Multiple hosts for replicas. The CLI creates one task per host (e.g., bridge@regional-eu-1)
• replicas (optional) - Replica count
• port (optional)
• grpc_port (optional)
• env_file (optional) - Per-service env file (overrides the top-level env_files)
• depends_on (optional)
• config (optional) - Inline key/value overrides (highest priority)

Privateer auto-deploy

When privateer has no host or hosts field, the CLI automatically deploys it to every non-edge host in the manifest. This ensures the WireGuard mesh covers all backend nodes before application services start, and it is now required anywhere internal gRPC service leaf certificates are staged under /etc/frameworks/pki/services/<service>/.

Environment variable merge order (later wins):

1. Auto-generated from manifest infrastructure (DATABASE_URL, KAFKA_BROKERS, QUARTERMASTER_GRPC_ADDR, etc.)
2. Top-level env_files
3. Per-service env_file
4. Inline config map

The merged result is written to /etc/frameworks/{service}.env on each host.

For profile: production, provisioning also forces secure internal gRPC defaults:

• BUILD_ENV=production
• GRPC_ALLOW_INSECURE=false
• GIN_MODE=release when not explicitly set or when set to debug

If navigator is enabled in a production manifest, the shared env_files must also provide:

• one complete managed CA source for Navigator:
• NAVIGATOR_INTERNAL_CA_ROOT_CERT_FILE
• NAVIGATOR_INTERNAL_CA_INTERMEDIATE_CERT_FILE
• NAVIGATOR_INTERNAL_CA_INTERMEDIATE_KEY_FILE
• or:
• NAVIGATOR_INTERNAL_CA_ROOT_CERT_PEM_B64
• NAVIGATOR_INTERNAL_CA_INTERMEDIATE_CERT_PEM_B64
• NAVIGATOR_INTERNAL_CA_INTERMEDIATE_KEY_PEM_B64

---

Edge Manifest

For multi-node edge fleets, use a dedicated edge manifest.

Format:

version: v1
channel: stable
root_domain: example.com
pool_domain: edge.media-eu.example.com
email: [email protected]
cluster_id: regional-production
cluster_manifest: cluster.yaml
enrollment_token: fw_edge_...
hosts_file: clusters/edge/hosts.enc.yaml
mode: docker
capabilities: [ingest, edge, storage, processing]
bandwidth_mbps: 2000
max_transcodes: 4
storage_capacity_bytes: 500000000000
nodes:
  - name: edge-us-east-1
    ssh: [email protected]
    subdomain: edge-us-east-1
    region: us-east
    apply_tune: true
    register_qm: true

SSH auth for edge nodes resolves from the operator’s machine. Pass --ssh-key on the frameworks edge provision command when an explicit key is needed.

Fields:

• channel (optional) - stable, rc, or an explicit version such as v0.2.0-rc3
• root_domain or pool_domain (at least one required)
• email (ACME)
• cluster_id (required when register_qm: true)
• cluster_manifest (optional) - Cluster manifest used to load platform SERVICE_TOKEN for register_qm; relative paths resolve from the edge manifest directory. Can also be passed as --cluster-manifest.
• enrollment_token (optional) - Token used for edge bootstrap
• hosts_file (optional) - SOPS-encrypted edge host inventory
• fetch_cert (deprecated) - Ignored for edge manifests; enrolled edge TLS is delivered by Foghorn ConfigSeed
• mode (optional) - Default node mode, docker by default or native
• capabilities (optional) - Default enabled Helmsman capabilities: ingest, edge, storage, processing. Empty means all capabilities enabled.
• bandwidth_mbps (optional) - Default node bandwidth limit advertised by Helmsman and reconciled into MistServer bwlimit.
• max_transcodes (optional) - Default local transcoding concurrency limit reported by Helmsman.
• storage_capacity_bytes (optional) - Default local storage capacity reported by Helmsman.
• nodes (required)

Edge telemetry ingress is control-plane managed. Enrolled edge nodes receive remote-write URL and auth material from Foghorn during preregistration and subsequent ConfigSeed updates.

Node fields:

• name (required)
• ssh (required unless provided by hosts_file) - user@host
• subdomain (optional)
• region (optional)
• apply_tune (optional)
• register_qm (optional)
• mode (optional) - Per-node override, docker or native
• capabilities (optional) - Per-node override for enabled capabilities
• bandwidth_mbps (optional) - Per-node bandwidth override
• max_transcodes (optional) - Per-node transcode limit override
• storage_capacity_bytes (optional) - Per-node storage capacity override
• labels (optional)

---

Validation Notes

The CLI validates:

• Required fields (version, type, hosts)
• Host references used by Postgres, ClickHouse, Kafka, services, interfaces, and observability entries
• If clusters is present, each entry must have name and type
• If a service sets cluster, it must reference a defined cluster key
• Cluster IDs (map keys) must be non-empty strings

Unknown keys are rejected by the parser. Keep manifests minimal and aligned with the schema above so typos and removed fields fail loudly.

Remote Manifests (GitHub App)

Instead of managing manifests locally, you can store them in a private GitHub repository and fetch them at provision time using GitHub App credentials.

Setup

Credentials are written to the canonical CLI config ($XDG_CONFIG_HOME/frameworks/config.yaml by default; run frameworks config path --kind config to confirm):

frameworks config set github.app-id 12345
frameworks config set github.installation-id 67890
frameworks config set github.private-key /path/to/app-private-key.pem
frameworks config set github.repo your-org/infra-repo
frameworks config set github.ref main  # optional, defaults to main

For CI or one-shot automation, the same values can be passed via environment variables: FRAMEWORKS_GITHUB_APP_ID, FRAMEWORKS_GITHUB_INSTALLATION_ID, FRAMEWORKS_GITHUB_PRIVATE_KEY, FRAMEWORKS_GITHUB_REPO, FRAMEWORKS_GITHUB_REF. Env values beat the stored config; per-invocation flags beat both.

Usage

# Provision using the remote manifest (cluster name required)
frameworks cluster provision --github-repo your-org/infra-repo --cluster production

# Override credentials via flags
frameworks cluster provision \
  --github-repo your-org/infra-repo \
  --cluster production \
  --github-app-id 12345 \
  --github-installation-id 67890 \
  --github-private-key /path/to/key.pem

The CLI fetches clusters/<cluster>/cluster.yaml from the repo, along with all referenced env_files, per-service env_file paths, and hosts_file. The standard provision pipeline then runs with the fetched data inside a tempdir that’s cleaned up when the command exits.

Configuration Keys

Key	Description
github.app-id	GitHub App ID
github.installation-id	GitHub App Installation ID
github.private-key	Path to PEM-encoded private key
github.repo	Repository in owner/repo format
github.ref	Git ref (branch/tag), defaults to main