Multi-Cluster Operations

What Is a Cluster?

A media cluster is a set of edge nodes plus the cluster-scoped services assigned to it, strictly the media plane. Every media cluster is served by a platform-run Foghorn control cell (one Foghorn instance or HA set). One cell can serve several virtual media clusters, including customer private clusters, so the Foghorn process cluster is never a substitute for an edge node’s cluster identity. Commodore, Quartermaster, and Purser remain central. Today the unsharded ClickHouse deployment also provides one logical Periscope metering source. When ClickHouse is split by wider region, each logical regional deployment runs local Periscope Ingest, Query, and Metering services; the regional metering workers publish to the same central Purser through Kafka. These services scale by logical data source, not one copy per media cluster.

While control and data plane services share infrastructure, they are registered under separate cluster IDs for operational visibility. The control plane cluster appears in the inventory and dashboard but does not participate in Foghorn federation.

Cross-cluster coordination is handled exclusively by the FoghornFederation gRPC protocol between Foghorn instances.

---

Deployment Models

Model	Description
Shared	Free shared capacity is currently unmetered; premium shared tiers are seeded for later enforcement. All tenants share the same cluster, edges, and services.
Dedicated	Enterprise tier. Isolated provider-operated capacity and routing for one tenant. To run Foghorn yourself, self-host the entire stack.
Hybrid	Tenant runs self-hosted Caddy/MistServer/Helmsman edges that connect to provider Foghorn and can fall back to managed capacity; the control/data plane remains provider-operated.
Open Marketplace	Operators publish listed clusters with access controls, invites, subscription requests, and pricing metadata. A vetted public marketplace program for third-party capacity is on the roadmap.

---

Cluster Lifecycle

Creating a Cluster

Via CLI provisioning (recommended for full deployments):

# Generate and validate GitOps-owned WireGuard identity from a local checkout.
frameworks mesh wg generate --manifest gitops/clusters/production/cluster.yaml
frameworks mesh wg check --gitops-dir gitops --cluster production

# Provisioning is read-only against GitOps.
frameworks cluster provision --gitops-dir gitops --cluster production

When the manifest has a clusters section, the CLI creates all clusters, registers nodes to the correct cluster based on role-based resolution, and generates per-cluster enrollment tokens. The manifest must already contain complete WireGuard identity for every managed Privateer host; cluster provision validates that state but does not write cluster.yaml or hosts.enc.yaml. See Cluster Manifest: clusters for the manifest format.

Via admin command (ad-hoc cluster creation):

frameworks admin clusters create \
  --cluster-id eu-prod \
  --cluster-name "EU Production" \
  --cluster-type edge \
  --base-url eu.frameworks.network \
  --foghorn-count 2 \
  --deployment-model shared

This:

1. Creates the cluster record in Quartermaster
2. Claims N Foghorn instances from the pool (--foghorn-count)

Navigator-managed DNS and certificates are driven separately from node/service inventory and certificate status, not directly by admin clusters create.

Service Pool

Cluster-scoped media services such as Foghorn, Chandler, and Livepeer gateway are assigned from shared pools to logical media clusters.

# View pool status (shows per-cluster grouping)
frameworks admin service-pool status --service-type foghorn

# Assign idle instances to a cluster
frameworks admin service-pool assign --service-type foghorn --cluster-id <CLUSTER_ID> --count 2

# Release an assigned instance back to the pool
frameworks admin service-pool add --service-type livepeer-gateway --instance-id <UUID>

# Drain an instance from its cluster (graceful)
frameworks admin service-pool drain --service-type chandler --instance-id <UUID>

Cluster Health

frameworks admin clusters cert-status --cluster-id <ID>

Shows cluster health including Foghorn HA status, edge count, and wildcard certificate readiness.

---

Federation

Foghorn instances discover peers via Quartermaster’s ListPeers RPC. The peer manager maintains long-lived PeerChannel connections for known peers, refreshes them periodically, and also accepts demand-driven peer hints from stream validation so new routing relationships do not wait for the next refresh.

How It Works

1. PeerChannel: Bidirectional gRPC streams between Foghorn peers exchange edge telemetry, stream advertisements, cluster summaries, heartbeats, and replication/artifact events on their own intervals
2. QueryStream: When a viewer’s cluster doesn’t have the stream, Foghorn asks the origin cluster for edge candidates
3. Origin-Pull: Foghorn arranges a DTSC pull from the remote origin into a local edge, and the viewer is served from there. Subsequent viewers are served locally. Viewers are never redirected to another cluster

Source advertisements carry both Mist’s native buffer health state and a separate media-playability signal. FULL, DRY, and RECOVER can all serve viewers; DRY means the booted buffer has a diagnostic issue, not that it is empty. This distinction matters for valid video-only streams, which Mist may flag with HLSnoaudio! while continuing to serve video. Federation refuses only sources whose media is not playable.

Leader-Only Peering

Each cluster elects one Foghorn instance (via Redis SET NX, 15s TTL) to run PeerChannel connections. This prevents duplicate peer traffic in HA deployments.

For architecture details, see docs/architecture/federation.md and docs/architecture/stream-replication-topology.md.

Artifact Command Routing

When a tenant deletes a clip, stops a DVR, or removes a VOD asset, Commodore routes the command to the cluster that owns the artifact — not necessarily the tenant’s primary cluster. This uses a push+forward model:

1. Push: Commodore tracks which cluster ingested each stream (active_ingest_cluster_id). Artifact operations read origin_cluster_id from the business registry and route directly to that cluster’s Foghorn.

2. Forward (safety net): If the command arrives at a Foghorn that doesn’t own the artifact (stale routing data, race condition), Foghorn forwards it to federation peers via ForwardArtifactCommand. The first peer that owns the artifact handles it. Each peer receives a fresh bounded attempt, so one unavailable peer does not consume the next peer’s timeout window.

Tenant suspension (TerminateTenantStreams, InvalidateTenantCache) fans out to all clusters the tenant has access to, ensuring streams are terminated and caches invalidated everywhere.

---

Marketplace

The marketplace allows third-party operators to publish clusters that other tenants can subscribe to.

Cluster Visibility

Clusters can be configured with different visibility levels:

Visibility	Who Can See	Access
private	Only the owner tenant	Direct access or invite token for another tenant
unlisted	Anyone with the direct link	Via the billing checkout flow
public	All tenants	Via subscription request

Subscription Lifecycle

Tenant discovers cluster (marketplace UI or invite link)
  → Public marketplace: SubscribeToCluster routes the decision through Purser
  → Monthly: verified checkout webhook materializes active marketplace access
  → Custom/approval listing: Purser materializes an inactive PENDING_APPROVAL row
  → Cluster owner approves/rejects (ApproveClusterSubscription / RejectClusterSubscription)
  → Private invite: RequestClusterSubscription validates the invite in Quartermaster
  → Unlisted direct link: Purser validates pricing and materializes marketplace access
  → Tenant can now route streams through the cluster

Platform-official capacity is provisioned from the tenant billing tier, and a tenant’s own private edge cluster uses owner authority. Neither enters marketplace checkout. v0.3 fails closed if a listing combines monthly checkout with owner approval; that combination needs a future two-proof lifecycle.

Invitation Flow

Cluster operators can invite specific tenants:

# Creates an invite token with optional expiry
frameworks admin clusters invites create \
  --cluster-id <ID> \
  --owner-tenant-id <OWNER_TENANT_UUID> \
  --invited-tenant-id <INVITED_TENANT_UUID> \
  --expires-in-days 7

The invited tenant sees the cluster in their dashboard and can accept with one click.

---

Dashboard Pages

The webapp provides dedicated infrastructure pages for multi-cluster operations:

Page	Route	Purpose
Infrastructure Overview	/infrastructure	Tenant info, platform performance (CPU/memory/nodes), service health summary, clickable cluster cards
Cluster Detail	/infrastructure/[clusterId]	Per-cluster metrics, node cards with live health, service instances, health checks
Node Detail	/nodes/[id]	Per-node CPU/memory/disk/streams, 5-minute performance history, service instances
Clusters	/infrastructure/clusters	Merged view — “My Clusters” tab (subscriptions, invitations, approvals, private cluster creation) and “Marketplace” tab (browse and connect to clusters)
Federation	/infrastructure/federation	Federation overview — topology map with peering/traffic relationships, traffic matrix, event type breakdown, recent federation events
Audience Analytics	/analytics/audience	Routing map with cross-cluster flow visualization (amber lines for cross-cluster routes), routing events tagged with local/cross-cluster badges

---

Edge Enrollment

Edge nodes enroll into a specific cluster using bootstrap tokens.

Token Creation

Via Dashboard (tenant owner):

1. Go to Infrastructure -> Clusters
2. Click Create Private Cluster or select an existing cluster
3. Copy the bootstrap token

Via CLI (admin):

frameworks admin bootstrap-tokens create \
  --kind edge_node \
  --tenant-id <TENANT_UUID> \
  --cluster-id <CLUSTER_ID> \
  --name "edge-node-1" \
  --ttl 24h

Provisioning with Token

frameworks edge provision \
  --ssh [email protected] \
  --enrollment-token enroll_xxx \
  --email [email protected]

The CLI sends the token to Bridge via the public bootstrapEdge mutation. Bridge validates the token via Quartermaster, finds the cluster’s assigned Foghorn, and proxies PreRegisterEdge. The response carries:

• Assigned node ID and edge domain ({node_label}.{cluster_slug}.{base}, where node_label is the node ID with a single edge- prefix)
• Pool domain for the cluster
• Foghorn gRPC address the edge will use at runtime for Helmsman’s control stream
• Internal CA bundle for initial gRPC trust bootstrap

The operator never needs to know cluster topology — the bootstrap token IS the cluster selector. After bootstrap, the edge talks directly to its assigned Foghorn over the public internet; it never joins the WireGuard mesh.

If you need to dial a specific Foghorn directly (for debugging), pass --foghorn-addr and the CLI bypasses Bridge for that one call.

---

DNS

Each cluster gets its own set of DNS records under {cluster_slug}.{base_domain}:

Record	Purpose
edge-ingest.{slug}.{base}	RTMP/E-RTMP/SRT/WHIP ingest
edge.{slug}.{base}	Edge pool (any edge in cluster)
foghorn.{slug}.{base}	Viewer routing / playback resolution
livepeer.{slug}.{base}	Livepeer gateway / transcoding endpoint
{node_label}.{slug}.{base}	Per-edge A records for direct addressing (node_label is the node ID with a single edge- prefix)

Navigator manages these records automatically. Wildcard TLS certificates (*.{slug}.{base}) are issued via ACME DNS-01 and distributed to edges via ConfigSeed.

See DNS and Cluster Routing for details.

---

Billing Attribution

Stream/viewer/artifact analytics carry cluster_id (the cluster that performed the work) and origin_cluster_id (where the stream was ingested, retained for audit). Delivery is always billed to the serving cluster; processing is billed to the executing cluster; storage is billed to its placement/provider cluster. Only truly tenant-wide meters such as API usage attach to the tenant’s primary cluster.

• Periscope Metering generates per-cluster, dimensioned usage reports
• Purser stores these as per-cluster usage records
• Settlement queries can identify cross-cluster traffic for infrastructure cost attribution

Inter-cluster DTSC bandwidth is infrastructure cost, not a tenant-facing billing item.

Cluster Pricing

Each cluster has a pricing model configured in purser.cluster_pricing:

Pricing Model	Description
free_unmetered	No metering (community tier)
metered	Pay-as-you-go resource billing
monthly	Fixed monthly subscription
tier_inherit	Inherit pricing from the tenant’s billing tier
custom	Operator-defined pricing

See docs/architecture/cross-cluster-billing.md and docs/architecture/billing-tier-provisioning.md for the full attribution and provisioning model.

---

Monitoring

Federation Health

Monitor peer connectivity and replication state:

• PeerChannel status: Each Foghorn logs peer connections and heartbeat latency
• Stream advertisements: Exchanged every 5 seconds via PeerChannel; stale advertisements indicate peer issues
• Active replications: In-flight cross-cluster DTSC pulls tracked in Redis (5-min TTL)
• Federation events: All cross-cluster operations (peering, replication, artifact access) emit geo-enriched events to ClickHouse (federation_events table). View them on the Federation dashboard page (/infrastructure/federation)
• Self-geo: each Foghorn resolves its own coordinates from infrastructure_nodes.external_ip at bootstrap and stamps them on the events it emits. Peers do not exchange coordinates over the PeerChannel, so a remote position is left unset rather than guessed; the topology map sources remote coordinates from infrastructure records instead

Per-Cluster Metrics

ClickHouse tables partition by cluster_id, enabling per-cluster dashboards for:

• Viewer hours and egress
• Stream health and QoE
• Edge node utilization
• Cross-cluster replication events
• Federation event geo data (local/remote lat/lon for topology visualization)