Architecture Overview

Overview

FrameWorks uses the same logical planes across environments, but the physical placement varies by deployment:

Local development: Docker Compose brings up the full stack on one host.
Provider-operated deployments: The FrameWorks CLI (frameworks cluster provision) drives Ansible roles against the hosts named in the cluster manifest. Release manifests live in a GitOps repository; the CLI reads them to pin versions and reconcile state. See CLI Reference and Ansible Provisioning.
Customer-operated edge capacity: The supported hybrid path runs MistServer and Helmsman on customer infrastructure while FrameWorks operates the control plane. A customer-operated control plane is a separate distribution model, not a deeper hybrid tier.
Coordination: Quartermaster remains the system of record for tenants, clusters, and service discovery. Management paths query it directly; Commodore compiles the media-serving subset with Purser and catalog policy into signed, versioned Foghorn-local authority so an established media cell does not require a live control-plane query for every request.

Multi-Plane Architecture

FrameWorks uses a multi-plane distributed architecture designed for scale, separating tenant-facing control logic from high-volume media, event data, and network automation. Plane names describe runtime responsibility; deployment tiers describe placement.

Control / API Plane: GraphQL aggregation, auth, stream management, tenant routing, and billing APIs.
Media Plane: Media ingest, processing, playback routing, edge orchestration, and media asset delivery.
Event & Analytics Plane: Event ingress, Kafka fan-out, real-time fan-out, and analytical storage/querying.
Network & Trust Plane: Public DNS, certificates, service identity, mesh networking, and ingress.
Support & Experience Services: Support workflows, AI consultant workflows, forms, newsletters, and stream interaction services.
Interfaces: Web apps, marketing sites, documentation, CLI, SDKs, and player/studio packages.
Observability & Operations: Metrics, dashboards, alerting, BI, and incident workflows.

flowchart TB
    I[Dashboard · SDKs · CLI · MCP] --> B[Bridge<br/>GraphQL and MCP]

    subgraph C[Control and API plane]
        COM[Commodore<br/>streams and auth]
        QM[Quartermaster<br/>tenants and topology]
        PUR[Purser<br/>billing]
        NAV[Navigator<br/>DNS and trust]
    end

    subgraph R[Regional media and event plane]
        F[Foghorn<br/>routing and orchestration]
        D[Decklog → regional Kafka]
        P[Periscope<br/>analytics]
        S[Signalman<br/>realtime events]
    end

    subgraph E[Edge capacity]
        H[Helmsman]
        M[MistServer]
        LG[Livepeer Gateway]
    end

    B --> COM
    B --> QM
    B --> PUR
    B --> P
    COM --> F
    QM --> F
    NAV --> E
    F <--> H
    H --> M
    M --> LG
    F --> D
    D --> P
    D --> S

Service Stack

The FrameWorks platform is composed of specialized services, each responsible for a distinct domain. These services communicate primarily via gRPC for internal traffic and expose a unified GraphQL API for external clients.

Core Services

Service	Port(s)	Tier	Plane	Purpose
Bridge	18000	Regional	Control / API	GraphQL API Gateway and MCP hub
Commodore	18001 (HTTP), 19001 (gRPC)	Central	Control / API	Auth, streams, API tokens, media registry, and business workflows
Quartermaster	18002 (HTTP), 19002 (gRPC)	Central	Control / API	Tenants, clusters, nodes, service discovery, and bootstrap state
Purser	18003 (HTTP), 19003 (gRPC)	Central	Control / API	Billing, usage, invoices, payments, and x402 state
Periscope Query	18004 (HTTP), 19004 (gRPC)	Central	Event & Analytics	Analytics and reporting API
Periscope Ingest	18005	Regional	Event & Analytics	Kafka event processing and ClickHouse writes
Periscope Metering	18021	Regional	Event & Analytics	Local ClickHouse aggregation and durable reports to Purser
Decklog	18006 (gRPC)	Regional	Event & Analytics	gRPC event ingress → Kafka
Decklog (metrics)	18026	Regional	Observability & Operations	Decklog Prometheus metrics
Signalman	18009 (WS), 19005 (gRPC)	Regional	Event & Analytics	Real-time event fan-out and WebSocket hub
Foghorn	18008	Regional	Media	Load balancer and media pipeline orchestrator
Foghorn (internal HTTP)	18027	Regional	Media	Internal management and authenticated mutation API
Foghorn (internal gRPC)	18019 (gRPC)	Regional	Media	Internal control API and HA relay over the internal CA
Foghorn (external gRPC)	18029 (gRPC)	Regional	Media	Edge bootstrap, Helmsman control, and FoghornFederation over ACME
Helmsman	18007 (HTTP)	Edge	Media	MistServer sidecar
Helmsman (management)	18017	Edge	Media	Loopback management mutations
Chandler	18020 (HTTP)	Regional	Media	Cluster-scoped static media asset server
Navigator	18010 (HTTP), 18011 (gRPC)	Central	Network & Trust	Public DNS, ACME certificates, and internal CA material
Privateer	18012 (HTTP)	Central/Regional	Network & Trust	WireGuard mesh agent and local `.internal` DNS
Deckhand	18015 (HTTP), 19006 (gRPC)	Central	Support & Experience Services	Support messaging API (Chatwoot adapter)
Skipper	18018 (HTTP), 19007 (gRPC)	Central	Support & Experience Services	AI video consultant API
Forms	18032	Central	Support & Experience Services	Contact form and newsletter handling

Infrastructure Components

Component	Role	Plane	Port(s)	Deploy Location
MistServer	Media processing, ingest, and edge delivery	Media	4242, 8080, 1935, 5554, 4200, 8889/udp, 18203/udp	Edge
Livepeer Gateway	Transcoding gateway and compute-marketplace integration	Media	8935	Edge (co-located with MistServer)
Livepeer Signer	Keystore-backed ETH transaction signer; keyless ticket-batch signing is not yet wired to gateways	Control / API	18016	Central (not included in local Compose)
PostgreSQL-compatible DB (Postgres/Yugabyte)	Service-owned state and configuration database substrate	Infrastructure	Operator-defined	Central
ClickHouse	Time-series analytics database	Event & Analytics	8123 (HTTP), 9000 (Native)	Central
Kafka	Regional event backbone; canonical topics replicate to the aggregator region with MirrorMaker 2	Event & Analytics	9092 (Int), 29092 (Ext)	One KRaft cluster per region
Foghorn Redis	Foghorn HA state sync	Media	6379	Regional
Nginx	Reverse proxy and local development routing	Network & Trust	18090	Central
Caddy	Edge TLS termination and reverse proxy	Network & Trust	80, 443, 18090	Edge
VictoriaMetrics + vmagent	Metrics collection and remote-write fan-in	Observability & Operations	Operator-defined	Central + every backend and edge host
Grafana	Metrics visualization	Observability & Operations	Operator-defined	Central
Metabase	BI analytics	Observability & Operations	Operator-defined	Central
Listmonk	Newsletter and mailing list manager	Support & Experience Services	9000 (service), 9001 (docker-compose host)	Central

Livepeer integration: The Livepeer Gateway is provisioned via the CLI and discovered via Quartermaster service discovery. MistServer sends segments to the gateway automatically when Livepeer transcoding is configured for a stream. Ingest auth stays in the media plane: the gateway calls Foghorn’s internal /webhooks/livepeer/auth webhook, and wallet/top-up automation reads wallet_address from Quartermaster service-instance metadata.

Interfaces

Component	Role	Path	Port	Deploy Location
Chartroom / Web Console	SvelteKit user dashboard	`website_application`	18030	Regional
Foredeck / Marketing	Sales / Marketing website	`website_marketing`	18031	Regional
Logbook / Documentation	Astro Starlight docs site	`website_docs`	18033	Central
NPM Player	Embeddable multi-protocol player	`npm_player`	N/A (npm package)	User websites
NPM Studio	Embeddable browser ingest studio	`npm_studio`	N/A (npm package)	User websites
Android App	Native mobile client (in development)	`app_android`	N/A	Roadmap

CLI Tool

Component	Role	Path	Install
FrameWorks CLI	Cluster management and provisioning	`cli/`	`go install` or binary

Deployment Tiers

Central: Commodore, Quartermaster, Purser, Navigator, Periscope Query, Deckhand, Skipper, Forms, PostgreSQL-compatible DB (YugabyteDB recommended), ClickHouse, Listmonk, Chatwoot + Redis.
Regional: Bridge, Foghorn, Decklog, Periscope Ingest, Periscope Metering, Signalman, Chandler, Kafka, Web Console, Marketing Site. Today the unsharded ClickHouse deployment is central, so there is one metering source there; when ClickHouse becomes regional, each logical deployment gets a local worker with a unique source ID.
Central/Regional: Privateer (WireGuard mesh agent runs on all backend infrastructure nodes).
Edge: MistServer, Helmsman, Livepeer Gateway.

Multi-Cluster Topology

FrameWorks supports multiple media clusters. A Foghorn control cell may serve more than one virtual media cluster, so the Foghorn process cluster is never used as a substitute for an authenticated edge node’s cluster identity. Commodore, Quartermaster, and Purser remain the central systems of record, but they distribute signed, versioned authority into each Foghorn cell. Already authorized media continues from that durable local authority through its signed validity bound during a central outage. Event collection is regional, and Purser remains the central financial authority.

Cross-cluster coordination is handled exclusively by the FoghornFederation gRPC protocol between Foghorn instances:

Peer discovery: Quartermaster maintains a registry of all clusters. Foghorn queries ListPeers to find peers and opens PeerChannel bidirectional streams for real-time edge telemetry exchange.
Cross-cluster viewer routing: When a viewer’s cluster doesn’t have a stream, Foghorn uses QueryStream to find it on a peer, then either redirects the viewer or arranges a DTSC origin-pull to serve locally.
Cross-cluster artifacts: clips, VODs, and DVR chapters on remote clusters are all accessed via PrepareArtifact. Origin returns a presigned S3 URL when the artifact is synced, or a short-lived peer-relay URL + opaque capability grant pointing at the specific origin node when the canonical file is on disk but S3 sync is still pending (the origin Foghorn authorizes each pull online; no signing key on the edge) — viewers don’t wait on sync. DVR chapter listing flows through the GraphQL chapter API (Commodore → origin Foghorn for the chapter row metadata); each finalized chapter is a regular VOD-shaped artifact with its own Commodore-minted, shareable playbackId, while the snapshotted parent-DVR policy still controls authorization. Chapter playback follows the same federation rules as any other VOD.
Leader-only peering: Each cluster elects one Foghorn instance (via Redis SET NX) to run PeerChannel connections, avoiding duplicate peer traffic.

Quartermaster is the system of record for cluster identity, ownership, and grants. Commodore compiles those facts into signed cell-local media authority; management routing still resolves dynamically and no service hardcodes Foghorn addresses.

Deployment models:

Model	Description
Shared	Free shared capacity is currently unmetered; premium shared tiers are seeded for later enforcement. All tenants share the same cluster, edges, and services.
Dedicated	Enterprise tier. Isolated provider-operated capacity and routing for one tenant. Customer-operated Foghorn is not part of this model.
Hybrid	Tenant runs self-hosted MistServer/Helmsman edges that connect to provider Foghorn and can fall back to managed capacity; the control/data plane remains provider-operated.
Open Marketplace	Operators publish listed clusters with access controls, invites, subscription requests, and pricing metadata. Tenants browse listings and subscribe through the dashboard or API.

Data Flow Architecture

Understanding how data moves through the system is critical for operations and troubleshooting. The diagrams below illustrate the key pipelines for analytics, control, and media.

Event-Driven Analytics Pipeline

graph LR
    MS[MistServer] --> H[Helmsman]
    H <-->|gRPC| F[Foghorn]
    F --> D[Decklog]
    C[Commodore<br/>signed authority] -->|durable delivery| F
    D --> K[Kafka]
    K --> PI[Periscope-Ingest]
    K --> S[Signalman]
    PI --> CH[(ClickHouse)]
    S --> FE[Frontend<br/>WebSocket]

Note: Helmsman maintains a persistent bidirectional gRPC stream with Foghorn:

Helmsman → Foghorn: MistServer triggers, heartbeats, clip/DVR progress updates
Foghorn → Helmsman: Clip pull requests, DVR start/stop commands, config seeds

Helmsman normally samples Mist stream state every 10 seconds and also maintains one authenticated MistController WebSocket per configured node. The socket detects status/input/output changes and requests an immediate, filtered refresh through the same lifecycle pipeline. It never replaces polling: viewer-only changes are ignored, and offline synthesis, source-lease reconciliation, health sampling, and boot readiness remain owned by the full poll. Ping/pong liveness closes and reconnects half-open controller sockets, and node replacement cancels any in-flight targeted refresh rather than waiting for the Mist HTTP timeout.

Node replacement and shutdown are generation boundaries. Replacement cancels the old generation, publishes the new node/client pair without waiting on an active transport write, and joins retired authoritative/accelerator HTTP clients, edge-API requests, queued full sweeps, and WebSocket work asynchronously. Shutdown joins all current and retired work before returning. Changing nodes clears node-scoped vanish, admission, and stream observation state; re-registering the same node preserves pending offline/reconciliation evidence while replacing the runtime generation. Observation IDs remain monotonic across both paths. Node metrics use a joined, deadline-bounded, one-slot-coalesced forwarding worker, so a slow control send cannot fill the monitor update queue. Every control write, including context-free heartbeat, relay, and outbox traffic, gets a 15-second transport watchdog beginning only after it owns the serialized send lane. The ordinary non-blocking send budget remains five seconds, so transient flow-control backpressure can outlive one caller without immediately recycling the shared stream. Caller cancellation controls lane acquisition. After ownership, explicit cancellation is not transport-failure evidence; a caller deadline may stop waiting while the serialized write retains the lane until it completes or reaches its independent transport watchdog. Non-blocking Mist webhooks are detached from request disconnects. If an active production gRPC write exceeds that watchdog, Helmsman cancels the owning control stream and reconnects. Node-runtime cancellation also reaches lifecycle sends, bounding replacement and shutdown when Foghorn is connected but not reading. Once shutdown marks the monitor stopped under its node-lifecycle lock, later desired-state add/remove callbacks are ignored and cannot register retirement work while shutdown waits.

The controller’s initial stream dump has a fixed deadline relative to connection time; continuous frames cannot keep bootstrap open indefinitely. Expiry schedules one authoritative full sweep; first-seen dump-tail frames remain snapshot data until it succeeds, while later changes to already observed streams remain actionable. Monotonic observation ordering is held through lifecycle-trigger send. A targeted read newer than an in-flight poll is merged into one stable presence snapshot for source leases, vanish detection, admitted runtime reconciliation, and freshness pruning. It suppresses absence only for that stream; the next authoritative poll confirms or removes it, while boot reconciliation still completes. Failed targeted refreshes fall back without reserving freshness or consuming their throttle slot. Failed full sweeps retry with bounded jittered exponential backoff; CAS contention waits for the already-running poll, whose successful inventory also completes bootstrap. Stream-end frames rely on the normal vanish diff instead of requesting an all-stream replay. If lifecycle-observation or WebSocket-dedup cardinality exceeds its cap, targeted rows remain disabled until a bounded authoritative poll clears the condition. Dedup overflow requests one accelerated sweep rather than one per subsequent frame; the normal poll and all reconciliation work continue throughout.

Stream-frame dedup survives reconnects, so a repeated initial dump does not refresh every unchanged stream. Authoritative inventories prune departed names from the reconnect dedup map. Successful fallback sweeps are paced to at most one every two seconds, and an authoritative poll advances freshness only to its own observation rather than discarding targeted refreshes that began while it was running. Targeted requests keep request-start ordering but register only after a successful Mist response, and inventory pruning preserves nudges newer than the poll snapshot. This prevents slow targeted responses from reviving removed streams or an in-flight poll from deleting a newer queued change.

Source reconciliation requires 10 seconds of continuous absence. Runtime-admission cleanup requires the existing 30-second minimum age plus 20 seconds of continuous absence. These elapsed-time windows are independent of how many accelerator sweeps run.

Monitor helmsman_stream_ws_connections, helmsman_stream_ws_reconnects_total, helmsman_stream_ws_nudges_total, and helmsman_stream_ws_refreshes_total together with normal control-trigger delivery. Closing the socket must only increase reconnect activity; lifecycle polling must continue unchanged.

Foghorn routes analytics to Decklog and performs media admission/routing from its signed local authority plus local runtime state. Connected control-plane lookups remain for new mutations, bootstrap, mixed-version shadowing, and explicit online-only operations such as new x402 settlement.

Analytics keeps physical placement separate from control-plane and content origin. cluster_id identifies the authenticated cluster of the node that served or performed work; origin_cluster_id identifies where the content originated; control_cell_id identifies the Foghorn/media-authority cell that made or observed the decision. One control cell may manage multiple media clusters, so operators should not treat these fields as aliases.

The API exposes three deliberately different views: anonymous callers receive official cluster-level network status; tenants receive viewer, routing, processing, storage, and federation facts for content they own; cluster owners receive private node/service topology and operational metrics for clusters they own. Marketplace/subscription access adds aggregate cluster status but never another tenant’s streams, sessions, URLs, or raw node telemetry.

Cluster owners can use the analytics.infra.clusterWorkload GraphQL field for the redacted operational overlay: viewer, ingest, processing, storage, and federation work grouped by owned cluster, node, and work kind. It includes counts, active work, bytes, media seconds, and errors, but deliberately omits tenant, content, stream, session, URL, and client identifiers. Each row says whether it represents activity in the requested window or a freshness-bounded current observation. Current storage rows also retain their hot/cold scope and observation time; resident bytes are never added to lifecycle-event byte flow.

Control / API Communications

graph LR
    FE[Frontend] --> B[Bridge]
    B -->|gRPC| COM[Commodore]
    B -->|gRPC| PUR[Purser<br/>billing]
    B -->|gRPC| QM[Quartermaster<br/>tenant info]
    B -->|gRPC| PER[Periscope Query<br/>analytics]

    H[Helmsman] <-->|gRPC| F[Foghorn]
    F -->|gRPC| COM2[Commodore<br/>validation]
    F -->|gRPC| QM2[Quartermaster<br/>enrollment]

Media Pipeline

graph LR
    IN[RTMP/E-RTMP/SRT/WHIP] --> MS[MistServer]
    MS <--> LP[Livepeer<br/>transcoding/AI]
    MS --> OUT[Multi-protocol outputs<br/>HLS / LL-HLS / DASH / WebRTC]
    OUT --> V[Viewers]
    MS --> H[Helmsman]
    H --> F[Foghorn]

Ingest Endpoint Resolution

Node-specific ingest recommendations include only protocols currently advertised by that node. RTMP/SRT URLs retain its listener ports; changing a Foghorn-wide default port does not override them. A healthy node with no ingest protocol report is not a valid recommendation. WHIP redirects select a WHIP-capable node before limiting the candidate list. The ingest gRPC request can also require WHIP, RTMP or SRT. Unsupported enum values are rejected, and no available node for the requested protocol means unavailable—not a recommendation for a different protocol. Omitting the protocol lists any advertised ingest protocols.

Listener reports expire after 30 seconds independently of heartbeats. Helmsman sends an empty snapshot when Mist stops advertising listeners, withdrawing their URLs. For WHIP, Foghorn uses the reported HTTP signalling endpoint, not WebRTC’s UDP listener port, and retains the public proxy path. Helmsman reconciles the managed HTTP public address into the scalar form needed by Mist’s listener report while preserving the node’s port and other listener options. Repository hydration does not renew listener evidence, and changing a node’s public address without a new report invalidates its old evidence.

Generic DNS ingest names remain manual entry points, not proof of a policy-selected node. These endpoint checks do not by themselves activate the federation placement policy engine.

The placement editor’s authorized cluster, operator and region options are a management catalogue, not a live routing result. “Connected” does not guarantee available capacity or permission for both ingest and delivery, and selecting marketplace capacity never subscribes to it. A changed catalogue requires a fresh search; the editor retains draft selections. Saving requested rules and observing them enforced at media destinations remain separate steps: a rule becomes effective once every cell holding the content acknowledges it.

Viewer Routing

Foghorn selects optimal edge nodes for viewers based on geography, load, and stream availability.

Routing Paths:

graph LR
    P[Player] -->|GraphQL| B[Bridge]
    B -->|gRPC| C[Commodore]
    C -->|gRPC| F[Foghorn]
    F -->|node list| C
    C --> B --> P
    P -->|connect| E[Edge Node]

Path	Flow	Use Case
GraphQL (primary)	Player → Bridge → Commodore → Foghorn	FrameWorks Player, SDK integrations
HTTP (direct)	Client → Foghorn `/play/{viewkey}`	CLI tools, direct URL access, 307 redirects

Foghorn HTTP Endpoints:

Endpoint	Response	Use Case
`GET /play/{viewkey}`	JSON with all endpoints	Direct integration
`GET /play/{viewkey}/hls/index.m3u8`	307 redirect to edge	Direct HLS playback
`GET /play/{viewkey}/webrtc`	307 redirect to edge	Direct WebRTC (WHEP)

There is no hostname-only viewer route. The MistServer compatibility surface is a node-bound source lookup on a signed per-node path, which answers a media node asking where to pull a stream from; it never returns a viewer’s playback destination. Any other path is a 404.

For URI patterns and player integration, see Playback & Embedding.

Client location behind a proxy: configure TRUSTED_PROXY_CIDRS on Bridge and Foghorn to name their immediate reverse proxies as narrowly as your topology permits. Forwarding headers from other peers are ignored. Keep GEOIP_MMDB_PATH configured on Foghorn for geographic lookup; public coordinate/country headers and lat/lon query parameters are not a substitute for GeoIP. Missing location means unknown geography. Authenticated node-to-node source routing has separate coordinate hints.

Call viewer resolution from the viewer’s browser/device where possible. A backend resolving on behalf of a viewer supplies its own network identity, not the viewer’s; setting arbitrary forwarding headers does not change that trust boundary.

Infrastructure Layers

graph TB
    subgraph "Infrastructure Layers"
        SOURCE["Release Source of Truth<br/>GitOps manifests"]
        CONFIG["Configuration Layer<br/>Ansible (driven by CLI)"]
        ORCH["Coordination Layer<br/>Quartermaster"]
    end

    subgraph "Components"
        VMS["Hosts (BYO or operator-provided)"]
        DNS["DNS (Cloudflare / Bunny via Navigator)"]
        CERTS["TLS Certificates"]
        MESH["WireGuard Mesh (Privateer)"]
        SERVICES["FrameWorks Services"]
        TENANTS["Tenants &amp; Clusters"]
    end

    SOURCE --> CONFIG
    CONFIG --> VMS
    CONFIG --> DNS
    CONFIG --> CERTS
    CONFIG --> MESH
    CONFIG --> SERVICES

    ORCH --> TENANTS
    ORCH --> SERVICES

Proxy & SSL Strategy

Traffic routing and TLS termination vary by deployment tier:

Development: Nginx (via Docker Compose) handles local routing with self-signed or no TLS.
Production (Edge): Caddy serves edge traffic. Bootstrap templates start with internal TLS and a 503 response; after enrollment, Foghorn pushes SiteConfig and TLS material to Helmsman, which rewrites and reloads Caddy for viewer traffic.
- Managed via CLI templates (cli/internal/templates/edge/Caddyfile.tmpl).
- Supports HTTP/2, HTTP/3, and WebSocket upgrades natively.
Production (Central/Regional): Navigator manages public DNS through Cloudflare for root/API/web/admin names and Bunny DNS for delegated media cluster zones.
- Root/API/web/admin names → Cloudflare A records or load balancers
- Multiple root/global nodes → Cloudflare load balancer with one proximity-steered pool per Quartermaster cluster
- Media cluster names → Bunny A-record sets under {cluster_slug}.{base_domain}
- Media cluster records use Bunny geolocation Smart Routing when Quartermaster has node coordinates
- Certificates issued via Let’s Encrypt DNS-01 through the provider authoritative for the name

Note: Cloudflare remains authoritative for root/API/web/admin/support names. Bunny DNS is authoritative for delegated media cluster zones. Traffic flows directly to FrameWorks services; provider-side proxies are not used for media traffic.

Mesh Networking

Internal communication between nodes uses a WireGuard mesh managed by the Privateer agent (api_mesh). This handles:

Secure inter-node communication
Internal DNS resolution
Backend infrastructure isolation

See Mesh Networking for bootstrap and runtime details.