DNS and Cluster Routing
Navigator reconciles public DNS and certificates from Quartermaster inventory. This page owns both root service DNS and media-cluster routing; avoid keeping a separate GeoDNS table elsewhere.
DNS Architecture
Section titled “DNS Architecture”FrameWorks uses two routing layers:
- DNS layer: Sends clients to the right public service or media cluster.
- Application layer: Foghorn applies health, capacity, stream availability, and cluster federation rules before returning or redirecting to a node.
Cloudflare remains authoritative for root/API/web/admin/support names. Media and edge traffic is cluster-scoped and may be delegated to Bunny DNS so edge growth does not consume root-domain load-balancing capacity.
How Navigator Works
Section titled “How Navigator Works”Navigator (api_dns) automatically manages DNS based on node inventory from Quartermaster:
- Quartermaster triggers a DNS sync (for example, when nodes are created/registered)
- Navigator fetches nodes for that service type and filters to those with external IPs
- Updates the authoritative provider:
- Root/API/web/admin names → Cloudflare A records or load balancers
- Multiple nodes on root/global hostnames → Cloudflare load balancer with one pool per origin node IP; proximity steering is enabled only when at least two pools carry coordinates
- Media cluster names → Bunny A-record sets with geolocation Smart Routing when coordinates are available
- Applies proxying rules (default set:
bridge,chartroom,chatwoot,foredeck,grafana,listmonk,logbook,metabase,steward; override viaNAVIGATOR_PROXY_SERVICES) - Handles ACME certificate issuance via DNS-01 challenges
Private staging deployments that use split DNS can set
NAVIGATOR_DNS_RECORDS_ENABLED=false. This suppresses public A records, aliases, Bunny records,
and Cloudflare load-balancer reconciliation while leaving certificate issuance and renewal active.
Tenant alias removals and label renames still complete in Navigator’s own state; provider records
are neither published nor cleared. Provider credentials are therefore still required for ACME DNS-01. Do not use the switch for a
public production cluster.
If no nodes are returned for a service type, Navigator skips changes to avoid removing working DNS during a transient outage.
Verifying DNS State
Section titled “Verifying DNS State”Use the CLI to verify public DNS matches your inventory:
frameworks dns doctor --domain yourdomain.comThis checks:
- Each subdomain resolves to expected IPs from inventory
- Reports mismatches between public DNS and Quartermaster
Pass --domain; default domains are for development and platform-owned checks.
Recommended DNS Structure
Section titled “Recommended DNS Structure”Public Endpoints
Section titled “Public Endpoints”| Subdomain | Purpose | Service |
|---|---|---|
example.com | Marketing site | Foredeck (Website) |
chartroom.example.com | Web console | Chartroom |
bridge.example.com | GraphQL API | Bridge |
logbook.example.com | Docs site | Logbook |
steward.example.com | Forms API | Steward |
listmonk.example.com | Newsletter/admin | Listmonk |
chatwoot.example.com | Support inbox | Chatwoot |
Cloudflare remains authoritative for root/API/web/admin/support names. Navigator can reconcile those records through Cloudflare, Media and edge traffic is cluster-scoped; Navigator delegates those zones to Bunny DNS when BUNNY_API_KEY is set, and otherwise publishes media cluster records through Cloudflare.
Internal Services (Private Network / Mesh)
Section titled “Internal Services (Private Network / Mesh)”These services communicate over the private network or WireGuard mesh (Privateer) and do not need public DNS. Edge nodes do not join the mesh.
- Commodore, Purser, Periscope, Quartermaster, Signalman
- PostgreSQL, ClickHouse, Kafka
Services reach each other through *_GRPC_ADDR variables on the 190xx gRPC ports (defaults such as commodore:19001 and purser:19003). To point a service at explicit private/mesh addresses:
COMMODORE_GRPC_ADDR=10.0.1.10:19001PURSER_GRPC_ADDR=10.0.1.11:19003Media Cluster DNS
Section titled “Media Cluster DNS”In multi-cluster deployments, each media cluster gets a delegated Bunny DNS zone at {cluster_slug}.{base_domain}:
| Pattern | Purpose |
|---|---|
{slug}.{base} | Default media-cluster entrypoint via Foghorn |
edge.{slug}.{base} | Edge pool domain for this cluster |
edge-ingest.{slug}.{base} | RTMP/E-RTMP/SRT/WHIP ingest for this cluster’s edges |
edge-egress.{slug}.{base} | Viewer egress for this cluster’s edges |
edge-storage.{slug}.{base} | Storage-capable edge pool for this cluster |
edge-processing.{slug}.{base} | Processing-capable edge pool for this cluster |
telemetry.{slug}.{base} | Cluster-scoped telemetry ingress |
chandler.{slug}.{base} | Cluster-scoped control helper |
foghorn.{slug}.{base} | Viewer routing / playback resolution |
livepeer.{slug}.{base} | Livepeer gateway for this cluster |
Navigator creates the Bunny zone, delegates {cluster_slug}.{base_domain} from Cloudflare with NS records, and reconciles each media service as a Bunny A-record set. The zone apex and foghorn.{slug}.{base} both point at the cluster’s assigned virtual Foghorn instance(s), so callers can use the cluster domain as the default playback/routing entrypoint. Records use Bunny geolocation Smart Routing when Quartermaster has node coordinates. Quartermaster remains the health source; Navigator removes unhealthy nodes from the desired record set instead of enabling provider-side health checks.
Media and edge names ALSO publish at global root entrypoints (see Tier-Based URL Surface below). Cluster-concrete records remain as operator-pin / debug surfaces.
Protocol Routing
Section titled “Protocol Routing”HTTP-based protocols can go through Foghorn because Foghorn can return JSON or issue redirects. Non-HTTP protocols connect directly to edge pools because they cannot follow HTTP redirects.
| Traffic class | Domain | Routing behavior |
|---|---|---|
| HLS, LL-HLS, DASH, WHEP/WebRTC, MP4, WebM/MKV, TS | {slug}.{base} or foghorn.{slug}.{base} | Foghorn resolves or redirects |
| Livepeer gateway API | livepeer.{slug}.{base} | Direct to gateway endpoint |
| RTMP/E-RTMP/SRT/WHIP ingest | edge-ingest.{slug}.{base} | Direct to ingest edge pool |
| RTMP/SRT/RTSP/DTSC playback when exposed | edge-egress.{slug}.{base} | Direct to egress edge pool |
DNS Health
Section titled “DNS Health”Quartermaster is the health source for desired DNS state. Navigator filters
unhealthy or addressless nodes before reconciling records; provider-side health
checks are an implementation detail of the authoritative provider integration.
Use frameworks dns doctor to compare public DNS with Quartermaster inventory.
The comparison is provider-aware: Bunny smart DNS may return any non-empty
geographically selected subset of the healthy authoritative members, while a
Cloudflare-proxied hostname is validated by successful resolution rather than
by comparing Cloudflare anycast addresses with origin IPs. An NXDOMAIN response
or a Bunny address outside Quartermaster’s healthy inventory still fails the
doctor check. Both per-cluster names and platform-official global Bunny
entrypoints are checked.
Wildcard TLS Certificates
Section titled “Wildcard TLS Certificates”Each cluster uses a wildcard certificate for its subdomain space:
*.{cluster_slug}.{base_domain}Certificates are issued via ACME DNS-01 challenges through the provider authoritative for the name. Cloudflare handles root/API/web/admin names; Bunny handles media cluster wildcards such as *.{cluster_slug}.{base_domain}.
Once issued, certificates are distributed to edge nodes via ConfigSeed — Foghorn pushes TLS bundles to Helmsman over the existing gRPC control stream. Edges do not need direct access to the ACME provider.
Tenant URL Surface
Section titled “Tenant URL Surface”FrameWorks exposes three URL forms per service. The webapp and SDKs pick the right one for each tenant; operators see all three in the dashboard.
1. Global root entrypoints — default UX for free / platform-official tenants
Each tenant-aliasable service has a single global record under
{base_domain}:
| Pattern | Purpose |
|---|---|
foghorn.{base} | Global HTTP playback (geo-routed to nearest platform Foghorn) |
edge-ingest.{base} | Global RTMP/SRT/WHIP ingest (geo-routed to nearest platform edge) |
edge-egress.{base} | Global viewer egress |
edge.{base} | Global edge pool |
edge-storage.{base}, edge-processing.{base} | Storage / processing variants |
chandler.{base} | Global control helper |
livepeer.{base} | Global Livepeer gateway |
These live as Bunny smart records and resolve to platform_official
cluster nodes only (third-party / marketplace / tenant-private edges
are excluded from global pools — see “Self-hosted trust boundary”
below). Certificates: one pool-assigned multi-SAN cert for
foghorn/chandler/livepeer; one platform-edge multi-SAN cert for the
five edge-* names. Both distributed via ConfigSeed only to
platform-operated nodes.
Livepeer gateways also have a stable physical identity at
livepeer-gateway.{node}.infra.{base}. That record remains present for a
provisioned running instance even while its process health is degraded; it is
the HTTPS path Quartermaster uses to probe the loopback-only gateway through
the node’s ingress. Pooled livepeer.* membership remains health- and
freshness-gated, so preserving physical identity does not route production
traffic to an unhealthy gateway.
2. Cluster-concrete URLs — operator pin / debug
Per-cluster names as documented in “Media Cluster DNS” above:
{service}.{cluster_slug}.{base}. Always available regardless of
tier. Used by self-hosted operators on free tier, by operators
wanting to pin traffic to a specific cluster, and by diagnostics.
3. Tenant aliases — entitled branded URLs that span subscribed clusters
Tenants with the explicit custom_subdomain_enabled billing entitlement get a per-tenant alias under the shared Bunny zone
cdn.{base}:
| Pattern | Purpose |
|---|---|
{tenant}.cdn.{base} | Tenant apex |
foghorn.{tenant}.cdn.{base} | Tenant playback (geo-balances across subscribed clusters of any kind) |
edge-ingest.{tenant}.cdn.{base} | Tenant ingest (RTMP/SRT/WHIP) |
edge-egress.{tenant}.cdn.{base} | Tenant egress |
edge-storage.{tenant}.cdn.{base}, edge-processing.{tenant}.cdn.{base} | Tenant storage / processing |
chandler.{tenant}.cdn.{base}, livepeer.{tenant}.cdn.{base} | Tenant control helpers |
A single per-tenant wildcard cert (*.{tenant}.cdn.{base} plus the
apex) covers every service. The cert is distributed via ConfigSeed
to every edge in clusters the tenant subscribes to — regardless of
cluster kind (platform-official, tenant-private, third-party
marketplace). Caddy SNI on each edge presents the right cert per
Host header. On Foghorn hosts (which serve
foghorn.{tenant}.cdn.{base} playback), Privateer materializes the
same tenant bundles under
/etc/frameworks/ingress/tls/tenant-alias/{tenant}/ and nginx
resolves the cert per handshake from $ssl_server_name, proxying to
the local Foghorn — one static catch-all vhost serves every alias
tenant. Bunny smart records under cdn.{base} resolve to the
union of edges across the tenant’s subscribed clusters.
This is the marketplace-spanning feature: one URL that geo-balances across whatever capacity the tenant has subscribed to while keeping service identity and certificate distribution tenant-scoped.
Tenant alias lifecycle
Section titled “Tenant alias lifecycle”| Status | Meaning |
|---|---|
cert_issuing | Quartermaster signaled entitled intent; Navigator is running ACME (LE primary, GTS fallback). |
cert_issued | Cert exists; tenant URLs become visible once Navigator has at least one ACK-gated DNS member. |
cert_failed | ACME error; check last_error. The worker retries on the next reconciliation pass. |
tearing_down | Tenant downgraded / cancelled; Navigator clears DNS, per-edge distribution, the tenant TLS bundle, and tenant certificates without active custom-domain authority. |
Additions are ACK-gated: Helmsman applies the ConfigSeed, Foghorn reports
the ACK to Navigator, and Navigator adds only successfully applied edges to
Bunny DNS. Removals are DNS-first: Navigator removes the cluster’s edges
from Bunny before later ConfigSeeds drop the tenant cert from those edges.
Navigator continues serving the existing certificate while teardown clears the
alias records. After the alias records are gone, Navigator removes credentials only if the alias
is still tearing_down; a concurrent reactivation fences that stale cleanup. The
same transaction removes the tenant TLS bundle and tenant certificate rows that
are not backed by an active custom-domain intent. Bundle persistence and renewal
also require live authority. During rename or issuance retry, edges continue
serving the last-good certificate under its original SANs; the old alias remains
in DNS until an edge ACKs the exact replacement bundle revision and Navigator
publishes the replacement alias. Quartermaster sends ordered cluster grants and
revocations through its durable Navigator outbox. Navigator stores active or
revoked authority for each tenant/cluster pair with the outbox sequence, locks
that decision while admitting an edge ACK, and excludes tombstoned pairs from
readiness and DNS publication. An edge ACK therefore cannot create or resurrect
authority. If a rolling-upgrade first-admission lookup is unavailable, Navigator
still records results backed by local authority before requesting durable replay
of the unresolved pairs. Quartermaster’s repair loop compares the full desired and applied
cluster sets. Foghorn does not
treat a shorter successful Quartermaster tenant list as immediate revocation: an
omitted tenant bundle remains in the durable edge seed only until its certificate
expires. Navigator’s explicit, error-free GetTLSBundle(found=false) is
authoritative and removes that bundle from the next seed. A response error is a
failed authority read and preserves valid local material. This preserves a still-valid local media
path through a list gap without retaining a torn-down credential. A completed
issuance cannot recreate an orphan after teardown.
Navigator advances a monotonic authority generation on rename, teardown, and
reactivation, and serializes issuance across replicas. Delayed ACME completion
therefore cannot cross a same-label reactivation or an a → b → a transition.
Replacement bundle persistence atomically publishes the revision, alias status,
and included custom-domain certificate metadata. First alias issuance performs a
separate fenced status transition after bundle persistence. Foghorn renders every certificate SAN and
pushes revision-only changes so Helmsman can ACK the exact material it applied.
Custom-domain removal follows the same rule: reactivation wins over stale
cleanup, late ACME writes cannot cross the lifecycle fence, and Navigator removes
the tenant ACME account once no custom domain in verified, pending_alias,
cert_issuing, cert_issued, or cert_failed remains. Pending-verification and
teardown rows cannot preserve an otherwise orphaned account.
A custom domain is served only as a SAN of the tenant’s multi-SAN bundle;
Navigator never issues a separate per-domain certificate. After CNAME
verification the domain waits in pending_alias until the tenant alias bundle is
issued, then joins the bundle (cert_issuing) and becomes cert_issued once the
edge-distributed bundle includes it. Only cert_issuing and cert_issued domains
join bundle orders, so one broken custom domain cannot block issuance or renewal
for the tenant alias and healthy SANs. A failed order marks only that domain
cert_failed (check last_error) and retries after next_attempt_at, 15 minutes
later. A domain whose CNAMEs have not verified within 7 days becomes
verification_failed and is no longer retried; a new EnsureCustomDomain for it,
such as the tenant removing and adding the domain again, restarts verification. No bundle is rebuilt, because the serving bundle never contained the
failed SAN. A failed renewal leaves every domain’s status and SAN in place while
the still-valid bundle keeps serving; GetCustomDomainStatus reports it as
last_renewal_error until the next successful bundle clears it.
During mixed-version rollouts, legacy edges may ACK without a bundle revision.
An already-published legacy row stays in DNS for continuity, but it does not count
as replacement readiness and an unproven edge cannot enter the pool. A legacy
failure may demote only legacy state without a newer sequenced result. An upgraded
Helmsman must ACK the exact current revision for promotion or readiness.
Self-hosted trust boundary
Section titled “Self-hosted trust boundary”- Cluster wildcards (
*.{cluster_slug}.{base}) and per-tenant wildcards (*.{tenant}.cdn.{base}) are distributed to all edges in the relevant cluster regardless of operator kind. Trust math: the operator only ever sees certs scoped to their own cluster’s namespace or to tenants who explicitly subscribed to them. - Global service entrypoints and the platform-edge multi-SAN cert
are ONLY distributed to
platform_officialcluster edges. Third-party / marketplace / tenant-private edges cannot impersonate the global names.
Billing entitlement and teardown rules
Section titled “Billing entitlement and teardown rules”Purser owns the effective values of custom_subdomain_enabled and
custom_domain_enabled. A subscription override wins over the tier default;
a missing or non-boolean catalog value is a bootstrap error, and an absent
effective grant is treated as false. Purser sends a timestamped observation
through Quartermaster’s service-only entitlement RPC. Quartermaster rejects
older observations, so webhook, bootstrap, and sweep races converge to the
newest billing state.
A platform tenant alias is desired only while the tenant is active, has
custom_subdomain_enabled=true, and has at least one usable cluster grant. A
BYO custom domain additionally requires custom_domain_enabled=true. Usable
means active, subscribed, known provenance, unexpired, and attached to an
active cluster. Entitlement loss, tenant deactivation, or loss of the final
usable cluster enqueues removal. The requested subdomain/custom-domain value is
preserved so re-entitlement can restore service without re-entering the name.
Quartermaster’s repair loop compares desired state with Navigator and restores
missing ensure/remove work after transient failures.
| Tenant state | Primary URL | Diagnostic / fallback |
|---|---|---|
| Free + platform-official only | Global root | Cluster-concrete |
| Free + self-hosted | Cluster-concrete | — |
| Alias-entitled (any sub mix) | Tenant alias | Global root, then cluster-concrete |
The Commodore CreateStreamResponse and GraphQL StreamingConfig
surface all three tiers ({service}_domain, global_{service}_domain,
tenant_{service}_domain). SDK/webapp prefer tenant > global >
cluster-concrete.
Provider Configuration
Section titled “Provider Configuration”Navigator needs credentials for the authoritative provider it is expected to manage. For Cloudflare-managed root names, use an API token with:
- Zone.DNS: Edit
- Account.Load Balancers: Edit (for multi-node setups)
Core environment variables:
CLOUDFLARE_API_TOKEN="your-api-token"CLOUDFLARE_ZONE_ID="your-zone-id"CLOUDFLARE_ACCOUNT_ID="your-account-id"BUNNY_API_KEY="your-bunny-api-key"NAVIGATOR_GOOGLE_TRUST_EAB_KID="your-gts-eab-key-id"NAVIGATOR_GOOGLE_TRUST_EAB_HMAC_KEY="your-gts-eab-hmac-key"Optional environment variables:
# Cloudflare proxy overrides (comma-separated service types)NAVIGATOR_PROXY_SERVICES="bridge,chartroom,chatwoot,foredeck,grafana,listmonk,logbook,metabase,steward"
# Certificate issuanceACME_ENV="production" # or "staging"NAVIGATOR_CERT_ALLOWED_SUFFIXES="example.com"NAVIGATOR_ACME_CA_ORDER="letsencrypt,google-trust" # override only