Skip to content

Livepeer network observations

The public Livepeer tab at /network?view=livepeer and GET /v1/network/livepeer API combine Livepeer discovery with real gateway transcoding observations. Unknown coverage remains unknown; regional performance is shown separately. The platform-operator webapp can enable private correlation through the authenticated diagnostics route. The default FrameWorks tab shows infrastructure topology. The Livepeer tab opens on a map: the toolbar sets the time window, history region and what the map shows (availability, observed errors or transcoding activity), and “Layers & filters” toggles the worker, gateway, prober and FrameWorks node layers. The side panel lists the observing regions and the orchestrator identities; choosing a region, orchestrator or prober site opens its detail with regional history, prices and measured performance. The view is kept in the URL, so a link reopens the same selection. Gateway map positions are cluster locations. Workers and probers without a known location stay listed. Prices and performance keep their units, region and output profile. When the platform runs no observer, the tab says the network view is not configured instead of offering a retry.

This feature requires platform v0.3.13 and CLI v0.3.13 or later. Upgrade existing clusters with the release’s ClickHouse expand migration and initialize the new observer database from its baseline.

Declare network-observer under the cluster manifest’s services, with an enabled entry, a host in the aggregator region, and that host’s platform cluster ID. Both native and docker modes are supported by the CLI. The observer is one coordinator per platform, not a per-gateway-cluster service, so it runs only where a manifest declares it; a self-hosted platform that does not want public Livepeer observations omits the entry. It is also available in the network-observations selection profile; the default profiles do not start an observer. External probers are not part of that profile or CLI enrollment.

The observer has no public hostname, ingress site or DNS record. Bridge reaches it over the WireGuard mesh with the internal service CA, and external probers reach it through Bridge’s public API.

Add the network_observer database to the selected managed SQL deployment’s database list. The release bootstrap creates its baseline and network_observer_runtime role before deploying the observer. The CLI derives its database connection from this declaration and the existing database credentials. For a manually managed service, build with make build-bin-network-observer and apply the same baseline to its own database using the normal database tooling.

For the new database, set owner: network_observer and runtime_role: network_observer_runtime. Supply the runtime password through the existing DATABASE_RUNTIME_PASSWORD secret, or the database-specific POSTGRES_NETWORK_OBSERVER_RUNTIME_PASSWORD; the latter takes precedence. Keep the owner/bootstrap credential separate. An omitted runtime_role preserves an owner DSN for compatibility; declaring it makes the observer connect with the runtime role.

The observer needs DATABASE_URL, NETWORK_OBSERVER_TENANT_ID, NETWORK_OBSERVER_CLUSTER_ID, NETWORK_OBSERVER_SECRET, LIVEPEER_ETH_RPC_URL, LIVEPEER_CONTROLLER, DECKLOG_GRPC_ADDR, SERVICE_TOKEN and KAFKA_BROKERS, with verified internal gRPC TLS as for other services. CLI provisioning renders all of them from the manifest and the shared env; the only operator inputs are the shared secret and an Arbitrum RPC endpoint. For isolated local use, the development docker-compose.yml runs the observer under the network-observations profile once the chain settings and shared secret are set, together with a network-prober that uses a signed, unfunded local discovery identity unless LIVEPEER_BROADCASTER_ADDRESS and LIVEPEER_BROADCASTER_SIGNATURE are set. NETWORK_PROBER_ALLOWED_PRIVATE_CIDRS lets a prober dial orchestrators on a private test network; leave it unset on real probers.

What CLI provisioning derives, with explicit env-file or service-config values taking precedence:

  • Tenant and cluster. The observer tenant is the resolved system tenant and its cluster is the manifest task’s cluster.
  • Namespace. The observation environment is the manifest profile (production, staging, and development for dev): the CLI renders it as NETWORK_OBSERVER_ENVIRONMENT on the observer and Bridge and as FRAMEWORKS_ENVIRONMENT on every Livepeer gateway, so one manifest’s producers and readers share a namespace and staging stays apart from production.
  • Chain and Controller. LIVEPEER_NETWORK_ID defaults to Arbitrum One (42161). LIVEPEER_CONTROLLER is the Livepeer Controller of that chain, taken from go-livepeer’s network defaults: 0xD8E8328501E9645d16Cf49539efC04f734606ee4 for 42161 and 0xf96d54e490317c557a967abfa5d6e33006be69b3 for Ethereum mainnet (1). Provisioning refuses any other chain unless LIVEPEER_CONTROLLER is set explicitly.
  • RPC. LIVEPEER_ETH_RPC_URL falls back to the existing ARBITRUM_RPC_ENDPOINT.
  • Analytics bindings. The aggregator Kafka binding, MIRROR_REGION_PREFIXES from inbound MirrorMaker links, and the aggregator-region DECKLOG_GRPC_ADDR. Manual deployments must set the source aliases themselves to include gateway discovery from other regions.
  • Internal identity. The observer gets an internal-CA leaf (/etc/frameworks/pki/services/network-observer/), which Privateer renews, and serves its API with it on NETWORK_OBSERVER_INTERNAL_PORT (18034). The plain NETWORK_OBSERVER_PORT (18024) listener answers health and readiness probes. The observer registers with Quartermaster through the rendered QUARTERMASTER_GRPC_ADDR, and every Bridge host’s mesh seed resolves network-observer.<aggregator-region>.internal to the observer host and peers with it.
  • Relay trust. NETWORK_OBSERVER_TRUSTED_PROXY_CIDRS is the mesh address of every Bridge host, so the observer takes a relayed prober’s source address from Bridge and from nothing else.
  • GeoIP. With the manifest’s geoip enabled, a declared observer always receives the MMDB and GEOIP_MMDB_PATH, including when geoip.services lists other services only. The MMDB supplies worker and prober coordinates for the map and the proximity-based prober assignment; missing geography does not prevent discovery.

NETWORK_OBSERVER_SECRET is a platform-generated shared secret kept in the SOPS-managed shared env. frameworks cluster secrets generate-shared includes it, and frameworks cluster secrets generate-network-observer --out <file> generates it alone for an existing deployment. Development provisioning generates it. A non-development manifest that declares the observer refuses to render the observer or Bridge without it. Provisioning renders it only to the observer and Bridge; external probers receive the same value from the operator.

When the manifest declares an observer, provisioning renders Bridge’s NETWORK_OBSERVER_URL as https://network-observer.<aggregator-region>.internal:18034 (Bridge verifies the leaf as network-observer.internal against GRPC_TLS_CA_PATH), its NETWORK_OBSERVER_SECRET, LIVEPEER_NETWORK_ID and NETWORK_OBSERVER_ENVIRONMENT from the observer’s values, and NETWORK_PUBLIC_TENANT_IDS as the observer’s tenant. A development manifest without Navigator has no internal CA, so Bridge uses the plain listener there. Gateway observations are stored under their cluster owner tenant, and platform clusters are owned by the system tenant, so this scope publishes the platform-operated dataset (the coordinator’s inventory and prober results and the platform gateways’ traffic) and never a tenant-owned cluster’s gateways. Set NETWORK_PUBLIC_TENANT_IDS on Bridge explicitly to publish a different scope. Publication scopes are server configuration, not caller-supplied query parameters. Periscope runs the canonical migration and rollup loop as part of its existing ingestion service.

Bridge relays the external probers’ two routes on its public API: GET /network-observer/v1/work and POST /network-observer/v1/observations. Bridge forwards the prober’s own Authorization header unchanged, so the observer authenticates the shared secret, and rejects a request without a bearer token before relaying it. It replaces any X-Forwarded-For with the source address it resolved, bounds request and response bodies at 8 MiB, and rate-limits each route per source address (work 60 per minute, observations 1200 per minute). The inventory and candidate routes are not exposed publicly.

The observer binary and prober binaries/images ship with the platform release; gateways use the separately released compatible go-livepeer fork. The platform’s release manifest provides artifact pins, not host placement or configuration. Complete these declarations before activating the feature:

LocationRequired inputs
Cluster manifestEnabled network-observer service on an aggregator-region host, and the network_observer database (owner: network_observer, runtime_role: network_observer_runtime) in the managed SQL database list.
SOPS-managed shared environmentNETWORK_OBSERVER_SECRET from frameworks cluster secrets generate-network-observer, and ARBITRUM_RPC_ENDPOINT (or an explicit LIVEPEER_ETH_RPC_URL). Preserve existing database credentials and the internal SERVICE_TOKEN.
Derived by provisioningController, chain, namespace, tenant and cluster on the observer; its internal leaf, registration and mesh DNS; Bridge’s URL, secret and publication scope; the observer’s trust in Bridge relays. Set a key explicitly only to override it.
Gateway configurationThe go-livepeer pin from the platform release and a verified Decklog connection; the CLI renders the gateway identity, namespace and telemetry spool. The CLI does not render candidate hints.
Lookout service environmentOptional LOOKOUT_OPS_SLACK_WEBHOOK_URL, LOOKOUT_OPS_DISCORD_WEBHOOK_URL, LOOKOUT_BUSINESS_PULSE_SLACK_WEBHOOK_URL, LOOKOUT_BUSINESS_PULSE_DISCORD_WEBHOOK_URL to split the existing destinations.

Keep credentials in encrypted env files referenced by the services. Generate the secret and import it with the GitOps SOPS helper, never by editing encrypted content, then delete the plaintext fragment:

Terminal window
frameworks cluster secrets generate-network-observer --out /secure/path/network-observer.env
gitops/scripts/sops-env.sh set <shared-secret-file> NETWORK_OBSERVER_SECRET <value-from-fragment>

Production and staging use separate secrets and, through the manifest profile, separate observation environments.

Activate in this order:

  1. Add the manifest and SOPS inputs above.
  2. Preview with frameworks cluster release plan and frameworks cluster release apply --dry-run. The dry-run renders the observer and Bridge environments and fails on a missing secret or an unknown chain.
  3. Apply the release. It initializes the observer database baseline and runtime role, applies the ClickHouse expand migrations, converges the Privateer mesh seeds that resolve the observer for Bridge hosts, deploys the observer, Bridge and the analytics services, and rolls the gateways to the release’s go-livepeer pin.
  4. Verify with frameworks cluster status, then GET /v1/network/livepeer on Bridge: inventory.network is 42161, inventory.environment is the manifest profile, and the inventory lists catalog resources. A 503 with code NETWORK_OBSERVATIONS_UNAVAILABLE (network inventory temporarily unavailable or network inventory namespace mismatch) means Bridge cannot reach or verify the observer. A 503 with code NETWORK_OBSERVATIONS_NOT_CONFIGURED means Bridge has no observer client, no analytics reader, or no valid NETWORK_PUBLIC_TENANT_IDS; check its environment. The external prober relay answers the same code when Bridge has no observer client.
  5. Install the external probers with NETWORK_OBSERVER_URL=<public Bridge URL>/network-observer and the same secret. The observer is ready once its database answers, the contract catalog has refreshed and its Kafka consumer group is joined with assigned partitions that are being consumed. A 503 lists each failing reason; one failed catalog refresh is tolerated, two consecutive failures or a group unassigned or stalled for two refresh intervals make it not ready.
  6. Confirm real discovery in the inventory, real gateway observations in the public API, and the dashboard’s observer coverage before retiring the old scripts.

Livepeer alerts are disabled; new Ops/Business Pulse webhook destinations require no Livepeer rollout.

Build with make build-bin-network-prober, or use the matching release’s Linux amd64 or arm64 network-prober binary after checking its release checksum. These hosts do not need the FrameWorks stack, Node/PM2 or CLI enrollment. Configure a unique stable ID, region, the Bridge relay URL (<public Bridge URL>/network-observer), shared secret and persistent spool directory using api_network_observer/deploy/network-prober.env.example.

The Linux release tarball includes install-network-prober.sh, network-prober.service and network-prober.env.example alongside the binary. Extract it on the prober host, rename the architecture-suffixed binary to network-prober for the commands below, and copy the example to a private network-prober.env. The installer and unit stay together in the extracted directory. A source checkout uses the same files from api_network_observer/deploy/. The supplied systemd unit grants writable state at /var/lib/frameworks-network-prober; use that path for NETWORK_PROBER_SPOOL_DIR with this installer.

Put the Ethereum broadcaster address and discovery signature in a separate private identity file:

LIVEPEER_BROADCASTER_ADDRESS=0x…
LIVEPEER_BROADCASTER_SIGNATURE=0x…

An existing valid pair works directly. In the legacy JavaScript configuration, CONF_BROADCASTER maps to the address; CONF_SIGNATURE is the same 65 signature bytes, encoded as hex rather than a JavaScript array. No private key belongs on the prober. To create another pair, run the optional helper on the wallet host:

Terminal window
./network-prober sign-discovery --keystore wallet.json --password-file wallet-password --output network-prober.identity.env

The password file must have mode 0600 or stricter. The helper writes a new mode-0600 identity file and refuses to overwrite an existing file. Discovery signs the checksummed address with go-livepeer’s Ethereum hash/signature convention. This command does not fund a wallet or send a transaction. The broadcaster may need sender reserve for discovery admission even though the prober submits no paid transcodes.

On the Linux prober host, make the configuration and identity files private, validate them offline, then install:

Terminal window
chmod 600 network-prober.env network-prober.identity.env
./network-prober check-config --env-file network-prober.env --identity-file network-prober.identity.env
sudo ./install-network-prober.sh --binary ./network-prober --config network-prober.env --identity network-prober.identity.env

Keep the installer beside its supplied service template. It creates the dedicated service user and persistent state directory, installs the files and starts frameworks-network-prober.service. Add --no-start to prepare a host without starting it. On later reconfiguration, omitting --identity preserves the installed pair. After replacing a running binary or configuration, restart the service explicitly with sudo systemctl restart frameworks-network-prober.service.

Each process resolves the whole catalog every five minutes and executes only its assigned endpoint discovery checks. It does not submit synthetic transcodes or require broadcaster funding for synthetic work. Orchestrator self-signed discovery certificates are accepted by default, matching Livepeer; the coordinator connection still verifies TLS. Keep the spool directory persistent and exclusive to this process.

Local port 18025 exposes /health with spool counts, successful_discovery_checks, last_discovery_success and admission_rejections. /ready checks recent coordinator connectivity only. Verify a real successful discovery and its arrival in the inventory before retiring an old prober. Authentication and sender-reserve failures are recorded separately and do not count as worker downtime. The shipped example sets BIND_ADDRS=127.0.0.1, so that endpoint answers only on the prober host itself. Observer heartbeat and DNS-cycle reports disclose queue depth and lost observations. An unavailable prober does not make the orchestrator unavailable.

CLI provisioning renders every gateway telemetry setting from the manifest: FRAMEWORKS_GATEWAY_ID (the host), FRAMEWORKS_GATEWAY_REGION, FRAMEWORKS_CLUSTER_ID, FRAMEWORKS_CLUSTER_OWNER_TENANT_ID, FRAMEWORKS_DECKLOG_GRPC_ADDR with its TLS mode, and the internal SERVICE_TOKEN. FRAMEWORKS_ENVIRONMENT is the manifest profile and FRAMEWORKS_NETWORK_ID is the chain ID of the gateway’s Livepeer network (arbitrum-one-mainnet is 42161; an offchain gateway reports 0). Provisioning refuses a network with no known chain ID unless FRAMEWORKS_NETWORK_ID is set explicitly. The telemetry spool is /var/lib/frameworks/livepeer-gateway/frameworks-telemetry, a state directory created for the service user and writable inside the unit’s sandbox, so undelivered observations survive restarts and upgrades.

Per-orchestrator work totals require the gateway build with accounting version 1. Apply the release’s 002_network_work.sql ClickHouse expand migration before upgrading Periscope ingestion and queries. Existing gateway observations remain readable, but their missing work and charge fields appear as unaccounted coverage. This extension uses the same telemetry credentials and spool; probers do not require new settings. The dashboard separates submitted work, accepted source minutes, reported pixels, service charges and expected ticket credit. These describe FrameWorks gateway traffic, not all work or redeemed earnings across the Livepeer network.

The optional FRAMEWORKS_NETWORK_OBSERVER_URL and FRAMEWORKS_NETWORK_OBSERVER_SECRET enable candidate hints. Provisioning does not render them: the observer serves gateways only on its internal-CA listener, and the gateway’s candidate client verifies the system roots, so hints stay off on managed gateways. When set, keep FRAMEWORKS_CANDIDATE_FEED_MODE=shadow initially; supplement adds fresh hints without replacing local discovery or relaxing local eligibility checks. HTTP overrides are for isolated local development only.

The public route accepts RFC3339 start/end (maximum 730-day window), resource (onchain address), ip, region, interval (1m, 5m, 1h) and include_inactive=true. Windows beyond raw retention use hourly snapshots and identify that in the response. Prices and raw checks retain 30 days. Whole-window summaries and bounded chart series are separate fields; inspect truncated before treating a series as complete.

Livepeer notifications and incident escalation are disabled in this version. Operational incidents and business pulse delivery can be routed separately using Lookout’s purpose settings in the configuration reference.