Livepeer network observations
The public Livepeer tab at /network?view=livepeer and GET /v1/network/livepeer API combine Livepeer discovery
with real gateway transcoding observations. Unknown coverage remains unknown; regional
performance is shown separately. The platform-operator webapp can enable private
correlation through the authenticated diagnostics route.
The default FrameWorks tab shows infrastructure topology. The Livepeer tab opens on a
map: the toolbar sets the time window, history region and what the map shows
(availability, observed errors or transcoding activity), and “Layers & filters” toggles
the worker, gateway, prober and FrameWorks node layers. The side panel lists the
observing regions and the orchestrator identities; choosing a region, orchestrator or
prober site opens its detail with regional history, prices and measured performance.
The view is kept in the URL, so a link reopens the same selection. Gateway map positions
are cluster locations. Workers and probers without a known location stay listed. Prices
and performance keep their units, region and output profile. When the platform runs no
observer, the tab says the network view is not configured instead of offering a retry.
This feature requires platform v0.3.13 and CLI v0.3.13 or later. Upgrade
existing clusters with the release’s ClickHouse expand migration and initialize
the new observer database from its baseline.
Coordinator
Section titled “Coordinator”Declare network-observer under the cluster manifest’s services, with an enabled
entry, a host in the aggregator region, and that host’s platform cluster ID. Both
native and docker modes are supported by the CLI. The observer is one coordinator
per platform, not a per-gateway-cluster service, so it runs only where a manifest
declares it; a self-hosted platform that does not want public Livepeer observations
omits the entry. It is also available in the network-observations selection profile;
the default profiles do not start an observer. External probers are not part of that
profile or CLI enrollment.
The observer has no public hostname, ingress site or DNS record. Bridge reaches it over the WireGuard mesh with the internal service CA, and external probers reach it through Bridge’s public API.
Add the network_observer database to the selected managed SQL deployment’s database
list. The release bootstrap creates its baseline and network_observer_runtime role
before deploying the observer. The CLI derives its database connection from this
declaration and the existing database credentials. For a manually managed service,
build with make build-bin-network-observer and apply the same baseline to its own
database using the normal database tooling.
For the new database, set owner: network_observer and
runtime_role: network_observer_runtime. Supply the runtime password through the
existing DATABASE_RUNTIME_PASSWORD secret, or the database-specific
POSTGRES_NETWORK_OBSERVER_RUNTIME_PASSWORD; the latter takes precedence. Keep the
owner/bootstrap credential separate. An omitted runtime_role preserves an owner DSN
for compatibility; declaring it makes the observer connect with the runtime role.
The observer needs DATABASE_URL, NETWORK_OBSERVER_TENANT_ID,
NETWORK_OBSERVER_CLUSTER_ID, NETWORK_OBSERVER_SECRET, LIVEPEER_ETH_RPC_URL,
LIVEPEER_CONTROLLER, DECKLOG_GRPC_ADDR, SERVICE_TOKEN and KAFKA_BROKERS, with
verified internal gRPC TLS as for other services. CLI provisioning renders all of them
from the manifest and the shared env; the only operator inputs are the shared secret
and an Arbitrum RPC endpoint. For isolated local use, the development
docker-compose.yml runs the observer under the network-observations profile once
the chain settings and shared secret are set, together with a network-prober that
uses a signed, unfunded local discovery identity unless LIVEPEER_BROADCASTER_ADDRESS
and LIVEPEER_BROADCASTER_SIGNATURE are set. NETWORK_PROBER_ALLOWED_PRIVATE_CIDRS lets
a prober dial orchestrators on a private test network; leave it unset on real probers.
What CLI provisioning derives, with explicit env-file or service-config values taking precedence:
- Tenant and cluster. The observer tenant is the resolved system tenant and its cluster is the manifest task’s cluster.
- Namespace. The observation environment is the manifest
profile(production,staging, anddevelopmentfordev): the CLI renders it asNETWORK_OBSERVER_ENVIRONMENTon the observer and Bridge and asFRAMEWORKS_ENVIRONMENTon every Livepeer gateway, so one manifest’s producers and readers share a namespace and staging stays apart from production. - Chain and Controller.
LIVEPEER_NETWORK_IDdefaults to Arbitrum One (42161).LIVEPEER_CONTROLLERis the Livepeer Controller of that chain, taken from go-livepeer’s network defaults:0xD8E8328501E9645d16Cf49539efC04f734606ee4for42161and0xf96d54e490317c557a967abfa5d6e33006be69b3for Ethereum mainnet (1). Provisioning refuses any other chain unlessLIVEPEER_CONTROLLERis set explicitly. - RPC.
LIVEPEER_ETH_RPC_URLfalls back to the existingARBITRUM_RPC_ENDPOINT. - Analytics bindings. The aggregator Kafka binding,
MIRROR_REGION_PREFIXESfrom inbound MirrorMaker links, and the aggregator-regionDECKLOG_GRPC_ADDR. Manual deployments must set the source aliases themselves to include gateway discovery from other regions. - Internal identity. The observer gets an internal-CA leaf
(
/etc/frameworks/pki/services/network-observer/), which Privateer renews, and serves its API with it onNETWORK_OBSERVER_INTERNAL_PORT(18034). The plainNETWORK_OBSERVER_PORT(18024) listener answers health and readiness probes. The observer registers with Quartermaster through the renderedQUARTERMASTER_GRPC_ADDR, and every Bridge host’s mesh seed resolvesnetwork-observer.<aggregator-region>.internalto the observer host and peers with it. - Relay trust.
NETWORK_OBSERVER_TRUSTED_PROXY_CIDRSis the mesh address of every Bridge host, so the observer takes a relayed prober’s source address from Bridge and from nothing else. - GeoIP. With the manifest’s
geoipenabled, a declared observer always receives the MMDB andGEOIP_MMDB_PATH, including whengeoip.serviceslists other services only. The MMDB supplies worker and prober coordinates for the map and the proximity-based prober assignment; missing geography does not prevent discovery.
NETWORK_OBSERVER_SECRET is a platform-generated shared secret kept in the SOPS-managed
shared env. frameworks cluster secrets generate-shared includes it, and
frameworks cluster secrets generate-network-observer --out <file> generates it alone
for an existing deployment. Development provisioning generates it. A non-development
manifest that declares the observer refuses to render the observer or Bridge without
it. Provisioning renders it only to the observer and Bridge; external probers receive
the same value from the operator.
When the manifest declares an observer, provisioning renders Bridge’s
NETWORK_OBSERVER_URL as https://network-observer.<aggregator-region>.internal:18034
(Bridge verifies the leaf as network-observer.internal against GRPC_TLS_CA_PATH), its
NETWORK_OBSERVER_SECRET, LIVEPEER_NETWORK_ID and NETWORK_OBSERVER_ENVIRONMENT from
the observer’s values, and NETWORK_PUBLIC_TENANT_IDS as the observer’s tenant. A
development manifest without Navigator has no internal CA, so Bridge uses the plain
listener there. Gateway
observations are stored under their cluster owner tenant, and platform clusters are owned
by the system tenant, so this scope publishes the platform-operated dataset (the
coordinator’s inventory and prober results and the platform gateways’ traffic) and never
a tenant-owned cluster’s gateways. Set NETWORK_PUBLIC_TENANT_IDS on Bridge explicitly
to publish a different scope. Publication scopes are server configuration, not
caller-supplied query parameters. Periscope runs the canonical migration and rollup loop
as part of its existing ingestion service.
Bridge relays the external probers’ two routes on its public API:
GET /network-observer/v1/work and POST /network-observer/v1/observations. Bridge
forwards the prober’s own Authorization header unchanged, so the observer
authenticates the shared secret, and rejects a request without a bearer token before
relaying it. It replaces any X-Forwarded-For with the source address it resolved,
bounds request and response bodies at 8 MiB, and rate-limits each route per source
address (work 60 per minute, observations 1200 per minute). The inventory and candidate
routes are not exposed publicly.
GitOps inputs and activation order
Section titled “GitOps inputs and activation order”The observer binary and prober binaries/images ship with the platform release; gateways use the separately released compatible go-livepeer fork. The platform’s release manifest provides artifact pins, not host placement or configuration. Complete these declarations before activating the feature:
| Location | Required inputs |
|---|---|
| Cluster manifest | Enabled network-observer service on an aggregator-region host, and the network_observer database (owner: network_observer, runtime_role: network_observer_runtime) in the managed SQL database list. |
| SOPS-managed shared environment | NETWORK_OBSERVER_SECRET from frameworks cluster secrets generate-network-observer, and ARBITRUM_RPC_ENDPOINT (or an explicit LIVEPEER_ETH_RPC_URL). Preserve existing database credentials and the internal SERVICE_TOKEN. |
| Derived by provisioning | Controller, chain, namespace, tenant and cluster on the observer; its internal leaf, registration and mesh DNS; Bridge’s URL, secret and publication scope; the observer’s trust in Bridge relays. Set a key explicitly only to override it. |
| Gateway configuration | The go-livepeer pin from the platform release and a verified Decklog connection; the CLI renders the gateway identity, namespace and telemetry spool. The CLI does not render candidate hints. |
| Lookout service environment | Optional LOOKOUT_OPS_SLACK_WEBHOOK_URL, LOOKOUT_OPS_DISCORD_WEBHOOK_URL, LOOKOUT_BUSINESS_PULSE_SLACK_WEBHOOK_URL, LOOKOUT_BUSINESS_PULSE_DISCORD_WEBHOOK_URL to split the existing destinations. |
Keep credentials in encrypted env files referenced by the services. Generate the secret and import it with the GitOps SOPS helper, never by editing encrypted content, then delete the plaintext fragment:
frameworks cluster secrets generate-network-observer --out /secure/path/network-observer.envgitops/scripts/sops-env.sh set <shared-secret-file> NETWORK_OBSERVER_SECRET <value-from-fragment>Production and staging use separate secrets and, through the manifest profile, separate observation environments.
Activate in this order:
- Add the manifest and SOPS inputs above.
- Preview with
frameworks cluster release planandframeworks cluster release apply --dry-run. The dry-run renders the observer and Bridge environments and fails on a missing secret or an unknown chain. - Apply the release. It initializes the observer database baseline and runtime role, applies the ClickHouse expand migrations, converges the Privateer mesh seeds that resolve the observer for Bridge hosts, deploys the observer, Bridge and the analytics services, and rolls the gateways to the release’s go-livepeer pin.
- Verify with
frameworks cluster status, thenGET /v1/network/livepeeron Bridge:inventory.networkis42161,inventory.environmentis the manifest profile, and the inventory lists catalog resources. A 503 with codeNETWORK_OBSERVATIONS_UNAVAILABLE(network inventory temporarily unavailableornetwork inventory namespace mismatch) means Bridge cannot reach or verify the observer. A 503 with codeNETWORK_OBSERVATIONS_NOT_CONFIGUREDmeans Bridge has no observer client, no analytics reader, or no validNETWORK_PUBLIC_TENANT_IDS; check its environment. The external prober relay answers the same code when Bridge has no observer client. - Install the external probers with
NETWORK_OBSERVER_URL=<public Bridge URL>/network-observerand the same secret. The observer is ready once its database answers, the contract catalog has refreshed and its Kafka consumer group is joined with assigned partitions that are being consumed. A 503 lists each failing reason; one failed catalog refresh is tolerated, two consecutive failures or a group unassigned or stalled for two refresh intervals make it not ready. - Confirm real discovery in the inventory, real gateway observations in the public API, and the dashboard’s observer coverage before retiring the old scripts.
Livepeer alerts are disabled; new Ops/Business Pulse webhook destinations require no Livepeer rollout.
External probers
Section titled “External probers”Build with make build-bin-network-prober, or use the matching release’s Linux amd64 or
arm64 network-prober binary after checking its release checksum. These hosts do not
need the FrameWorks stack, Node/PM2 or CLI enrollment. Configure a unique stable ID,
region, the Bridge relay URL (<public Bridge URL>/network-observer), shared secret and
persistent spool directory using api_network_observer/deploy/network-prober.env.example.
The Linux release tarball includes install-network-prober.sh,
network-prober.service and network-prober.env.example alongside the binary. Extract
it on the prober host, rename the architecture-suffixed binary to network-prober
for the commands below, and copy the example to a private network-prober.env.
The installer and unit stay together in the extracted directory. A source checkout
uses the same files from api_network_observer/deploy/.
The supplied systemd unit grants writable state at /var/lib/frameworks-network-prober;
use that path for NETWORK_PROBER_SPOOL_DIR with this installer.
Put the Ethereum broadcaster address and discovery signature in a separate private identity file:
LIVEPEER_BROADCASTER_ADDRESS=0x…LIVEPEER_BROADCASTER_SIGNATURE=0x…An existing valid pair works directly. In the legacy JavaScript configuration,
CONF_BROADCASTER maps to the address; CONF_SIGNATURE is the same 65 signature bytes,
encoded as hex rather than a JavaScript array. No private key belongs on the prober.
To create another pair, run the optional helper on the wallet host:
./network-prober sign-discovery --keystore wallet.json --password-file wallet-password --output network-prober.identity.envThe password file must have mode 0600 or stricter. The helper writes a new mode-0600 identity file and refuses to overwrite an existing file. Discovery signs the checksummed address with go-livepeer’s Ethereum hash/signature convention. This command does not fund a wallet or send a transaction. The broadcaster may need sender reserve for discovery admission even though the prober submits no paid transcodes.
On the Linux prober host, make the configuration and identity files private, validate them offline, then install:
chmod 600 network-prober.env network-prober.identity.env./network-prober check-config --env-file network-prober.env --identity-file network-prober.identity.envsudo ./install-network-prober.sh --binary ./network-prober --config network-prober.env --identity network-prober.identity.envKeep the installer beside its supplied service template. It creates the dedicated
service user and persistent state directory, installs the files and starts
frameworks-network-prober.service. Add --no-start to prepare a host without starting
it. On later reconfiguration, omitting --identity preserves the installed pair.
After replacing a running binary or configuration, restart the service explicitly with
sudo systemctl restart frameworks-network-prober.service.
Each process resolves the whole catalog every five minutes and executes only its assigned endpoint discovery checks. It does not submit synthetic transcodes or require broadcaster funding for synthetic work. Orchestrator self-signed discovery certificates are accepted by default, matching Livepeer; the coordinator connection still verifies TLS. Keep the spool directory persistent and exclusive to this process.
Local port 18025 exposes /health with spool counts, successful_discovery_checks,
last_discovery_success and admission_rejections. /ready checks recent coordinator
connectivity only. Verify a real successful discovery and its arrival in the inventory
before retiring an old prober. Authentication and sender-reserve failures are recorded
separately and do not count as worker downtime. The shipped example sets
BIND_ADDRS=127.0.0.1, so that endpoint answers only on the prober host itself.
Observer heartbeat and DNS-cycle reports disclose queue depth and lost observations.
An unavailable prober does not make the orchestrator unavailable.
Gateway
Section titled “Gateway”CLI provisioning renders every gateway telemetry setting from the manifest:
FRAMEWORKS_GATEWAY_ID (the host), FRAMEWORKS_GATEWAY_REGION, FRAMEWORKS_CLUSTER_ID,
FRAMEWORKS_CLUSTER_OWNER_TENANT_ID, FRAMEWORKS_DECKLOG_GRPC_ADDR with its TLS mode,
and the internal SERVICE_TOKEN. FRAMEWORKS_ENVIRONMENT is the manifest profile and
FRAMEWORKS_NETWORK_ID is the chain ID of the gateway’s Livepeer network
(arbitrum-one-mainnet is 42161; an offchain gateway reports 0). Provisioning refuses a
network with no known chain ID unless FRAMEWORKS_NETWORK_ID is set explicitly. The
telemetry spool is /var/lib/frameworks/livepeer-gateway/frameworks-telemetry, a state
directory created for the service user and writable inside the unit’s sandbox, so
undelivered observations survive restarts and upgrades.
Per-orchestrator work totals require the gateway build with accounting version 1.
Apply the release’s 002_network_work.sql ClickHouse expand migration before upgrading
Periscope ingestion and queries. Existing gateway observations remain readable, but
their missing work and charge fields appear as unaccounted coverage. This extension
uses the same telemetry credentials and spool; probers do not require new settings.
The dashboard separates submitted work, accepted source minutes, reported pixels,
service charges and expected ticket credit. These describe FrameWorks gateway traffic,
not all work or redeemed earnings across the Livepeer network.
The optional FRAMEWORKS_NETWORK_OBSERVER_URL and
FRAMEWORKS_NETWORK_OBSERVER_SECRET enable candidate hints. Provisioning does not
render them: the observer serves gateways only on its internal-CA listener, and the
gateway’s candidate client verifies the system roots, so hints stay off on managed
gateways. When set, keep FRAMEWORKS_CANDIDATE_FEED_MODE=shadow initially; supplement
adds fresh hints without replacing local discovery or relaxing local eligibility checks.
HTTP overrides are for isolated local development only.
API filters
Section titled “API filters”The public route accepts RFC3339 start/end (maximum 730-day window), resource
(onchain address), ip, region, interval (1m, 5m, 1h) and
include_inactive=true. Windows beyond raw retention use hourly snapshots and identify
that in the response. Prices and raw checks retain 30 days. Whole-window summaries and
bounded chart series are separate fields; inspect truncated before treating a series
as complete.
Livepeer notifications and incident escalation are disabled in this version. Operational incidents and business pulse delivery can be routed separately using Lookout’s purpose settings in the configuration reference.