FrameWorks Docs | Blog

Restreams Send the Original Video Wherever It Fits

Tue, 29 Sep 2026 00:00:00 GMT

A broadcaster publishes one high-bitrate 1080p60 H.264 stream to FrameWorks and restreams it to two places: YouTube and Kick. YouTube takes the stream as it is. Kick documents a hard ceiling of 8 Mbps for video, and this stream is above it. One input, two destinations with different rules.

Each restream now picks its video separately. YouTube gets the original video, untouched. Kick gets a rendition that fits under its ceiling, taken from the renditions the stream already produces for adaptive playback. Nothing is encoded a second time for either destination.

Why passthrough matters

Every encode between the broadcaster and a destination adds delay, costs compute, and loses some quality, because a lossy encoder working from already-compressed video cannot get back what the first encode dropped. When the original video already meets a destination’s rules, the best thing to send is those exact bytes. For YouTube in this scenario that is what happens: the audience there sees what the broadcaster’s encoder produced.

The capped destination still needs something it will accept. A stream with a live video ladder is already producing lower renditions so that viewers on slower connections can play it. Choosing one of those for Kick means the restream costs no encode of its own. The decision is made per destination, from the tracks the stream actually has and the limits that destination documents, when the restream starts.

Audio is chosen on its own. A browser publishing over WebRTC sends Opus audio, and RTMP destinations expect AAC. The restream can take the original video and a converted AAC track together, so a browser-published stream can still go out with its video untouched.

When a stream does not need a ladder

Some streams have no use for lower renditions, such as a stream that exists mainly to be restreamed to destinations that accept the original. For those, the live video ladder can now be turned off per stream. Audio conversion, thumbnails, and recording keep working, and the stream no longer produces video renditions nobody watches.

The trade-off is direct. With the ladder off, there is no rendition to fall back on, so every destination has to accept the original video. In the scenario above, Kick would have nothing it can take. If you need to be sure which kind of video a destination receives, each restream target can also be set to require the original or require a rendition, instead of choosing automatically.

What it does not check

Destination limits are applied where the platform documents a hard ceiling. For YouTube, Twitch, and custom RTMP destinations, the restream checks that the codec is accepted but does not enforce their recommended bitrates, because those are recommendations rather than limits. Whether a publisher’s encoder keeps a constant bitrate or a fixed keyframe interval is not visible from the stream’s track information, so a destination that requires either still depends on how the encoder is configured. And a destination can refuse a publish for reasons of its own.

The multistreaming guide lists each preset’s limits, the per-target video setting, and how to turn off the live video ladder for a stream.

Moving From Livepeer Studio: Webhooks, Playback Gates, and URL Import

Thu, 24 Sep 2026 00:00:00 GMT

Teams moving from Livepeer Studio told us two things take the longest to port: VOD and webhooks. We read Studio’s source (it is open) and went through every webhook event, payload field, and playback-policy option it offers, then compared each one against FrameWorks. This release closes the gaps that would have forced a Studio integration to change its logic.

The goal was narrow: a team should be able to point its existing webhook receiver and playback gate at FrameWorks, change the parts that are named differently, and keep the logic behind them.

A 204 used to lock out every viewer

The clearest example came from the playback gate. A webhook playback policy asks your server whether a viewer may watch: FrameWorks sends the request, and your answer decides. Studio treats any 2xx as “allow”. We only accepted 200.

That difference is invisible until you migrate. Plenty of access endpoints answer 204 No Content, because there is nothing to say beyond yes. Pointed at FrameWorks, such an endpoint denied every viewer of every gated stream, and nothing in its own logs looked wrong: it had answered yes each time. Any 2xx now allows.

The same comparison turned up two smaller gaps in the gate. A policy can now store a small JSON context that is sent back with every access request, which Studio calls webhookContext. It lets your endpoint know which course or product a video belongs to without keeping its own lookup table. And a policy can list the websites allowed to embed the content.

What an origin check can and cannot stop

Allowed origins stop a player copied onto someone else’s site. The check runs twice: when the player looks up the stream, and again when the media session opens on the edge. For the second check, our MistServer build now passes each viewer’s Origin and Referer to the access decision, and requests from a different site get a separate session.

It is not DRM. A client that is not a browser can send any Origin header it likes. RTMP and SRT players send no origin at all, so they are refused unless the list allows every site. Edges running a MistServer build from before this release cannot report the origin, so origin-restricted content does not play from them until they update. If you run your own edges, update them before you turn this on.

Events carry what Studio events carried

Studio’s stream webhooks carry the stream’s playback ID, and ours now do too. Recordings now send an event when capture starts and another when it stops, the two points Studio reports.

One difference is deliberate. Clip, recording, and upload events do not carry a playback ID. A playback ID can be rotated, so a copy inside an event that is retried or replayed hours later can already be out of date. Key your records on the artifact ID and look up the current playback ID when you publish the video.

The migration guide maps each Studio event and field to its FrameWorks equivalent, including the one habit that does not carry over directly: how a Studio player passes the gate’s key.

Import a video from a URL

Studio lets you create an asset from a URL, and so does FrameWorks now. You pass a public link and get the asset and its playback ID back right away, with no upload step. The file is read from the source server and processed like any upload, with the same upload events, so a receiver that already handles uploads needs no new code.

The fetch connects only to public addresses, checked again on every connection and redirect, so an import URL cannot be used to reach a private network. The source server has to support range requests. For IPFS or Arweave content, pass a public gateway’s https link. The VOD guide has the details.

What stays different

Webhook signing uses Standard Webhooks rather than Studio’s Livepeer-Signature header, so a receiver needs a new verification step, which existing libraries cover. Retries run for about three days instead of one. There is no Task API, no per-stream transcode profiles, and no creatorId or userTags on streams yet. The gap list in the migration guide tracks what is still missing, and the playback access guide covers webhook policies and allowed origins.

SDKs for TypeScript, Go, and Python

Sat, 19 Sep 2026 00:00:00 GMT

Until now, a FrameWorks integration outside the browser started with a GraphQL client of your choice and a copy of the schema. The player and StreamCrafter packages covered playback and browser ingest, but creating streams, uploading files, signing viewer tokens, and checking webhook signatures from a backend meant writing that code yourself. That changes today with three packages:

@livepeer-frameworks/api on npm, for Node 22 and later and for browsers.
github.com/Livepeer-FrameWorks/sdk-go for Go.
livepeer-frameworks on PyPI, with a synchronous and an asynchronous client.

One set of operations, three languages

Each SDK generates its types from the public GraphQL schema and the same operation documents, using a standard code generator per language. When an operation changes, CI regenerates all three and fails if the committed code differs. The first release shipped 44 hand-written operations covering streams, clips, DVR, VOD uploads, playback access control, multistreaming, and developer tokens. The current 0.3 line adds a generated operation for every public root field that is not deprecated, 227 in total, so the whole public API has typed methods.

Creating a stream from a Node backend looks like this:

import { createClient, CreateStreamDocument, expectResult } from "@livepeer-frameworks/api";

const client = createClient({
  url: "https://api.example.com/graphql",
  token: process.env.FRAMEWORKS_API_TOKEN,
});
const created = await client.request(CreateStreamDocument, { input: { name: "Launch stream" } });
const stream = expectResult(created.createStream, "Stream");
console.log(stream.playbackId);

Most mutations return a union of the result and typed errors. expectResult either hands back the stream or raises an error that names what went wrong and which input field caused it, so a validation failure is something you can match on rather than a field that is quietly missing.

The same runtime in every language

Around the generated code, each SDK has a small hand-written runtime that does the same jobs: retrying transient failures with backoff, raising errors with the same names in every language, iterating over the API’s list shapes, and checking the server’s version before the first call so an operation the server does not know yet fails early instead of being sent. A mutation that may already have run is not retried behind your back.

Three helpers cover work that is not GraphQL. The upload helper sends a large file in parts with bounded concurrency and cleans up if anything fails. The playback token helper signs viewer tokens for content protected by a JWT playback policy. Webhook verification wraps the official Standard Webhooks library for each language, so the SDK carries no signature code of its own, and an event type the SDK does not know yet comes back as a raw event rather than an error. A new event type on the server never breaks an older SDK.

Two gateway changes shipped with the SDKs so they could rely on them. Errors from backend services now carry a machine-readable code, and a subscription with an expired token is closed before it starts instead of continuing as an anonymous connection. The SDKs report that as an authentication error at once.

Agreement between SDKs is tested, not assumed

Three runtimes written by hand drift apart unless something checks them. The repository has a directory of language-neutral JSON fixtures covering retries, error shapes, pagination, version checks, subscriptions, uploads, and webhooks, including the Standard Webhooks reference vector. Each SDK’s test suite loads the same files, so a change that makes one language behave differently fails CI.

For playback tokens, agreement between the SDKs was not enough. Three SDKs can agree with a fixture and all be wrong about what the edge accepts. So each SDK’s token for a shared test key is recorded, and a Go test runs all three through the same policy evaluation the media edge runs when a viewer connects. The first run of that test failed, and the fault was in the fixtures themselves: a token’s not-before time was still in the future. Every SDK had produced exactly what the fixture asked for, and the edge would have refused it.

Versions and what is not included

The SDKs stay at 0.x until the platform reaches 1.0, and all three always share a version. Each minor line names the oldest FrameWorks release it supports and supports every stable release from there on; the 0.3 line needs v0.3.11 or later. Within a line, operations are only added, and a breaking change starts a new line. On the server side, CI checks every operation of each supported line against the schema of each supported release, so a schema change that would break a field an SDK uses fails before it ships.

There are no SDKs for PHP, Ruby, Elixir, Java, or C#. Those integrations keep using the GraphQL API directly, with a Standard Webhooks library for signatures.

The SDK guide has install instructions, runnable examples in all three languages, the full list of error types, and the version table.

Webhooks and a Public Event Catalog

Sat, 19 Sep 2026 00:00:00 GMT

Until now, a FrameWorks integration that needed to react to a stream going live or a clip becoming ready had two options: keep a GraphQL subscription open, or poll. Neither works well for a serverless function or a backend that restarts during deploys. Webhooks are the missing piece, and before we could send them we needed a set of events that is safe to hand to a third-party server and stable enough to write code against.

Both now exist. Your account’s events are published as a versioned catalog, available live over a subscription and as signed HTTPS requests to endpoints you register.

An event is a promise about something that already happened

The event catalog covers streams going live or idle, stream keys being rotated, clips, recordings and uploads becoming ready or failing, multistream target status, API tokens, billing, account suspension, and custom domains. Each type has a protobuf message in a versioned public package, and that message is the payload.

Two rules make the catalog safe to build on. First, an event is written in the same database transaction as the change it describes. A billing.invoice_paid event exists only if the invoice really is paid; there is no window where your backend hears about a change that later rolled back. Second, a payload carries only identifiers and values your account already sees through the API. It does not carry internal stream names, node IDs, storage paths, IP addresses, or raw error text; a failed clip reports a coarse reason rather than a stack trace.

Within the v1 package fields are only added. A breaking change ships as a v2 message, and each webhook endpoint stays on the version it was created with, so a receiver you wrote today keeps parsing what it receives.

The same events are available over the tenantEvents GraphQL subscription, filtered by type and stream. The dashboard’s media library uses it to refresh when an artifact finishes processing.

What a delivery looks like

Under Developer → Webhooks you add HTTPS endpoints and pick the event types each one receives. Each event becomes a POST:

webhook-id: 0199a3c2-7e41-7d2a-9b3e-5f8c1d2e4a67
webhook-timestamp: 1789826608
webhook-signature: v1,NSZG8yLlem8faHh7EcOnDIbFIp4Wa/3hhM320Atl5LM=

{"id":"0199a3c2-7e41-7d2a-9b3e-5f8c1d2e4a67","type":"clip.ready","api_version":"v1","created_at":"2026-09-19T14:03:27.412Z","data":{"artifact":{"artifactId":"c7f3a9e2b41d4f6a8e0b5d2c9a1f7e34","kind":"ARTIFACT_KIND_CLIP","streamId":"5b0e6a2c-3f1d-4c8e-9a7b-2d4f6e8a0c13","playbackId":"k3v9x2m7q4w8"},"durationMs":"30000","sizeBytes":"7340032"}}

That is a real request body and signature, produced by the delivery code with the example secret whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw. The data object uses the protobuf JSON mapping, so 64-bit values such as sizeBytes are strings.

Why we did not invent a signature scheme

The signature follows the Standard Webhooks specification: HMAC-SHA256 over <webhook-id>.<webhook-timestamp>.<body>. We chose it over a scheme of our own because verification libraries already exist for most languages. In Node:

import { Webhook } from "standardwebhooks";

const event = new Webhook(process.env.FRAMEWORKS_WEBHOOK_SECRET!).verify(rawBody, headers);

We checked the example above against the official TypeScript, Go, and Python libraries; all three accept it. The timestamp is part of the signed content, so a captured request cannot be replayed against you days later.

The webhook-id is the event ID, and it stays the same across every attempt, every replay, and every endpoint. That is the one field a receiver needs to handle duplicates: store the IDs you have processed and ignore a repeat. Failed deliveries are retried for about three days, every attempt is logged for 30 days, and you can replay one delivery or a whole time range after an outage on your side; the webhooks guide has the schedule and the replay calls.

A slow receiver only delays itself

Delivery runs apart from the systems that validate stream keys and serve playback. A customer endpoint that hangs or returns errors delays its own deliveries and nobody else’s.

Endpoint URLs must resolve to public addresses, and that check runs again on every connection. A DNS record changed to point at an internal address after the endpoint was created is refused at send time, so a webhook cannot be turned into a way to reach our private network.

The limit worth knowing up front: an endpoint receives events from the moment it is created, with no backfill, and events carry no ordering guarantee across retries. Design the receiver to look up current state when order matters.

The webhooks guide covers endpoints, verification in TypeScript, Go, and Python, retries, auto-disable, secret rotation, and replay. The events reference lists every event type, when it fires, and its payload fields. Agents connected over MCP can list, test, and replay deliveries with an explicit confirmation.

Incidents: One Failure, One Thing to Work On

Thu, 17 Sep 2026 00:00:00 GMT

A server in one of your clusters loses power. It was running five services, and every one of them stops answering health checks at the same moment. Each produces its own “service unreachable” alert. Handled as plain notifications, that is five near-identical messages at once, the same messages again every few hours while the machine stays down, and no shared place to record that someone is already on it.

FrameWorks now turns alerts into incidents. That power loss becomes one incident for the cluster, listing every service that went dark, with a timeline, an owner and a status that stays correct until the last alert clears.

What the node failure looks like now

The failing services still fire individually, and Alertmanager still does what it is good at: evaluating rules, grouping related alerts, applying silences, and retrying delivery. Alerts of the same kind in the same cluster and region arrive as one group, and each group becomes one incident.

Alertmanager waits briefly for related alerts, then sends the five “service unreachable” alerts as one group. One incident opens with five firing alerts. When Alertmanager repeats the group a few hours later because the machine is still down, the existing incident is updated rather than a new one opened. An on-call engineer acknowledges it, which records who took it without silencing anything; if a sixth service fails, the incident still shows it. Notes and assignment build a timeline as the work happens. When the server comes back and every alert in the group resolves, the incident resolves itself.

If someone resolves an incident by hand, repeats of the same alerts do not reopen it. A genuinely new alert opens a new incident. That keeps a closed incident closed without hiding a second failure.

Every state change is pushed through a live subscription, so the dashboard updates without polling. After a reconnect, the incident itself is the durable record to read, not the stream of changes.

Incidents belong to whoever owns the cluster

A tenant running their own cluster should see incidents about their infrastructure and nobody else’s. So each incident is scoped to the tenant that owns the cluster the alert came from, and a tenant can only read or change incidents on clusters they own. Asking for another tenant’s incident returns the same answer as asking for one that does not exist.

Ownership can change while alerts are arriving, and the system that records ownership can itself be part of the outage. The rule for that case is simple: when ownership cannot be confirmed, the incident stays visible to platform operators only. A tenant might see their incident late, but never someone else’s, and operators still see the outage while the control plane is down.

Resolved incidents stay with the tenant that owned the cluster when they happened. Moving a cluster to a new owner does not hand over its history.

Delivery keeps working during the failure it reports

An incident system is most needed when things are broken, which is also when it is most likely to be affected. We designed delivery around that.

Platform incidents notify operators by email, Slack and Discord. Tenant incidents feed the tenant’s live dashboard and Skipper. An incident and the obligation to announce it are saved together, so a notification service that is briefly down delays the message and retries it later rather than losing it. When more than one copy of the incident service is running, each delivery is claimed by one copy at a time, so a restart or a failover does not leave a message unsent.

Critical alerts also have a second path that bypasses the incident system entirely and goes straight to email. If the incident service is the thing that failed, the page about it still goes out. And an alert that fires permanently is sent to an external heartbeat monitor once a minute; if the alerting pipeline itself stops, the heartbeat goes quiet and the external monitor raises the alarm.

The same workflow for people and agents

Tenants work incidents under Infrastructure → Incidents. Platform operators have a cross-tenant view under Platform Admin → Incidents. The same actions are available through GraphQL and the MCP tools: list, inspect, acknowledge, assign, resolve and add a note.

Skipper reads the tenant incident feed and can attach an investigation report to the timeline. It never acknowledges or resolves anything on its own; analysis and state changes stay separate, and a person decides when an incident is over.

Which alert rules exist is still an operator decision, and incidents do not merge alerts of different kinds into one record: a failed server that also trips a second, different alert rule produces two incidents, one per rule.

Start with the incidents API guide. Operators deploying incidents should read Alerting and Incidents, which covers notification channels, the heartbeat and an end-to-end delivery test.

Your Stream, Your Edge: Media Placement Policy

Wed, 09 Sep 2026 00:00:00 GMT

When a broadcaster goes live in Europe and someone opens the stream in the United States, that viewer should be served from a healthy US edge. Until this release, the answer depended partly on which part of the platform happened to receive the viewer’s request: arrive at a European coordinator, and “nearby” was measured from Europe.

Media placement treats three locations separately: where a request is answered, where a stream originates, and where a viewer receives it. The viewer’s distance is measured from the viewer, and accounts can now state their own rules about which capacity their streams may use.

Nearby means near the viewer

Federation gave clusters a way to find streams and source them from one another. Placement decides where that capability should be used.

Every eligible edge is compared on the same facts: health, usable capacity, distance to the viewer, and whether the stream is already there or needs a source connection. A cluster boundary is not a distance penalty. An edge in Virginia does not become less attractive to a viewer in Virginia because the request reached a coordinator in the EU.

When the chosen edge does not have the stream yet, it is pulled there. The first viewer may pay some setup latency; later viewers reuse the copy. If the chosen edge cannot accept the work, the next permitted candidate is tried, rather than whatever node sits next to the coordinator that answered.

My own servers first

Nearby is only part of the answer. Some accounts run their own edge servers and want the platform to treat them as the first choice, not one option among many.

Take a broadcaster who operates two edge servers in their own facility in Frankfurt and also has access to platform capacity. Their account rule says: serve viewers from my servers first, and use platform capacity only when mine are full or the viewer is too far away.

A viewer in Berlin is served from the broadcaster’s own servers, as they would expect. A viewer in Chicago is beyond the distance threshold the broadcaster configured, so the rule permits spillover and that viewer is served from a platform edge in the US, which pulls the stream across once. During a large event the Frankfurt servers fill up, and new European viewers spill onto platform capacity while existing sessions stay where they are.

The preview for that rule shows which capacity groups qualify, why a candidate was excluded, and which condition would permit spillover. It is careful about evidence: a self-hosted cluster that cannot be reached is not treated as proof that it is full, and a missing price is not treated as free.

The same rule could be stricter. “Prefer” and “never” mean different things: a preference can fall back, a prohibition cannot. If an account says never to use platform capacity and its own servers have no room, the stream is reported unavailable rather than moved onto paid capacity. Ingest and serving have separate rules, because the best place for a broadcaster to publish is not necessarily the best place for every viewer to watch. Account defaults cover the ordinary case; stream-specific rules can narrow them.

The choice is two-sided: a cluster owner can offer viewer capacity without accepting publishers, and your preference for a cluster does not grant you access to it.

Enforced where media actually starts

Placement is enforced at the two points where media starts: when a publisher connects and when a viewer connects.

For publishers, the policy is checked against the protocol the edge actually observed accepting the connection, never against the URL the encoder sent, because the encoder controls that URL and RTMP lets a client label a connection however it likes. A publisher that lands on a node the policy forbids is refused, and nothing is held for a session that never starts.

For viewers, the same policy that chose the destination admits the connection at the edge. A denied HLS request gets a complete refusal and a closed connection instead of a response that never finishes, so players fail over instead of stalling.

The rules reach the edge as signed authority, which matters during outages. A control-plane outage does not turn a deny into an allow. A publisher who is already live stays live, and edges keep enforcing the last rules they received. During an outage, availability can be limited, but policy is never widened.

What it does not cover yet

Saving a rule is not the same as every connection following it. A new rule takes effect once every cell that holds your content confirms it enforces placement, and the rollout status shows where that stands. A publisher who is already live stays on their current ingest cluster; changing the rule does not migrate them.

This release covers ingest and viewer delivery. Where recordings are stored and where processing jobs run are separate decisions on the roadmap, and viewer-serving rules do not control them today.

The media placement guide covers writing, reviewing and previewing rules, including the self-hosted-first example in the API Explorer demo. Cluster owners should read media placement controls for capacity permissions and rollout status.

FrameWorks 0.3.0

Sun, 05 Jul 2026 00:00:00 GMT

FrameWorks 0.3.0 is out. Until this release, running a FrameWorks platform meant answering many everyday questions from private dashboards and a database console. 0.3.0 gives operators and tenants a supported view of what is happening, keeps viewers watching through control-plane blips, lets you say where your streams may run, and puts guardrails around what AI agents can do. Operators upgrading self-hosted edges should read the note at the end first.

Operators and tenants can see what is happening

Platform admin is the most direct change. A cross-tenant admin API and admin console give operators one supported place to inspect tenants, clusters, billing, and content. Access to it is an explicit grant, and every use lands in an audit log. That matters for hosted operations and for anyone running more than one cluster.

Analytics is moving out of the internal-only bucket. Playback quality of experience, tenant activity, operator activity, stream monitoring controls, and a consolidated set of dashboard charts put the platform’s state in front of the people who can act on it. The goal is answering normal operational questions without opening three private dashboards and a SQL console. For programmatic access, see the analytics API.

Playback and placement hold up under failure

Viewers who are already watching keep watching when the control plane is briefly unreachable. New viewers are admitted from signed answers the media cluster already holds, so a short central outage does not turn into refused playback. One viewer’s approval is never reused for another: an earlier yes says nothing about the new viewer’s location, their permission, or a policy that changed since. Viewers keep watching when the control plane blips explains the rule and its limits. Alongside it, clusters tell each other which streams they can serve, so a viewer can be served through a peer cluster, and native edge installs on Linux and macOS are smoother.

Media placement is the other routing change. Where a request is answered, where a stream originates, and where a viewer receives it are three different places, and 0.3.0 stops treating a cluster boundary as a distance penalty. On top of that sits a placement policy per account and per stream: prefer your own capacity, spill over only when you run out of capacity or have no coverage in a region, never use platform-official capacity, or rank marketplace capacity by effective price. A policy only takes effect in a cluster that has confirmed it enforces it. Your stream, your edge tells the full story, and the media placement docs show how to preview a policy. Storage and processing placement are not in this release.

Agents can act, with guardrails

Skipper and MCP ship in 0.3.0 because AI agents are becoming part of how people operate streams. The useful work is diagnosis from real stream state, schema-aware API help, a billing preflight before expensive actions, confirmation before anything changes, and labels that say whether an answer came from docs, schema, metrics, or model inference. Billing changes and private-infrastructure tools now require an owner or admin identity as well as the matching API token scope, so an agent linked to a regular member account no longer sees those tools at all. The MCP docs cover connecting an agent.

What is not in 0.3.0

Chat and conferencing are next, because FrameWorks already has pieces of the real-time stack: support conversations, message lifecycle events, live subscriptions, and Skipper conversations. Conferencing still needs room state, participant identity, moderation, recording rules, pricing, and media routing built around active participants. 0.3.0 tightens the real-time and operator foundations that work depends on.

Composer stays on the roadmap. Overlays, lower thirds, server-side ad insertion, transcription, object detection, picture-in-picture, and automated clipping are among the more interesting media features we could build, and each carries a product contract: API shape, placement, lifecycle, billing meters, permissions, dashboard controls, and MCP access. It ships once those contracts are defined.

Bring-your-own-compute also gets a clearer boundary. You can run your own edge nodes with MistServer, and FrameWorks keeps running routing, federation, transcoding, storage, DNS, analytics, billing, and the control plane. There is no deeper tier where a customer runs part of that coupled stack. Teams that want all of it self-host the entire stack as an independent platform.

Operator note: self-hosted edges must be redeployed

This release raises the minimum protocol version between edges and the control plane, with no window where old and new versions work together. The control plane upgrades first. From that moment, edges on the older protocol are refused, drop out of routing, and cannot receive the automatic update that would fix them.

If you run edge nodes yourself, redeploy them with the 0.3.0 edge image after the control plane rollout. The same applies to an older edge that was offline during the rollout and comes back later: it cannot reconnect until it has been updated. This one-time interruption was a deliberate trade: an edge that cannot show it speaks the new protocol is refused rather than trusted. The upgrade guide has the redeploy command, and the self-hosted edge rollout runbook covers release order and recovery.

Viewers Keep Watching When the Control Plane Blips

Fri, 03 Jul 2026 00:00:00 GMT

Picture a stream in the middle of a broadcast. The encoder is pushing, the edge nodes are serving video, and people are watching. Then the control plane becomes unreachable for a few seconds: a deploy, a database failover, a network partition between regions. Nothing about the media changed. What matters is what happens to the people already watching, and to the people who press play during those seconds.

The rule FrameWorks follows: viewers who are already watching keep watching when the control plane is briefly unreachable. New viewers are admitted from a signed, durable answer the media cluster already holds. If no valid answer exists, the new viewer is refused cleanly. A missing answer is never treated as permission.

A failure to answer is not an answer

Every new playback request on an edge node goes through a hook: before MistServer starts serving, it asks the FrameWorks edge agent what the requested playback ID resolves to and whether this viewer may have it. The agent asks the cluster’s regional load balancer, which holds the policy.

The weak spot was what happened when that question could not be answered. MistServer’s hook interface had a single fallback value for “the handler failed”, and for playback that fallback was a string that read as a stream name. It could not tell an explicit denial apart from a handler that never replied, and different call sites interpreted the same fallback differently. A control-plane problem could reach the viewer as an arbitrary answer rather than as a failure.

Our MistServer fork now accepts an explicit action for each hook. For playback, a handler that cannot answer produces a denial, and an authoritative rejection produces the same denial. No sentinel value is involved. The edge agent also keeps the hook’s definition repaired continuously, because a hook that has gone missing skips the decision entirely instead of failing it.

Each media cluster holds its own signed answers

Refusing new viewers is still a poor outcome when the media cluster already knows the right answer. So the control plane distributes signed, versioned authority for tenants and media to every cluster that serves them. The cluster verifies each update and commits it to its own database before acknowledging it.

A decision past its soft expiry keeps serving while the cluster requests a refresh. Hard expiry, a signed revocation, or a failed signature check never turns into an allow. Because the answers live in the cluster’s database, restarting the regional load balancer does not force a round trip to the central control plane before viewers can be admitted again.

The same signed authority covers ingest ownership, pull sources, processing policy, multistream targets, and restart configuration. Changes and revocations reach each cluster independently. What still needs a live control plane is new management work, such as creating a stream, and new x402 settlement. Media that is already authorized does not.

Why an earlier viewer’s approval is not reused

There is a tempting shortcut: when the regional load balancer is unreachable, remember the last successful answer for a playback ID and hand it to the next viewer who asks for the same one. We do not do this.

An approval for an earlier viewer says nothing about the new viewer’s location, about whether the destination that viewer would be sent to still permits them, or about a tenant policy that changed since. Consider a geofenced event, or a signing key that was just revoked. Replaying an earlier yes in those cases would admit exactly the viewers the policy exists to stop. Every new viewer gets a fresh decision from the regional load balancer, based on the signed authority it holds.

What still fails

Signed local authority protects a media cluster from outages further upstream. Losing every reachable regional load balancer for a cluster is a different failure. New requests then have to reconnect or fail over before they can get a decision, and if no valid answer exists, MistServer applies the explicit denial.

That affects the admission of new viewers only. Playback that is already established is not terminated because the control plane went quiet. The honest boundary is expiry: stored authority serves through its signed validity window, and never beyond it.

The architecture overview describes how signed authority reaches each cluster, and playback access control covers the policies it carries.

DVR, Clips, and VOD: What Happens After the Live Edge

Mon, 08 Jun 2026 00:00:00 GMT

A live stream, at the edge, is a ring buffer. MistServer keeps a bounded window of segments on disk, old ones fall off the back, and when the stream ends the buffer eventually stops mattering. Everything that outlives the broadcast (the seekable archive, the highlight clip, the replay a viewer opens next week) has to be copied out of that buffer before it disappears.

Short streams make that easy to get roughly right. Long ones do not. This post follows one always-on channel through three months of recording, a plan change and a highlight clip.

A channel that never stops is one recording

Picture a 24/7 channel, such as a venue camera that nobody turns off. It goes live on the first of the month and stays live for 90 days.

On FrameWorks that is one recording. A session produces one DVR artifact with no per-artifact length cap, so a channel that streams for 90 days straight is one artifact. If the stream drops and comes back an hour later, that is a new session and a new recording.

We got there by separating three clocks that look related but are not:

The live DVR window is how far back a viewer can scrub while the stream is up. It is sized by plan and kept short so live playlists stay small.
The session is the recording itself, as long as the ingest stays live.
Retention is how long the archive survives after the stream ends, counted from the moment the session finishes. An active recording is invisible to retention: a channel on day 89 cannot lose day 1 to an expiry timer.

Underneath, every recorded segment is tracked with its position in stream time and where it is stored. Playlists are generated from that record when something needs one, so there are no manifest files in object storage that can drift out of step with the media they describe.

Replaying three months without downloading three months

Nobody wants to open a 90-day file. So replay works through chapters: bounded slices of the recording, each finalized into its own ordinary VOD asset. Chapters can match the length of the live window or follow fixed UTC intervals such as hourly or daily, so an application that wants “yesterday afternoon” converts that to a UTC range and lists the chapters in it. Each chapter keeps wall-clock timing, so a player can show the real time of day for any position.

A chapter becomes playable as soon as it is finalized, and is later uploaded to object storage, after which the temporary per-segment copies are removed. If a chapter cannot be built because its source segments were lost, it is marked failed rather than published as a shorter file that pretends to be complete.

Because chapters are ordinary VOD, they follow the same playback path, the same access policy and the same rules across clusters as any uploaded video. A remote cluster never asks for “the 90-day archive”; it asks for a chapter. If that chapter is already in object storage, the viewer gets a signed download from storage. If it is not uploaded yet but the node that recorded it still has the complete file, it is served from there. If neither is true yet, the request is refused with a retryable answer, and a later attempt succeeds once the upload finishes.

What a plan change does to an archive in progress

On day 40, the channel’s owner changes plan. The obvious question is what happens to the recording that has been running for 40 days.

Nothing, until it ends. The retention that applies to a recording is resolved when the recording starts and stored with it. A downgrade halfway through never quietly shortens an archive that is already recording, and the plan in force when the stream finally stops is never looked up again to recompute it. The new plan applies from the next session.

Re-resolving retention at the end would let a plan change reach back and shorten weeks of footage someone was counting on, so we chose to resolve it once, at the start. If you want a particular archive kept longer, you can set a retention override on that finalized asset afterwards, within your plan’s cap.

The same rule applies to chapter settings: changing how a stream cuts chapters takes effect from the next broadcast, and a recording in progress keeps the setting it started with.

Clips come from the same place

On day 63 something worth sharing happens. A clip can be cut from the live buffer, from the rolling DVR window, or from a chapter that has already been finalized. It goes through the same processing path as chapters and produces the same kind of VOD asset. A clip that fails is marked failed and shows up that way in the dashboard, instead of a spinner that never resolves.

What it costs

A long recording is only as durable as its path to object storage. Edges evict local segments under disk pressure, oldest first, because the durable copy lives in object storage. Every cleanup step deletes only after that durable copy is confirmed, so a crash in the middle of cleanup leaves extra data behind, never missing data.

The live window is also not the archive. A viewer who joins on day 90 can scrub back only as far as the live window allows; everything older is reached through chapters.

Docs: DVR chapters, clips, recordings, storage and retention, and the operator’s view in DVR operations.

Going Multi-Region: Local Latency, One Bill

Tue, 26 May 2026 00:00:00 GMT

A viewer in Virginia opens a stream. In a single-region platform hosted in Europe, the routing decision, the session bookkeeping and the event traffic for that viewer all cross the Atlantic before the first frame arrives.

FrameWorks now runs two regions, eu-west and us-east. That viewer in Virginia gets a US edge, a US routing decision, and a US event pipeline absorbing the telemetry their session produces. The tenant who owns the stream still gets one dashboard and one bill covering both regions. Most of the design work went into keeping that second sentence true.

Nothing on the viewer’s path crosses the ocean

Each region is a complete media cell. It has its own edge nodes, its own routing and load balancing, its own transcoding capacity, its own cache, its own realtime layer and its own Kafka cluster. A broadcaster pushing into us-east and a viewer watching from us-east never wait on a request to Europe: stream startup, routing and playback admission are all answered inside the region.

What stays central is the state that has to be single: the control plane (accounts, provisioning, billing, DNS), the database behind it, and ClickHouse for analytics. None of those sit on the path between a viewer pressing play and video arriving.

Why events converge in one region

The media is regional, but the questions a tenant asks are not. “How many people watched last week?” and “what do I owe?” need every region’s data in one place.

We use Kafka’s MirrorMaker 2 to copy each region’s durable event streams (analytics, service events, billing usage reports and the dead-letter queue) into a single aggregator cluster in the EU, where ClickHouse and billing already live. A narrower link in the other direction carries only the realtime events that dashboards subscribe to, so a GraphQL subscription stays connected to its nearest region and still sees changes from both. Mirroring durable data toward the consumers that need it was a much smaller commitment than teaching every consumer about every region.

graph LR
    subgraph "us-east"
        USV["Viewers and broadcasters"] --> USE["Edges and routing"]
        USE --> USK["Regional Kafka"]
    end
    subgraph "eu-west (aggregator)"
        EUK["Aggregator Kafka"] --> CH[("Analytics")]
        EUK --> BILL["Billing"]
    end
    USK -- "durable + realtime events" --> EUK
    EUK -- "realtime only" --> USK

Producers need no knowledge of the topology. A service in us-east writes to its regional pipeline and is done; the mirror moves the event later. The mesh is declared explicitly in our public cluster manifest, and the CLI refuses a manifest with a missing or misplaced link. Two regions make a full mesh cheap. Before a third region, we will measure realtime volume and decide whether broadcasting every realtime event everywhere still makes sense.

At-least-once mirrors, exactly-once bills

Mirroring is at-least-once. Regions can partition. A mirror can deliver the same event twice after a restart. Billing sits downstream of all of it, and a tenant should never pay twice for one viewer session.

Every event carries a unique ID and the region and cluster it came from. Analytics deduplicates on that ID, so a replayed or double-mirrored event collapses into the row that already exists. Events that record a state change, such as billing usage or an artifact being created, are written in the same database transaction as the change itself and shipped afterwards, so an outage in the regional event pipeline delays them without losing them. Loss-tolerant telemetry such as load-balancing samples skips that step and accepts a small loss window.

The finalized facts billing depends on, like a completed viewer session, needed one more change, and it came from a race we found rather than one we predicted. ClickHouse offers table engines that deduplicate rows in the background, which looks like exactly the right tool. It is not. When a Kafka consumer group rebalances, two parser instances can briefly process the same partition. Combine that with the timing of background merges, and a finalized fact can disappear without any error being raised.

So those facts are stored append-only, and the current truth is computed when it is read. When two writers disagree about the value of a billed fact, the disagreement is written to an audit table and raised as a metric, and supported cases become explicit corrections on the bill. Nothing is silently overwritten, and every change to what a tenant owes can be traced back to the events that caused it.

What it costs and what it does not do

Cross-region dashboard updates arrive a little later than local ones, because they travel over the mirror. Analytics for a US stream reaches ClickHouse after the mirror delivers it, not the instant it happens. Keeping one central control plane means an EU-side control-plane outage is felt in both regions for account and billing operations, although viewers already watching keep watching.

In return, a tenant sees one bill that covers usage in every region, counted once, and viewers see latency set by their own region. Viewers whose stream lives in the other region are served through federation: a nearby edge pulls the stream across once, and later viewers watch it locally.

The operator’s view, including how regions and mirror links are declared, is in multi-cluster operations and the cluster manifest reference.

An AI Agent Walks Into a Streaming Platform

Mon, 18 May 2026 00:00:00 GMT

Most video platforms assume a person is at the keyboard. Signup wants an email address and a click on a verification link, billing wants a card, and when something goes wrong the answer is a support ticket. An AI agent can do none of those things. It can hold a wallet, sign messages, call an API, and read structured errors.

FrameWorks lets an agent become a paying customer with only those abilities. It signs in with a wallet, sets up what it needs at zero balance, and pays for usage in USDC when a billable call asks for it.

One agent’s first run

Say an agent has been asked to publish a finished video on FrameWorks. It has never used the platform and has no account.

It starts with an EVM wallet. It asks FrameWorks for a one-time login message, signs it with the wallet’s key, and sends the signature back. That signature is the whole signup. The first successful sign-in creates an account tied to that wallet, with no email address and a balance of zero.

At zero balance the agent can already do a lot: read its account status, create API credentials, configure a stream, and look up what the platform offers. None of that costs anything, so none of it needs money.

Then it asks to create the upload. Processing a video is billable work, and the balance is empty, so the call comes back with HTTP 402 Payment Required and a payment offer bound to this account: the amount, the token, and the networks it can pay on.

The agent signs a USDC transfer authorization for that amount and submits it as a top-up. FrameWorks relays the transfer, so the agent does not need to hold gas, waits until the chain confirms it, and credits the balance. The agent retries the upload, the usual balance and account checks run again, and the upload is created.

No step in that run needed a person. The agent never read an email, typed a card number, or waited for support.

Why payment is part of the API

The x402 protocol gives the 402 status code a machine-readable body: what to pay, how, and where. An agent that hits a billable call without funds gets a structured answer that names the fix and carries the offer to pay it, so it does not have to parse an error message written for humans.

We settle on Base and Arbitrum. We skipped Ethereum mainnet, because a few dollars of gas per transaction makes no sense for topping up a streaming balance. Agents, or people, who prefer a plain transfer can use a deposit flow instead: ask for a quote, get a deposit address, and send ETH or USDC to it.

What a wallet-only account can and cannot do

A wallet costs nothing to create, so a wallet-only account is always prepaid: it has to fund its balance before it consumes anything billable. That is what stops someone from minting thousands of wallets and running work on credit. Adding and verifying an email address later opens the other billing options.

Payment and identity stay separate. Paying never signs an agent in, and a payment does not override a suspended account; after the top-up, the retried call goes through the same checks as any other request. Usage is metered and itemized the same way whether the caller is a person or an agent.

The cost of this model for the agent’s operator is that the wallet is the account. Whoever holds the key controls the account and its balance, so the key belongs somewhere the agent can use it and nothing else can read it.

One MCP surface for the whole platform

Once signed in, the agent works through the MCP server: tools for streams, clips, DVR, VOD, playback, quality diagnostics, billing, and support, plus read-only resources for account and stream state. A small part of it is public, covering sign-in and playback lookup, because an agent has to be able to find out how to become a customer before it is one.

Skipper, the AI video consultant, is available on the same surface for questions such as why a stream is buffering. It can search documentation and run diagnostics but cannot change anything on that path, so asking it a question never deletes a stream as a side effect. Changes go through the dedicated tools, each with its own authorization. We wrote about Skipper when it launched.

The agents overview is the place to start. Wallet authentication and agent payments cover this post’s flow step by step, including the payment networks, the deposit flow, and the errors an agent should expect.

Trusting an Edge Node on Hardware We Don't Own

Thu, 02 Apr 2026 00:00:00 GMT

With bring-your-own-compute, the machine that delivers your viewers’ video may sit in your rack, a colocation cage, or a rented server we have never logged into. That machine still has to talk to the FrameWorks control plane, and the control plane has to believe it is the edge it claims to be. The obvious shortcuts each carry a risk. A secret baked into every edge image leaks the first time one box is compromised. Handing every edge our internal certificate root turns the anchor for service-to-service traffic into a file anyone with a VM can copy. A hosted mesh coordinator puts our network topology and the power to add peers in someone else’s hands.

The design we settled on keeps two trust roots apart. Provider services trust each other through a FrameWorks-owned internal CA. Anything that faces the internet, edges included, uses public web PKI. An edge verifies the control plane with a certificate its operating system already trusts, and identifies itself with a token minted for one tenant.

An edge you don’t own never sees the internal root

Every TLS listener in FrameWorks belongs to exactly one of two issuers. The internal CA covers gRPC between provider services on the private network. Public ACME certificates cover everything a browser, a viewer, or an edge node touches, including per-tenant custom domains, which get the same issue-and-renew lifecycle.

The control-plane listener that edges connect to presents a publicly trusted certificate for the cluster. A customer edge therefore needs no FrameWorks CA material at all: it does not receive the provider trust root, does not join the backend mesh, and holds no federation credentials. That listener must never present the internal certificate, so in production the service refuses to start if the public certificate is unavailable, rather than serve the wrong identity on a public port.

Clients on our side are just as strict. Each one dials with the address to connect to and the identity the server must prove as two separate inputs. A DNS change can send a client somewhere else, but it cannot make the client accept a certificate for a different name.

How an edge proves who it is

A new edge presents an enrollment token. The token is minted for one tenant, can be scoped to a specific cluster, and can carry a usage limit for provisioning many nodes at once. Tokens are consumed atomically, so a single-use token that has already been spent cannot be replayed by whoever finds it later. A node that has enrolled before is recognized from its network identity and hashed hardware identifiers and does not need a new token.

After enrollment, every call from the edge carries a credential that the control plane checks and logs. Authentication rides on top of server-authenticated TLS rather than inside it. The only unauthenticated surfaces are health checks and the enrollment entry point itself, which validates the token before doing anything else.

The edge connection guide walks through creating a token and enrolling a node.

Why we did not use Tailscale

Inside the provider plane, backend nodes talk over a WireGuard mesh. The question was who coordinates it. The mesh coordinator decides which keys count as trusted peers, which makes it part of the trust root: whoever runs it can add a node to the network.

We looked at Tailscale first. Its coordination plane is a hosted service, so our topology would be visible to a third party, and pricing is per device. Headscale removes the hosted dependency but places an external project at the center of our trust model. Both would have put the decision about who joins our network outside the platform operator’s control.

So mesh coordination is first-party. A small agent on every backend node builds the WireGuard configuration from three sources: the host’s static identity in the deployment manifest, a set of seed peers for bootstrap, and peers, mesh DNS, and certificates synced from the control plane. It caches the last known mesh locally, so a node that reboots while the control plane is down comes back with the peers it had. Hand-written tunnel configs are not a supported input, so every peer traces back to the manifest or the control plane. The manifest itself is public, with host addresses and keys encrypted.

What this model does not do yet

Authentication is tokens over TLS, not transport mTLS. We have not built per-edge client certificates or SPIFFE-style workload identity; per-edge client certificates are an open design item. What exists today is encrypted transport with a pinned root on the inside, publicly verifiable certificates on the outside, and a credential on every call that can be checked and logged.

Shared and dedicated provider capacity use the same architecture. Teams that self-host the entire stack run their own CA, mesh, DNS, and control plane as an independent platform with its own trust root; that platform does not join the FrameWorks network.

Operators can find the setup details in mesh networking, DNS and certificates, and the architecture overview.

Our Production Topology Is Public on Purpose

Mon, 16 Mar 2026 00:00:00 GMT

Last week we merged our private gitops repo into the public one. The commit message is one line: “merge gitops-private for deployment transparency.” Since then, the FrameWorks gitops repo has contained our actual production deployment: every service, database, message broker, cache, and host-to-role mapping, in plaintext, for anyone to read.

The position follows from how we operate the product. FrameWorks publishes the topology of its managed provider plane rather than hiding how production is wired. The cluster manifest that drives our deployment is published as-is, and each cluster in it carries an explicit public_topology: true flag. The same frameworks CLI and GitOps layout are what anyone self-hosting the entire stack uses to deploy and upgrade their own independent platform.

The boundary

The repo splits into two kinds of files.

Plaintext: the cluster manifest (topology, service placement, database and broker wiring, per-service configuration), the edge-node topology, the non-secret operator environment, and the release manifests with their channel pointers. SOPS-encrypted files also keep their YAML keys and structure visible, so you can see that there’s a WireGuard private key per host and what the file layout is, just not the value.

Encrypted: the hosts file (server addresses, SSH targets, WireGuard private keys) and the secrets env files (API keys, passwords, tokens). Everything encrypted uses SOPS with age keys.

The interesting property is what the plaintext half enables. Anyone evaluating FrameWorks can answer “how is this actually deployed” from the source instead of from a marketing diagram. An operator setting up their own cluster has our production manifest as a working reference for the manifest format. And our own deployment discussions happen against a file everyone can see.

The guards

A public repo with an encryption boundary is one careless commit away from being a public repo without one. Three pre-commit hooks enforce it:

No plaintext secrets. Staged secrets files must actually be SOPS-encrypted: env files need the SOPS version marker, encrypted YAML needs its sops: metadata block. A decrypted file that someone forgot to re-encrypt fails the commit.
No leaked addresses. Staged plaintext manifests are grepped for IP addresses that belong in the encrypted hosts file, with only documentation ranges and loopback allowed through.
No age keys. Any staged file containing an age secret-key prefix fails the commit, so a private key can’t ride along in a config or a note.

Secret edits go through a small wrapper script that decrypts to a temp file, applies the change, and re-encrypts. Nobody hand-edits encrypted files. Key management is documented in the repo itself: adding a teammate is a keypair plus sops updatekeys; a compromised key means removing it and rotating every value it could decrypt, on the theory that once the ciphertext is public, revoking the key alone protects nothing.

What it costs

The downside: our infrastructure is easier to case. An attacker can read which services exist, where the trust boundaries are, and what a valuable target would be. We accepted that because most of it is discoverable anyway: port scans, DNS, TLS certificates, and job postings reconstruct a topology diagram slowly but reliably, while the part that gives an attacker traction (addresses, credentials, keys) stays encrypted. What we got in return is a deployment reference that can’t go stale, since the deploy tooling reads the same file you do.

FrameWorks infrastructure operators start from the deployment guide; customer edge operators should use the hybrid edge guide.

Federation: Serving a Stream From a Cluster That Doesn't Have It

Wed, 18 Feb 2026 00:00:00 GMT

A FrameWorks stream is pushed into exactly one cluster. Viewers show up wherever they are. For a long time those two facts forced an awkward choice whenever a viewer’s best edge lived in a cluster that had never seen the stream: send the viewer somewhere worse, or copy the stream somewhere it might not be needed.

As of the v0.1.0 cycle, clusters that serve the same tenant talk to each other directly. Each one knows which live streams its peers hold, and when a viewer needs a stream that lives elsewhere, the nearby cluster pulls it once and serves it locally. The stream is copied only when someone is watching it there.

One viewer, two clusters

Take a broadcaster in Amsterdam who pushes over RTMP into a cluster in Europe. Their tenant also has access to a second cluster in North America. A viewer in Toronto opens the stream.

The North American cluster does not have the stream, but it already knows who does. Peer clusters continuously advertise the live streams they hold, so the viewer’s cluster can answer “where is this stream” from its own memory rather than by asking a central service on every play.

Before it uses that answer, the viewer’s cluster checks that the origin is still a peer the tenant is allowed to use. That check is made fresh, so a cluster whose access was revoked cannot keep supplying streams on the strength of an old record.

It then asks the European cluster for a good source edge. The origin scores its own edges and replies with candidates. One of the North American edges opens a pull from that source over DTSC, MistServer’s native server-to-server transport. The control plane only decides which node pulls from which source; the media moves directly between MistServer instances.

The viewer in Toronto waits for that pull to start, which is the setup cost of the first cross-cluster view. The second viewer in Toronto, and the hundredth, are served from the same North American edge with no additional traffic across the ocean.

graph LR
    subgraph "Cluster A (origin, Europe)"
        PROD["Broadcaster"] --> EA["Edge A"]
    end
    subgraph "Cluster B (North America)"
        EB["Edge B"]
        V["Viewers in Toronto"]
    end
    EA -. "stream advertisements" .-> EB
    EB -- "one DTSC pull" --> EA
    EB -- "HLS / DASH / WebRTC" --> V

A copy is made once, from the origin only

The obvious failure of a system like this is a cascade: every cluster opening its own pull, or clusters pulling from each other’s copies until nobody knows where the original is.

Two rules prevent that. When a cluster starts pulling a stream, it tells its peers, so other clusters can see that a copy exists and where it came from. And replication is strictly single-hop: an edge pulls from the origin cluster, never from another cluster’s replica. A pull chain has no path along which to loop.

Federation state is also built to expire rather than to be cleaned up. If a peer crashes or is cut off, its advertisements stop and its streams drop out of the directory, so nobody is routed toward a copy that no longer exists. When a peer is unreachable, requests to it fail quickly with an explicit error and the caller falls back to whatever local answer it has, instead of blocking a viewer on a dead connection.

Who pays for the extra hop

Cross-cluster serving raises an accounting question that had to be answered precisely. Every event is stamped with the cluster that did the work and with the stream’s origin cluster. Delivery is billed to the cluster that actually served the viewer, so a tenant’s bill reflects where their audience watched. The DTSC traffic between clusters is our infrastructure cost and never appears as a tenant line item. If a rated usage record ever arrives without a cluster, billing refuses to guess and fails closed.

Routing decisions land in analytics with the serving and origin clusters recorded, so cross-cluster serves are visible in the dashboard rather than hidden inside an aggregate.

What federation does not decide

Federation answers “can this cluster get the stream”. It does not by itself decide which edge a viewer should use.

The limits are straightforward. The first viewer in a new cluster pays the setup latency of one pull. Federation only runs between clusters that share a tenant, so a stream cannot spread into a cluster its owner has no relationship with. And because replication is single-hop, every cluster that needs the stream pulls from the origin, which carries one outgoing copy per serving cluster.

How edges are chosen across clusters has changed since this post. Media placement now compares eligible edges in every permitted cluster by health, capacity and distance to the viewer, and lets accounts state rules such as “my own servers first”. Your Stream, Your Edge explains that change.

For the operator’s view of peering, artifact routing across clusters and billing attribution, see multi-cluster operations. For how routing choices are made and previewed, see the media placement guide.

Skipper: An AI Video Consultant With Real Diagnostics

Sat, 07 Feb 2026 00:00:00 GMT

A question we see often, in some form, is “my viewers keep buffering, what’s wrong?” No docs page answers that. Answering it means looking at the stream (rebuffer ratio, packet loss, which edges the viewers landed on, whether the encoder is keeping up) as well as the documentation.

Skipper does both. It is an AI video consultant built into FrameWorks: it searches curated streaming documentation, runs diagnostics against your live streams, generates API operations from the current schema, and, in authenticated dashboard chat, can manage streams and billing for you.

One question, answered with your stream’s data

Take the buffering question. Ask Skipper “why is stream X buffering” and it pulls the stream’s real numbers: rebuffering patterns, packet loss, buffer health, and where viewers were routed. The diagnosis that takes 30 minutes of dashboard-hopping by hand comes back as one answer with the evidence attached, so you can see whether the problem is the encoder’s uplink, one region’s viewers, or a player falling behind.

When the answer is a configuration change, Skipper looks it up before it tells you. Knowledge questions go to a curated corpus that covers FrameWorks, MistServer, FFmpeg, OBS, SRT, HLS, DASH, WebRTC, nginx-rtmp, and Livepeer, and source-backed answers include citations. Topics the corpus does not cover fall back to web search where one is configured.

Ask “give me a Python script that creates a stream” and it reads the live GraphQL schema at query time, generates the operation, and wraps it in code. The output matches your current API rather than an API shape from the model’s training data.

In authenticated dashboard chat it can also act: create streams, refresh keys, clip the last 30 seconds of a broadcast, start a DVR recording, upload VOD assets, check your balance, or start a supported crypto top-up. Each action is a call to the same GraphQL API you would use directly, and each one shows up in the chat as a card with what ran and what came back.

It tells you when it is guessing

A consultant that sounds equally sure about everything is hard to use, because you cannot tell which part to check. Skipper tags each section of an answer with one of four levels:

Verified: grounded in the curated knowledge base, with source links.
Sourced: backed by live web search results.
Best guess: from the model’s general knowledge, not checked against a source.
Unknown: nothing supporting it was found.

Mixed answers are the common case. In that Python snippet, the GraphQL operation comes from the real schema, while the Python around it is model-written glue code and is tagged as a best guess. The tag tells you where to spend your review time. Skipper is also instructed never to guess CLI flags, codec parameters, or configuration values without a source, which is where a confident wrong answer does the most damage.

Your agents get the same access, and no more

Skipper connects through the same MCP gateway that external agents use: the same GraphQL API, the same tool catalog, the same authorization checks. It has no private path into the platform. Anything it can do in your account, your own agents can do too, and anything your account cannot do, Skipper cannot do either.

That boundary shapes where it runs and what it may change:

Dashboard: authenticated chat with account-scoped diagnostics and management actions where enabled.
Docs site: a chat panel on every page (Cmd/Ctrl+J) that can explain and inspect but never modifies anything.
Your own agents: any MCP client can call ask_consultant for the full answer with confidence tags. Changes are blocked on that path; an agent that intends to create or delete something calls the dedicated platform tools, which carry their own authorization.

Keeping changes out of the consultant path means an agent asking Skipper a question cannot end up altering your account as a side effect of the answer.

Watching streams when nobody is asking

Where it is enabled, Skipper also checks the health of your active streams and infrastructure on a schedule. It learns what normal looks like for your account and runs a full investigation only when something departs from it; the report comes back with a root cause and recommendations over email or the channels you have configured. Accounts whose streams are healthy use no model calls at all during these checks, so monitoring does not turn into a background token bill.

What it does not do

Skipper is a consultant, not an operator with its own authority. It can only act where your account can, on the dashboard, and with the tier and rate limits of your plan. Standard usage billing applies to stream operations it performs on your behalf. Its answers are only as current as the knowledge base and the schema it reads, which is why the confidence tags exist: verify a best-guess section before acting on it.

The Skipper guide covers where to find it, what it can do, and how agents call it. Operators running their own deployment can pick models, add knowledge sources, and tune retrieval and monitoring in the operator guide.

What Seven AI Agents Found in Our Streaming Platform

Sat, 31 Jan 2026 00:00:00 GMT

Live video systems collect awkward edge cases. A viewer connects from a location that sits on the wrong side of a geofence boundary. A DVR recording starts at the same moment a stream is shutting down. Two usage events for the same tenant arrive close enough together that billing code has to prove it is actually idempotent. None of those cases are hard to understand on their own. The failures come from these cases interacting across services.

FrameWorks is split across routing, ingest, VOD, edge orchestration, analytics, billing, auth, MCP agent access, DNS, Skipper, WebSocket routing, and tenant management. Each subsystem has tests and reviews, but most of the failures we found sit between them.

A finding without evidence did not count

We ran the first audit pass by hand: seven roles, 26 batches, 90 tasks, 12 subsystems. One agent traced a domain, another checked the report, another wrote a fix if the finding held up, another reviewed the patch. CI and a human reviewer still owned the merge.

Every finding had to include evidence. A report had to name the files and functions, describe the failing input or interleaving, explain the tenant or operator impact, and say what test would prove the fix. “This might race” went nowhere. “Two prepaid usage reports can reserve against the same tenant balance before the ledger state is visible” was at least reviewable.

That rule removed most of the noise. Agents had to keep reading until they could point at the code instead of reporting general impressions of the codebase.

Most failures sat between subsystems

The findings were mostly the kind you expect after a platform gets broad enough:

Category	Count	Example
Race conditions	12	Prepaid balance deduction under concurrent Kafka messages
Tenant isolation	8	Stream context cache keyed by `internal_name` only, missing `tenant_id`
Data loss risks	7	Event ingest batch flush interrupted by producer crash
Stale cache behavior	6	GeoIP cache stampede under bursty traffic
Protocol edge cases	5	Player protocol blacklist leading to “no playable protocol” dead-end
Storage consistency	5	S3 upload succeeds but local delete fails, leaving duplicate artifacts
Auth bypass vectors	4	GraphQL complexity bypass through deep nesting with small page sizes
DNS propagation	3	Stale DNS pointing at decommissioned nodes

Some of the best results were not bug fixes. A reviewer would chase a claim and find the lower layer already enforced the invariant. That turned a hidden assumption into something we could preserve during the next change.

Agents traced well and overstated severity

The agents were good at long, boring traces. GraphQL to gRPC to a service event. Kafka to ClickHouse to the billing ledger. Load balancer state to the edge’s playback rewrites to the player fallback path. A person can do that too, but it is easy to stop when the first function looks fine; the agents kept following each path to the end.

The weak point was severity. LLMs like to make findings sound bigger than they are. The review pass had to push hard on impact: can a tenant hit this, does it cross a trust boundary, is there data loss, is it only a noisy log, is the bad state already repaired one layer later?

The first run also taught us what a finding report should look like. Early reports came padded with process notes and architecture recaps, which did not help decide whether a bug was real. The format we settled on has one paragraph for the claim, the exact files and functions, the failing interleaving or input, the tenant impact, the smallest reasonable fix, and the test that proves it. Batch size mattered too. Three tasks per batch kept the human reviewer ahead of the queue; bigger batches just moved the bottleneck to whoever had to merge.

What changed in our reviews

The lasting change was in review questions. We now ask more often whether a cache key includes tenant_id, whether a handler survives Kafka retry, whether stream lifecycle events can arrive out of order, and whether an edge can keep serving a previously approved playback path when the control plane is temporarily out.

Next up is lightweight automation: track when each subsystem was last audited, weight the riskier areas like billing and auth, and open work only when there is a useful review to run. Most audit cycles should work like Skipper’s automated monitoring: check the current state and open work only when something needs attention.

Welcome to the FrameWorks Blog

Fri, 06 Dec 2024 00:00:00 GMT

Most streaming platforms keep their architecture behind closed doors. You get API docs, a dashboard, and a support email. How the ingest pipeline actually works, how viewer routing decisions are made, how billing enforcement interacts with stream lifecycle: that stays internal.

We publish ours. If you run live video on a platform, you should be able to see how it works: the API surface and the design decisions underneath it.

This blog is where we write about that. Expect architecture deep-dives into how subsystems are designed and the trade-offs behind them, technical explanations of what shipped and how it works under the hood, and notes on how we test, audit, and operate a multi-tenant streaming platform.

We publish when something ships or when a design decision needs explaining.