Media Storage & Playback Paths

This page is for operators sizing storage, debugging playback, or deciding how much S3 egress to budget. It explains the three storage tiers, how each media type moves between them, and which paths are S3-backed vs Mist-native.

For DVR-specific operations (tier ceilings, chapter sweeper, retention), see 24/7 DVR — Operations.

Storage Tiers

Tier	What lives here	Persistence	Cost driver
S3	Durable archival copy of every clip, DVR segment, and processed VOD	Forever (until retention)	Storage + egress
Edge disk	Per-node warm cache (recent + active artifacts, relay block cache)	Until evicted under pressure or retention	Local SSD capacity
Mist shm	Mist’s in-process look-ahead window (~120s) for active playback	Per-session	Edge RAM

S3 is the only authoritative tier. Edge disk and Mist shm are caches/working memory — they can disappear at any time without data loss. Helmsman’s admission and cleanup logic keeps edge disk usage bounded by an intent-priority hierarchy (active recordings and processing outputs evict viewer caches first).

The Two Lanes: Live vs Archival

FrameWorks splits playback into two completely separate transport paths. Operators monitoring bandwidth, S3 egress, or edge-disk pressure need to know which lane drives which metric.

Live lane — Mist-native, no S3

• Cross-node ingest distribution: Mist DTSC (origin Mist pushes to, or peer-pulls between, edge Mist instances).
• Active stream playback: viewers pull HLS / LL-HLS / WebRTC / DASH from the nearest edge that has the DTSC pull running.
• Active DVR playback (the rolling DVR surface for an in-flight recording): the origin node owns the local rolling manifest; other edges DTSC-pull dvr+<dvr_internal_name> from the recording origin. No S3 reads — the per-segment S3 objects exist as recovery-bridge durability only, not playback infrastructure.

S3 sees zero traffic on the live lane.

Archival lane — relay through S3 → edge disk → Mist

• VOD playback, clip playback, finalized DVR chapter artifact playback, and processing job input all flow through Helmsman’s read-through artifact relay at http://<edge>:18007/internal/artifact/.... Chapter artifacts are ordinary VOD artifacts (origin_type='dvr_chapter', library_visible=false) — they share the VOD code path completely.
• Mist sees a stable HTTP URL per artifact and reads it like any other HTTP input source. Behind that URL Helmsman materializes bytes from edge disk (warm), or from S3 into edge disk (cold + healthy), or from S3 straight to socket (cold + memory-pressured).
• The relay route is internal to each Helmsman; viewers and Mist never talk to it directly. It is mounted on the existing Helmsman listener; it is not a MistServer trigger and won’t fire STREAM_SOURCE.

                    ┌───────────────────────┐
   live ingest ─DTSC▶│ Mist (origin/edges) │── live playout ─▶ viewers
                    └───────────────────────┘
                                ▲
                     active DVR │ (recorder owns local bytes)
                                │
                    ┌───────────────────────┐
   clip / VOD /   ──▶│ Helmsman relay        │── stream ─▶ Mist ─▶ viewers
   finalized DVR    │   /internal/artifact/  │           (shm look-ahead)
   playback         │   S3 → disk cache      │
                    │   stream-first tee     │
                    └───────────────────────┘
                                ▲
                                │ S3 range GET (presigned)
                                │
                            ┌───────┐
                            │  S3   │
                            └───────┘

Per-Asset Behavior

Live stream

• Ingest lands on the origin node’s MistServer.
• Distribution to other edges is Mist DTSC; no S3 involvement.
• Recording (if DVR is enabled for the stream) writes segments to the origin’s local disk and the freeze pipeline uploads each segment to S3 in the background.
• Viewer playback comes from any edge with the DTSC pull running.

S3 egress: none. Edge disk: only on the recording origin, bounded by the DVR rolling window.

Active DVR (recording in progress)

The active DVR stream is its own Mist playback identity (dvr+<dvr_internal_name>), distinct from the live ingest stream (live+<playback_id>). It is fed by the rolling HLS manifest that Mist’s HLS push writes to the recording origin’s disk (storage/dvr/<stream>/<dvr_hash>/<dvr_hash>.m3u8).

• Foghorn detects recording state on viewer resolve and routes the public DVR playbackId to any healthy edge.
• On the chosen edge, Mist’s STREAM_SOURCE resolves dvr+<dvr_internal_name>: the recording origin returns the local rolling manifest path; any other edge returns dtsc://<recording-origin>/dvr+<dvr_internal_name> and pulls the rolling stream via DTSC.
• Viewer playback fan-out happens at the chosen edge’s outputs, the same way it does for live streams. The recording origin is not a per-viewer bottleneck.

Chapter playback (finalized chapter artifacts)

Chapters are produced by the finalization queue at each boundary close as canonical .mkv VOD artifacts:

• The processing job runs on the recording origin, reading source TS segments from local disk (or fetching from the recovery-bridge S3 when local copies are gone) and pushing a single MKV.
• The resulting artifact lives at the standard VOD prefix and serves through the standard archival lane.
• library_visible=false keeps the chapter artifact out of tenant-wide VOD library listings even though it shares the VOD storage tree. The Commodore VOD registry row still carries stream_id, origin_type='dvr_chapter', and origin_id=chapter_id, so stream-scoped VOD artifact queries can include finalized chapters without joining through Foghorn.
• Policy enforcement walks chapter artifact → parent DVR (origin_type='dvr_chapter' + origin_id=chapter_id) so a protected DVR’s chapters inherit the parent’s auth policy transparently.
• The recovery-bridge S3 segment objects are deleted by the chapter reclaim sweep once the chapter artifact + .dtsh are durably on S3 (state='frozen').

Clip

• Always created from a finalized stream window. The clip artifact has a single canonical file (clips/<hash>.<container>).
• Cold playback flows through the relay’s block cache: edge disk fills in 32 MiB blocks aligned to the file, only the blocks covering the requested byte range are fetched.
• .dtsh index sidecar generated by Mist on first cold play and persisted alongside the media so subsequent cold plays skip the per-block header scan.

VOD (uploaded video)

• Uploads land in S3 first (direct upload via presigned PUT) and are registered with Foghorn.
• Playback identical to clip: read-through relay, 32 MiB block cache on edge disk, .dtsh sidecar on first cold play.
• Processed VOD output (the canonical container produced by a processing job) replaces the original upload for playback once the new file is durably synced to S3. Until sync completes, the relay refuses to serve the artifact from S3 and falls back to warm-node lookup; this prevents an edge without a local processed copy from accidentally streaming the original raw upload.

Processing job input

• Wrapper-safe containers (mp4, mov, mkv, webm, ts) read directly from the relay’s upload path. Admission forces memory-only — the relay pass-through streams from S3 to Mist’s process input without writing to edge disk. A multi-GB upload contributes 0 bytes to edge disk during a remux; only the output ever lands locally.
• Wrapper-unsafe containers (avi, flv, m4v) cannot be opened by Mist over HTTP. Helmsman fully stages the upload to local disk first, then runs the job against the staged file. Staged files become cleanup- eligible after the job completes.
• Processing output is pushed to local disk by Mist and only registered with Foghorn after the job finishes. The relay never serves a job’s output while it’s still being written.

Edge Disk Pressure: Eviction Order

Helmsman’s storage admission ranks intents by priority. When disk fills and a high-priority write needs room, low-priority bytes are evicted first:

Intent	Priority	Behavior under pressure
DVR recording	Highest	Must get disk. Evicts everything below to make room.
Processing output	High	Must get disk. Evicts everything below to make room.
Unsafe wrapper staging	High	Disk required (Mist’s FLV/AV inputs are file-only).
Playback cache (relay)	Medium	Opportunistic; degrades to memory-only above ~95% full.
Processing input	Memory-only by default for safe wrappers; doesn’t compete for disk.
Warm/aged cache	Lowest	First to evict.

Operationally: a single very large VOD or DVR chapter can never starve a live recording, but a sustained burst of cold VOD playback can exhaust edge disk and degrade to S3-direct (memory-only) until cleanup catches up. The two relevant metrics:

• storage_node_out_of_space lifecycle events from Helmsman — recording or processing wrote-then-failed because cleanup couldn’t keep up.
• Relay block cache hit ratio vs cold fetch latency at the edge — rising cold-fetch share usually means pressure-driven memory-only or routine cache turnover; the eviction pass treats relay block caches as the priority-zero candidates before touching warm canonical files.

S3 Layout (quick reference)

s3://bucket/clips/{tenant_id}/{stream_internal_name}/{clip_hash}.{container}
s3://bucket/vod/{tenant_id}/{vod_hash}.{container}
s3://bucket/dvr/{tenant_id}/{stream_internal_name}/{dvr_hash}/segments/{segment_name}
s3://bucket/uploads/{tenant_id}/{upload_hash}.{container}

Sidecars (.dtsh, .gop) live next to their media object with the sidecar suffix appended. Don’t hand-manipulate these prefixes — use the artifact APIs and the freeze flow so Foghorn’s ledger stays consistent.

Storage Backend Identity & Adoption

Each media cell is bound to exactly one S3 backend, described by a four-part tuple: bucket / endpoint / region / prefix. That tuple is the cell’s identity and it is immutable — repointing a live cell at a different bucket or prefix would strand every artifact already written under the old descriptor, so the platform refuses it.

Three components hold a copy and they must agree:

• Quartermaster persists the descriptor on the cluster row (infrastructure_clusters.s3_bucket/s3_endpoint/s3_region/s3_prefix). This is the authority.
• Chandler reads the full descriptor tuple from Quartermaster (GetCluster) at boot and serves under it. There is no env-serving fallback: without a usable backend Chandler starts but is not ready — it answers /ready with 503 and serves 503 for assets. Only credentials come from env. /health reports process liveness (the process is up); /ready proves this instance can read its immutable backend by successfully reading a known readiness sentinel object (thumbnails/.readiness) — a real, provisioned object, so an absent or access-denied response cannot look ready, and it needs only object-read permission (no bucket listing). /ready is the check that gates deployment and service discovery — a Chandler that is up but cannot read its bucket is never rolled out or advertised as serviceable. Foghorn writes the sentinel at boot (it owns writes to the bucket); Chandler only reads it, so there is no request-time Foghorn handshake.
• Foghorn establishes an immutable cell identity on first boot from the Quartermaster descriptor: it adopts the descriptor and refuses to start if its env disagrees. It does not self-establish from STORAGE_S3_* env — an S3-enabled cell whose Quartermaster row carries no descriptor fails closed until the descriptor is established.

The Quartermaster descriptor is established by the desired-state bootstrap: the operator declares the tuple (bucket/endpoint/region/prefix) in the manifest cluster config, and provisioning writes it onto the Quartermaster cluster row.

Credentials (STORAGE_S3_ACCESS_KEY / STORAGE_S3_SECRET_KEY) are never part of the descriptor — they stay in per-cluster env files.

The pre-deploy gate

frameworks cluster upgrade refuses to move an S3-enabled Foghorn whose cluster declares no descriptor: without one, Chandler has no backend and Foghorn would establish an unproven identity. The gate also refuses a Foghorn whose env descriptor disagrees with the cluster row (a repoint). Populate and reconcile the descriptor before upgrading.

Quartermaster is the sole source of truth for the descriptor: set it (bucket/endpoint/region/prefix) on the cluster in your source gitops manifest so the Foghorn pre-deploy gate agrees. cluster upgrade (and cluster upgrade --all) moves every host a service runs on, not just the first.

Dev

S3 is opt-in in the compose stack: STORAGE_S3_BUCKET is empty by default (storage disabled). Set the STORAGE_S3_* variables in .env to bind the local cell to a backend. The storage initializer writes the descriptor tuple (bucket/endpoint/region/prefix) onto the Quartermaster cluster row — the authority. Foghorn then adopts that descriptor as its immutable cell identity at first boot, and Chandler reads the same descriptor from Quartermaster to serve; only the access/secret credentials come from env, for both. The example CLI manifest (cli/examples/cluster-dev.yaml) shows the cluster-descriptor shape on the demo-media cell.

Multi-Cluster Sourcing

When a viewer’s edge isn’t the origin cluster for the artifact, the relay’s resolve step routes through Foghorn’s federation client. Origin returns one of two URL shapes:

• Synced: a presigned URL against the artifact’s home S3 bucket. The edge streams from that S3 directly, not from the home cluster’s Helmsman.
• Hot-but-unsynced: a short-lived peer-relay URL pointing at the origin node’s Helmsman, with an opaque capability grant (5min TTL, bound to that specific node + artifact + allowed media/.dtsh paths). The edge fetches blocks directly from that origin node with the grant id as Authorization: Bearer, and caches them locally identically to the S3 path. Once S3 sync catches up, the next cold miss naturally falls back to the presigned S3 URL.

The grant is an opaque random id, not a signed token: the serving edge holds no key. When it receives the pull it asks its own Foghorn to authorize the grant (against the node, artifact, and path the grant was minted for). This keeps S3 credentials and signing keys inside their home cluster — and off the edges entirely — while still letting freshly-recorded artifacts serve viewers across the federation immediately; a grant’s access ends when its short TTL expires (there is no explicit revoke).

Operationally, this means cross-cluster playback adds S3 egress at the home cluster’s S3 region but does not add cross-cluster bandwidth between Helmsman processes. For repeated cross-cluster playback of the same artifact, the requesting edge’s block cache absorbs it after the first cold pass.