24/7 DVR — Operations

What you’re operating

FrameWorks ships a continuous DVR archive: one DVR artifact per stream session, with no per-artifact lifetime cap. Live viewers seek across a tier-bounded rolling Mist window. Replay viewers navigate the recording sliced into chapter VOD artifacts that the chapter finalization queue produces from the per-segment ledger (foghorn.dvr_segments) at each boundary close. Chapter playback uses the chapter artifact’s normal VOD playback path — there is no dvr+<chapter_id> token anymore.

If you’re triaging a DVR issue, the engineering reference is docs/architecture/dvr-continuous-archive.md. This page is the operator-facing surface.

Cluster ceilings — env vars

The Foghorn process running in each cluster reads two env vars to ceiling the DVR window/manifest size for tenants on that cluster:

Env var	Meaning	Default
DVR_CLUSTER_MAX_WINDOW_SECONDS	Maximum live DVR window any tenant on this cluster can request, in seconds	0 (no cluster cap; tier ceilings stand)
DVR_CLUSTER_MAX_ENTRIES	Maximum HLS playlist entries (rolling-window manifest size)	0 (no cap)

Set these in the cluster’s gitops env file. Topology assumption: one Foghorn process per cluster, so process env IS the per-cluster surface.

Enterprise tenants have dvr_allow_cluster_extension=true in their tier entitlements — for them the cluster setting can raise their max DVR window up to the platform ceiling (72h). For non-enterprise tenants the cluster setting only acts as a ceiling.

Tier defaults

Resolved at DVR start by pkg/dvrpolicy.Resolve. Snapshot stored on foghorn.artifacts.dvr_window_seconds so tier changes mid-stream don’t affect the in-flight session.

Tier	Default window	Max window	Segment duration	Max manifest entries
Free	30 min	1 h	6s	600
Supporter	2 h	6 h	6s	3,600
Developer	4 h	12 h	6s	7,200
Production	4 h	1 d	12s	7,200
Enterprise	4 h	3 d (cluster opt-in)	24s	10,800

Defined in api_billing/internal/bootstrap/catalog/billing_tiers.yaml as tier entitlements.

Chapter pipeline overview

Three Foghorn-side workers run the chapter lifecycle:

Worker	Cadence	Job
chapter_sweeper.go	60s	Rotates chapter boundaries on active DVRs: closes the open chapter, opens the next.
chapter_finalization_queue.go	30s	Picks up state='closed' chapters and dispatches a dvr_chapter_finalize processing job to the recording origin.
chapter_reclaim_sweep.go	60s	Once a chapter reaches state='frozen', deletes its source TS segments locally and the recovery-bridge S3 objects.

Chapter state machine

open → closed → finalizing → finalized → frozen → reclaimed
           ↓        ↓
           └───→ failed_source_missing | failed_permanent

State	Meaning
open	Recording in progress; the rolling DVR surface serves viewers.
closed	Boundary reached; finalization queue will pick it up.
finalizing	Processing job in flight on the recording origin (Mist remuxes TS → canonical .mkv).
finalized	PUSH_END fired; chapter artifact exists locally. Waiting on freeze + .dtsh sync.
frozen	Chapter artifact + .dtsh durably on S3. Safe to reclaim source segments.
reclaimed	Source segments deleted; row remains as range metadata. Playback uses the canonical .mkv.
failed_source_missing	Recovery exhausted — at least one source segment was missing from both local and the recovery-bridge S3.
failed_permanent	Unrecoverable input (max retries exceeded, ledger invariants violated).

Active DVR durability boundary

Per-segment S3 uploads stay as a recovery bridge, not playback infrastructure. Active DVR playback always reads the local rolling manifest on the recording origin (other edges DTSC-pull from that origin). The S3 segment objects exist only so that chapter finalization can recover from local segment loss (disk corruption, eviction edge case) and so the recording survives a recording-node loss until the chapter finalization queue produces the canonical .mkv.

Recording-node loss before chapter finalization can lose history depending on S3 recovery state; recording-node loss after a chapter reaches state='frozen' cannot.

Artifact statuses (DVR state machine)

requested → starting → recording → finalizing
              ↓             ↓
              └────────────→ completed | completed_partial | failed → deleted

Status	Meaning
requested / starting	Sidecar accepting DVR start; transient
recording	Active recording; Mist push running, segments writing to ledger
finalizing	Stream ended; Foghorn running bounded retry on pending segments before classification
completed	All segments uploaded; archive complete
completed_partial	Some segments are lost_local; chapter finalization recovers from S3 where it can, otherwise the chapter is marked failed_source_missing
failed	No playable segments; recording produced nothing usable
deleted	Past retention; soft-deleted by the retention job

Triage: start with cluster diagnose dvr

For any DVR problem, first read the recording as its origin cell’s Foghorn sees it:

frameworks cluster diagnose dvr <dvr_hash>

Commodore resolves the hash to the recording’s origin cluster and the command asks the manifest Foghorn entry that serves it (--service foghorn-eu picks an entry explicitly; --output json returns the full report). It needs a platform-operator login (frameworks login) and reaches Commodore and Foghorn through the context’s SSH tunnel. The report shows:

• the recording: status and error, start/end, storage location, sync status with last attempt, node, failure count, and error, .dtsh state, retention_until, and the chapter policy snapshot;
• the segment ledger: count, first/last media time, counts per segment status, and the gaps (first 100 plus the total), each with the sequence it precedes;
• every chapter: state, [start, end) bounds, finalize attempts, latest finalize job ID, assigned node, and last failure reason;
• the finalize queue: chapters in closed or finalizing with queued/started times.

The sections below use that output. The GraphQL dvrChapters(dvrId: "<dvr_hash>") query is the tenant-facing view of the same chapters; use it to confirm what a viewer’s application sees.

Triage: what completed_partial means

A completed_partial artifact has at least one segment that was force-evicted from local disk before the sidecar could upload it (disk pressure or retention edge case during recording). Chapter finalization first tries to recover those segments from S3; chapters that overlap an unrecoverable segment are marked failed_source_missing and produce no playback artifact (operator triage path).

To inspect, run frameworks cluster diagnose dvr <dvr_hash>. The segment summary counts segments per status (force-evicted segments appear as lost_local) and lists the gaps in the ledger; chapters in failed_source_missing overlap a lost segment, with the reason in last_failure. Through the GraphQL API, dvrChapters(dvrId: "<dvr_hash>") shows the same chapters with hasGaps: true or state: FAILED_SOURCE_MISSING.

Cross-reference with disk-pressure metrics on the recording sidecar (storage_node_out_of_space lifecycle events) to find root cause.

Triage: chapter not appearing for a viewer

1. Confirm the recording has historical chapters enabled. The chapter mode is snapshotted from the Stream (Stream.dvrChapterMode) at StartDVR. If dvrChapters(dvrId: "<dvr_hash>") returns no chapters for a recording that has been running longer than one chapter interval, the recording was started with historical chapters off: the active rolling DVR window still records and serves live time-shift, but no finalized replay chapter artifacts will appear after media rolls out of that window. Configure dvrChapterMode on the Stream via updateStream; the change applies to the next recording, not this one.

2. Inspect the chapter’s state with frameworks cluster diagnose dvr <dvr_hash>, which lists each chapter’s state, bounds, finalize attempts, job ID, node, and last failure. The dvrChapters query shows the tenant’s view: each DVRChapterRef carries state, startMs/endMs, isCurrent, playbackId, hasGaps, segmentCount, and lastFailureReason.

3. Map the state to action:

   State	What to do
   open / closed	Recording still in flight or queue hasn’t picked up yet. Wait one finalization tick (30s).
   finalizing	Processing job dispatched. Check the recording origin’s Helmsman logs for chapter finalize entries.
   finalized / frozen / reclaimed	Chapter is playable. Use its Commodore-minted playback_id as the player input; the parent DVR’s snapshotted policy still applies.
   failed_source_missing	Source segments lost from both local AND the recovery freeze. Chapter is unrecoverable. Inspect lastFailureReason.
   failed_permanent	Repeated retries exhausted (5 finalize attempts) or invalid ledger input. Inspect lastFailureReason.

Triage: chapter finalization stuck

Chapter finalizations that hang in state='finalizing' past the dispatch deadline (default max(2*chapter_duration, 30 min) capped at 24h) are auto-re-queued by the next tick. To check a recording, run frameworks cluster diagnose dvr <dvr_hash> and read its finalize queue: chapters that stay in finalizing, their started time, the attempt count, the job ID, the node running it, and last_failure. Running it again shows whether the attempt count is climbing.

While a finalization is active, Helmsman renews an attempt-fenced progress lease through recovery and readiness and reports five-second progress while pushing. A healthy long remux therefore remains owned even when its media position has not advanced yet. If the attempt number in the job ID keeps climbing, check Foghorn for rejected progress/failure fences and Helmsman for expired dispatches, superseded local attempts, admission failures, disk pressure, or an unavailable source. Correlate both services with the job ID chapter-finalize-v2-<attempt>-<chapter_id>.

Triage: storage node out of space during DVR

1. dropPressuredDVRSegments on the recording sidecar will start evicting uploaded-and-aged segments first (safe local cleanup; the recovery-bridge S3 object remains intact).
2. If pressure persists, the sidecar may be forced to evict unsynced segments — these emit DVRSegmentDropped(was_uploaded=false) and become lost_local ledger rows. Chapter finalization for overlapping chapters will surface this as failed_source_missing once recovery is exhausted.
3. If the sidecar can’t keep up at all, it stops the push cleanly and emits storage_node_out_of_space. The artifact then finalizes as completed_partial if some segments survived, or failed if none did.

Lifecycle event types to alert on:

• DVRSegmentDropped(was_uploaded=false) — data loss event; investigate disk capacity
• storage_node_out_of_space — push aborted; node needs attention
• dvr_terminal rejection rate spiking — Foghorn-side state machine racing with sidecar; usually transient

S3 layout

s3://bucket/dvr/{tenant_id}/{stream_internal_name}/{dvr_hash}/segments/{segment_name}
s3://bucket/vod/{tenant_id}/{chapter_artifact_hash}/{chapter_artifact_hash}.mkv
s3://bucket/vod/{tenant_id}/{chapter_artifact_hash}/{chapter_artifact_hash}.mkv.dtsh

The DVR segment prefix holds the recovery-bridge TS segments uploaded during recording. These are NOT playback objects: viewers never read them. Chapter finalization fetches them (or recovers from local disk) to build the canonical .mkv. Once a chapter reaches state='frozen' the reclaim sweep deletes both the local TS file and the corresponding S3 segment object.

Chapter playback artifacts live at the normal VOD prefix. They’re regular VOD artifacts whose origin_type='dvr_chapter', origin_id=chapter_id, and source stream_id are registered in Commodore. library_visible=false keeps them out of the tenant-wide VOD library, but stream-scoped VOD artifact queries can still return them for a stream’s media overview.

Don’t manually manipulate any of these prefixes; let the chapter finalization + reclaim sweep do their work.

Retention

Retention only acts on terminal artifacts. An active DVR (status recording / finalizing) is never deleted by the retention job, regardless of age.

retention_until is computed at FinalizeDVR as ended_at + dvr_retention_days*24h, where dvr_retention_days is snapshotted on foghorn.artifacts at DVR start. Commodore resolves that value through the per-class cascade (per-stream override → tenant per-class default → 30-day system default → tier cap) before passing it to Foghorn; once stamped, the tenant’s plan is never re-resolved at end. A dvr_retention_days of 0 or NULL means “keep forever” and the retention job skips the artifact entirely.

To extend or shorten retention on a specific finalized recording, apply a per-asset override with the updateMediaRetention mutation (targetType: DVR, targetId: "<dvr_hash>", and retentionDays or retentionUntil); resetMediaRetentionOverride clears it. See Storage and retention. Active recordings are rejected.

The retention job picks up the new value on its next tick (default hourly).