What Seven AI Agents Found in Our Streaming Platform
Live video systems collect awkward edge cases. A viewer connects from a location that sits on the wrong side of a geofence boundary. A DVR recording starts at the same moment a stream is shutting down. Two usage events for the same tenant arrive close enough together that billing code has to prove it is actually idempotent. None of those cases are hard to understand on their own. The failures come from these cases interacting across services.
FrameWorks is split across routing, ingest, VOD, edge orchestration, analytics, billing, auth, MCP agent access, DNS, Skipper, WebSocket routing, and tenant management. Each subsystem has tests and reviews, but most of the failures we found sit between them.
A finding without evidence did not count
Section titled “A finding without evidence did not count”We ran the first audit pass by hand: seven roles, 26 batches, 90 tasks, 12 subsystems. One agent traced a domain, another checked the report, another wrote a fix if the finding held up, another reviewed the patch. CI and a human reviewer still owned the merge.
Every finding had to include evidence. A report had to name the files and functions, describe the failing input or interleaving, explain the tenant or operator impact, and say what test would prove the fix. “This might race” went nowhere. “Two prepaid usage reports can reserve against the same tenant balance before the ledger state is visible” was at least reviewable.
That rule removed most of the noise. Agents had to keep reading until they could point at the code instead of reporting general impressions of the codebase.
Most failures sat between subsystems
Section titled “Most failures sat between subsystems”The findings were mostly the kind you expect after a platform gets broad enough:
| Category | Count | Example |
|---|---|---|
| Race conditions | 12 | Prepaid balance deduction under concurrent Kafka messages |
| Tenant isolation | 8 | Stream context cache keyed by internal_name only, missing tenant_id |
| Data loss risks | 7 | Event ingest batch flush interrupted by producer crash |
| Stale cache behavior | 6 | GeoIP cache stampede under bursty traffic |
| Protocol edge cases | 5 | Player protocol blacklist leading to “no playable protocol” dead-end |
| Storage consistency | 5 | S3 upload succeeds but local delete fails, leaving duplicate artifacts |
| Auth bypass vectors | 4 | GraphQL complexity bypass through deep nesting with small page sizes |
| DNS propagation | 3 | Stale DNS pointing at decommissioned nodes |
Some of the best results were not bug fixes. A reviewer would chase a claim and find the lower layer already enforced the invariant. That turned a hidden assumption into something we could preserve during the next change.
Agents traced well and overstated severity
Section titled “Agents traced well and overstated severity”The agents were good at long, boring traces. GraphQL to gRPC to a service event. Kafka to ClickHouse to the billing ledger. Load balancer state to the edge’s playback rewrites to the player fallback path. A person can do that too, but it is easy to stop when the first function looks fine; the agents kept following each path to the end.
The weak point was severity. LLMs like to make findings sound bigger than they are. The review pass had to push hard on impact: can a tenant hit this, does it cross a trust boundary, is there data loss, is it only a noisy log, is the bad state already repaired one layer later?
The first run also taught us what a finding report should look like. Early reports came padded with process notes and architecture recaps, which did not help decide whether a bug was real. The format we settled on has one paragraph for the claim, the exact files and functions, the failing interleaving or input, the tenant impact, the smallest reasonable fix, and the test that proves it. Batch size mattered too. Three tasks per batch kept the human reviewer ahead of the queue; bigger batches just moved the bottleneck to whoever had to merge.
What changed in our reviews
Section titled “What changed in our reviews”The lasting change was in review questions. We now ask more often whether a cache key includes tenant_id, whether a handler survives Kafka retry, whether stream lifecycle events can arrive out of order, and whether an edge can keep serving a previously approved playback path when the control plane is temporarily out.
Next up is lightweight automation: track when each subsystem was last audited, weight the riskier areas like billing and auth, and open work only when there is a useful review to run. Most audit cycles should work like Skipper’s automated monitoring: check the current state and open work only when something needs attention.