Incidents: One Failure, One Thing to Work On
A server in one of your clusters loses power. It was running five services, and every one of them stops answering health checks at the same moment. Each produces its own “service unreachable” alert. Handled as plain notifications, that is five near-identical messages at once, the same messages again every few hours while the machine stays down, and no shared place to record that someone is already on it.
FrameWorks now turns alerts into incidents. That power loss becomes one incident for the cluster, listing every service that went dark, with a timeline, an owner and a status that stays correct until the last alert clears.
What the node failure looks like now
Section titled “What the node failure looks like now”The failing services still fire individually, and Alertmanager still does what it is good at: evaluating rules, grouping related alerts, applying silences, and retrying delivery. Alerts of the same kind in the same cluster and region arrive as one group, and each group becomes one incident.
Alertmanager waits briefly for related alerts, then sends the five “service unreachable” alerts as one group. One incident opens with five firing alerts. When Alertmanager repeats the group a few hours later because the machine is still down, the existing incident is updated rather than a new one opened. An on-call engineer acknowledges it, which records who took it without silencing anything; if a sixth service fails, the incident still shows it. Notes and assignment build a timeline as the work happens. When the server comes back and every alert in the group resolves, the incident resolves itself.
If someone resolves an incident by hand, repeats of the same alerts do not reopen it. A genuinely new alert opens a new incident. That keeps a closed incident closed without hiding a second failure.
Every state change is pushed through a live subscription, so the dashboard updates without polling. After a reconnect, the incident itself is the durable record to read, not the stream of changes.
Incidents belong to whoever owns the cluster
Section titled “Incidents belong to whoever owns the cluster”A tenant running their own cluster should see incidents about their infrastructure and nobody else’s. So each incident is scoped to the tenant that owns the cluster the alert came from, and a tenant can only read or change incidents on clusters they own. Asking for another tenant’s incident returns the same answer as asking for one that does not exist.
Ownership can change while alerts are arriving, and the system that records ownership can itself be part of the outage. The rule for that case is simple: when ownership cannot be confirmed, the incident stays visible to platform operators only. A tenant might see their incident late, but never someone else’s, and operators still see the outage while the control plane is down.
Resolved incidents stay with the tenant that owned the cluster when they happened. Moving a cluster to a new owner does not hand over its history.
Delivery keeps working during the failure it reports
Section titled “Delivery keeps working during the failure it reports”An incident system is most needed when things are broken, which is also when it is most likely to be affected. We designed delivery around that.
Platform incidents notify operators by email, Slack and Discord. Tenant incidents feed the tenant’s live dashboard and Skipper. An incident and the obligation to announce it are saved together, so a notification service that is briefly down delays the message and retries it later rather than losing it. When more than one copy of the incident service is running, each delivery is claimed by one copy at a time, so a restart or a failover does not leave a message unsent.
Critical alerts also have a second path that bypasses the incident system entirely and goes straight to email. If the incident service is the thing that failed, the page about it still goes out. And an alert that fires permanently is sent to an external heartbeat monitor once a minute; if the alerting pipeline itself stops, the heartbeat goes quiet and the external monitor raises the alarm.
The same workflow for people and agents
Section titled “The same workflow for people and agents”Tenants work incidents under Infrastructure → Incidents. Platform operators have a cross-tenant view under Platform Admin → Incidents. The same actions are available through GraphQL and the MCP tools: list, inspect, acknowledge, assign, resolve and add a note.
Skipper reads the tenant incident feed and can attach an investigation report to the timeline. It never acknowledges or resolves anything on its own; analysis and state changes stay separate, and a person decides when an incident is over.
Which alert rules exist is still an operator decision, and incidents do not merge alerts of different kinds into one record: a failed server that also trips a second, different alert rule produces two incidents, one per rule.
Start with the incidents API guide. Operators deploying incidents should read Alerting and Incidents, which covers notification channels, the heartbeat and an end-to-end delivery test.