What incident management actually is
Incident management is the practice of detecting unplanned interruptions to a service, co-ordinating a response, restoring the service, and learning from what happened. The restoring-service part is the definition’s load-bearing clause: incident management is about getting the service back, not about finding the root cause. Those are different jobs with different urgencies, and conflating them is the commonest reason incidents run long.
Four things get confused with each other constantly, and the distinctions matter because they route differently.
| Type | What it is | Goal | Urgency |
|---|---|---|---|
| Incident | Something is broken that should not be | Restore service | Now |
| Service request | Someone wants something they are entitled to | Fulfil it | Scheduled |
| Problem | The underlying cause of one or more incidents | Remove the cause | Deliberate |
| Change | A planned modification to the environment | Make it safely | Planned |
If your access-request queue and your production outages arrive in the same inbox, that is the first thing to fix. They have nothing in common except that somebody typed them.
ITIL or SRE: two worldviews, and which parts of each survive
Search this topic and most of what you find is written from the service-desk tradition: ticket categorisation, priority matrices, SLAs, the relationship to problem management. It is a coherent and well-tested body of practice, and it was designed for an environment where a central IT function supported the business and incidents arrived as reports from users.
The other tradition comes from site reliability engineering: on-call rotations, incident command, blameless postmortems, error budgets. It assumes the people who built the service operate it, and that detection comes from telemetry rather than from a user noticing.
Most teams running a cloud estate are somewhere in between and get told they must pick. They do not.
| Worth keeping from ITIL | Worth keeping from SRE |
|---|---|
| The incident / problem / change / request distinction | Detection from telemetry, not from users |
| Defined severity and priority, written down | Incident command as a role structure, not a job title list |
| Formal major-incident declaration and comms | Blameless postmortems with real action items |
| A record that survives the people involved | Ownership by the team that built it |
| Problem management as a deliberate follow-on | Error budgets as the rule for when to stop shipping |
What does not survive contact with a cloud estate: approval ceremony on the restoration path, categorisation taxonomies with forty leaf nodes, and the assumption that a central team can hold enough context to triage everything. Those were reasonable when infrastructure changed monthly.
The incident lifecycle, as a loop
Every article on this subject draws a straight line: identify, log, categorise, prioritise, respond, close. Real incidents do not behave that way. You contain something, discover the containment made it worse, re-triage at a higher severity, and go round again. A model that cannot represent that is a model you will abandon at 3am.
- Detect. Telemetry, a synthetic check, or a customer. If it is routinely the third, that is the finding.
- Triage. What is affected, how many, is it getting worse. Minutes, not a form.
- Declare. Assign a severity and say it out loud in a channel. Under-declaring is the more expensive error — it is cheap to stand people down, expensive to summon them late.
- Contain. Stop the bleeding. Roll back, fail over, shed load, disable the feature. Do not debug yet.
- Resolve. Service restored to normal. Not root-caused — restored.
- Learn. Postmortem, action items with owners and dates, and the honest question of whether detection worked.
The loop is between triage, declare and contain. Expect to go round it two or three times in a long incident, and build the process so re-declaring a severity upward is normal rather than an admission of failure.
Severity levels: a matrix you can copy
Six of the highest-ranking pages on this topic do not contain a usable severity table between them. Here is one. Adapt the response targets to your own commitments; keep the shape.
| Level | Customer impact | Example | Response | Who is woken |
|---|---|---|---|---|
| Sev 1 | Service unusable for most or all customers, or data is at risk | Checkout down, region failure, suspected breach | Immediate, 24×7 | On-call, incident commander, engineering leadership, comms |
| Sev 2 | Major feature unusable, or severe degradation for a subset | Search returning errors, one large tenant down | Immediate during the day, 30 min overnight | On-call plus the owning team |
| Sev 3 | Degraded but usable; a workaround exists | Elevated latency, a failing async job with retries | Next business hours | Owning team only |
| Sev 4 | Minor or cosmetic; no meaningful customer impact | Dashboard panel broken, log noise | Backlog | Nobody |
Three rules that make a matrix like this work in practice. Severity is about impact, not effort — a one-line fix for a total outage is still Sev 1. Anyone can declare, and nobody is criticised for declaring too high. And severity is reviewed during the incident, not fixed at declaration. Most incidents that run badly were declared one level too low and nobody revisited it.
Who does what: the incident command model
Most articles list roles as job titles, which is not the same as a command structure. Under pressure, what you need is explicit ownership of four distinct jobs — they may collapse into fewer people on a small incident, but they should never collapse silently.
| Role | Owns | Explicitly does not |
|---|---|---|
| Incident commander | Co-ordination, severity, who does what next | Debug. The moment the IC is in a terminal, there is no IC. |
| Operations lead | Making the changes — the hands on the system | Communicate externally |
| Communications lead | Status page, internal updates, stakeholder questions | Interrupt the ops lead for updates |
| Scribe | Timeline: what was done, when, by whom | Participate in the fix |
For incidents running past about two hours, hand over explicitly — the outgoing commander briefs the incoming one on state, actions taken and open threads, and the handover goes in the channel. Fatigue causes more bad decisions in hour four than inexperience does in hour one.
Major incident management
A major incident is not just a bigger Sev 1. It is the point where the normal process stops being sufficient because the volume of co-ordination and communication exceeds what the responding team can carry while also fixing the problem.
Declare one when any of these is true: the incident affects the majority of customers, it has run past a duration you have agreed in advance, it requires three or more teams, or it has regulatory or contractual consequences.
What changes on declaration: a dedicated commander who does nothing else, a comms lead publishing on a fixed cadence whether or not there is news, a scribe, and stakeholder updates at agreed intervals. The fixed cadence matters more than the content — silence is read as loss of control, and teams that only update when they have news get interrupted constantly by people asking for news.
Alert fatigue, with arithmetic
Alert fatigue is named in every article on this topic and quantified in none, which is unhelpful, because the way out of it is arithmetic.
Take your own numbers for a single week. Count the pages that reached a human. Count how many resulted in somebody doing something — not acknowledging, doing. That ratio is your actionability rate, and it is the single most diagnostic number in your operation.
Below roughly 30%, the rotation has learned that most pages are noise, and the cost is not irritation but latency: response time degrades on every alert because the first assumption is that this one is also nothing. That is how a Sev 1 ends up acknowledged in eleven minutes. Alert fatigue does not announce itself as tiredness; it shows up in your response metrics.
What actually reduces volume, in order of effect:
- Delete alerts nobody acted on. Take last quarter’s alerts, find the ones that never produced an action, and remove them. Expect to remove more than half. This is uncomfortable and it is the highest-yield hour you will spend.
- Alert on symptoms, not causes. One alert for “checkout error rate above threshold” beats nine for the components underneath it. The nine are for the dashboard you open after the one fires.
- Route by severity, not by volume. Sev 3 and below belong in a queue, not on a phone. Most estates page for things nobody intended to be paged for, inherited from a default.
- De-duplicate and group. One failure in a dependency should produce one incident, not thirty alerts from thirty dependents.
A reasonable target for a healthy rotation is a small number of actionable pages per on-call week and close to zero outside working hours that were not worth waking for. If you are an order of magnitude above that, no tooling purchase fixes it — the alert set does.
Metrics that survive scrutiny
MTTA — mean time to acknowledge. The cleanest early signal, because it degrades as trust in alerts degrades.
MTTR — mean time to restore. Universally reported and routinely misleading, because it is a mean. One nine-hour incident among thirty twenty-minute ones produces a comfortable average and tells you nothing about the nine-hour one, which is the only one anybody remembers. Look at p90, and look at the distribution.
MTBF — mean time between failures. Useful for a component, nearly meaningless for a whole platform.
Actionability rate — the alert-fatigue number above. Not standard, and more useful than the standard ones.
SLOs and error budget — the only metric that converts reliability into a decision. An error budget gives you an agreed, unemotional rule for when feature work stops and reliability work starts, which is otherwise an argument held repeatedly and settled by seniority.
Incident management on AWS
No widely-ranking guide on this topic says anything cloud-specific, which is strange, because the estate changes the mechanics considerably.
| Job | AWS service | What to watch for |
|---|---|---|
| Detection | CloudWatch alarms, composite alarms, synthetics | Composite alarms are how you stop one failure paging nine times |
| Routing and response | EventBridge, Systems Manager Incident Manager | Response plans and escalation live here rather than in a wiki |
| Runbook execution | Systems Manager Automation | Runbooks that execute beat runbooks that are read aloud at 3am |
| Provider-side faults | AWS Health Dashboard | Wire it into your alerting — otherwise you debug your own code during someone else’s outage |
| Multi-account visibility | Organizations, cross-account CloudWatch | Thirty accounts without aggregation means thirty places to look |
Two cloud-specific things worth planning for. Blast radius is an architectural property — multi-AZ, multi-region and cell-based designs decide in advance how large your worst incident can be, and no amount of response process changes that ceiling. And incidents have a bill. The scaling event during the incident, the failover capacity, the cross-region data transfer, and the over-provisioning nobody rolled back afterwards all land on next month’s invoice. Add “reverse the emergency capacity” to the resolution checklist; it is the most commonly skipped step in cloud incident response.
The blameless postmortem
Everyone asserts blamelessness. Few describe the artefact, so here it is: a timeline reconstructed from logs and chat rather than memory, the contributing factors — plural, because single root causes are mostly a storytelling convention — what made detection slow or fast, and action items with a named owner and a date.
Blameless does not mean nobody is accountable. It means the question is “what about the system made this action reasonable at the time” rather than “who did it”. An engineer who deployed a bad config at 18:00 on a Friday is a signal about your deployment guardrails, not a culprit.
The failure mode is action items. A postmortem that generates twelve items of which two get done is worse than one that generates three that all get done, because it trains everyone that the exercise is theatre. Cap them, assign them, and review them in the following month’s operations meeting.
Automated incident management: what to automate, what never to
The sceptical question deserves a straight answer, so here is the line we would draw.
| Safe to automate | Keep a human in |
|---|---|
| Enrichment — attaching logs, metrics, recent deploys and ownership to the alert | Declaring an incident and setting its severity |
| De-duplication, grouping and correlation | Customer communication of any kind |
| Routing and escalation against a defined policy | Rollback decisions with data consequences |
| Bounded, reversible remediation — restarting a stateless service, clearing a known queue | Anything whose blast radius exceeds the incident’s current one |
| Status-page updates once a human has written the words | Declaring an incident resolved |
| Postmortem timeline assembly from logs and chat | Writing the contributing factors |
The principle underneath: automate what reduces the time a human spends assembling context, and keep the human on anything that changes state unpredictably or reaches a customer. Enrichment is where most of the realisable time saving is anyway, because a meaningful share of incident duration is spent working out what is going on rather than fixing it.
Where agents fit, if you are considering them, is in that assembly step and in proposing the remediation for a person to approve — which is how CloudOpsGPT works on AWS: it builds the plan, a person approves it, execution happens inside your own account through roles you connect, and every action is logged. That shape is worth demanding of anything in this space, ours included; an agent that acts without an approval gate is an unsupervised script with better marketing.
Tools, by job rather than by brand
| Job | Common choices |
|---|---|
| Detection and observability | CloudWatch, Datadog, Grafana, New Relic, Honeycomb |
| Alerting, on-call and escalation | PagerDuty, Opsgenie, Systems Manager Incident Manager, Grafana OnCall |
| Collaboration during response | Slack or Teams with an incident bot; a dedicated channel per incident |
| Customer communication | Statuspage, Instatus, or a static page you control and can publish when your platform is down |
| Postmortems and tracking | Jira, Linear, incident.io, or a template in your wiki that is actually used |
Most teams have all five already and the gap is not a product. Before buying anything in this category, run the actionability calculation above — in most estates the alert set is the problem, and a new tool inherits it.
Frequently asked questions about incident management
What are the 5 stages of incident management?
Commonly: detect, triage, respond, resolve and learn — though declaration deserves its own step because assigning a severity determines who is involved. The stages are a loop rather than a line; long incidents cycle back through triage and re-declare at a higher severity as understanding improves.
What is the difference between incident management and problem management?
Incident management restores service as quickly as possible, accepting a workaround. Problem management removes the underlying cause so the incident does not recur. They run on different clocks: incidents are urgent, problems are deliberate. Trying to root-cause during an incident is the commonest reason incidents run long.
What is the difference between an incident and a service request?
An incident is something broken that should be working — the goal is restoration and the urgency is immediate. A service request is someone asking for something they are entitled to, such as access or a new environment — the goal is fulfilment and it can be scheduled. They should not share a queue.
What are incident severity levels?
A scale, usually Sev 1 to Sev 4, defining customer impact and the response it triggers. Sev 1 is service-wide failure or data at risk, warranting immediate 24×7 response. Sev 4 is cosmetic. Severity should reflect impact rather than how hard the fix is, and should be reviewed during the incident rather than fixed at declaration.
What does an incident commander do?
Co-ordinates the response: sets severity, decides what happens next, assigns the other roles, and keeps the incident moving. Critically, the incident commander does not debug. The moment they are in a terminal fixing something, nobody is holding the overall picture, and that is when incidents stall.
What is a good MTTR?
Less useful a question than it appears, because MTTR is a mean and one long incident among many short ones hides inside it. Look at p90 and the distribution rather than the average, and compare your own trend over time rather than against a published benchmark from a different estate.
What is alert fatigue and how do you fix it?
Alert fatigue is what happens when most alerts are not actionable, so responders stop treating them as urgent. It shows up as degraded acknowledgement time rather than as complaints. Fix it by measuring what proportion of alerts produce an action, deleting those that never do, alerting on symptoms rather than causes, and routing anything below Sev 2 to a queue.
Can incident management be automated?
Parts of it, safely. Enrichment, de-duplication, routing, and bounded reversible remediation automate well. Declaring an incident, communicating with customers, deciding on rollbacks and declaring resolution should keep a human. The biggest realisable saving is in context assembly, which is where much of an incident’s duration actually goes.
What is a blameless postmortem?
A review that asks what about the system made an action reasonable at the time, rather than who made the mistake. It contains a timeline rebuilt from logs and chat, contributing factors rather than a single root cause, an honest assessment of detection, and a small number of action items with named owners and dates.
What is the difference between incident response and incident management?
Incident response is the technical activity of containing and resolving a specific incident. Incident management is the wider practice around it — severity definitions, on-call structure, communication, postmortems and the metrics that tell you whether any of it is working.
Where to go from here
Three places to go next:
Sources
- AWS Systems Manager — Using Systems Manager tools, AWS documentation
Cut the time spent working out what is going on
CloudOpsGPT assembles context across your AWS accounts and proposes an itemised plan. A person approves it, execution runs inside your own account through roles you control, and every action is logged.
No credit card required. Your trial clock starts when the Runtime connects.