Major incident management is the process for incidents that cause a big business impact, and it is handled separately from the normal incident process. A company-wide outage cannot wait in a queue for someone to pick up, like a password reset. One person owns the incident, and all responders work in one channel, called a war room. The updates also have a fixed interval to keep stakeholders informed about the status.
IT major incidents are also about preparation beyond speed. The criteria, who declares the incident are agreed before the first outage happens. This guide covers when to declare one, the process step by step, the roles, the communication templates, the post-incident review, and a checklist.
Key Takeaways
- A major incident is one that has a significant impact on the business, and the response to it is handled separately from the normal process so it can move faster.
- Ownership goes to one major incident manager. The technical teams stay on the fix.
- On the communication side, the updates follow a fixed cadence in one channel, and the templates for them are written before anything breaks.
- Every major incident closes with a blameless post-incident review, and a problem record is opened from it.
- Small teams can combine the roles. The person fixing the incident and the person sending the updates should still be two different people.
What Is Major Incident Management?
Major incident management is used when the impact of an incident reaches a certain level. This level is agreed before any outage happens. It depends on how many people are affected and which server has stopped working. In most ITSM documentation, it is written as major incident management (MIM), and the short form is used in the tooling as well.
ITIL major incident management is handled as a separate procedure inside incident management, and it is kept for the incidents that have the highest impact. It doesn’t replace the normal incident process, the same logging and diagnosis steps are used, with a declared owner working on them. A review is also required after the service comes back.
Major incidents belong to the wider IT operations management work, which covers the monitoring and alerting that pick up an outage in the first place.
Major Incident vs Standard Incident
The same incident can start as a standard one and turn into a major incident later. The change happens the moment somebody declares it, which is when the owner and the war room get set up.
When to Declare a Major Incident
Simply put, any incident that meets the criteria a team has written in advance can be marked major. The criteria should be written in advance and checked against all the tickets getting logged.
- Are many users affected, or is a whole site down?
- Has a critical service stopped, like email, VPN, ERP, or payments?
- Are customers affected as well?
- Is any data at risk?
- Is the service unusable with no workaround to fall back on?
- Will an SLA be breached in the next hour?
An organization can decide how many of these have to be true before an incident is declared a major one. You can declare the checklist earlier and downgrade later when needed. The incident gets moved back to the standard process, and the war room is closed.
The criteria are mapped onto the priority levels, and a P1 is the level the declaration is attached to. The same document holds the escalation matrix, which lists who gets contacted at each level.
Major Incident Examples
Major incident management examples in IT are built around shared services, because an outage in one of them reaches everybody at the same time.
The Major Incident Management Process
The major incident process has eight steps, and the declaration at the second step is what separates it from the normal flow. The stages remain identical in the incident management lifecycle, with the declaration and the review added on top.
- Detect and log: An alert from monitoring or a report from a user starts it. The incident gets logged in the normal way, and the service request vs incident check decides which queue it starts in.
- Declare: The criteria get checked against what is known at that point. The major incident manager makes the call, and on an out-of-hours outage, the on-call lead does it instead. The decision is made on partial information.
- Assemble the team: A technical team lead is assigned to run the diagnosis. The communication lead handles the updates, and resolver groups own the affected services.
- Open the war room: One channel is created for the responders to keep technical conversation inside it. Stakeholders are in another channel where they can get answers to status questions.
- Communicate: The first response is sent before the cause is known. The cadence is agreed in advance, for instance, a first notice within 15 minutes and then an update every 30 minutes while a P1 is open. The cadence is held even when there is nothing new to report.
- Diagnose and restore. The first step is going through the workaround if one exists, and the next one is finding the permanent fix. Any change during an outage is updated as an emergency change, with approval recorded. When engineering expertise is needed to fix the issue, the work moves across as a Jira issue linked to the incident, with the comments and the status synced both ways.
- Resolve and close: Service restoration gets confirmed with the users who reported it. The resolution summary is shared afterward, and the ticket is marked closed.
- Review: A post-incident review follows within five business days. The write-up covers the timeline and the actions, and a problem record is opened so the cause can be removed.
Major Incident Roles and Responsibilities
A major incident is run by one owner, and there are a few other roles around that owner. The names of these roles are different in different organizations. The major incident roles listed by Emory University IT use crisis manager and incident technician for the same jobs.
Major Incident Manager
The major incident manager, which is also called the incident commander in some organizations, is responsible for the incident until it is closed. The declaration is made by this person, the war room is chaired by the same person, and the updates are also controlled from here. Debugging is not a part of this role, because a person who is looking into the logs cannot do the coordination at the same time.
Technical Lead
The technical lead is responsible for the diagnosis work, and what gets tried and in what order is decided by this person. On the outages that are engineering-led, the role is given to the team that owns the service, and a DevOps service desk is set up on the same split.
Communications Lead
All the updates that go out of the war room are handled by the communications lead. The wording for the staff is not the same as the wording for the customers, and the leadership also gets a different version. One person keeping all of them helps to avoid two different status messages going out.
Scribe
The scribe is used for writing down what happened and at what time. The post-incident review is prepared later on the basis of this log.
Service Desk
The users contact the service desk during an outage. The status that has been declared is repeated to anyone who asks about it, so the war room does not get interrupted. The escalation into the second line and the third line happens on the same path that the support tiers use on a normal day.
Problem Manager
The cause of the incident is taken up by the problem manager after the service comes back. A problem record is opened out of the review, and the permanent fix gets tracked on that record.
Executive Sponsor
An executive sponsor is named on the outages that reach the customers or the ones that run for a long time. The spending and the emergency changes which the major incident manager is not able to approve alone are signed off by the sponsor.
RACI is a grid that is used to show who is responsible for an activity, who is accountable for it, who has to be consulted, and who is only informed.
The major incident manager is accountable for all the rows, and that is how the single owner rule works in practice.
Major Incident Management for Small IT Teams
The roles we have discussed so far are applicable for a full suite team. On a team of two to eight people, the roster cannot be filled, and the same jobs have to be covered by fewer people.
The on-call lead is the one who becomes the major incident manager. One person is kept on the fix. Another person is made responsible for the updates and also for the timeline. The leadership is informed on one channel, and the same channel is used for all of the updates that go to them.
There are a few combinations that work when there are not enough people, such as:
- The scribe work and the communications work can be given to the same person, because both of them are about the writing.
- The major incident manager can also take the communications when there are only two people who are available.
- The service desk role is handled by whoever is on the queue at that time, and a separate person is not needed for it.
The fixing and the communicating should be kept with two different people. A person who is working on the fix is not able to answer questions from the business at the same time, and the updates get delayed when both are tried together.
Who is allowed to declare outside the business hours should be decided in advance. The name is written into the on-call rota, so an approval does not have to be sought during the outage.
Small team, big outage? Suptask gives the incident one Slack ticket with an owner, a priority, and a thread for updates. High-priority tickets can be paged to on-call through PagerDuty. See major incident handling in Slack
Major Incident Communication Templates
The templates are written before the outage, so the wording does not have to be worked out while the service is down. The three below cover the first notice, the updates in between, and the resolved message.
First notice
Subject: [Service] is unavailable
We are aware of a problem with [service] which started at [time]. [Who is affected] cannot [what is not working].
The team is working on it now. There is no workaround available at the moment.
The next update will be sent at [time].
Regular update
Subject: [Service] update, [time]
[What has changed since the last update.]
[The workaround, if there is one, or a line saying there is none yet.]
The next update will be sent at [time].
Resolved
Subject: [Service] is back
[Service] was restored at [time]. The cause was [one line on the cause].
[Anything the users have to do now, if there is something.]
A review of the incident will be held, and the actions coming out of it will be shared by [date].
Three versions of the same update are sent, one for the staff, one for the leadership, and one for the affected customers.
In Suptask, post these updates in the incident ticket’s thread so everyone following the ticket sees them.
Running a Major Incident in Slack
When the team already works in Slack, the war room does not have to be set up anywhere else. One channel is created for the incident, and only the responders are kept in it. A status message is pinned at the top of the channel, and the same message is edited every time something changes.
The updates are posted by one person only. The others keep their findings in a thread, so the main channel does not fill up. The stakeholders are kept in a separate announcement channel, and the decisions which get taken during the call are written into a thread for the review.
A Slack help desk turns the same coordination into a ticket. In a Slack-native service desk like Suptask, the major incident is a ticket which has an owner, a priority, and a thread for the updates. The alerts that are posted in Slack can open the ticket on their own, and a high-priority ticket can also create a PagerDuty incident to page the on-call person. The thread is kept with the ticket, so service desk incident management in Slack leaves a record that the review can use.
The Post-Incident Review
A post-incident review is a meeting that is held after the service is back, and the purpose of it is to understand what happened and stop the same thing from occurring again. The meeting is blameless, which means the discussion stays on what happened in the system. The names of the people are kept out of the findings.
The review is held within five business days of the closure. Everyone who was in the war room attends it, along with the service owner and the problem manager. The log that the scribe kept is used as the basis for the timeline.
A short agenda is enough, such as:
- The timeline is read out first.
- What worked gets covered next, which includes the early detection or a workaround that was held.
- What did not work is written down with the reason for each delay.
- The cause is identified and handed over to problem management as a problem record.
- Action items come at the end, and each one of them gets a name with a date against it.
The problem record keeps the cause open after the incident is closed, and when an incident becomes a problem is the point where that handover happens.
The actions are checked in the next review, and the ones that are still open get carried forward.
Major Incident Management Metrics
The measurement of major incidents is done on a small set of numbers.
- Time to detect is the gap between the failure starting and the first alert.
- The declaration has its own number, which is the time taken between the logging and the call being made.
- Mean time to restore (MTTR) is counted until the service becomes usable again, starting at the beginning of the outage.
- Whether the update cadence was kept is recorded for each incident.
- The post-incident review completion rate shows how many of the declared incidents actually got a review.
- Repeat major incidents are counted separately, because the same cause coming back points to the problem record not being closed.
The restore time is the one that goes into the SLA reporting, and the rest of them are used within the team. The team’s own earlier numbers are used for the comparison.
Major Incident Management Best Practices
The major incident management best practices get written into the process document and followed during an outage.
- The criteria should be written down before they are needed, and a one-page sheet with the yes or no checks is enough.
- Making the call early is the safer option, because a declaration can be withdrawn in a few minutes.
- One owner and one channel should be fixed, and a second channel for the same outage should not be opened.
- Even in a small team, the fixing and the communicating should stay with two people.
- The templates should be kept in a document that the on-call person can open, because wording takes time, which is not available during the outage.
- Twice a year, a tabletop exercise should be run, and an old incident can be used as the scenario.
- The review should be run on every declared incident.
Tools are now used in the detection step, and the grouping of alerts before a person sees them is one part of AI incident management.
Major Incident Management Checklist
The checklist below can be copied into the process document.
Before
- ☐ The declaration criteria are written down and agreed.
- ☐ The person who can declare outside business hours is named in the on-call rota.
- ☐ The major incident manager role is assigned, with a backup for it.
- ☐ The three update templates are saved where the on-call person can open them.
- ☐ The naming for the war room channel is decided in advance.
During
- ☐ The incident is declared, and the priority is set.
- ☐ One owner is named, and the technical lead is kept separate from the communications lead.
- ☐ The war room is opened, and the stakeholders are kept on another channel.
- ☐ The first notice goes out before the cause is known.
- ☐ The updates are posted at the agreed interval, even when nothing has changed.
After
- ☐ Service restoration is confirmed with the users who reported it.
- ☐ The resolved notice is sent.
- ☐ The post-incident review is held within the agreed window.
- ☐ A problem record is opened for the cause.
- ☐ The action items have a name and a date against them.
The incident management tools are chosen after the process is written, because the process decides what the tool has to support. If your team already handles outages in Slack, try running major incidents as incident tickets in Slack.
FAQs
Who declares a major incident?
The declaration is made either by the major incident manager or by the on-call lead. In small teams, anyone can declare one when it meets the criteria agreed earlier. The declaration should be made before the impact gets worse. Waiting for more information keeps the responders on a normal queue while the outage is running.
What is a P1 incident?
P1 is universally the highest priority level on a standard scale. A critical service is down at that level, a large number of people are affected, and no workaround is available. A P1 and a major incident are not the same thing on paper, because the priority is a field on the ticket and the major incident is a procedure. In practice, the two go together, and a P1 is what gets handled through the major incident process.
How does major incident management relate to problem management?
Major incident management is used to get the service back as quickly as it can be done. Problem management comes after that, and the cause gets found and removed so the same incident does not return. The handover between the two happens at the post-incident review, where the cause gets written into a problem record. The incident gets closed before the problem is closed, and the two records run on different timelines.
What makes a good major incident manager?
A good major incident manager keeps the coordination going when the information is incomplete. Decisions get made on what is known at that moment, and a decision is revised later if it turns out to be wrong. The communication has to be clear enough for a non-technical manager to follow. The people who are working on the fix are coordinated by this person, and the fixing is not done by the same person.
