Marek’s research group had rented time on a remote quantum processor for a month, and on day nineteen the job queue stopped moving. The cloud console showed green. Two postdocs were messaging the provider, a supervisor was asking for a status every ten minutes, and nobody could say who owned the next decision. Marek is invented, but the scene is familiar to anyone who runs experiments on shared, fragile infrastructure. What would have helped is a plain audit of their incident habits, the kind of check the field calls IncidentOps. This article gives you that audit as a checklist you can complete in an afternoon.
Why emerging-tech teams need this early
Quantum and other frontier platforms are young, they change often and they depend on suppliers. Disruptions are normal. What differs between teams is whether they respond with a shared routine or with improvisation. The Ops Atlas entry on IncidentOps defines the practice as the coordinated handling of an unplanned disruption, from declaring it to learning from it, and describes the label as informal and overlapping with incident management. You can ignore the name and still copy the habits.
How to use this checklist
Mark every line yes, partly or no, and be candid with yourselves. Each “no” turns into a small job on someone’s list. Closing two gaps a month is a healthier pace than trying to fix everything in a week.
Section A: declaring
- A1. Do you have written criteria for calling something an incident? The Atlas cites triggers such as another team being needed, users or collaborators noticing, or an hour passing without progress.
- A2. Can any team member declare, without asking permission?
- A3. Is it understood that declaring a small incident is cheap, whereas declaring late is expensive?
- A4. Do you avoid waiting for your provider’s status page before acting? The Atlas notes that supplier pages can lag, so declare on your own criteria.
Section B: roles
- B1. Is there a named coordinator for each incident, someone whose job is to keep the picture rather than to fix?
- B2. Is the group touching the experiment stack small, ideally no more than five people per supervisor, as the emergency-management rule of thumb in the Atlas suggests?
- B3. Does someone own communication, separate from both coordination and repair?
- B4. Is there a scribe recording times and decisions?
- B5. Do people understand that roles are hats, and that a doctoral student can coordinate if they hold the right standing on the call?
Section C: acting
- C1. For each major dependency, do you know the fastest safe reversal? Examples include pinning an earlier software version, switching to a simulator, or moving a job to another backend.
- C2. Have you written those steps where a tired colleague can find them?
- C3. Does the team accept that finding the faulty component is enough to act, without a full explanation?
For Marek’s group, C1 was the revelation. Their notebooks could run on a classical simulator with a few edits. Nobody had written that down, so for two days nobody tried it. A one-page fallback would have kept the work moving.
Section D: communicating
- D1. Is there a standing rhythm for updates? The Atlas suggests an opening note within roughly five minutes, a scope note shortly after and refreshes no less often than every twenty minutes at first.
- D2. Is your update channel separate from your working channel, so that observers do not flood the people doing the work?
- D3. Does your closing message say plainly whether any results or data were lost?
- D4. Does your notice board live somewhere that will not fail together with the platform it reports on? The Atlas lists the paging path and status page as two things that should never fail alongside the systems they describe.
Section E: handing over
- E1. If an incident crosses time zones or shifts, is command passed with an explicit spoken or written confirmation?
- E2. Does the incoming person receive the timeline, not just a summary?
Section F: reviewing
- F1. Is an owner named for the review before the response ends?
- F2. Does the review happen within days, led by someone who was not deeply involved?
- F3. Is the action list short, with one owner and one date per item?
- F4. Does the analysis end at conditions in the setup rather than at the phrase “human error”? The Atlas leans on a safety scholar’s caution about hindsight.
- F5. Do you close actions on evidence rather than by downgrading them?
Section G: measuring
- G1. When you quote a repair time, do you say which phase and which incidents it covers? The Atlas notes at least four meanings.
- G2. Do you look at the distribution of durations instead of only the average? In the Atlas’s lab example, one severe incident shifted a mean by about 45 minutes.
- G3. Do you track how heavy the on-call or watch load is on your people?
Running the audit as a group
Do not fill the checklist in alone. Print it, sit with three or four colleagues, and have each person mark it silently before comparing. The disagreements are the valuable part. When one person says “yes, we declare early” and another says “no, I waited two hours last month”, you have found a gap that no document would reveal. Assign a note-taker, and keep the session to ninety minutes.
A good habit is to attach evidence to each yes. If you claim a fallback exists for item C1, someone should be able to open the page and show it. If you say updates go out regularly, point to the last incident channel and look at the timestamps. Claims without evidence count as “partly”.
Scoring
Count your yes answers out of 26. A score under ten suggests you are running on goodwill. Between ten and twenty means you have a base and some gaps. Above twenty-five, you probably have a routine; spend your effort on rehearsal. As a rough guide for next steps, the Atlas sketches a three-stage ramp over ninety days: policy, role cards, hosted notice page and a tabletop first; trained coordinators and a first drill next; punctual reviews and better metrics last.
Small fixes that pay off quickly
Several items can be closed in under an hour. Write three sentences defining an incident and pin them where the group works. Create a one-page card listing the four roles. Draft three update templates, one each for the early, middle and closing stages of a disruption. Agree that the first person to notice a problem posts in the shared channel, even if only to say “looking at it”. None of this needs budget, and together these steps close perhaps a third of the checklist.
Larger items, such as moving the notice board to separate hosting or building a rota that respects human limits, deserve a plan. On the second point, the Atlas does the arithmetic for one example: half a dozen people splitting a 24-hour watch into two weekly slots each would be on duty about a third of the time, above the quarter Google treats as a ceiling, and would need roughly eight people to sit within it. Small research groups should find that sobering and plan shared cover with neighbouring labs.
What happened to Marek
He ran the audit with his group on a Friday. They scored eleven. The first gaps they closed were A1, C1 and D2. Three weeks later, when a calibration run failed overnight, a postdoc declared it within ten minutes, a colleague moved the jobs to a simulator and a short note went to the supervisor on a schedule. Nobody lost a weekend.
If your team also tracks the technical side of emerging platforms, the explainer on quantum approaches to hard optimisation puzzles offers context on why these jobs matter, and the beginner roadmap for platform engineering covers the engineering foundations beneath them.