Software Incident Drill Checklist

An incident drill tests whether a team can detect, communicate, contain, recover, and learn from a software failure before a real customer or staff impact occurs. The drill should expose missing access, unclear ownership, stale documentation, and unsafe assumptions.

Choose a credible scenario

Use a failure that matters to the business: form delivery, booking, login, integration, inventory, payment, notification, data mismatch, provider outage, or release issue. Define what is simulated and what must not be touched in production.

Set objectives and success

Choose detection, escalation, communication, workaround, restore, reconciliation, or decision objectives. Record acceptable time, required evidence, user protection, and who decides the drill is complete.

Assign roles

Name incident lead, technical investigator, business owner, support, communicator, vendor contact, data or security reviewer, and observer. Give each role authority, contact, and expected action.

The postmortem checklist and communication checklist cover the evidence and updates.

Test detection and access

Verify alerts, dashboards, logs, provider accounts, runbooks, backups, status page, support queue, and escalation contacts. Record what was unavailable or required a person who was not scheduled.

Practice communication

Send simulated internal, customer, vendor, or leadership updates through the intended channels. State impact, known facts, workaround, owner, and next update without exposing private or security-sensitive details.

Practice recovery and reconciliation

Run the safe restore, rollback, retry, queue, failover, or manual process as appropriate. Verify records, duplicate behavior, notifications, permissions, reports, and customer-facing state after recovery.

Record gaps and improvements

Capture timeline, decisions, confusion, missing access, failed step, action, owner, due date, evidence, and next drill. Do not score the drill only by whether the team eventually recovered.