Claude Incident Recovery Playbook

Official References: Best Practices · Hooks · GitHub Actions · Security

Why this playbook exists

During incidents, teams lose time when ownership and evidence standards are unclear.

This playbook keeps Claude-assisted recovery deterministic under pressure.

Incident severity matrix

First-15-minutes protocol

• declare severity and incident owner
• freeze unrelated merges on impacted surface
• capture first evidence snapshot (logs, failing checks, scope)
• assign next owner per lane

No owner + no snapshot means no reliable control.

Recovery lane split

Run lanes in parallel with explicit boundaries:

• Investigation lane: root-cause hypothesis and reproduction
• Mitigation lane: rollback/hotfix preparation
• Communication lane: status timeline and stakeholder updates
• Verification lane: regression test + rollback safety checks

Lane overlap without ownership usually creates duplicate work and blind spots.

Decision ladder

1. Feature-flag disable (fastest reversible option)
2. Targeted hotfix (when scope is narrow and testable)
3. Full rollback (for high uncertainty or safety-critical surfaces)

Record why options were rejected, not only why one was chosen.

Evidence packet template

Every decision checkpoint should include:

• current severity + owner
• impacted scope
• command/log evidence
• residual risks
• next action + deadline

Post-incident retro minimum

Within 24 hours, write:

• trigger and detection gap
• what slowed mitigation
• what made recovery faster
• guardrail change proposal
• owner + due date

Incidents are expensive lessons. Capture them while details are fresh.

Advanced anti-patterns

One-lane hero mode

Fast at first, fragile later. Split lanes early.

Hotfix merged without timeline communication

You ship code, but teams still operate on stale assumptions.

Incident closed before fresh verification rerun

A green statement without rerun evidence is not closure.

Quick checklist

Before closing:

• severity + owner documented
• mitigation decision recorded
• verification rerun evidence attached
• follow-up guardrail task assigned

Claude helps with speed. This playbook preserves control while moving fast.

Scenario library (real pressure patterns)

Use this library to practice the playbook under realistic pressure instead of ideal paths.

Scenario A — Auth token failure after release

• Signal: sudden spike in 401/403 from one service after deployment.
• Risk: support load surge + customer trust degradation.
• Decision branch: feature-flag rollback vs targeted hotfix.
• Evidence minimum: failing endpoint list, first bad deploy hash, rollback safety check output.
• Owner handoff: investigation owner → mitigation owner → verification owner.

Scenario B — Billing state mismatch

• Signal: dashboard revenue events diverge from transaction ledger.
• Risk: financial and legal exposure.
• Decision branch: immediate write freeze vs selective replay.
• Evidence minimum: mismatch sample size, blast radius estimate, safe replay boundary.
• Owner handoff: data owner + incident commander dual sign-off required.

Scenario C — Permission leak in edge flow

• Signal: low-volume but severe authorization bypass report.
• Risk: security incident escalation.
• Decision branch: emergency rollback and access revocation first, then patch.
• Evidence minimum: reproducible minimal case, revocation completion proof, audit trail snapshot.

War-room artifact bundle (copy/paste)

### Incident Command Snapshot
- Incident ID:
- Severity:
- Incident commander:
- Current phase: detection | containment | recovery | hardening
- Latest stable revision:
- Candidate mitigation path:
- Risk if we wait 30 minutes:
- Next checkpoint at:
- Next owner:

### Checkpoint Decision Record
- Timestamp:
- Evidence reviewed:
- Decision: continue mitigation | rollback | escalate+pause
- Why this decision now:
- Rejected alternatives:
- Owner for execution:
- Owner for verification:

Escalation message templates

Internal engineering escalation

[INCIDENT][SEV-X] <short summary>
Impact: <users/systems>
Current decision: <path>
Immediate ask: <approval/resource>
Next update: <time>
Owner: <name>

Stakeholder update

Status: investigating/mitigating/recovered
Customer impact: <plain language>
Current mitigation: <plain language>
Known unknowns: <top 1-2>
Next committed update time: <time>

Closure acceptance criteria (hard gate)

Close incident response only when all conditions are true:

1. mitigation path executed and verified with fresh command evidence
2. rollback trigger and owner are still documented for 24h watch window
3. residual risks are explicit (not "none" by default)
4. hardening backlog item owners and due dates are assigned
5. communications timeline is complete and reviewable

If one condition is missing, status stays active.

30-minute post-closure review

Run a short review immediately after closure:

• what signal was first but ignored?
• which decision checkpoint took longest?
• where did ownership become ambiguous?
• what one guardrail would have reduced resolution time most?
• what one metric should be added or redefined?

Turn answers into backlog items before context fades.

Provider-specific readiness checks

Before running this playbook, enforce these prechecks:

• Repository state: identify last known-good revision and confirm rollback path is executable now.
• Owner graph: commander, mitigation owner, verifier owner, and communications owner are all named.
• Evidence channel: one canonical thread/document where all checkpoint records are written.

Minimal precheck command set

git rev-parse --short HEAD
git log --oneline -n 5
# add your service health check here

Store command output in the first checkpoint record.

Decision quality guardrails

For each checkpoint decision, require three statements:

1. reversibility statement — how quickly can we undo this decision?
2. blast-radius statement — what can get worse if this is wrong?
3. verification statement — what exact signal proves this worked?

If one statement is missing, decision quality is below advanced standard.

Handoff acceptance test

Before handing to next owner, verify:

• scope boundary is explicit (what is included/excluded)
• unresolved unknowns are listed (not hidden in chat history)
• next checkpoint time is committed
• failure trigger for immediate rollback is written

Use this quick test to reduce context loss across lanes.

60-minute command timeline blueprint

Use this timeline when severity is unclear but impact is real.

• T+00–10: confirm severity hypothesis, freeze risky merges, assign commander + deputy.
• T+10–20: establish mitigation branch options and rollback readiness.
• T+20–35: execute one branch decisively; avoid parallel contradictory mitigations.
• T+35–50: run verification and compare against pre-incident baseline.
• T+50–60: publish decision note and commit the next checkpoint schedule.

The point is not speed alone; it is synchronized decision quality under pressure.

Dependency risk matrix

Classify affected dependencies before choosing mitigation.

This matrix prevents overconfident “single-fix” decisions.

Contradiction handling protocol

If two owners provide conflicting mitigation recommendations:

1. capture both recommendations in one decision record
2. score reversibility and blast radius for each
3. choose the more reversible path unless evidence disproves it
4. schedule a rapid reassessment checkpoint (<=10 minutes)

Conflicts are normal. Unstructured conflicts are dangerous.

Decision confidence ladder

Tag each checkpoint decision with confidence:

• L1 (low): incomplete evidence, reversible action only
• L2 (medium): partial evidence, bounded mitigation allowed
• L3 (high): consistent evidence, broader rollout or closure allowed

Never close incident response with L1 confidence.

Recovery branch strategy matrix

Choose branch strategy deliberately:

Do not maintain all three branches active without one branch owner.

Incident closure review package

Before final closure, produce one package containing:

• timeline (key timestamps + decisions)
• mitigation diff summary
• verification command bundle
• residual risk register
• hardening backlog links

This package should let a new owner understand the incident in 10 minutes.

Leadership handoff note template

### Incident Leadership Note
- What happened:
- Why this decision path was chosen:
- Current confidence level (L1/L2/L3):
- What remains uncertain:
- What we need from leadership:
- Owner for next 24h watch:

Short, explicit leadership notes reduce re-litigation of decisions.