## Introduction Post-Incident Review (PIR) is a structured, blame-free review held after a service incident, delivery miss, security event, or other impactful disruption. This guide is for CTOs, CIOs, VPs of Engineering, product leaders, and program managers who need a repeatable management practice that turns disruption into durable value. Practical takeaway: establish clear triggers, run a well-facilitated session using strong evidence, produce high-quality actions with owners and dates, and track closure across a visible operating cadence so findings influence roadmaps, risk, architecture, and budgets. ## When to run (and not run) a PIR Run a PIR when severity and impact justify organizational learning and change: - Severity triggers: - Sev-1 (critical customer impact, safety, or data loss): always. - Sev-2 (material customer impact, SLA breach, or multi-hour degradation): always. - Sev-3 (localized or short-lived impact): at least one per category per quarter, or when similar incidents recur 2+ times in 30 days. - Security events: as required by policy or regulation. - Timing: schedule within 24–72 hours after stabilization. Hold a short “hot wash” within 24 hours to capture raw facts, then a full PIR once evidence is collected. Consider a 2–3 week follow-up to check action outcomes. - When not to: do not use PIR to assign blame, evaluate individual performance, or dissect trivial alerts without customer or business impact. Use lightweight ticket hygiene or team retrospectives for small issues. - Prerequisites: a named incident record, severity, owner, start/stop times, and a repository for artifacts (dashboards, logs, chat transcripts, tickets, change records). ## Roles, facilitation, and psychological safety Assign explicit roles for consistency and speed: - Executive sponsor: sets expectations, unblocks funding, receives summaries. - Facilitator: neutral chair; enforces timebox, safety, and structure; keeps the group on facts and decisions. - Scribe: captures timeline, insights, actions, owners, and due dates in the standard template. - Incident lead(s) and SMEs: present facts, supply artifacts, answer questions. - Action owners: accountable for each corrective or preventive action. - Comms owner: prepares stakeholder updates and executive summary. Psychological safety rules: - Focus on conditions and decisions, not people; ban counterfactuals like “should have.” - Prefer quotes and data (“at 09:12 on-call said…”) over interpretations. - Encourage surfacing uncertainty and trade-offs made under time pressure. Facilitation tips: - Timebox: 60 minutes for Sev-2/3; 90 minutes for Sev-1. - Parking lot: capture tangents as potential actions; resolve offline. - Evidence first: demo dashboards or logs for key moments before debating causes. ## Structure and terminology: what goes where Use consistent definitions to avoid mixing analysis and decisions: - Timeline: factual sequence of signals, actions, and decisions with timestamps and sources. No analysis here. - Contributing factors: conditions that increased the likelihood or impact (e.g., demand spikes, misaligned thresholds, unclear ownership). Multiple can apply. - Root causes: the most influential contributing factors you decide to address. Avoid single-cause thinking; “root” means strongest leverage, not sole origin. - Corrective actions: reduce current risk by mitigating the specific issue that occurred (e.g., configuration change, playbook, alert). - Preventive actions: reduce future likelihood or impact by changing systems, process, or governance (e.g., capacity guardrails, readiness reviews, dependency contracts). - Learning review: a broader reflection on how the organization thinks, decides, and coordinates; used to shape training, standards, and architecture principles beyond a single incident. ## Evidence collection Before the meeting, assemble: - Monitoring graphs for the affected services and dependencies. - Change records (deploys, config changes, feature flags) ±4 hours around onset. - On-call logs, chat transcripts, tickets, and customer support summaries. - Capacity/traffic baselines and SLO/error budget reports. - Vendor status pages or contracts relevant to the event. Store artifacts with the PIR record so future teams can reuse them. ## Operating cadence and governance Make PIR an operating rhythm, not an occasional ritual: - Triggering and scheduling: incident owner opens a PIR record at declaration for Sev-1/2, or upon escalation for Sev-3 recurrences; facilitator schedules within 72 hours. - Weekly action review: program manager or facilitator reviews the action register for status, blockers, and escalations; publish a short status note. - Monthly trend review: aggregate actions and incidents to identify recurring themes, dependencies, and systemic gaps; propose cross-team initiatives. - Quarterly executive summary: highlight top risks addressed, open risks, reliability trends vs SLOs, and investment decisions taken. - Escalation: any Sev-1 action overdue by >7 days or Sev-2 by >14 days triggers sponsor review; re-baseline scope or secure resources. - Closure tracking: actions are Open, In progress, In validation (success test pending), or Closed (evidence attached). No “Done” without proof. ## Templates you can copy PIR document (single record per incident): - Summary: incident ID, severity, services, start/stop, customer and business impact. - Expected vs actual: what should have happened, what did. - Timeline: timestamped events with sources. - Contributing factors: list with short explanations. - Root causes selected: 1–3 with rationale and expected leverage. - Corrective actions: owner, due date, success test, status. - Preventive actions: owner, due date, success test, status. - Communications: stakeholders notified, external statements if any. - Evidence links: dashboards, logs, tickets, change records. - Learnings: decisions, standards, or training to update. Action register (portfolio view): - Action ID, linked incident(s), type (corrective/preventive), owner, team, effort (S/M/L), priority score, due date, status, success test, evidence link. Executive summary (one page): - What happened and customer impact (non-technical). - Why it mattered (business, risk, compliance). - What we changed (top 3 actions and dates). - How we will prevent recurrence (structural investments, guardrails). - What leadership should know (decisions, trade-offs, funding asks). Recurring-trend review (monthly): - Top themes by count and impact (e.g., capacity, deployment, vendor). - Cross-cutting risks and dependencies. - Action portfolio health (closure rate, lead time, overdue items). - Proposed initiatives (owners, timelines, expected risk reduction). ## Prioritization, ownership, and action quality Set clear criteria so actions compete fairly with feature work: Priority = (Impact + Likelihood + Time sensitivity) / Effort. - Priority score (1–5 scale each): - Impact: customer/business harm avoided. - Likelihood: chance of recurrence within 90 days. - Time sensitivity: cost of delay if deferred one quarter. - Effort: inverse weight (High effort lowers score). - Ownership: one accountable owner per action; supporting teams listed as consulted (RACI pattern). - Quality bar (use SMART): specific, measurable, achievable, relevant, time-bound; define a simple success test (e.g., canary shows <1% 5xx at 2x peak). - Funding: ring-fence reliability capacity (e.g., 10–20% of team time) or set guardrail budgets based on error-budget burn; use PIR themes to propose roadmap epics and architecture work. - Closure: do not close without attached evidence (screenshot, query, test run, or procedure link) and a passing success test. ## Metrics: define precisely - Leading metrics: indicate rising risk before failure (e.g., saturation, queue depth, error-budget burn rate, paging volume per on-call shift). - Lagging metrics: measure outcomes after events (e.g., time to detect, time to mitigate, time to restore, customer minutes impacted, change failure rate for affected area). - Diagnostic metrics: help locate the issue (e.g., request error ratio by dependency, cache hit rate, retry storms). - Guardrail metrics: constrain behavior (e.g., max concurrent risky changes, blast-radius limits, allowed SLO error-budget consumption per week). - Vanity metrics: look good but do not change decisions (e.g., number of pages created without usage, word count in PIRs). Avoid or demote. Track also: action closure rate, average action lead time, recurrence rate of similar incidents, and estimated impact minutes avoided next quarter (based on tests and simulations). ## Realistic example: SaaS sign-in outage Scenario (illustrative): a Sev-2 sign-in outage increases support tickets and delays revenue recognition. - Context: SLO for sign-in availability is 99.95%; error budget 22 minutes/month. A partner campaign doubles peak auth traffic for 30 minutes. - Timeline (simplified): - 09:05 Monitoring shows spikes in auth 5xx; first user reports failed login. - 09:12 Support opens incident; on-call begins triage. - 09:28 On-call identifies auth service rate-limit saturation; flags partner traffic spike. - 09:40 Mitigation: raise per-client limit and enable autoscaling step; errors fall. - 10:15 Backlog clears; service stable; 2.5 hours partial impact, 12% of users affected, 420 tickets created. - Contributing factors: - Partner-driven demand spike not forecasted in capacity plan. - Rate-limit thresholds tuned to historical weekday peaks. - Fragmented ownership between growth and platform for campaign readiness. - Limited scenario testing around rate-limit behavior. - Root causes selected (leverage): - Missing campaign readiness process and dependency contract. - Inflexible rate-limiting strategy for bursty partner traffic. - Actions and influence on the organization: - Corrective: tune auth rate limits; add autoscaling guardrails; targeted alerts on auth error ratio and queue depth. Owners: Auth TL (7 days). Roadmap: hotfix now, minor hardening task in current sprint. - Preventive: institute growth-platform readiness checklist; add partner traffic segment dashboard; run load test at 2x expected campaign peak; adopt token-bucket rate limiting per client. Owners: Growth PM, Platform EM (30 days). Roadmap: create “Campaign Readiness” epic; Architecture: document rate-limiting standard. - Risk register: add “partner campaign capacity” as a known risk with mitigation and trigger points; tie to monthly trend review. - Budget: allocate 2 team-weeks to hardening and load testing; ring-fence 15% reliability time this quarter. - Reliability objectives: tighten sign-in SLO burn alerting; set guardrail of max 25% monthly error budget consumed by a single event; if breached, auto-escalate to sponsor. - Closure tracking: success tests include synthetic sign-in at 2x peak with <1% 5xx and autoscaling within 2 minutes; attach load-test report to the PIR record before closing. ## Common pitfalls and how to avoid them - Treating PIR as blame assignment: enforce safety rules and focus on conditions. - Skipping evidence: require artifacts before analysis; “show, then tell.” - Action sprawl without prioritization: use the priority score, limit WIP, and ring-fence reliability capacity. - No owner or success test: do not accept actions without both. - Over-fixating on a single “root cause”: list multiple contributing factors and pick leverage points. - Holding the meeting too soon: do a short hot wash, then a full PIR once evidence is ready. - No follow-through: run weekly action reviews and escalate overdue items. ## Next steps - Adopt the templates above and create a shared PIR repository. 2) Define your severity triggers and 72-hour scheduling rule. 3) Appoint facilitators and scribes; train them on safety and structure. 4) Start a weekly action review and a monthly trend review. 5) Tie PIR themes to roadmaps, SLOs, risk registers, and budget planning. 6) Measure closure rate, lead time, and recurrence for the first 90 days. ## Conclusion PIR is a management practice, not a meeting. With clear triggers, strong facilitation, evidence-first analysis, and disciplined action tracking, you will reduce recurrence, speed recovery, and make better investment decisions that show up in roadmaps, reliability, and stakeholder trust.