Why you need a written plan
During an incident, your brain does not work at full capacity. Stress narrows your focus. You forget steps. You skip communication. You fix the symptom but not the cause.
A written incident response plan removes the thinking from the first 10 minutes. You follow a checklist. You know what to do and in what order. The plan handles the process so your brain can handle the debugging.
This template is designed for small teams (2-10 people). It assumes you do not have a dedicated SRE team or an on-call rotation with 15 people. It assumes the person who gets paged is also the person who fixes the issue.
The incident response plan
Step 1: Detect
Your monitoring tool alerts you. You get a notification: “Monitor [name] is down. 3 consecutive failures. HTTP 500.”
Do not panic. Take 30 seconds to verify. Check the site yourself. Check your monitoring dashboard for other alerts. A single monitor down might be a false alarm. Multiple monitors down means a real incident.
If you use PingWatchdog, incidents are created automatically when monitors go down. You get a timeline of what happened and when. This is your starting point for debugging.
Step 2: Declare
If this is a real incident (not a false alarm or a planned event), declare it. Tell your team. Tell your users.
For your team: Post in your designated incident channel (Slack, Discord, Teams). Include what you know: which monitors are down, when they went down, what errors you are seeing. Tag anyone who might need to be involved.
For your users: Update your status page. Post an “Investigating” update. You do not need to know the root cause. You just need to acknowledge the issue. “We are investigating reports of elevated API errors. Next update in 15 minutes.”
Your status page should update automatically from your monitoring data. If a monitor is down, the status page should show it. The manual update tells users you are on it.
Step 3: Investigate
Start with the simplest explanation. Did you deploy recently? Did a dependency change? Did a third-party service have an incident?
Check your monitoring data. What errors are the probes reporting? Connection refused? DNS failure? Timeout? HTTP 500 with a specific error message? The probe data tells you what the outside world sees.
Check your logs. Application logs, database logs, web server logs. Look for errors that started around the same time the monitor went down.
Check your infrastructure. Is the server up? Is the database accepting connections? Is memory or disk usage spiking? Is CPU utilization abnormal?
Step 4: Mitigate
Once you know what is wrong, stop the bleeding. This might not be the permanent fix. That is fine. The goal is to restore service first, then fix the root cause.
Common mitigations:
- Deploy a rollback: If a recent deploy caused the issue, roll back to the previous version. Deploy the fix after the incident is resolved.
- Restart a service: Database connection pools, cache servers, and message queues sometimes need a restart. Restart the service and monitor for recovery.
- Scale up resources: If the issue is resource exhaustion (CPU, memory, connections), scale up. You can scale back down after the incident.
- Fail over: If a single server or region is affected, route traffic elsewhere.
Step 5: Communicate
During the mitigation, post another status update. “Issue identified as database connection pool exhaustion. We are rolling out a fix.”
After the fix is deployed, post again. “Fix deployed. Monitoring recovery.”
After confirming recovery, post the final update. “Service restored. Incident duration: 23 minutes.”
Do not go more than 30 minutes without an update, even if the update is “Still investigating. We have narrowed the issue to the database tier.” Silence creates uncertainty. Updates create trust.
Step 6: Resolve
Wait for your monitors to confirm recovery. Do not declare the incident resolved based on a quick manual check. Let the automated checks verify that the service is stable.
PingWatchdog auto-resolves incidents when monitors return to “up” status for a sustained period. This ensures you do not close an incident prematurely.
Step 7: Post-mortem
Within 24 hours of the incident, write a brief post-mortem. This does not need to be a formal document. A Slack message or a short doc works. Answer these questions:
- What happened? Describe the incident in plain language.
- What was the impact? How long were services affected? Which services? How many users?
- What was the root cause? What specifically triggered the failure?
- How was it detected? Did monitoring catch it? Did a user report it?
- How was it resolved? What steps were taken to fix it?
- What will prevent recurrence? What action items come out of this?
The goal of the post-mortem is not to assign blame. It is to learn and improve. Every incident is a chance to make your system more reliable.
Setting up your incident toolkit
To execute this plan, you need a few things in place before an incident happens:
Monitoring: Uptime monitoring, SSL monitoring, and heartbeat monitoring. You need to know about issues before users report them. Set up PingWatchdog monitoring.
Alerting: Alerts routed to the right channels. Critical monitors go to PagerDuty or phone notifications. Non-critical go to Slack. Configure alert channels.
Status page: A public status page where users can check service health. Updates during incidents should be quick and honest. Set up a status page.
Incident management: Auto-created incidents with timelines. When an incident starts, you should not have to manually create a ticket. The monitoring tool should do it for you. Learn about incident management.
Communication channels: A designated Slack channel, Discord room, or Teams channel for incidents. Everyone knows where to go during an incident.
Practice the plan
Run a fire drill. Pick a non-critical service. Take it down intentionally. Follow the plan. Time how long each step takes.
Most teams find that the hardest part is not the technical fix. It is the communication. “I fixed the issue in 5 minutes but spent 15 minutes figuring out what to say on the status page.” Practice makes communication faster.
Start building your plan
The template above works as-is. Customize the channel names, the escalation paths, and the tool references for your team. Keep it short. If the plan is longer than one page, your team will not read it during an incident.
Start monitoring free with PingWatchdog. Every plan includes incident management with auto-created and auto-resolved incidents.