Security Incident

Last Reviewed: YYYY-MM-DD Last Tested: YYYY-MM-DD

Incident Commander Required

As with all major incidents at PagerDuty, security incidents will also involve an Incident Commander who will delegate the tasks to relevant responders. Tasks may be performed in parallel as assigned by the IC. Page one at the earliest possible opportunity !ic page.

Not Sure it's a Security Incident?

Trigger the process anyway. It's better to be safe than sorry. The Incident Commander will make a determination on if response is needed.

Checklist#

Details for each of these items are available in the next section.

  1. Stop the attack in progress.
  2. Cut off the attack vector.
  3. Assemble the response team.
  4. Isolate affected instances.
  5. Identify timeline of attack.
  6. Identify compromised data.
  7. Assess risk to other systems.
  8. Assess risk of re-attack.
  9. Apply additional mitigations, additions to monitoring, etc.
  10. Forensic analysis of compromised systems.
  11. Internal communication.
  12. Involve law enforcement.
  13. Reach out to external parties that may have been used as vector for attack.
  14. External communication.

Attack Mitigation#

Stop the attack as quickly as you can, via any means necessary. Shut down servers, network isolate them, turn off a data center if you have to. Some common things to try,

Cut Off Attack Vector#

Identify the likely attack vectors and path/fix them so they cannot be re-exploited immediately after stopping the attack.

Assemble Response Team#

Identify the key responders for the security incident and keep them all in the loop. Set up a secure method of communicating all information associated with the incident. Details on the incident (or even the fact that an incident has occurred) should be kept private to the responders until you are confident the attack is not being triggered internally.

Isolate Affected Instances#

Any instances which were affected by the attack should be immediately isolated from any other instances. As soon as possible, an image of the system should be taken and put into a read-only cold storage for later forensic analysis.

Identify Timeline of Attack#

Work with all tools at your disposal to identify the timeline of the attack, along with exactly what the attacker did.

Compromised Data#

Using forensic analysis of log files, time-series graphs, and any other information/tools at your disposal, attempt to identify what information was compromised (if any),

Assess Risk#

Based on the data that was compromised, assess the risk to other systems.

Apply Additional Mitigations#

Start applying mitigations to other parts of your system.

Forensic Analysis#

Once you are confident the systems are secured, and enough monitoring is in place to detect another attack, you can move onto the forensic analysis stage.

Internal Communication#

Delegate to: VP or Director of Engineering

Communicate internally only once you are confident (via forensic analysis) that the attack was not sourced internally.

Liaise With Law Enforcement / External Actors#

Delegate to: VP or Director of Engineering

Work with law enforcement to identify the source of the attack, letting any system owners know that systems under their control may be compromised, etc.

External Communication#

Delegate to: Marketing Team

Once you have validated all of the information you have is accurate, have a timeline of events, and know exactly what information was compromised, how it was compromised, and sure that it won't happen again. Only then should you prepare and release a public statement to customers informing them of the compromised information and any steps they need to take.


Communicating During an Incident#


Additional Reading#