Building a culture of incident response

At Vanta, our goal is to nurture a positive security culture in everything we do—which is especially critical given that helping our customers improve their security and compliance posture starts with our own. Employees are the key to our security resilience, so we strive to build and support a strong culture of incident response in tandem.

Here’s what that means to us at Vanta:

• Knowing how to sound the alarm: Your company has a clear and easy-to-understand way to file an incident, with consistent messaging for what might constitute an incident (bonus: you have keywords or short links to reduce employee time spent searching for company wide incident-response resources).
• Declaring potential incidents: Every employee, across all roles and teams, is encouraged and empowered to declare potential incidents. For instance, technical support teams are often on the front lines of discovering that something isn’t working quite as expected, and should be the first to submit potential incidents.
• Sharing opportunities for feedback and improvement: Every employee, across all roles and teams, is encouraged to share their observations and feedback to help improve the overall incident response process—not only process owners and incident commanders who are often part of engineering organizations.

To build and cultivate an active culture of incident response, let’s start with designing the program. First, you’ll want to think about your tooling (hint: we use incident.io!). Another critical component is recruiting, training, and supporting teammates who can help lead incidents—it’s helpful to mix both brand-new folks who bring new perspectives and long-time incident commanders who bring a wealth of past experience.

This includes deciding and documenting what it means to be an incident commander. Help your incident commanders understand their role and ensure they’re consistent with a clear list of responsibilities and playbooks. It also helps to build out a lightweight training program for your incident commanders—including resources, timelines, expectations, and a structure for shadowing and reverse-shadowing to ensure they have feedback mechanisms built in before they’re ready to operate solo.

To foster an environment where your incident commanders are motivated to help mature your program, invite them to contribute feedback, whether asynchronously or through quick debriefs when needed. This supports a healthy environment where anyone can contribute their feedback, knowledge, and perspectives regardless of role or seniority. Examples of contributions could include things like refining severity levels and criteria, proposing revised steps, and streamlining processes.

When structuring rotations, keep an eye out for the overall energy and scalability of your incident responders; after all, they have quite a bit on their plates already. As you grow and scale your rotation, assess your overall bench strength. Consider training new incident commanders, supporting existing team members, and potentially even designing tiers of responders who are able to serve as back-up to one another and be paged for incidents of different severity, domains, and complexity.

Next, let’s talk about the launch phase for your program. Keep in mind that culture isn’t built overnight with only one action or communication. Nurturing strong cultures of security and incident response requires motivating and enabling your employees with every policy, communication, behavior, and interaction they have with your security and incident response teams.

Here are our suggestions for how to approach this:

• Set the tone that every employee plays an important role in a company’s security—from filing potential incidents to partnering on incident response and investigations when needed. Be sure to have your security and leadership teams emphasize this message wherever possible.
• Recognize when your employees do the right thing with reporting and helping with incidents, and do so in ways that are meaningful for the employee and your company. This could be sharing recognition with their manager or kudos in team/company wide channels or meetings. In addition, recognize the contributions of your incident responders and teammates who respond to incidents to keep the company secure—though be sure to stay away from normalizing hero culture, which could lead to consequences such as burnout, inability to scale, and silos of knowledge.
• Communicate clearly so your employees understand how to declare an incident and are empowered to do so. If and when an incident occurs that requires wider communication, make sure impacted teams have the visibility needed and are informed to the degree necessary about what’s going on—this can help customer-facing teams understand what to communicate and create a greater sense of camaraderie and support around incident response.
• Normalize declaring incidents and communicate (and emphasize wherever possible) that you’d rather have employees err on the side of declaring a prospective incident.
• Hold blameless postmortems that focus on identifying the root cause of the incident without pointing the finger at any teams or individuals. Important questions to ask include things like

Incidents can be stressful, so it helps to view incident response as a company muscle to exercise and strengthen over time. To know what to improve, we suggest measuring your overall incident response culture in ways that are meaningful to your organization. To ensure employees know how to report a potential incident, start by looking at the number of internal searches and hits and keywords for incident response documentation, such as in a centralized wiki or hub.

To understand your current culture around filing incidents, look at incidents filed by severity level and any patterns with incident reporting—such as where incidents were filed and by which teams and orgs. For instance, how many filed incidents were closed as a false positive? If that number is zero, it may be a symptom that your employees aren't erring on the side of filing when in doubt.

And lastly, to help foster an environment where all employees can share their feedback, take a look at their questions, suggestions, and feedback around incident response for the depth of their engagement, the percentage of suggestions that have been implemented, and more. A lightweight internal survey can also help reveal shared observations and actionable recommendations for your overall program.

To close, remember that building a culture of incident response, like security, takes ongoing support and investment—we’re never quite finished, and we’ll continue learning from one another along the way.

Vanta is trusted for continuous security monitoring and compliance by thousands of established companies. Learn more about how Vanta can revolutionize your security.