We have all heard about Key Performance Indicators (KPIs) and how critical they can be for your security program, but confusion remains surrounding what KPIs are important to track and how they can be used to measure and improve the organization’s security program. Tracking KPIs is great, but if those KPIs are not relevant and actionable; if you are tracking KPI’s just for the sake of tracking KPIs and they are not being used to inform your security program, your KPIs will become more of a detriment than an enabler for your security program.
At its core, a KPI is a way of measuring the success or failure of a business goal, function or objective, and a means of providing actionable information on which decisions can be based. Quality KPIs serve as a security program enabler and driver for continuous improvement. This is true of both the tactical functions of security operations – looking for attack patterns and trends of malicious activity, as well as the strategic functions of security operations – identifying program gaps and making long-term program decisions.
KPIs should focus on assessing a goal or function and providing actionable information on which decisions can be made. The most effective way to develop meaningful KPIs is to start by identifying which security operations goals or functions are the most critical to the security operations program, then developing KPIs to measure those critical goals or functions. KPIs which will not inform the decision-making process in some way are unnecessary and should be avoided, they will serve only to muddy the waters.
When choosing KPIs to measure, quality should be valued above quantity. Each KPI should have a meaning to the organization and add value to the security program. There are many different methods for evaluating the effectiveness of a KPI; here we will use the acronym SMART. Each KPI should be:
- Simple– KPIs should not be overly complicated to measure. It should be clear what the purpose of each KPI is and how it impacts the security program.
- Measurable– A KPI must be able to be measured in some way, quantitatively or qualitatively. The method by which each KPI is measured should be clearly defined and consistent.
- Actionable– KPIs should be used as a driver for decisions. The purpose of a KPI is to measure performance, and if necessary, take some action based on the results. A KPI which is not actionable serves little to no purpose.
- Relevant– Each KPI should be a measurement of the function being assessed; in this case, the security program. KPIs which are simple, measurable and actionable, but are not relevant to the function being assessed will be of little value.
- Time-Based– KPIs can and should be used to show changes over time. An effective KPI should be able to be collected and grouped by various time intervals to show variations and patterns.
There will never be a set of “correct” KPIs to measure; the goals and objectives for each organization will always be different, and the organization’s KPIs should reflect the individual priorities. The key to choosing KPIs which will have a real, actionable impact on the organization’s security program is to ensure that the KPIs are SMART, focus on the six most common components of a successful security operations program, and are used to further the security program.
For more detailed information on how your organization can use KPIs to enhance your security program, check out our whitepaper “ Key Performance Indicators (KPI’s) for Security Operations and Incident Response” here
The DNA sequence for each human is 99.5% similar to any other human. Yet when it comes to incident response and the manner in which individual analysts may interpret the details of a given scenario, our near-total similarity seems to all but vanish. Where one analyst might characterize an incident as the result of a successful social engineering attack, another may instead identify it as a generic malware infection. Similarly, a service outage may be labeled as a denial of service by some, while others will choose to attribute the root cause to an improper procedure carried out by a systems administrator. Root cause and impact, or incident outcome, are just a couple of the many considerations that, unless properly accounted for in a case management process, will otherwise play havoc on a security team’s reporting metrics.
Poor Key Performance Indicators can blind decision makers
What is the impact of poor KPI’s? All too often the end result leads to equally poor strategic decisions. Money and effort may be assigned to the wrong measures, for example into more ineffective prevention controls instead of improved response capability. In a worst case scenario, poor KPI’s can blind decision makers to the most pertinent security issues of their enterprise, and the necessary funding for additional security may be withheld altogether.
Three best practices are required to address this all too common problem of attaining accurate reporting:
- A coherent incident management process is necessary in order to properly categorize incident activity. Its definitions must be clear, taking into account outliers, clarifying how root causes and impacts are to be tracked, and providing a workflow to assist analysts in accurately and consistently determining incident categorization.
- The process must be enforced to guarantee uniform results in support of coherent KPI’s. Training, quality assurance, and reinforcement are all necessary to ensure total stakeholder buy-in.
- Security teams must have the technologies to support effective incident response and proper categorization of incidents.
There are several ways that the IncMan platform supports the three best practices:
First, IncMan provides a platform to act as the foundation for an incident management program. It provides customizable incident forms allowing for complete tailoring to an organization and the details it must collect in support of its unique reporting requirements. Custom fields specific to distinct incident types allow for detailed data collection and categorization. These custom fields can be coupled with common attributes to track specific data, thereby providing a high level of flexibility for security teams in maintaining absolute reporting consistency across the team’s individual members.
Next, playbooks can be associated with specific incident types, providing step-by-step instructions for specialized incident response activities. Playbooks enforce consistency and can further reinforce reporting requirements. However, playbooks are not completely static, and while they certainly provide structure, IncMan’s playbooks also offer the ability to improvise, add, remove or substitute actions on the fly.
The platform’s Knowledge Base offers a repository for reference material to further supplement playbook instructions. Information collection requirements defined within playbook steps can be linked to Knowledge Base references, arming analysts with added information, for example with standard operating procedures pertaining to individual enterprise security tools, or checklists for applicable industry reporting requirements.
IncMan also includes Automated Responder Knowledge (ARK), a machine learning driven approach that learns from past incidents and the response to them, to suggest suitable playbooks for new or related incident types. This is not only useful for helping to identify specific campaigns and otherwise connected incident activity but can also highlight historical cases that can serve as examples for new or novice analysts.
Finally, the platform’s API and KPI export capabilities enable the extraction of raw incident data, allowing for data mining of valuable reporting information using external analytics tools. This information can then be used to paint a much clearer picture of an enterprise’s security posture and allow for fully-informed strategic decision-making.
Collectively, the IncMan features detailed above empower an organization with the means to support consistency in incident categorization, response, and reporting. For more information, please visit us at https://www.dflabs.com