As with most topics of cyber security we could go on to write multiple books on the topic of vulnerability triage. However, for the purpose of keeping this short and sweet we are going to make some assumptions about your current security processes.
The main assumption we will make is that you perform some level of active security testing. This can range from full penetration tests and prolonged adversary emulation exercises to monthly vulnerability scans with one of the many vulnerability scanning products and solutions out there. It doesn’t really matter which, so long as you’re actively making some effort to identify security vulnerabilities across assets you control. Whether it’s hardware, software, process or people the activity of identifying vulnerabilities is ultimately going to leave us with a list of issues that we need to address.
If you work with a good penetration testing supplier, a lot of the hard work is likely done for you. Penetration testers will use a combination of the Common Vulnerability Scoring System (CVSS) and some form of impact & probability / risk metric to grade the severity of any identified vulnerabilities based upon the information they have available to them. Combined with the testers experience, report items will then be sub-ordered to ultimately provide a top down technical report itemising the most severe to least severe vulnerabilities.
While a lot of the hard work is done for us there will be times, particularly early in your cyber security journey, that will result in a handful of High & Medium impact vulnerabilities being discovered that at first glance, all appear as critical as each other and need to be addressed as a matter of urgency. For someone in a position of responsibility early in the cyber security process this can be a daunting task, particularly when individuals further up the chain of command are demanding immediate results.
Stop. Take a breath and orientate yourself with the information you have.
The first thing we need to do before any level of active remediation effort takes place, is to come up with an effective remediation plan to ensure a structured approach is followed when deploying changes, as a result of security issues. Without these, one of two things are likely to happen:
1) “The task is too daunting; I have uptime and availability to worry about and this is not causing us an immediate operational impact. I will deal with this when I have time” – Spoiler Alert… The to do list never gets any smaller. The report will likely sit in a file share or a desk drawer until the next test comes around, a customer asks for evidence of security tests or, should the planets align you actually end up with the time to look into this.
2) “Let’s just start at the top and work down”. While this is indefinitely better than the first outcome, due to the fact we’re actually trying to address issues, this can still present problems. Problems can arise particularly where remediation efforts are allocated as a secondary task to normal day to day function. While we in no way want to disrupt the smooth operating of the organisation, if the task is not given the proper attention. We are likely going to start with the best intentions, but as other tasks get in the way the efforts tail off. The end result of this is that we remediate some or all of the high criticality vulnerabilities but don’t consistently remediate all vulnerabilities.
If you fall into either of these categories, it’s likely that you have two core problems. The first is that your vulnerability management process lacks enough allocated responsibility and/or resources. The second is that there is no process to analyse and triage the output of vulnerability identification exercises, so that we can produce an actionable plan that allows us to best align our efforts and yield the highest output in the shortest time. So, let’s dive right in…
While not the primary objective of this blog post, it’s still worth touching on the topic of responsibility. Responsibility, or more accurately the lack of, is a common issue that particularly plagues SMEs. By reading this post you are already showing that you understand you have some level of responsibility for the protection of your customers data. But with a million other things to do and budget dictating that hiring dedicated resource is unfeasible, you are going to have to work with your existing teams to be sure that these vulnerabilities get fixed. It’s worth remembering that while the responsibility for deploying fixes at a tactical level can be delegated to relevant teams, the overall responsibility for the protection of data still lies with you. Make sure you track your team’s efforts and allocate timelines that are both achievable and ensure remediation is completed before something bad happens. Once fixes get tested and deployed, ensure that efforts are made to retest so that we can ensure fixes are effective and deployed correctly. Too many times changes are made without proper understanding of the issue, or worse, with a slap dash “just get it done” approach. The result of either of these is often the continued existence of the security vulnerability under the false understanding that the issue has been fixed. This can all be managed with a proper “Test & Deploy” process, but that’s for another post here.
However, before we allocate tasks to the relevant teams, we must perform our triage process. For this we need to delegate a separate team whose core function within the vulnerability management process is to rapidly analyse and prioritise incoming vulnerability data. This team will be known as the ‘Vulnerability Triage Group’ (VTG) and should hold between its members experience in the organisation’s IT management, general cyber security risk and wider business risk.
The bottom line is that while you are responsible for the overall protection of your commercial, staff and customers data, you will likely need to work with several teams to implement a complete vulnerability management process. Prioritisation is the responsibility of a core group (the VTG) who have enough significant experience to understand the issues and their impact within your business context.
As always, the NCSC have outlined vulnerability management steps here, and it is advised material for anyone who wants to understand the vulnerability management process. As such we’ll lean on its core principles below.
At this point we’ve hopefully allocated responsibility to asset owners for the technical remediation of both current and future vulnerabilities identified across systems under their control. We’ve informed them of this responsibility and the need for this process. We’ve made sure they understand that dedicated time will be allocated to fixing security issues, but they also understand there may be occasions when these fixes take priority, particularly when threat intelligence activities identify malicious actors actively exploiting the vulnerability in the wild or where exploitation is trivial.
We’ve also allocated a core team that hold the correct level of experience and knowledge to understand your vulnerability identification output within the context of your business; the VTG. This group is responsible for the prioritisation of vulnerabilities prior to the delivery to asset owners.
With our team now preparing for incoming tickets we can begin to work through our identification process output. As mentioned, there are situations where, despite the best efforts of the security tester, we’re left with a load of vulnerabilities to remediate, many with high and medium criticality findings and we’re not quite sure where to start.
Like any good strategy we want to make the maximum impact in the least number of moves.
The process of ordering these vulnerabilities by criticality is called “Triage” which can be loosely defined as “The process whereby previously identified vulnerabilities are evaluated, impacted, prioritised and remediation tasks assigned.”
We’ve already said vulnerabilities are ordered by CVSS.. So why invest the time and resources into triage when it’s basically done for us? Surely, we can just get stuck in.
No.. We can’t.
Let’s look at an example: AcmeOrg has around 20 members of staff, and a small IT team of one or two people. For the sake of argument an experienced technician and a junior in training. The CEO has been alarmed by the increasing number of cyber-attacks they see on the news and as such has allocated some budget to perform a penetration testing exercise. Having just performed their first penetration testing exercise across all their assets, from both external and internal perspectives, the team gets dumped with 2 very large reports with a load of red and orange all over them. The CEO reads the report and understands that the issues are not a reflection of the current teams’ efforts, but rather the nature of security coupled with a resource issue… After all, their company is growing and so should their IT team to support them. They tell you that you can sit down with them, go through the Cyber Security Skills Matrix and create a job posting to fill a more permanent position. …………This could not be going any smoother… what’s the catch?
“In the meantime, we need you to sort all of this report out ASAP. Go get me a timeline”
Whelp. We still only have a team of two and there’s a lot of vulnerabilities… Where do I start? The report is ordered by CVSS so I’ll just start at the top and work down and come up with the arbitrary figure of 2 weeks because I really don’t know how long it’s going to take and you can get a lot done in 2 weeks.
The reality is that across the two reports, it’s likely some vulnerabilities with CVSS 9 are going to be less exploitable than some vulnerabilities with a CVSS of say 7.5. For example, that locally exploitable missing Microsoft patch on the machine in the corner that’s sat on its own VLAN with no outbound access is going to be less critical than the directory traversal on the public web server.
So, if we take this top down approach the obvious result is that were spending time remediating vulnerabilities that are not likely to be exploited instead of fixing vulnerabilities that are more likely to be exploited for direct impact or used as part of a wider kill chain. Which is not ideal. Don’t forget that the vulnerability might have been sat there for months already prior to testing, so taking a little bit of extra time to come up with an effective plan is going to be much more efficient than just spraying update commands across everything we can see.
Instead, we could take a morning to assemble our VTG and go through the reports to come up with an effective plan.
One objective of the VRT is to categorise vulnerabilities under three labels: Fix, Acknowledge, Investigate.
We’re going to prioritise and deploy fixes for these issues. Regardless of the CVSS, these vulnerabilities pose a risk to our organisation and our ability to maintain Confidentiality, Integrity or Availability (CIA) of our data and systems.
We understand these issues exist, but they aren’t a priority right now, we’ll justify why we aren’t fixing them (In documentation!) and we’ll re-evaluate the issues at a later date.
We are not sure if we need to fix or acknowledge this vulnerability. There are unknowns that are blocking further action. We’re going to allocate a completion date and investigate in the interim to get more information, so we can move the vulnerability to either the Fix or Acknowledge categories.
A note on Acknowledgment – It can be tempting at times to just “accept the risk” and start lumping everything into the Acknowledge category. Ultimately this is a business decision, but whenever something ends up in this category you need to document exactly WHY this issue is being acknowledged with sound enough justification. So that if you are compromised and this vulnerability gets exploited you can justify why you didn’t fix it to third parties… remember… by knowing it’s there you’re no longer ignorant and your decisions now could have serious ramifications down the line.
Anyways, we’re making progress, we’ve started to document a vulnerability management process that we can use to align our efforts and make a real impact on our organisation’s security. But how do we decide what vulnerability goes into what box?
To prioritise vulnerabilities, it naturally boils down to two questions:
1) How exploitable is this vulnerability?
2) If it gets exploited what’s the worst that’s going to happen?
This line of thinking naturally dictates that externally facing vulnerabilities that will allow an attacker to gain access to data, accounts, systems and anything else we don’t want them to access are going to be a priority. However, ensure you don’t get tunnel vision. On the ground, networks are seldom built with the classic network boundary we all know and love. We’re integrating systems all over the place. From Azure to AWS and Online CRMs the traditional network boundary is becoming blurred. So, make sure you really take time to think about the vulnerability impact and how it will be exploited. This is where the teams wider experience comes into play since the impact of exploitation is at both a technical and business level. For example, we at a tactical level, care an awful lot more about the Remote Code Execution (RCE) vulnerability than the Local file Inclusion (LFI). However, in the wider operating environment we care an awful lot more about the LFI on systems holding our Critical Data that we do about the stand-alone printer in the guest network. Some simple questions that can help when triaging individual vulnerabilities are:
- Does this system hold critical data, or is it connected to systems that do?
- Is there exploit code available for this vulnerability and, if so, is it public?
- Is it being exploited actively in the wild? (good threat intelligence can really help with this whole process)
- What does successful exploitation give an attacker access too, what systems can they see?
- What are the chances this has been exploited already?
- How easy is it to detect exploitation of this vulnerability?
- If this issue is exploited, does it make any other vulnerabilities exploitable?
- How old is the vulnerability?
- How much is it going to financially cost to fix this issue?
- How much time needs dedicated to fixing the issue?
- Who is responsible for the asset?
- Who is responsible for the fix?
- Do we need to engage any third parties to address this issue?
- Are there any temporary mitigations we can put in place pending further investigation?
- Has this vulnerability been identified previously, and risks accepted? – if so, what is the Risk ID? and importantly has the risk changed?
On a side note: you can see that one of the questions is “What are the chances this has been exploited already?”. Its again not the focus of this post, but there comes a point where you have to decide if you need to move into an incident response phase and start looking for Indicators of Compromise and adversary activity.
By answering the above questions, we can start to gain a deeper understanding of the vulnerabilities that exist across our assets and as such start to come up with an actionable plan that allows us to make the biggest impact in the least amount of time. Understandably though, these questions require a level of both business and technical input, which in many situations we may not possess. Don’t be afraid to lean on your preferred security consultancy for help in understanding the issues. But, if you’re really stuck, flustered and don’t have the time to perform a full vulnerability triage exercise the following guidelines from the NCSC are a good place to start:
Priority 1: Fix Internet services and off-the-shelf web applications that can be exploited automatically across the Internet with no user (or attacker) interaction.
Priority 2: Fix bespoke/niche web applications that can be exploited across the Internet with no user (or attacker) interaction.
Priority 3: Fix Issues that can be exploited across the Internet with minimal user interaction (workstation vulnerabilities, drive-by downloads, email-based attacks).
Priority 4: Fix issues that can be exploited across the Internet with social engineering of users (malicious applications downloaded from the web or sent via email). These attacks require your users to play a part.
It’s a fact of life (for the majority of us at least) that we can’t afford everything we want, sometimes closing a security vulnerability in the most ideal way is going to cost far too much. It’s also a fact of life that phishers are going to phish, no matter what you’re told by a security vendor you aren’t going to stop 100% of those emails hitting your users. Finally, it’s also a fact of life that no matter how much we all want to pretend it isn’t true, some systems just can’t be upgraded.
When you find yourself in any of these positions we can’t just say “well we can’t fix it so why bother?”; we need to make the best of a bad situation. We can’t implement the ideal fix, so we need to introduce compensating controls. These are changes we can make to reduce and contain the overall risk posed by the vulnerability to an acceptable level. Be under no illusion compensating controls can often be more difficult to implement than the ideal fix. They are in no way the easy and cheap option and should only be used when there is an absolute blocker on the preferred remediation.
We’ve shown today that a structured approach to vulnerability triage is an essential part of an efficient vulnerability management programme and, while it can take some time and effort it’s well worth it.
If you are looking for any guidance on how to best implement Vulnerability Triage across your organisation please get in touch at firstname.lastname@example.org and we’d be happy to share any guidance where we can.