Winston Churchill would have made a great cybersecurity practitioner. “The era of procrastination, of half-measures, of soothing and baffling expedients, of delays is coming to its close,” he said. “In its place we are entering a period of consequences." Any company that has been hacked after failing to patch its software in time will doubtless agree.
Patching is a process, often lengthy depending on the size and complexity of an exposed network. It always starts with the discovery of a bug or flaw, which needs to be fixed to maintain security and compliance. Once identified and disclosed either by a researcher or developer, it then needs to be remedied by the software vendor that created it and released to the public for deployment.
The time between the publication and application of a software security patch is critical. Those who don't patch quickly leave a window of opportunity for hackers to exploit their systems. This article looks inside that window and advises how to close it.
Understand what automation does
Because media and entertainment often muddy the waters between automation and artificial intelligence (think of sentient robots, or manufacturers replacing their workforce with robots) it is important to define what automation does and doesn’t do.
Automation has been with us for a long time already, and it isn’t going away. The fact is, some jobs are so tedious, time consuming, or even dangerous that it’s advantageous to have machines doing them rather than people. When we use machines to do the drudge work, people can concentrate on what we should be doing – planning and implementing creative and imaginative strategies to improve productivity. Software engineers can design new products rather than having to spend hours manually pushing out updates, for instance. Other departments can focus more deeply on customers, products, or services.
The importance of responsible disclosure
The security industry relies on responsible disclosure, which is a code of honor for releasing vulnerabilities. This theoretically stops security flaws from becoming public before a product's developers have a chance to fix it. When researchers discover a security bug, simply publishing it is frowned upon because that leaves users with no immediate defense against attackers.
The right way to tackle a security flaw is to notify the vendor and give them time to fix it. If the vendor acknowledges the notification and agrees to patch the flaw, researchers should coordinate the release of the details to take place after the fix. This gives the vendor time to release and publicize the product patch.
In a poll conducted by 451 Research for Veracode, 90% of respondents see the disclosure of security vulnerabilities as a public good, opining that the identification of such vulnerabilities increases transparency and is positive for everyone’s overall security posture. This acknowledges the fact that attackers might already have discovered the flaw and may soon exploit it.
The window of opportunity
If all goes to plan, the product's users get notified of the software patch at the same time the hackers do. At this point, the race is on between attackers and defenders. The product's users will (hopefully) rush to implement the patch, but the attackers will also move quickly to exploit unpatched systems.
This is a high-stakes race, as each vulnerable system an attacker finds is a potential goldmine. They can use it to install malware, dwelling in systems for months or even years and silently expanding their footprint. They can steal data and sell it online or use it to extort its owners. In some cases, they can even use systems for more destructive purposes, especially when targeting critical national infrastructure organizations.
The value of an exploit correlates directly with its speed. The faster the attacker works, the more vulnerable systems they are likely to find. With this mind, they have developed a sophisticated process chain to wring as much value from each vulnerability as possible.
Black Hat hackers scan for outdated systems looking for the new vulnerabilities, which they will exploit by deploying malware payloads. Then they employ sophisticated mechanisms to find vulnerable systems at scale, often running bots that scan thousands of IP addresses on the web.
Multiple players participate in this criminal value chain. There are data brokers who trade stolen credentials often purchased by access brokers who find and compromise vulnerable systems. They then sell access to ransomware groups and other attackers on cybercrime forums.
Industry best practices and established regulatory guidelines do exist for researchers and software developers to follow, specifically in the International Standards Organization’s (ISO) standards – particularly ISO/IEC 29147 and ISO/IEC 30111.
- ISO/IEC 29147:2018 gives guidelines for the disclosure of potential vulnerabilities in products and online services. It details the methods a vendor should use to address issues related to vulnerability disclosure.
- ISO/IEC 30111:2019 gives guidelines for how to process and resolve potential vulnerability information in a product or online service.
This might sometimes help to delay the development of an exploit, but it isn't a panacea. When the patch is public, the cat is essentially out of the bag, as attackers can reverse-engineer the patch to develop an exploit.
Zero-days can widen the window of opportunity for criminals
While responsible researchers and vendors do all they can to close an attacker's window of opportunity, several things can widen it. Some researchers are irresponsible and publish details of a vulnerability without consulting with software vendors. Worse, black hatters are also constantly researching vulnerabilities in code and software, then releasing the code into the wild. This adds to the number of what are known as zero-day bugs.
This leaves vendors behind the curve as they struggle to catch up with attackers who might already be exploiting these flaws. If they're lucky, customers might be able to follow workarounds such as configuring a product to turn off vulnerable features until a patch can be issued. In the worst-case scenario, there might be no workaround at all, leaving the customer helpless and having to disable the product altogether - if their business operations permit them to do so.
The barriers to effective patching
In the best-case scenario, the customer gets the patch before the details of the flaw are public. Even then, though, many organizations fail to apply those patches. This has led to some catastrophic security incidents. The Wannacry ransomware outbreak disrupted computers worldwide in May 2017, bringing public and private-sector organizations (including, notably, the UK National Health Service) to their knees. Microsoft had patched the vulnerability three months prior.
Microsoft patched another vulnerability, CVE-2019-0708 (BlueKeep) in 2019. This exploit was wormable, spreading via the company's Remote Desktop Protocol (RDP) implementation. A year later, almost half of hospital systems running Windows were still vulnerable to the RDP bug.
The problem with patching
So, what stops organizations from acting quickly when patches are available? It's overly simplistic to blame tardy patching entirely on laziness or a lack of attention. While these may sometimes play a part, there are other factors. The lack of patching in hospitals is a case in point. Many systems running in healthcare are provided directly by medical equipment vendors and are highly regulated. Only the vendor is allowed to patch them, even if they run on Windows, and the testing process is intense.
What about administrative systems that are not subject to the same constraints? Complexity is a key barrier there. Some patches are so complex that they take multiple tries to get right. The flaw in the Log4J Java logging library is a good example. It is an open-source library that is embedded in many different apps. These apps might contain all or part of Log4J's functionality. This makes it difficult for many people to find what needs patching in the first place and is at least partly why so many systems remain unpatched against Log4Shell. As of October 2022, 72% of organizations were still vulnerable to the flaw a full year after it was disclosed.
Complex product portfolios also make it more difficult to triage patches according to risk. The number of documented software vulnerabilities is escalating. Two decades ago, there were just 2,156 annual common vulnerabilities and exposures (CVEs). In 2022, there were almost 25,000.
Security professionals and administrators struggle to keep up with the rising tide of patches, especially when running a lot of software products. The simplest solution is to prioritize each relevant CVE based on its severity by checking its Common Vulnerability Scoring System (CVSS) score, but this isn't entirely effective. Risks vary based on the importance of a vulnerable system to business processes, its proximity to sensitive resources, and its possible role in an attack path. Evaluating these factors takes time, which further expands the window of opportunity for cyber criminals.
The need to test these patches further taxes already overworked security analysts and admins. Ideally, they'd test every patch in a dedicated test environment, but in practice they often lack the infrastructure or time. Instead, they must rely on a best-effort approach by rolling out patches to groups of users and monitoring the results before expanding their scope. Manually managing this process - and rolling back patches that cause problems - is burdensome and error-prone.
Another resource constraint makes this more difficult still: network bandwidth. Traditional patching solutions rely on hub-and-spoke systems that draw patches from a central point. They either force endpoints to download those patches from a server over the WAN, chewing up valuable bandwidth, or they rely on large numbers of local servers that redistribute the patches. That might free up WAN bandwidth, but at a cost; a fleet of local servers is expensive and difficult to manage.
How to slash your patching times
These challenges can extend patching times for all endpoints to weeks, which widens the window of opportunity for criminals. The key is to automate the heavy lifting of the patching process as much as possible. Some security pros and admins have valiantly tried to do this using custom automation, typically using Bash or PowerShell scripts. They quickly find out that this doesn't scale. The more complex the endpoint fleet becomes, the more closely the scripts must be managed, imposing yet another burden on overworked staff.
The solution lies in two combined approaches: edge computing with a peer-to-peer (P2P) architecture for deploying, (also known as edge computing) software and automatic patching tools.
Edge computing with a P2P architecture abandons the hub and spoke approach for a faster, more reliable distribution model. Instead, it takes advantage of each endpoint’s processing power and unused storage to distribute patches to its peers during its downtime. This speeds up patching while reducing the burden on network bandwidth.
The second solution manages the complex process of deploying patches automatically. Gathering all third-party patches directly from the vendors before applying and verifying them unburdens administrators. Managing that deployment progressively to predefined groups - and automatically managing any rollbacks - gives IT staff feedback that the task has been executed correctly and that the infrastructure is compliant.
Security pros and admins have struggled for far too long with these patching problems, but the stakes are now so high that more efficient, automated patching mechanisms are becoming mandatory. It's time for IT to move away from security tools that simply monitor for problems but don't remediate them, closing the circle by ensuring that problems are both identified and fixed.
Talk to Adaptiva to find out more about how our new OneSite Patch solution can help to tame your patching problems.
This is the latest entry in our Adaptiva Practitioner Blog Series. In these blog posts, we share what we know about managing endpoints. Stop by to hear from our own in-house subject matter experts. We are excited to discuss best practices, technical how-tos, and other topics we think you'll find valuable. Our solution architects, product experts, and own IT practitioners have seen and done it all. We are adding new content regularly and are happy to have you here.
