Patch Management in the Post-SolarWinds Era

The SolarWinds breach, in which hackers inserted malware into software updates sent to thousands of customers and created a backdoor to their IT systems, suggests organizations need to seriously rethink patch management.

Until recently, installing patches and keeping systems up to date was purely a risk reduction exercise, but IT security professionals understand that attackers will eventually find a way in, either by accident or by focusing their efforts on a known weakness.

This requires a different approach to updates and patch management; roll-out tests will still be necessary and valid, but the documentation from the vendor about the patches delivered will also need to increase.

Jayant Shukla, co-founder and CTO of K2 Cyber Security, explained that in most cases, an exploit is not readily available for a vulnerability, and that perpetuates a lax attitude towards patching because of the need to establish a balance between usability and availability.

“However, in the case of SolarWinds, the exploit was available, and its impact was devastating,” he said. “This is a scenario where the patching must be done immediately.”

Shukla pointed out that NIST’s guidelines on patch management can be of service to IT security leaders in a post-SolarWinds world, as having a framework and guideline is always important as a starting point when implementing or updating a process or tool.

“While having a guideline like the NIST SP800-40 Revision 3 is important, it’s also one that is dated, as many have pointed out,” he said. “It hasn’t actually been updated in over seven years, so it’s important to review the guidelines and update them as necessary for your organization’s specific needs.”

Shukla explained that with improvements in software testing for vulnerabilities and regular patching, it is slightly more difficult for attackers to exploit vulnerabilities, so placing backdoors in third-party code provides an easier way to execute a breach.

“We can expect more of these incidences,” he said. “These backdoors may not be detectable earlier in software development via static analysis, and the focus may shift to better monitoring at runtime to detect them.”

It’s also the responsibility of IT security leaders to confirm the integrity of their third-party code and guard against flawed software updates.

Companies producing products that end up in the technology ecosystem should not only have a complete understanding of third-party components used, but should also have published standards or best practices that were followed during the creation of the product.

For Asaf Karas, co-founder and CTO of Vdoo, identifying supply chain risks requires a high degree of transparency into the security state of the acquired software.

To identify known and potential vulnerabilities, security leaders need a software bill of materials (SBOM) for software and devices deployed into their environment, as well as for new updates and patches.

“This is only possible with full cooperation from the software or device vendor who provides the SBOM, or by using automated scanning tools that generate the SBOM from a binary image, firmware or artifact,” Karas said.

However, Karas pointed out security leaders are now responsible for dealing with more sophisticated attacks that introduce backdoors into the supply chain, which was the case in the SolarWinds incident.

“Efforts to combat these attacks can be done through manual analysis effort on critical software and devices, through malware scanning of files or with the use of dedicated gatekeeping binary analysis tools that can detect such malicious behavior,” he said.

Shukla argued there is no easy way to validate the integrity of third-party code, and while care should be taken in selecting third-party vendors and partners, the SolarWinds incident shows that even trusted third-party code may be compromised.

“The responsibility of IT security leaders will be to pay special attention to vulnerabilities that can be exploited and apply patching for those vulnerabilities, regardless of the need to balance between usability and availability,” he said.

Furthermore, organizations must enhance their runtime protection and XDR capabilities as a safeguard against attacks on unpatched or flawed patching.

According to Dirk Schrader, global vice president of security research at New Net Technologies, there is a side effect to this situation, which, so far, seems to be underestimated.

“IT security folks will have to cope with the suspicion that each and every feed, patch, update or upgrade might be compromised,” he said. “[As a] consequence, some will take the stance of, ‘Let others make the move and see what happens,’ and might delay a patch, causing another known issue. It won’t be an easy choice for some,” Schrader said.

Future software testing environments should analyze software in its final form, which means examining the binary image, ready to be deployed, Karas added.

“This is the only source of truth, as this is what is out there in the field,” he said. “Source code analysis isn’t enough, as there could be an injection of malicious code through different build pipeline stages, from malicious build scripts to modification of the build tools themselves.”

Another significant requirement would be the detection of backdoor activity, and not just software vulnerabilities introduced in error. Karas noted that current static application software testing (SAST) tools cannot detect backdoors or malicious behavior introduced by the developer.

“New tooling and processes are needed, as manual code review isn’t a viable option for detecting such malicious modifications of the source code – if done at the source code level, at all,” Karas said.

Another potential remedy could be increased testing for edge cases and obtaining complete coverage, which has the potential to expose backdoors and vulnerabilities that are missed using the current approaches, which primarily rely on common use cases.

Unfortunately, as Shukla noted, covering all edge cases will exponentially increase the number of test cases and the time required for testing and vulnerability detection.

“While it may not be possible for all edge cases to be covered, the ones that could have severe impact should be included, and would have likely exposed the SolarWinds backdoor earlier,” he said.

Nathan Eddy

Nathan Eddy is a Berlin-based filmmaker and freelance journalist specializing in enterprise IT and security issues, health care IT and architecture.

nathan-eddy has 243 posts and counting.See all posts by nathan-eddy