Home » Security Bloggers Network » Prevalent Threats Targeting Cuckoo Sandbox Detection and Our Mitigation

Prevalent Threats Targeting Cuckoo Sandbox Detection and Our Mitigation

by Fortinet Blog | Latest Posts on January 3, 2018

Introduction

An Application Programming Interface (API) is a set of function definitions and subroutines that enable application software to interact with the underlying operating system. These function definitions are commonly monitored by security and monitoring software, such as Anti-Virus, in order to understand and evaluate the behavior of executable programs running on the target machine. Generally, behavioral monitoring can be done in a variety of ways, with one of the most widely used approaches being the API hook.

For the sake of simplicity and stability, many behavior analysis systems, such as malware analysis sandboxes, leverage the user-mode API hook, also known as ring-3, which is the first layer of the API function that the user-mode program interacts with. Cuckoo is an example of a sandbox analysis system adopted by many security researchers to automate malware analysis that uses this API hook. In fact, it is one of a number of core components used by FortiGuard Labs’ Kadena Threat Intelligence System (KTIS).

In this blog post, we will discuss the history of sandbox detection. We will then unveil the malware families that KTIS has observed from spear-phishing emails that attempt to bypass the user-mode API hook in order to evade sandbox detection. And finally, we will share the mitigation method we use to harden the Cuckoo sandbox against this bypass technique.

History of Cuckoo user-mode API hook bypass

Sandbox bypass, particularly in terms of bypassing the user-mode hook, is definitely not something new. It has been discussed many times. For instance, VB2016’s paper documented a comprehensive list of techniques used to detect the presence of the Cuckoo sandbox, discussed the weaknesses in Cuckoo’s user-mode hook techniques, and also proposed some possible solutions at the same time. It also includes a number of other random technical blog posts from a variety of security enthusiasts on this topic.

One of the interesting suggestions mentioned in VB2016’s paper, this one from Check Point’s researchers, to tackle the user-mode hook bypass attack is to replace the user-mode API hook with a kernel-mode driver to monitor the lower level API called by the program. This approach cannot be easily tampered by malware from the user-mode. In fact, because of its robustness, kernel-mode monitoring has long been adopted by most of the sophisticated commercial monitoring and security software on the market, including Fortinet’s FortiSandbox. As a matter of fact, an open-source zer0m0n driver, which is the kernel-mode module extension for Cuckoo, has even been introduced to mitigate this user-mode bypass issue on Cuckoo.

Sponsorships Available

Unfortunately, this driver is not yet officially supported by Cuckoo, and as a result integrating zer0m0n to the latest version of Cuckoo might lead to some stability and compatibility issues. But the good news is that the Cuckoo team is working around the clock with that feature and will enable kernel-mode monitoring in their coming version of Cuckoo.

Malware families that bypass the user-mode hook

By analyzing spear-phishing emails collected by KTIS, we believe that bypassing user-mode API hooks has become a norm for malware in order to evade behavior analysis – whether sandbox analysis systems or Host Intrusion Prevent Systems (HIPS). We have documented a spike in malware families that attempt to detect the presence of user-mode API hooks, and as a result, we decided to dive deeper into these malware families to see how this could affect Cuckoo’s detection capabilities.

Based on our analysis, we determined that the user-mode API hooks bypass technique is usually utilized by malware wrapped with a Visual Basic custom packer named VBCrypter. By searching into our backend telemetry we identified a number of malware families protected by VBCrypter, such as Fariet, Lokibot, NanoCore, NetWireRec, and Remcos to name a few. The distribution of these malware families is shown in Figure 1.

Figure 1. Top 5 Malware Families protected by VBCrypter

User-mode hook bypass effect on Cuckoo’s capabilities

Within the context of the user-mode hook, Cuckoo uses a series of additional hooks to analyze the activities of the target process. For example, Cuckoo will determine if there’s a new process being created in order to keep track of the behavior of the child processes. The behavior monitoring component of Cuckoo, Cuckoo Monitor, establishes this process tracking through the CreateProcessInternalW API hook, which enables it to intercept the process creation routine. In a nutshell, when a child process is spawned, the hook handler ensures that it executes the target application in suspended mode, injects the Cuckoo Monitor component, and then resumes the execution of the application.

Since Cuckoo relies on function hooking to monitor process behavior, the user-mode hook bypass may cause inaccurate results for tracking process behaviors.

Under the hood of the unhooking operation

The most common API unhooking routine that we observed by the malware families we mentioned above is by unhooking those API functions intercepted by the monitoring software. One way to do this is by restoring its original opcode. But first, it needs to determine if the function is hooked.

In the following section, we examine an example of an unhooking operation typically carried by the malware in its ntdl.dll library. Basically, Cuckoo Monitor implements API hooks as a trampoline which adds a jump at the beginning of a specific function that acts as a transition point to its hook handler. The API hooks can then be determined by comparing the first byte of the function to 0xE9 (JMP opcode). If it matches, the malware may assume that the function is hooked. It then looks for the next 0xB8 (MOV EAX opcode) and gets the DWORD value next to it, which is a system index number. This DWORD value will be decremented by one to get the original function number of the hooked API. Afterwards, the malware modifies the memory protection state of the hooked API address to writable. It is then able to restore the hooked API by replacing the trampoline opcode to its original value. Figure 2 shows this unhooking technique found on the actual malware.

Figure 2. Unhooking a hooked API

Unfortunately, the default API hooks mechanism implemented by Cuckoo Monitor is not immune to this attack. So we decided to look into possible ways to harden Cuckoo in order to get around the unhooking mechanism leveraged by the malware.

Mitigation of Cuckoo user-mode API unhooking

Due to the nature of the unhooking routine manipulated by VBCrypter, we figure out a simple mitigation strategy. As explained in the previous section, before the unhooking routine is carried out by the malware the memory protection state of the hooked API must be adjusted to writable, since by default the memory protection state of DLL library loaded in the program is readable and executable only:

0:000> !address kernel32!CreateFileW

Usage: Image

Allocation Base: 766d0000

Base Address: 766d1000

End Address: 76796000

Region Size: 000c5000

Type: 01000000 MEM_IMAGE

State: 00001000 MEM_COMMIT

Protect: 00000020 PAGE_EXECUTE_READ

More info: lmv m kernel32

More info: !lmi kernel32

More info: ln 0x7671cc56

Listing1: Default memory protection of API function in kernel32.dll

So we came up with the following mitigation strategies that can be basically divided into two steps:

Restricting memory protection modification on Cuckoo’s hooked API functions in specific DLL libraries, like ntdll.dll and kernel32.dll, which are the DLL libraries where most critical API functions are intercepted by Cuckoo Monitor in order to monitor the behaviors of the sample.
Registering an exception handler to handle the STATUS_ACCESS_VIOLATION error code triggered from specific DLL libraries with a memory-write ASM instruction.

In the first step, we can forcibly restrict the memory protection modification on Cuckoo’s hooked API functions in specific DLL libraries (for instance, ntdll.dll and kernel32.dll), which in turn could render the hooked API function into a non-writable state and effectively protect the hooked API from being tampered with by malware. We can deploy this logic to Cuckoo’s existing NtProtectVirtualMemory hook that can be found in $CUCKOO_MONITOR/sigs/process_native.rst, which is a reStructuredText template file that consists of the definitions and logics of the API functions that will be hooked by Cuckoo Monitor.

However, we then run into another problem; the sample would throw an access violation when attempting to write to the non-writable address that could yield a program error. So in the second step, we need to circumvent the access violation issue before actually adapting this mitigation strategy. Fortunately, there are several convenient ways to capture the exception error triggered within the program through structured exception handling.

It is important to take note that one of our objectives is to minimize code changes and make it as readable as possible, and afterwards push our implementation to the Cuckoo Monitor upstream. So we thoroughly analyzed the Cuckoo Monitor source code and decided to use Vectored Exception Handling to catch the access violation error that occurs when the malware attempts to write to the non-writable address.

The next important implementation in the second step is the exception handler logic, which is the operation that we wanted Cuckoo Monitor to perform when the exception handler takes control. For the sake of readability, we reused most of the existing code and functions; for example, we reused the disasm() function to get the ASM instruction from the exception address. When the exception got triggered, the exception handler then parses the ASM instruction and looks for the destination address (eg: mov [DEST_ADDR], 0B8) in the write-instruction. The destination address is then inspected to determine if it falls within the range of the ntdll.dll and kernel32.dll address space. If the condition is met, the exception handler will return to the initial program with the exception code EXCEPTION_CONTINUE_EXECUTION, indicating transfer control to the point at which the exception occurred.

Before transferring control back to the program, the exception handler also needs to adjust the instruction pointer (eg: EIP) to the address followed by the exception address, which effectively skips the previous faulty instruction from recurring.

A high-level overview of the mitigation mechanism we implemented is depicted in Figure 3. You can also check out the patch of the hardened Cuckoo Monitor in our repository on GitHub.

Figure 3: High-level overview of Cuckoo user-mode unhooking mitigation

Result of unhooking mitigation

After deploying the patch, we are able to accurately track some of the malware’s behavior that were not seen with the unmodified cuckoo monitor, as shown in Figure 4, below. On the right side of Figure 4 we can see that this chunk of behavior is part of the process hollowing method that replaces the new process memory with the malicious code. We can create a signature based on this behavior. Furthermore, we can detect the malware proactively when Cuckoo Monitor is able to track the target process effectively.

Figure 4: Comparison report of unmodified and modified cuckoo monitor

Conclusion

The ultimate goal of this blog post is to create awareness of the prevalence of malware exploiting user-mode hooks in Cuckoo sandbox, which could be helpful to the security industry, specifically to security researchers who rely on Cuckoo sandbox to automate malware analysis. At the same time, however, we do not believe this mitigation approach is bulletproof, as every remedy can be counteracted. This is especially true for open-source software, which has great transparency to not only defenders but attackers as well. Looking at the bright side, the open-source software also allows and encourages open collaboration among user communities to contribute their ideas and solutions to problems in order to make the software more stable and robust through their joint efforts.

As a side note, based upon our testing, the user-mode bypass techniques used by the malware mentioned above do not stop Fortinet’s FortiSandbox from blocking them.

Signing off,

FortiGuard Lion Team