Deobfuscating Python Bytecode

Introduction

During an investigation, the FLARE team came across an interesting
Python malware sample (MD5: 61a9f80612d3f7566db5bdf37bbf22cf ) that is
packaged using py2exe. Py2exe is a popular way to
compile and package Python scripts into executables. When we encounter
this type of malware we typically just decompile and read the Python
source code. However, this malware was different, it had its bytecode
manipulated to prevent it from being decompiled easily!

In this blog we’ll analyze the malware and show how we removed the
obfuscation, which allowed us to produce a clean decompile. Here we
release source code to our bytecode_graph module to help you analyze
obfuscated Python bytecode
(https://github.com/fireeye/flare-bytecode_graph). This module allows
you to remove instructions from a bytecode stream, refactor offsets
and generate a new code object that can be further analyzed.

Background

Py2exe is a utility that turns a Python script into an executable,
which allows it to run on a system without a Python interpreter
installed. Analyzing a Py2exe binary is generally a straightforward
process that starts with extracting the Python bytecode followed by
decompiling the code object with a module such as
meta
or
uncompyle2
.
Appendix A contains an example script that demonstrates how to extract
a code object from a Py2exe binary.

When attempting to decompile this sample using uncompyle2, the
exception shown in Figure 1 is generated. The exception suggests the
bytecode stream contains code sequences that the decompiler is not expecting.


Figure 1: Uncompyle2 exception trace

Obfuscation that breaks decompilers

To understand why the decompiler is failing, we first need to take a
closer look at the bytecode disassembly. A simple method to
disassemble Python bytecode is to use the built-in module
dis
. When
using the dis module, it is important to use
the same version of Python as the bytecode to get an accurate
disassembly. Figure 2 contains an example interactive session that
disassembles the script “import sys”. Each line in the disassembly
output contains an optional line number, followed by the bytecode
offset and finally the bytecode instruction mnemonic and any arguments.

Figure 2:  Example bytecode disassembly

Using the example script from Appendix A, we can view the
disassembly of the code object to get a better idea what is causing
the decompiler to fail. Figure 3 contains a portion of the disassembly
produced by running script on this sample.

Figure 3: Bytecode disassembly

Looking closer at the disassembly, notice there are several
unnecessary bytecode sequences that have no effect on the logic of the
code. This suggests that a standard compiler did not produce the
bytecode. The first surprising bytecode construct is the use of NOPs, for example, found at bytecode offset 0. The
NOP instruction is not typically included
in compiled Python code because the interpreter does not have to deal
with pipelining issues. The second surprising bytecode construct is
the series of
ROT_TWO
and
ROT_THREE
instructions. The ROT_TWO instruction
rotates the top two stack items and the ROT_THREE rotates the top three stack items. By
calling two successive ROT_TWO or three
ROT_THREE instructions, the stack is
returned to the same state as before the instruction sequence. So,
these sequences have no effect on the logic of the code, but may
confuse decompilers. Lastly, the LOAD_CONST
and POP_TOP combinations are unnecessary.
The LOAD_CONST instruction pushes a constant
onto the stack while the POP_TOP removes it.
This again leaves the stack in its original state.

These unnecessary code sequences prevent decompiling bytecode using
modules such as meta and uncompyle2. Many of the ROT_TWO and ROT_THREE
sequences operate on an empty stack that generates errors when
inspected because both modules use a Python List object to simulate
the runtime stack. A pop operation on the
empty list generates exceptions that halt the decompilation process.
In contrast, when Python interpreter executes the bytecode, no checks
are made on the stack before performing operations on it. Take for
example ROT_TWO from
ceval.c
in Figure 4.        

Figure 4: ROT_TWO source

Looking at the macro definitions for TOP, SECOND,
SET_TOP
and SET_SECOND from
ceval.c

in Figure 5, the lack of sanity checks allow these code sequences to
execute without stopping.

Figure 5: Macro definitions

The NOPs and LOAD_CONST/POP_TOP sequences stop the
decompilation process in situations where the next or previous
instructions are expected to be a specific value. An example debug
trace for uncompyle2 is shown in Figure 6
where the previous instruction is expected to be a jump or a return.

Removing the obfuscation

Now that the types of obfuscation have been identified, the next
step is to clean the bytecode in hopes of getting a successful
decompile. The opmap dictionary from the
dis module is very helpful when
manipulating bytecode streams. When using opmap, instructions can be referenced by name
rather than a specific bytecode value. For example, the NOP bytecode binary value is available with dis.opmap[‘NOP’].

Appendix B contains an example script that replaces the ROT_TWO, ROT_THREE and LOAD_CONST/POP_TOP sequences with NOP instructions and creates a new code object.
The disassembly produced from running the script in Appendix A on the
malware is shown in Figure 6.

Figure 6: Clean disassembly

At this point, the disassembly is somewhat easier to read with the
unnecessary instruction sequences replaced with NOPs, but the bytecode
still fails to decompile. The failure is due how uncompyle2 and meta deal
with exceptions. The problem is demonstrated in Figure 7 with a simple
script that includes an exception handler.

Figure 7: Exception handler

In Figure 7, the exception handler is created using the SETUP_EXCEPT instruction at offset 0 with the
handler code beginning at offset 13 with the
three POP_TOP instructions. Both the meta and uncompyle2
modules inspect the instruction prior to the exception handler to
verify it is a jump instruction. If the instruction isn’t a jump, the
decompile process is halted. In the case of this malware, the
instruction is a NOP because of the
obfuscation instructions were removed.

At this point, to get a successful decompile, we have two options.
First, we can reorder instructions to make sure they are where the
decompiler expects them. Alternatively, we can remove all the NOP
instructions. Both strategies can be complicated and tedious because
absolute and relative addresses for any jump instructions need to also
be updated. This is where the bytecode_graph
module comes in.  Using the bytecode_graph
module, it’s easy to replace and remove instructions from a bytecode
stream and generate a new stream with offsets automatically updated
accordingly. Figure 8 shows an example function that uses the bytecode_graph module to remove all NOP
instructions from a code object.

Figure 8: Example bytecode_graph removing NOP instructions

Summary

In this blog I’ve demonstrated how to remove a simple obfuscation
from a Python code object using the bytecode_graph module. I think you’ll find it easy
to use and a perfect tool for dealing with tricky py2exe samples. You
can download bytecode_graph via pip (pip install bytecode-graph) or from the FLARE
team’s Github page: https://github.com/fireeye/flare-bytecode_graph.

An example script that removes the obfuscation discussed in this
blog can be found here:
https://github.com/fireeye/flare-bytecode_graph/blob/master/examples/bytecode_deobf_blog.py.Hashes
identified that implement this bytecode obfuscation:

        61a9f80612d3f7566db5bdf37bbf22cf
        ff720db99531767907c943b62d39c06d
       
aad6c679b7046e568d6591ab2bc76360
        ba7d3868cb7350fddb903c7f5f07af85

Appendix A: Python script to extract and disassemble Py2exe resource

Appendix B: Sample script to remove obfuscation

This is a Security Bloggers Network syndicated blog post authored by Nick Harbour. Read the original post at: Threat Research Blog