SBN

Demystifying A Docker Image

Six months ago ForAllSecure started analyzing Docker images. What does this mean?  Imagine we have a user who wants us to fuzz their application. How do they give it to us?  Do they tar it up? Do they give us access to an environment where it’s running?  Do we integrate into their build pipeline? Applications are an entire ecosystem — they require specific library versions, environment variables, users, etc. While it may seem like a small limitation conceptually, this added barrier can contribute to the friction between development and security teams, especially as organizations look to incorporate security as a part of their build cycles.

 This is where Docker comes into play. We wanted Docker as a packaging solution for our users because it’s accessible and easy to use, but we didn’t want the overhead of the Docker daemon and all the other fancy features that come with it. We ended up building our own lightweight version of Docker, allowing ForAllSecure to accept Docker images, while running them with the barebones RunC runtime. This allows us to analyze code without requiring changes to developer behavior. In this blog, we’ll focus on the first part of the problem: how to ingest Docker images.  

Accompanying this post is the open sourcing of Rootfs Builder, the tool we use to extract a rootfs from a Docker image. A Docker image provides a portable, efficient format. Instead of sending a 4GB rootfs across the wire, users can simply give us a string like “ubuntu:latest” and ForAllSecure servers can pull the image and extract the rootfs. This value prop doesn’t just apply to ForAllSecure. Rootfs Builder allows any run time to ingest a Docker image and extract the rootfs. We chose Runc, but the extracted rootfs is vanilla (i.e. there is no Docker specific information) and will work with rkt, NSJail, etc.

It’s worth noting that there were a few already existing solutions for building a rootfs from an image. Unfortunately, they do not handle whiteouts correctly (explained further below). I also want to give a shout out to Makisu and Kaniko (written by Uber and Google respectively), which do provide functionality for extracting an image from a rootfs. They solve the problem of building Docker images in an environment not suitable for Docker, namely Kubernetes. We chose to not use their software because it was still a bit too feature-full for us.

Now that you understand the problem we are trying to solve, we can dive into the question, what is a Docker image?  How do we go from a Docker “image” which is just some string like “alpine:latest” to a running instance of Alpine?  In short, an image is a glorified tarball.  It consists of various layers, which when merged together, form the rootfs of the container. To understand these layer, we need to make a quick detour to discuss the underlying technology, OverlayFS (OFS).

OverlayFS

OverlayFS layers two directories on a single Linux host and presents them as a single directory. The first directory, referred to as the “lower” directory, is read-only and usually provides the base file system. The second directory, referred to as the “upper” directory, reflects any changes made to the lower directory, while leaving the lower directory itself unchanged. If a file is removed, a “whiteout” file is created in the upper directory, to simulate the removal. The mount point is the 2 merged directories. Note that OFS requires support for extended attributes in order to store metadata regarding whiteouts. 

OFS is the storage driver for Docker and, as you can imagine, is well-suited for containers. The lower directory is the filesystem, and then each layer on top is a snapshot of the container filesystem at a given time. OFS is an efficient way to generate and store diffs to a filesystem.  

Try it out yourself:

# Create a tmpfs because a tmpfs has support for extended attributesroot@5e2bb73f7afd:/tmp/tmpfs# Mount -t tmpfs tmpfs /tmp/tmpfs# Create the lowerdir, upperdir, workdir and the merged dirroot@5e2bb73f7afd:/tmp/tmpfs# cd /tmp/tmpfsroot@5e2bb73f7afd:/tmp/tmpfs# mkdir  lowerdir, upperdir, workdir, merged# I moved the alpine base filesystem into the lowerdir to make the example more meaningfulroot@5e2bb73f7afd:/tmp/tmpfs# mv /alpine /lowerdir# Create the OFS mountroot@5e2bb73f7afd:/tmp/tmpfs# mount -t overlay -o lowerdir=/tmp/tmpfs/lowerdir,upperdir=/tmp/tmpfs/upperdir,workdir=/tmp/tmpfs/workdir overlay /tmp/tmpfs/merged# Notice that now the merged directory, previously empty, reflects the lower directoryroot@5e2bb73f7afd:/tmp/tmpfs# ls merged/bin  dev  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var# If we create a file in the merged directory, it gets reflected in the upper directoryroot@5e2bb73f7afd:/tmp/tmpfs# ls upperdir/root@5e2bb73f7afd:/tmp/tmpfs# touch merged/helloroot@5e2bb73f7afd:/tmp/tmpfs# ls upperdir/hello# If we remove a file from merged, it’s also reflected in the upper directory as a whiteout fileroot@5e2bb73f7afd:/tmp/tmpfs# rm merged/bin/archroot@5e2bb73f7afd:/tmp/tmpfs# ls -la upperdir/bin/total 0drwxr-xr-x 2 root root   60 Sep 16 20:47 .drwxr-xr-x 3 root root   80 Sep 16 20:44 ..c--------- 1 root root 0, 0 Sep 16 20:47 arch

 

What’s in an image?

Now that we understand the tech underlying a Docker image, we can look inside and better understand its contents. The Docker imasge contains 3 components:

  1. Manifest.json: points to all the layers and the config.json.  
  2. Config.json: contains metadata necessary for running the container. Think Docker version, environment variables, mounts, etc.
  3. Layers: These are OFS layers as described above and are named using the hash of their contents. When merged together, they form the rootfs.

Let’s step through this using Docker to shed some more light on this:

Start by Docker pulling and saving the image. `docker save` saves the images to a tar archive.

marli 9:32:50 /tmp () docker pull httpdmarli 9:32:57 /tmp () docker save httpd:latest -o httpd.tarExtract the tar archive and look around.marli 9:33:10 /tmp () mkdir httpd && tar -C httpd -xvf httpd.tar && cd httpdmarli 9:33:51 /tmp/httpd () ls19459a87219415cc5751fb48e9aa37ea580b98eed1fe784e76c4bc3c9b3b0741.json4354893d890be3cc8574d2a43a153f01d3dbf2c3b6680bb54ad91e56e2103b1949adc30abd56426f5889a7edbe19c463d6b5c4d0e515b531ef09b33d7839476b55185aeb1145d787ef29c0c917b0a169737cc479df5c799805b65f22f0be848e5f6bd574c212bf1b00fa21bb12b588712d32bca72866be4061268498d90140ff83be7a564d0c2bad81aca09479229afba3cb114a10cc05a28774e166653e2aeamanifest.jsonRepositories

 

We see the manifest.json, which points to the config, as well as each layer.

marli 9:33:51 /tmp/httpd () jq . manifest.json[	{			"Config": "19459a87219415cc5751fb48e9aa37ea580b98eed1fe784e76c4bc3c9b3b0741.json",            RepoTags": [					"httpd:latest"             ],            "Layers": [             	"5f6bd574c212bf1b00fa21bb12b588712d32bca72866be4061268498d90140ff/layer.tar",                 "49adc30abd56426f5889a7edbe19c463d6b5c4d0e515b531ef09b33d7839476b/layer.tar",                 "55185aeb1145d787ef29c0c917b0a169737cc479df5c799805b65f22f0be848e/layer.tar",                 "4354893d890be3cc8574d2a43a153f01d3dbf2c3b6680bb54ad91e56e2103b19/layer.tar",                 "83be7a564d0c2bad81aca09479229afba3cb114a10cc05a28774e166653e2aea/layer.tar"             ]		} ]

 

I also suggest taking a look at the config.json, but it’s a bit large to include here.

Let’s check out the base layer. We see a complete file system.

marli 9:58:08 /tmp/httpd/5f6bd574c212bf1b00fa21bb12b588712d32bca72866be4061268498d90140ff () lsVERSIONjsonlayer.tarmarli 9:58:11 /tmp/httpd/5f6bd574c212bf1b00fa21bb12b588712d32bca72866be4061268498d90140ff () mkdir layer && tar -C layer -xvf layer.tarmarli 9:59:44 /tmp/httpd/5f6bd574c212bf1b00fa21bb12b588712d32bca72866be4061268498d90140ff () ls -la layertotal 0drwxr-xr-x  21 marli  wheel   672 Sep 16 09:59 .drwxr-xr-x   6 marli  wheel   192 Sep 16 09:59 ..drwxr-xr-x  72 marli  wheel  2304 Sep  9 17:00 bindrwxr-xr-x   2 marli  wheel    64 Aug 30 05:31 bootdrwxr-xr-x   2 marli  wheel    64 Sep  9 17:00 devdrwxr-xr-x  69 marli  wheel  2208 Sep  9 17:00 etcdrwxr-xr-x   2 marli  wheel    64 Aug 30 05:31 homedrwxr-xr-x   7 marli  wheel   224 Sep  9 17:00 libdrwxr-xr-x   3 marli  wheel    96 Sep  9 17:00 lib64drwxr-xr-x   2 marli  wheel    64 Sep  9 17:00 mediadrwxr-xr-x   2 marli  wheel    64 Sep  9 17:00 mntdrwxr-xr-x   2 marli  wheel    64 Sep  9 17:00 optdrwxr-xr-x   2 marli  wheel    64 Aug 30 05:31 procdrwx------   4 marli  wheel   128 Sep  9 17:00 rootdrwxr-xr-x   4 marli  wheel   128 Sep  9 17:00 rundrwxr-xr-x  66 marli  wheel  2112 Sep  9 17:00 sbindrwxr-xr-x   2 marli  wheel    64 Sep  9 17:00 srvdrwxr-xr-x   2 marli  wheel    64 Aug 30 05:31 sysdrwxr-xr-x   2 marli  wheel    64 Sep  9 17:00 tmpdrwxr-xr-x  10 marli  wheel   320 Sep  9 17:00 usrdrwxr-xr-x  13 marli  wheel   416 Sep  9 17:00 varM

 

Check out the top layer. Look familiar? This is simply an OFS upper layer described above.

marli 10:02:02 /tmp/httpd/83be7a564d0c2bad81aca09479229afba3cb114a10cc05a28774e166653e2aea () ls -la layertotal 0drwxr-xr-x  3 marli  wheel   96 Sep 16 10:02 .drwxr-xr-x  6 marli  wheel  192 Sep 16 10:02 ..drwxr-xr-x  3 marli  wheel   96 Sep  9 17:00 usr

 

Building a rootfs

Docker merges all the layers to create a single rootfs. The merging itself is pretty straightforward. We do it ourselves in Rootfs Builder, which takes the name of a Docker image, pulls the tarball, and extracts it. For every layer, we iterate through each tar header. We make 2 passes. The first pass is to remove whiteouts, recall these are files or directories that were removed in a layer. In the second pass, we read the tar header for metadata about the file or directory, specifically the mode, uid, and gid. If the file doesn’t exist we create it, otherwise we simply replace it. We also have logic to update the uid and gid. This is necessary if you want to unshare user namespaces. For example, you may want to appear to be root in the container, but outside the container you are an unprivileged user. This requires creating a subuid mapping. The mapping looks something like:

root@21d94d3c4539:/workdir# cat /etc/subuidfas:100000:65536

 

This mapping reserves the first 65536 uids starting at 100000 under fas’s namespace.  According to this mapping, uid 0 inside the container maps to 100000 outside the container.

Development Speed or Code Security. Why Not Both?

Find out how ForAllSecure can bring advanced fuzz testing into your development pipelines.

Request Demo Learn More

Next Steps

Developers use Docker images every day, and now you know, they are just glorified tarballs.  There’s plenty of room for improvement with Rootfs Builder. Outstanding features we hope to add will allow the user to specify:

  1. The number of layers to untar.
  2. A layer to omit when untarring.
  3. A binary the user is interested in.  Instead of returning an entire rootfs, this will just return the binary.  

But for now, hopefully Rootfs Builder will help users introspect into Docker images. You can get started with Rootfs Builder here: https://github.com/ForAllSecure/rootfs_builder


*** This is a Security Bloggers Network syndicated blog from Latest blog posts authored by Marlies Ruck. Read the original post at: https://forallsecure.com/blog/test/demystifying-a-docker-image