“Trust” — the Biggest Security Vulnerability in the Cloud

In this episode of the Security on Cloud Podcast, we’re joined by the well-known white-hat hacker, Robert Hansen of Bit Discovery. As a 26-year veteran in the computer security industry and known by insiders as “RSnake”, Robert shares how he became a security researcher and white-hat hacker. As a white-hat hacker at SecTheory, he hacked into the back ends of over 21 hundred banks, credit card processors, flight control systems, and SCADA control networks. We discuss some of the biggest security issues in the cloud as well as get Robert’s take on the move to Zero Trust in the cloud. We also get insight from Robert on questions like: 

• How does he see cloud security differing from the old data center model?  
• Does he think the CI/CD build environment is a new attack vector of focus for 2021?  
• Is he seeing people still move to the cloud, especially after 2020?  
• For a threat researcher, what would he do to protect his cloud environment if you had one? 

Tune in for this fascinating discussion about current security concerns and issues that could affect you when moving and securing your apps in the cloud. 

 

More Ways to Listen: Apple Podcasts | Spotify | Pandora | Google Podcasts | Deezer 

 

---

Episode Transcript 

 

John Vecchi: Hey, everybody, you’re listening to the Security on Cloud Podcast, live on Anitian Radio. I’m your host, John Vecchi. 

Scott Emo: And I’m Scott Emo. There’s so much going on in the security arena today that it’s hard to keep up. We hope that we make it just a little bit easier for you to stay informed and keep pace with the latest trends in security topics. 

John Vecchi: That’s right, there’s so much to cover in this security and cloud landscape. We always want to bring you the most interesting guests and topics on this podcast, as well as great discussions about current security concerns, and issues that could affect you and moving and securing your apps in the cloud. 

Scott Emo: And with that, I’d like to introduce today’s guest. He’s a 26-year veteran in the computer security industry, known to many on the inside by his handle, “RSnake”. He started his career at eBay where he was responsible for anti-fraud and anti-phishing technologies. His work at eBay was later built into every modern web browser and is now protecting every internet user. As a white-hat hacker at SecTheory, he hacked into the backends of over 2,100 banks, credit card processors, flight control systems, and SCADA control networks. Most recently, his corporate intelligence platform, OutsideIntel, was acquired by Bit Discovery, where’s he’s now the CTO. He was a floating CCO for many companies and sits on advisory boards of multiple technology and security companies. His hacker.org website was at one point responsible for a third of all top-ranked web vulnerabilities. He’s also authored a security column in Dark Reading for years. It’s our pleasure to have with us today my friend, Robert Hansen. 

Robert Hansen: Friend? Oh, now we’re friends. I don’t know… (laughing) 

Scott Emo: Well, I thought that sounded good, at least. (laughing) 

Robert Hansen: No, Scott, you’re definitely a good friend.  

Scott Emo: How are you doing, Robert? 

Robert Hansen: Yeah, I’m doing great. And for those of you don’t know, Scott and I’ve worked together for, I don’t know, probably about 15, 20 years ago, I guess it was. 

Scott Emo: 15, 20 years ago in security. When cloud was just a dream. 

Robert Hansen: Yeah. It really was when people talked about it back then. We were like, “Yeah, that’s kind of wishful thinking.” (laughing) 

Scott Emo: Yeah. But here we are! 

John Vecchi: Yeah, exactly. Well, look, Robert, thanks for joining us today. This is a treat for me and Scott. I mean, over the course of my career in this industry, and I’m sure Scott’s as well, I’ve had the privilege to work with and actually help build some top security research and threat teams at places like McAfee and Zscaler and Solera, Blue Coat, Checkpoint, and you know the rest. And it’s always represented one of the most fascinating areas for me in security. So, look, for our audience, maybe we can start with a really simple question. How did you become a security researcher and white-hat hacker? And what does that mean to even become that in your life? 

Robert Hansen: Yeah. I think it didn’t start so much white hat, it started just being a miscreant kid and trying to figure out how things worked. And the very first piece of code I ever wrote on my own was a Trojan horse. The very first thing I ever broke into on my own was a bank. I didn’t start simple. I went for the big stuff. And after I’d done some of that, I started a nonprofit called EHAB, Ethical Hackers Against Pedophilia, and we had one of the largest busts of child pornographies. At that time, I think it might’ve been even the largest bust in history. And that’s when I changed my whole attitude towards everything and I just became a white hat. I’m like, “I cannot be doing this and that other thing at the same time.” 

Robert Hansen: So, I switched just overnight, and I’m just like, “Okay, I’m going to start doing this for the good guys.” And that’s around the same time that I actually met Scott, which was probably a year or two after that. And I really was more on the product management side, but I had been doing security in one form or another for quite a while, even at that point, and later went on to eBay and so on. And I guess from my perspective, I’ve always thought the most interesting parts of security are the infrastructure, the things that make things work. And so, I focused on really big picture things like programming languages and operating systems and browsers and networks. So, when people say like, “Oh, you found a bug, what’s the CDE number on it?” I’m like, “No, no, no, no. That’s not the kind of thing I work on.” The kinds of thing I work on, you have to forklift upgrade the internet to fix. So, that’s how I got my start. 

Scott Emo: Well, you talk about infrastructure, and cloud is basically today’s largest infrastructure. Right? So, I’m going to shift the conversation to cloud. What do you see as the biggest security issues in the cloud? 

Robert Hansen: Yeah. It really is the same thing as hosting in your own environment in a lot of ways, so the same kinds of things that plague you when you’re doing yourself, plague you on cloud-only. A lot of people assume that in the cloud, things are done for you, but a lot of times, they aren’t done for you. For instance, configuration management is probably the single biggest problem, simple things like, “Do you have ACLs open to the internet?” Now, there’s no one telling you, “Hey, your ACLs are messed up.” There’s no centralized audit out there that can say, “Hey, this is all messed up,” because everyone’s ACLs are… Their needs are specific to whatever they’re doing. So, there’s no right or wrong way to do it, but there’s certainly a way to get hacked. Similarly, how people construct wherever they’re cashing or whatever, the S3 buckets. It’s like, well, it’s like the equivalent of an old Apache open directory or something. You shouldn’t have it open, but you have it open. That’s an easy configuration change, but people just don’t think about it in that way. They think it’s all done correctly for them, and it’s definitely not. 

I think that’s probably the single biggest gap, but then, secondarily, it’s the administration of it. Someone’s got to get into this thing in some way. And once upon a time, in the old model of a data center, you could literally have a guy just go over to the rack and that was the only way to access that particular machine. You could have two or three knock monkeys just sitting there typing in commands all day. It’s an awful job, but you could do it. 

Nowadays, that’s just not possible. You’re not sending people to Amazon to go sit down on and rack somewhere. This is all got to be done from some people’s houses, especially in the days of COVID. And so, people need to be able to contact these machines, typically, over a VPN or SSH, but oftentimes, they have administration consoles that are hanging out there. They’re just not paying attention to the fact that I’m reaching these things, can somebody reach me and then reach those things? Or can someone bypass what I think is good restrictions and just walk right through my network? So, it’s all that kind of configuration and administration mess, I think, is probably the biggest risk. 

John Vecchi: And I think we’ll certainly talk a little bit about some of that risk. And just a little bit ago, you mentioned the days in Covid, and certainly, it’s interesting times, right? Like I say, we woke up one day in 2020 and work was something we did as opposed to a place we went, and so everything moved to remote and offline. We always talk about how that had an impact on cloud adoption, cloud digital transformation in cloud. What do you see? I mean, do you actually see a transformation driven by the Covid pandemic to the cloud, and what are some of the things people should keep in mind if that is the case?  

Robert Hansen: This is a really interesting question. There’s a lot of nuance to it. So, my day job is actually the CTO of a company called Bit Discovery. That was the company that got acquired we talked about a little bit in the intro. And one of the cool things about the datasets that we have is we can actually see trends over time for different classes of industries or whatever. So, we can take the Fortune 500 and just ask the question, “From data point A to data point B, do we see a migration into the cloud? Do we see people moving into the cloud?” And the weird nuance to that is yes and no. They are certainly putting new infrastructure in the cloud. That is not a question. It’s definitely happening and at a pretty accelerating rate, actually. 

However, the question, “Are they moving to the cloud?” almost entirely the answer is no. There are some apps that are definitely moving into the cloud, but for the most part, if something isn’t in the cloud, they’re not moving it. It’s just going to stay wherever it’s at and it’s just going to live there forever. There’s going to be this dog that never gets updated and no one ever thinks about, no one cares and feeds for it. It just sits there languishing over time. So, from that perspective, I don’t see people moving to the cloud in the strictest sense, but people are building a lot of new services in the cloud. 

To your other question about, how does COVID change people’s reaction to the cloud or moving into it? I think this is really funny because right now we have this gigantic ice storm happening in Texas and knocking out power, knocking out roads, and there’s stuff I need to do in the data center, but I’m not driving out there. There’s no way I’m driving all the way out to the data center. It’s not by the airport. It’s probably like 20 miles from here or whatever. But for all the stuff that’s cloud-based, that I can reach accessible from the internet no problem. I can just SSH right in. And COVID is just a macroscopic version of that same thing. People aren’t really able to leave their houses, so are they going to just stop working? Of course not. They got to get stuff done. 

So, the number one thing is from a compliance perspective, a lot of people were not allowed to be in the cloud, for whatever reason. They have some SLA or there’s something that says they’re not supposed to do it or whatever. So, I think, very quickly, a lot of companies had to figure out how to be compliant in the cloud overnight, which is kind of interesting. 

Number two, there’s a lot of cloud-based solutions to solving things like Zero Trust, for instance. So, now, we’re going to have to basically button hook everybody through the cloud and then back out to the internet to do stuff like making sure people aren’t getting malware on their computers. And there are already companies that, you mentioned Zscaler, I used to work there, that already handle some of this stuff. And I see a lot of companies going, “Ah, crap, we really have to do this today. Not six months from now, not a year from now, we’ve got to solve this problem right now.” A lot of BYOD problems like, “Well, are you really going to have all the sensitive information on your computer? Or why don’t you just use Citrix and put it all in the cloud? There’s no reason you have to have this on your personal computers.” So, a lot more of that, getting stuff off of your local environments. The local environment isn’t the biggest choke point and risk in people’s networks. 

Scott Emo: That’s a great point because, in fact, one of the questions that I wanted to ask you was how cloud security was different from the old data center model. And you mentioned remote access is a major, major one in terms of security. One of the things that I’d like to double click on is like, maybe from a DevOps or a dev’s perspective, just developing applications, is there anything people, you think, should think about differently in terms of developing applications for the cloud, as opposed to when they were on-prem being able to just walk over to that rack and access it? Is there anything that you’ve got to think about that our listeners ought to know about? 

Robert Hansen: Yeah, there’s a lot, and it’s actually more pro-security than negative security, thankfully. So, one thing is caching, which turns out to be wildly more efficient in the cloud. Previously, you’d have to cache locally, and there’d be a big infrastructure cost associated with it. Now, there are companies like Cloudflare out there with, basically, key-value pairs, and you can basically put your entire website, literally the entire thing, dynamic website, inside Cloudflare and literally not have a backend behind it at all, which is really crazy. I’ve seen entire websites done in a caching layer and dynamic-looking, fully functional. You don’t get to post to it and do stuff to it, but you can still navigate it and it still acts and feels like it’s interactive, whatever. So, that’s really compelling because that makes the surface area of attack much, much smaller. 

It’s also much easier to scale up. So, the traditional service of just a basic overload of your site, now it’s good and bad. You can scale up very quickly, which makes it very difficult for adversaries to find and attack one particular part of a network and take it down as easily because you can scale up to meet that demand. One thing I really like about the cloud is the ability to overwrite a site that’s been compromised, so you basically have a known good state like this particular instance looks good right now, and I don’t know about it, and then in a week from now, I have no idea if it’s good or bad. So, who cares? Don’t worry about stressing out about what’s on it or what’s changed about it. Just install it over and just keep going. Just re-install it and just get it back up and running. So, all that Docker container stuff, Kubernetes. There’s really no reason that you have to worry too much about the site staying compromised. It will get re-compromised, which is a different problem but staying compromised, those compromises tend to go away very quickly. 

And then the entire idea of server-less compute and microservices has definitely improved the overall ability to compartmentalize code. So, instead of having everything being in this one monolithic code base like you used to have, now you can send out these microservices. They have very limited access. A simple example would be like a DAL, a database access layer, where you say, “Okay, here’s a hash and a username. Tell me if it is valid or not valid. If it’s valid, great, then I’ll set a cookie. If it’s not valid, then whatever, then I can’t do anything.” That means that the code, the original monolithic code base, the thing that you call a login, the login page, no longer can say, “Hey, select star from the database and give me all the database hashes.” I can only ask one question at a time, and that makes it much harder to compromise, like wildly, wildly more complicated. So, there’s a lot of really cool tricks you can do in the cloud that traditionally were much more difficult, and now they’re almost just built into the way you design things. 

John Vecchi: That’s very cool. And Scott mentioned in his previous question the term DevOps, right? And teams that are now developing applications and securing workloads in the cloud, and bringing those applications out to their customers. Everyone’s saturated a bit with the SolarWinds attack at this point, but one of the things that I found interesting in the attack that might be relevant in the discussion about cloud security and DevOps and DevSecOps, is just the idea that what, I think, CrowdStrike actually found, that the attackers actually created a piece of malware, a backdoor called Sunspot, that they use to actually breach the build process, the build environment. Or, in some cases, some might call it the CICD pipeline, which then they were able to insert Sunburst for that into Orion, and on we go with the rest is history. 

What do you see with that? Given the whole idea of shifting security left, the idea that security is oftentimes an inhibitor to these teams who are building dynamically, these applications, bringing them to the cloud, is that a new attack vector that we should be concerned about as we move forward in 2021? 

Robert Hansen: I certainly wouldn’t call it new. It is certainly been around for at least a decade, at least. Probably more than that. Probably quite a bit more. There are definitely issues in the supply chain, and that can come from a vendor, or it can be just something that’s totally just off the shelf, just works fine one day and suddenly something happens, it phones home to the wrong thing, or some server somewhere gets compromised and suddenly you’re uploading a malicious binary. I had a buddy of mine, he was legitimately… I don’t know if he ended up doing it, I never followed up on it, but he’s like, “I want to install malware into NPM packages. I want to do it. I don’t want to do a lot because I want to prove that people will just do this thing.” 

I’ve had another buddy of mine said he wanted to do the same thing with Docker, Docker containers, just put malware in there and have people run them. They’re not going to check, they’re not going to look at the detail of what’s going on and see, “Is this thing mining Bitcoin?” or whatever. They’re just going to have it run and it seems like it works, and it’s just going to work. I had another friend talk about doing the same thing with AMIs. “Hey, this thing runs WordPress, great. That’s what I want. I want a WordPress AMI. Click a button, run it,” and they don’t have any idea what’s going on under the hood. 

Now, those are all examples where it started off bad like the maintainer was bad, but there are lots of other variants of that where people are going along one day, everything works fine, and then all of a sudden the package they had was… The maintainer, for whatever reason, lost control or allowed the wrong person to insert a little one or two lines of code. It did something much more complicated and much more impossible for them to identify unless they were really good at auditing code, and all of a sudden they’ve got this little micro backdoor that can be used for all kinds of stuff. And it’s very hard to know which is which, because code can be obfuscated to the point where even the people who originally wrote it may not know what it does anymore. Just gets too complicated. 

Like Dan Geer probably had one of the best quotes about this. I was at a dinner with him once, and he was saying like, “When I get a computer and it’s fresh off the boat, just got there, I can tell you what’s on it. I can tell you what’s running on it. I can tell you what the operating system does. I can tell you the software that’s on it, et cetera. The second I plug that thing into the internet, I have no idea what’s happened anymore. It’s downloaded all these packages, my browser starts doing stuff, and cookies and JavaScript and blah, blah, blah.” And there’s just no way to keep track of it after it’s been up and running for a while. 

And in cloud, it’s just the same. In fact, in many ways worse, because a lot of DevOps people are hard-coded to run this script, and then this thing will magically happen. And if you actually look at the script, it’s just Sudu, some batch script, so it’s running as root and it runs the batch script that pulls some stuff down from GitHub. And did you audit that thing? Did anyone audit that thing? Yeah, it seems like it works when somebody put it on some forum like, “Hey, it worked for me when I did blah,” but it’s pretty dangerous, actually, the way current DevOps people think about the pipeline. 

Scott Emo: So, Robert, so being a threat researcher, with all the issues that you just brought up in that last question, what should a DevOps individual think about when he’s coding an application? What types of things should he make sure that they do in the process to make sure that they’re not going south? 

Robert Hansen: Well, unfortunately, there aren’t a whole lot of great answers. Number one, audit the code. Audit it as much as possible with as much as you can afford to audit it because obviously, there are diminishing returns. Number two, understanding the surface area as much as possible because when you’re installing stuff, which you generally don’t think about is, “Hey, I’ve got this test install. I’ve got this dev install. I’ve got this random other thing sitting there,” that’s also vulnerable, that’s also just not protected in the same way, because you want your marketing team to take a look at it before you push it to prod or whatever. So, it’s not behind the firewall or it’s not behind the web application firewall or whatever. 

And then the third is compartmentalization, so this comes back to the Zero Trust thing, like really nothing that you’re doing should trust anything else. And as much as possible, you should assume that every single piece of code that you’ve written or that you’ve downloaded will eventually become malicious. And once that information is known to the attacker, what can they do with it? Is there anything that can pivot to, or is that pretty much it, and they’re stuck there in their little bubble? Because that really definitely matters. And the more you can assume that people are compromised… 

In fact, that’s how I built hackers.org once upon a time. Every single thing assumed every single other thing was going to be bad. If I was using my website or whatever, my browser had to assume that everything about the website was compromised. The website assumed that I was compromised, the operating system assumed that the database and the web application were both bad, the database assumed that somebody had local access, et cetera, et cetera. So, everything was built with the assumption that everything else was going to be a problem. And we survived tens and tens of thousands of attacks over its lifetime. We were getting many hundreds or thousands of attacks per day every day for years, and the only reason it survived that was not because we had great code. Our code was crap. We were running WordPress 2.2. What saved us was by virtue of this Zero Trust concept, like nothing trusted anything else. 

John Vecchi: It’s interesting. Right? I mean, we’ve seen Zero Trust now come from just what people call the buzzword to, it’s real, and I think it’s an approach that’s getting implemented. So, it sounds like it’s probably good advice, Zero Trust approach, Zero Trust security, whether it’s Zero Trust access and east-west traffic, and all the things you care about in today’s world. Is it safe to say that’s smart for people to think about Zero Trust technologies? 

Robert Hansen: I think trust, in general, is a vulnerability, and the more things that you have to trust, the more likely you are to be vulnerable. So, if you unwind that and say it in reverse, just say, “Don’t trust anything,” and then you’ll be much, much better position in terms of security. There’s really no reason that you need to be doing this extra fun, cool widget-y thing if there’s a way to do it that doesn’t have to involve a third-party. I see a lot of people phoning home and pulling in jQuery, for instance. But why? Why are you pulling the current version of jQuery off of jquery.com, instead of just pulling it down to local? You can have a little cron job to say, “Hey, is this the current version of jQuery?” and update. There’s no reason you have to go pull this from a third-party website, but it’s not part of the developer mantra. Developers are trying to build things quickly, they’re not really thinking about these third-party dependencies. They’re not thinking about these APIs that they pull in. 

A simple question like, “What if this API does something you didn’t expect it to do? Do you have any error logic to handle that?” Like a simple question like, “What if it’s not there anymore? How does your application handle that, just simply not being there? What if it goes away? Or what if it gives you too much information? Or what about not enough? Or what if it’s badly formatted?” or a hundred other questions. And then you start getting to the crux of not trusting things. You’re like, “Okay, well, I better verify what I’m getting in return. It better be good data. If it isn’t, I better have a good fallback.” 

John Vecchi: Right. That’s fantastic advice. And we’ve covered a lot and relative to the question around, “Is the build environment going to be a new vector?” Your response is true, right? Well, it’s been one for a long, long time. There are so many vectors that continue to be leveraged and exploited. There are new techniques, tactics, procedures exploiting those. When you look, Robert, today, in 2021, as we move into this year, are there certain ones that just funnel to the top as top ones that either are coming back around or are they just surface at the top of your head as, “Man, these are probably some of the top things that are happening right now,” and will probably continue to happen through this year and onward? Right? 

Robert Hansen: I’m extremely pragmatic when it comes to stuff like this. A lot of hackers like to get prophetic and whatever. I just look at where the money is. So, the money, if you look at the insurance industry, the things that they’re spending their own dollars on because they get compromised, they are web-based attacks, they’re RDP, and they’re email. Those are the ways that people get in. You shouldn’t have remote desktop protocol running. If you do, they’re probably not going to insure you. You’re probably going to get compromised through your email or through your website. So, if you’re doing a good job on those three fronts, you’re probably going to survive the bulk of most attacks. 

John Vecchi: There you go, listeners. You heard it, you heard it directly from Robert, so heed that advice. Right, Scott? 

Scott Emo: It’s excellent. Now, I also heard, and I’m going to quote you again, Robert, because this was money, “Trust in general is a vulnerability.” That was beautiful. I actually had to write that one down. It was beautiful. So, Robert, we’re running short on time. Is there anything you’d like listeners to know about? Is there something you’re working on that maybe they’re interested in? Is there something you’d like to let our listeners know? 

Robert Hansen: Yeah. So, Bit Discovery is the current thing I’m working on, and the entire premise of the company is, it’s a very weird thing to say, but most companies don’t know what they own. They don’t know websites they’ve got out there, they have no idea what their infrastructure looks like, they have no idea what the service levels of those different services are running, they don’t know where their vulnerabilities are because they don’t even know where their assets are. 

Bit Discovery is all about finding those weird websites, those weird IP printers or telephones or whatever’s on the public internet that a company owns that they don’t realize they own, and basically surfacing those things so they can query it and making a querying language around it that allows them to ask simple questions like, “Hey, are we GDPR-compliant across all our websites? Hey, are any of our websites phoning home and sending malware to somebody?” or, “Hey, do we have a CVE anywhere across all these websites?” Common vulnerabilities that any scanner could find. It’s not the complicated stuff, but the scanner’s only going to find it if they know that their website exists in the first place. 

So, we’re just basically helping companies find that the assets exist, whether they’re in the cloud or whether they’re in their data center somewhere or under some guy’s desk at the office or whatever, and basically making those things publicly accessible so that other companies can find it if they’re trying to do insurance modeling on your company. Or you can find it if you’re trying to protect yourself, or a red team can find it if they’re trying to find vulnerabilities in your site to show you where your issues are. That kind of thing. Yeah, bitdiscovery.com. If any of you are interested, please let me know. I’ll give you a free inventory or whatever. 

John Vecchi: That’s fantastic. Robert, it’s just fascinating and so insightful. Thanks so much for joining us today. And if any of our listeners want to get in touch with you, is there a way for them to reach out to you? Do you have a social site or a handle or anything that our listeners can reach out to you? 

Robert Hansen: Yeah. I write pretty often on blog.bitdiscovery.com, and on Twitter, I’m @RSnake. 

John Vecchi: Thanks, Robert. It’s been great having you today. 

Robert Hansen: Thanks for having me. 

 

---

About Our Guest 

Robert Hansen – CTO at Bit Discovery
Robert “RSnake” Hansen is the data-whisperer who brings all of Bit Discovery’s “bits” together. Another information security lifer, Mr. Hansen started his career at eBay, where he was responsible for authentication as well as most anti-fraud systems and anti-phishing technologies. Since then, Robert has worked in nearly every bit of web application security for many of the world’s largest organizations, including over 2,100 banks, credit card processors, flight control systems, SCADA (water and power) control systems, and other security companies. It’s safe to say that nearly no one knows the Internet as well as he. In his spare time, Robert is a frequent keynote speaker and is on the speaker selection committee for Black Hat and Hack in the Box.