With claims of internet traffic encryption levels at 95% and cloud communications up to 100%, it’s increasingly difficult to see malicious activities happening on our networks. Encryption is a good thing, of course, until it hides nefarious network traffic and payloads. Then we want to know about it, be able to diagnose it and take appropriate action. Encryption can be a double-edged sword.
Steve Perkins, CMO of Nubeva, joins us on this DevOps Chat to talk about Nubeva’s solution for out-of-band decryption of next-generation TLS communications. It makes sense that cybersecurity teams desire this new level of visibility and DevOps teams working with APIs and cloud encrypted communications also want the option of taking an “outside-in” view for troubleshooting, addressing compound performance problems, etc.
We explore the benefits and use cases for Nubeva, how it works and how Nubeva provides this level of visibility while maintaining the integrity and security of TLS communications. Join us as we explore new and unprecedented levels of access to encrypted communications.
As usual, the streaming audio is immediately below, followed by the transcript of our conversation.
Mitch Ashley: Hi, everyone, this is Mitch Ashley with DevOps.com and you’re listening to another DevOps Chat podcast. Today, I’m joined by Steve Perkins, head of product at Nubeva. Our topic today is decrypting next-gen TLS. Security—my favorite topic. Steve, welcome to DevOps Chat.
Steve Perkins: Hey, thanks for letting me be here.
Ashley: Absolutely. Love having you on. Well, first of all, let’s start by having you introduce yourself and tell us a little bit about you and what Nubeva does.
Perkins: Yeah. So my name’s Steve Perkins. I’m the CMO and head of product for the company. Nubeva was a company that was created about three years ago, by some pretty large companies, who saw what was happening in the world of encryption – SSL and TLS encryption – and were realizing that they’re gonna start to lose significant visibility. So they funded us to go solve that problem and we built a company out of it. Today, Nubeva has released, into general availability, a breakthrough solution. It’s really the industry’s only true out-of-band solution for decrypting all the new next generation of encryption that’s happening out there on the internet and in private data centers.
Ashley: Well, seeing inside that encrypted stream, I know, is often sometimes what organization wanna do. What are some of the use cases or why go tackle this particular problem?
Perkins: Yeah, it’s pretty big. I mean, there’s a lot of use cases. So let’s start with the security side and it’s actually on security and DevOps, we have customers using this. So let’s start here with the basics, right? So SSL became TLS. Ninety-five-plus percent of the traffic in the internet and in the cloud is all encrypted. In fact, in the cloud, it’s almost 100 percent, so this is your session encryption between clients and servers and servers and services. Right?
So, from a security perspective, the need to look at network traffic is pretty important, right? The network is the source of truth of what’s really happening with your systems. Systems can be hacked. Systems don’t always report or log everything that’s going on. So, while you can do header-level monitoring or network-level flow monitoring, if you can’t see the payload, you can’t see the API call, you can’t see the data in motion. You can’t see the malware in motion. You can’t see the command and control in motion. You can’t see the data leakage and exceltration in motion.
So the first use case is detection, is, by allowing – decrypting traffic and allowing advanced detection systems to be able to look deeper and see more threats, we actually help companies reduce risk. But there’s a second part, which is, when there is something that happens, how do you go and respond? Even if you’re not monitoring everything, you might hear that something is wrong. Somebody has an anomaly and security operation centers, they go and they respond and they start doing research. Well, trying to research what’s going on without being able to see what’s happening on the wire in detail is sorta like looking at a redacted document, right? It’s all blacked out and say, “Figure out the case, Columbo.” Right? So you’re sitting there, saying, “I gotta infer what’s really happening,” and it takes ten times longer, which means the threats run longer and it’s all bad.
So, on the security side, we actually just help responders go faster by seeing the full picture. We help detection systems see what’s going on. On the DevOps side, we actually have people using us there too. And that is that developers who are developing encrypted applications or who are using, typically, cloud services that are all being encrypted, they – sooner or later and from time to time, you need to diagnose and troubleshoot from the outside in. And, without something like us, you can’t.
Then, on the support side and the ops side of DevOps, right – and support is, when you have complex applications that are using complex clusters of servers and services from third parties, which is pretty much everyone today, how do you diagnose a compound performance problem if you actually can’t look inside “What’s happening with that API call? Were the parameters right or wrong?” Seeing it from the inside doesn’t always work. So we have people on those four, yeah.
Ashley: Now the bad guy, the black hat version of this, would be the man-in-the-middle attack. What’s the white hat, the good guy version, of your architecture or your product?
Perkins: Yeah, well, so let me say it – maybe answer it a little bit differently, right? So, traditionally, there’s been two ways to be able to decrypt traffic and one is is you stick a device in the middle, in the middle of a connection, and it actually proxies on both sides for the connection, what you call a “man in the middle.” It could be used nefariously, as a man-in-the-middle attack, or it’s used all the time with next-generation firewalls or load-balancers, like advanced load-balancers, to actually – when it’s sitting there in the middle, it’ll actually take out a copy of the traffic in the middle and send it off to analysis and monitoring tools. So that’s the traditional – one of the traditional ways.
The second traditional way – and I think I’m getting your question, but just bear with me here for a sec –
Ashley: Oh, no, it sound good. You’re on track.
Perkins: Yep. The other way is you pull copies of traffic, mirror, TAP, SPAN, port, TCP dump – it doesn’t matter – however you can get your hands on a copy of traffic, in out-of-band sense, you actually then push those over to a decryptor system and a decryption system then goes and decrypts and then pushes the traffic into a tool that can analyze.
Perkins: Now so what happened, though, and kinda setting up to your white hat/black hat, so enterprises today, most enterprises, monitor their traffic. Most enterprises decrypt traffic, especially in private data centers. Not so much in cloud. It’s been really hard to do, to date, in cloud, in public cloud, but, in private data center, private cloud, they have these mechanisms ’cause you own the routers, you own the switches, you own the traffic, et cetera, and you own the certificates and the private keys to decrypt.
Well, the problem was is that nation state monitoring and hacking started happening, where all the big guys, these big bad guys, would go and capture your traffic for five years and, one day, they figure out how to compromise your master key set, your asymmetric key pair, and they could decrypt every session from the dawn of time. So some pretty big people got hacked this way and pretty big carriers and even internet providers – you think of nation state and Snowden type stuff, right? – so they set about to say, “We gotta fix this. Nobody should be able to sit on a black hat and sit on the sidelines and do this every again.”
So they started writing new standards and they’ve been leaking out and new protocols – actually, new protocols that have been starting to be adopted and now they’re being standardized. And, in some areas – I just met with a bank yesterday and they failed an audit and their auditor said, “You’re going to adopt these latest – the state of the art,” and, instantly, the security team went blind. Every one of their decryption systems failed. They woke up one day and saying, “I have zero visibility into what’s going on anymore.”
And so that’s really the backdrop which we step in – did I answer your question?
Ashley: You did. You did. I’m curious too – can you say a little bit more about what some of those encryption standards are or protocols? This is like TLS 1.3 or is it other things that organizations are adopting?
Perkins: Yeah, so it actually starts with – it’s interesting. So the reality is is TLS 1.3 is a standard, but, pre-that, the state of the art was TLS 1.2, but that’s not the issue. The issue is this thing called “perfect forward secrecy.” And what it is, without getting into it a lot – I mean, some of your people probably know this really well, but just to baseline everybody – it used to be, in the old days, with the old handshake, the handshake you and I – let’s say you and I wanted to communicate. We would communicate. We would authenticate each other, using the public certificate and the private key, and then we’d actually derive our final secret together that we’re gonna use to encrypt our traffic back and forth.
Ashley: Yep. Diffie-Hellman.
Perkins: Yep. Well, actually, not Diffie-Hellman. The RSA model actually transferred secrets back and forth and, if you have the original keys and the public certificate and the private key and if you could get your hands on the packet, after the fact, you could derive our session key.
Ashley: Ah, okay.
Perkins: Now what happens in Diffie-Hellman, with the Diffie-Hellman algorithm, that can’t be refigured out. You and I never exchange and share our secrets. We actually go through this crazy mathematical sequence where we both arrive at the same key but never – it’s never on the wire and never shared and never stored and never kept. It’s _____ –
Ashley: One-way derived keys.
Perkins: Exactly. ‘Cause the end of the day is if you can get your hands on that final session key, you can decrypt. So, in the new model, perfect forward secrecy, also called “PFS,” born out of Diffie-Hellman – sometimes you’ll hear “elliptic curve Diffie-Hellman ephemeral” – is because – I mean, it doesn’t matter; you could read all about this on the internet – the net is that perfect forward secrecy is the result. And that is being used everywhere with TLS 1.2 and every cloud provider, if you go to Amazon today, AWS, and you make a cloud call to an API REST call, it’s coming back with Diffie-Hellman turned on, elliptic curve Diffie-Hellman ephemeral, basically perfect forward secrecy. And, on top of it, they don’t do just standard AES. They do AES GCM cipher on top of that.
So all of that means that your traditional means to say, “Hey, I can figure out the key so I can decrypt,” is all broken. And that’s the real core problem here. And it’s really not a decrypt problem; it’s a key problem. You used to be able to figure out your keys and now you can’t. Now the good news is the bad guys can’t figure it out. That’s the good news, I should say, right? The bad guys can’t figure out the keys anymore.
Ashley: That’s the intention. I guess it’s a consequence of that _____ _____ _____ _____.
Perkins: Right. But the bad news is now you can’t either, right? So now how do you look at your stuff anymore and how do you troubleshoot, how do you debug, how do you respond, how do you detect, right?
Ashley: Mm-hmm. Very interesting. So you mentioned you were speaking with banks. I, of course, can imagine why they might be interested. Are there other particular industries that are super interested in this or is this more of a cloud problem or a private data center problem where everybody has this issue?
Perkins: Yeah, that’s a great question. So let me tackle it two different ways. So everybody has this problem – it’s a matter of severity and acuteness – and let me talk one extreme. Let’s say you live in a private data center on bare metal and you don’t use –
Ashley: Ouch. [Chuckles]
Perkins: Yeah. And you don’t use any external services. And you’re not regulated.
Perkins: And you have the right to set your cypher suites where you want. You don’t have to adopt any of this.
Ashley: Pretty small universe, yes.
Perkins: Right. ‘Cause the next part is is that pretty much everybody’s in some form of hybrid-cloud computing and your clusters now are becoming high-performance, and so it doesn’t make sense to jam all your traffic through these very expensive, old decryption mechanisms. And just think of this: a simple Web app. You actually use Google Analytics, I’m assuming, so Google Analytics is actually a perfect forward secret call out to the internet. Now, generally, you’re gonna trust them, but how many other services do you start using that are third-party connections into your system that you can’t monitor?
Ashley: A lot.
Perkins: Right. Yeah. Now the other end of this, the severity is, if you’re born in the cloud or going all cloud, you’re virtually blind. And so we’ve met with people saying, “I didn’t think this was possible. I was gonna try to figure this out with logs, with application logging, and just bear the burden there,” because they were saying it’s not possible in the way it’s going. So there’s this spectrum of severity, but, certainly, as everybody’s going to the cloud, that’s where we’re catching people in various levels.
There’s another – answer to your question is is that there’s two drivers, right? Is, one, you’re using cloud services and those are on critical connections you think you wanna respond to. In fact, there’s been a very famous – a fairly few – a few very famous breaches lately and they were all – they were not on client-to-server; they were actually on server command-and-control traffic to cloud. Right? That back-side traffic, that tier-two layer connection into cloud-owned services.
Ashley: Machine-to-machine kinda thing.
Perkins: Yeah. And east-west traffic but it was not east-west between two servers you control; it was east-west between your server or a system and the cloud.
Ashley: The cloud service.
Perkins: So how do you actually respond and look at that? And that’s what we unlock, right? And the last part is there’s some regulatory – you know, government is being imposed to use TLS 1.3, which is the most extreme implementation of all this. Other companies are adopting it just for higher security ’cause that’s what it does.
Ashley: Well, I think one of the natural questions, of course, is gonna be then, “If you have this capability through your product, through your service, how is it protected so that others can’t get access to those keys to be able to look at the – have visibility into the encrypted stream?”
Perkins: That’s a great question too. Probably, there’s two levels to it – there’s what I’ll tell you now and then there’s probably, if you wanna know more, there’s a whole hourlong conversation we’d love to dive into.
Ashley: I’ll bet. I’m sure you get this a lot.
Perkins: Yeah. But, well, it starts – maybe we should talk about what the solution is and then you can understand it because, in the context of our architecture, it represents the – the architecture itself is secure and then the elements within it are secure, so if you want me to explain –
Ashley: Great. Let’s go there.
Perkins: Yeah, okay. So, the end of the day, the goal here is is you’re trying to figure out if you can – there’s a million – to decrypt, you need copies of packets and you need the session keys and you need a decryptor function, right? So there’s a million ways to get packets. In private data centers, you can do anything from – well, you’ve got packet brokers – TAP, SPANs. In the cloud now, there’s AWS got packet-tapping mirrors, et cetera. You can always use TCP DUP. There’s many ways to get packets. And, as we covered, in the old days, you used to be able to derive the session keys from the packet streams and your certificates. Really compute-intensive. That’s broken.
So you really have a key problem. You need to get your hands on the keys. And then you need a new kind of decryptor that actually needs to be able to function with those keys in a different way. So that’s what we basically did, is we built a very, very lightweight memory probe, a microservice, if you will, a sensor, a TLS sensor that you can deploy on a workload. It works on VMs. It can work in – there’s a container version. There’s a SChannel version and a Linux version. And you deploy it on a VM. And it can actually – well, what it does is it actually looks at memory in a very sophisticated way, in a read-only process, and it looks for a session setup – essentially, what’s called a “client hello.” And, based off of that client hello, it knows where in memory the key will actually be generated and written. And we take a copy of it.
So we’ve actually created the ability to discover and extract the session key from the host and do that, without putting the system in debug mode, without putting the system in – modify your libraries, without having to put in taps or shims into your code. We basically commercialized being able to do that with manageability and simplicity and scale.
Ashley: It’s really runtime. Effective when it happens _____.
Perkins: Yeah, it’s full runtime. And what’s beautiful about this is it does that at the end of the – after the setup. So it goes and extracts the keys – and I know we’re gonna run out of time here, so let me go faster. So we have this ability to discover those keys and give ’em back to you. And then we write those into a secure cloud database that you own and operate, very tightly controlled. Think of it as a depo where you can write and store those or you can erase ’em if you never need ’em.
And then we created a decryptor that, as soon as you point traffic at it, it says, “Hey, I see an encrypted session. It’s the new kind. Let me go pull the key,” and, actually, it decrypts. And it decrypts and outputs to a file. It can output to a tool. It can output to Wireshark and it can output to an IDS system. It’s tool-agnostic; it’s packet-agnostic; it’s cloud-agnostic. It actually is – it almost creates a new key plane in your environment that you can control again. And it’s interesting ’cause it has a much lower risk profile because it’s – if a person ever gets a hand on a key, it’s just one key to one session and they need the exact packet string. Right? Versus, before, if you got the private key, you had the keys to the kingdom.
Ashley: Interesting. So how do you manage access to that key vault? Is it strictly programmatic access? Is there a manageability layer around it, that lets you kinda control it?
Perkins: So we use cloud databases _____ first issue, so access, first of all, every right – the only – it’s the customer’s database. We give ’em a template to set it up that has extremely limited access rights, but it is ultimately the customer’s INM rules, right? So we write over an encrypted channel. We pull and read from a decrypted channel. The only rights are for the agents to be able to write a key and for the decryptor to be able to read and there’s a single admin write. And it leaves in a customer’s EPC, inside their cloud.
If they’re doing it on-prem, they actually set up – we have customers who set up a private, dedicated subscription just to run that cloud database. And we chose a cloud database because it’s easy. Infinitely scalable. Cheap. Secure. Right? So, yeah. And then, like I said, most customers have a retention schedule that says, “Hey, I didn’t do any packet capture today. Let me throw away and wipe the key base tonight.” And you only keep the keys if you actually needed the key packets.
Ashley: I’m very interested too about what the setup process is, how long does it take to get sort of time-to-value, where you’re getting visibility into encrypted streams? What does that take? How hard is this to get going?
Perkins: You can actually be operational in literally about two minutes, so think of the – the simplest case is – let’s take a very simple case. A developer is actually trying to troubleshoot something. I know this is a DevOps channel, right? So you just go and you put a container on the host – well, you go in a management system. And we actually have a SAS; we have a private option. You go into a SAS console. You set up – you push a button and launch a cloud-formation template to build your cloud database. It’s in your subscription you took.
You literally then go and you deploy a key extraction agent on the host. That takes about 30 seconds by the time you cut, paste, and post it into, let’s say, Docker. And then same thing – the container – the decryptor’s a container and, when you deploy these, they have a key string that you set in that points them to and from the database. You’re done. You capture traffic and point it at the decryptor VM and you’re done. And you can be operational in a couple minutes.
And I call that out because one of the cases is as simple as a single developer saying, “I gotta troubleshoot something right now. This should not be any harder than deploying TCP dump and trying to figure out how to get the keys.” Right?
Ashley: Mm-hmm. Yeah, I was thinking the very same thing of how could a development teamer, like you said, a single developer, use this to very quickly be able to start debugging a service they’re either building or that they’re using from third-party cloud?
Perkins: Yeah. And we’re really developing out that use case and the pricing models. I mean, this is software and it’s borne out of these independent operating agents. Nothing is bottlenecked. The data plane and the key plane never leaves your subscription and your control. We actually just buy the agent software. And so you can start very small. I mean, this thing starts for – right now, we have sort of an entry pack which is ten agents. It doesn’t matter. Ten decryptors, ten key agents. It’s $100.00 bucks a month. We’re thinking of even bringing that down lower to allow even a lower price point just to have it in every DevOps and SecOps analyst kit, right?
But, at the same time, then we scale up with massive discounting on volume, so that we have large customers doing 100,000 simultaneous workloads and containers being monitored and pulling out keys at high scale, on the order of 1 billion a month. So this thing scales up and scales down, whereas, before, the entry points for decryption used to be crazy – tens of thousands of dollars – just to get started for a pretty static, cumbersome system.
Ashley: Is there a panic button, a red button, you can say, “Stop this,” if something’s happening that I don’t want to have visibility into?
Perkins: You got that. You also have the ability to wipe your database. The covering the rest of the security, right? So – and, again, I’d invite people who are interested, “Come to us. Engage with us. We’ll talk to you about it.” But the memory probe is a read-only function. It’s not a write. And it actually doesn’t have code in it to receive inbound connections. It only does REST outbound-to-post statistics. So it is – I mean, obviously, if a machine gets owned, a machine gets owned and any code could be owned, including us. But, as a security architecture, the keys are more secure; the database is more secure. The key model of being individual symmetrics and then the key ultimately of the key extractor is actually really inert. It doesn’t even fire off antivirus systems.
Ashley: Interesting. Well, we’ve run out of our time here and, of course, security’s one of those topics that has layers and layers of interesting paths to pursue. I really appreciate you being on, Steve. I know that our listeners might be interested in finding out more. They can go to www.nubeva – N-U-B-E-V-A – .com to get some videos and other things there that I saw, that were real helpful.
Perkins: Yep. We have free trials. We have cloud architects, network security architects, all ready to engage and answer your questions for you. We released this product three months ago. It’s in GA now. It’s blowing up; it’s taking off. If you wanna be a part of us, help us make the product great, come in. Let us know what you need. We’ll be happy to engage.
Ashley: Good time to get involved. Well, Steve, thank you very much for being on. Appreciate having you on the podcast today.
Perkins: Yeah, thank you, Mitch. It’s been great.
Ashley: Great. It’s been great having Steve Perkins, head of product from Nubeva, for joining us today here on the podcast. And, of course, you’ve listened to another DevOps Chat. We like to thank you, our listeners, for joining us too. This is Mitch Ashley with DevOps.com and you’ve listened to another DevOps Chat. Be careful out there.