DevOps Chat: Cloud-Native AppSec, with Manicode Security
RSA Asia Pacific & Japan Conference 2019 in Singapore promises exciting and engaging sessions. A likely popular and possibly controversial talk will be “The Future of AppSec is Cloud Native.”
Co-presented by Jimmy Mesta, CEO and co-founder of KSOC and CTO at Manicode Security, and Jim Manico, founder and trainer at Manicode Security, this talk makes the bold assertion that how we build cloud-native applications will establish the new benchmark for application security.
In this episode of DevOps Chat, Jimmy and Jim preview their RSA APJ 2019 talk and set out the case for why cloud-native is the future for AppSec. We hope to see everyone at their talk at RSA APJ 2019.
As usual, the streaming audio is immediately below, followed by the transcript of our conversation.
Transcript
Mitch Ashley: Hi, everyone, this is Mitch Ashley with DevOps.com, and you’re listening to another DevOps Chat podcast. Today, I’m joined by two people—Jim Manico, founder and trainer at Manicode Security; and Jimmy Mesta, CEO and co-founder at KSOC and CTO at Manicode Security.
Gentlemen, welcome to the podcast.
Jim Manico: Hey, thank you so much for having us on the show.
Jimmy Mesta: Thanks for having us.
Ashley: Great that you’re here. Our topic to here is, “The Future of AppSec is Cloud Native.” Now, this is a talk that both Jim and Jimmy are doing at RSA Asia Pacific & Japan Conference 2019 at the Marina Bay Sands Hotel in Singapore. The session is on July 17th at 4:30, so we’re giving a little bit of a preview here and letting folks now about what you’re gonna be talking about. Of course, you’ll do the full talk then.
Let’s first start by having Jim and Jimmy introduce yourselves. Jim, would you start out?
Manico: Certainly. My name is Jim Manico, I’m the founder of Manicode Security. I’m an author, OAuth volunteer, software developer, and I just love application security.
Ashley: You can hear that in your voice, you sure can. [Laughter] Jimmy, jump right in, how about you introduce yourself?
Mesta: Alright, yeah, my name is Jimmy Mesta, I am a founder of a brand new company called KSOC, trying to address some issues around Kubernetes security and visibility. I’ve been doing AppSec and infrastructure security specifically for 10, 11 years now. So, here we are.
Ashley: Awesome. Excited to talk with both of you. So, the future of AppSec is Cloud Native—okay, tell us why that’s the case. Give us a preview of what you’re gonna be talking about.
Mesta: For sure. I’ll kick it off. So, I’ve spent a number of years doing defensive security, right? So, typical security engineering roles, AppSec engineering. I’ve watched DevOps come and go and transformations being attempted to be made throughout a number of organizations. And there’s a lot of friction at times between development, operation, and security—hence, our lovely acronym that we like to throw around in the community called DevSecOps.
Ashley: Mm-hmm.
Mesta: So, I’ve spent a lot of time in this space with Jim as well—you know, in the classroom as well as hands on, in the weeds with these new technologies, and seeing how they’ve changed our worldview on what application security is and how that either plays nicely or doesn’t play at all with these new Cloud Native technologies.
So, this talk will be, you know, a mix of lessons learned, observations after being, you know, on site with hundreds if not thousands of software developers and infrastructure folks talking specifically about my near and dear passion—that is, Kubernetes and Kubernetes security. But everything in between in the whole Cloud Native stack and how AppSec is kind of getting swallowed up by this new space.
Ashley: Alright. Jimmy, jump in—your perspective on this? I’m sorry, Jim. [Laughter]
Manico: No, no problem. I know, there’s a Jim and a Jimmy. I think Cloud Native development technologies, these are obviously here to stay. It radically transforms the culture of how people write software and how applications are built.
And I’m gonna push it back to Jimmy—the big question I have is, we have this battle between Kubernetes, and I think between serverless technologies. Which one do you think—so, I wanna throw it back to Jimmy. So, Jimmy, where do you think serverless technology comes into play when it comes to Kubernetes and what do you think Kubernetes’ long term survival is? What’s special about Kubernetes that is gonna leave it within the different tools that developers use to do Cloud Native development as opposed to other technologies that are competing with it?
That’s what—I’m gonna throw it back to Jimmy. That’s my question.
Mesta: Yeah, no—happy to address that. I totally agree, serverless is also here, it’s also a very hot topic. The security around each of those technologies, Kubernetes or serverless, kind of differs, but I think Kubernetes, as far as my observations have gone, we’ve kinda had this evolution five years ago, maybe, now where everyone was preaching the, “We must write our applications to run using Docker, right, or some sort of container technologies.”
Manico: Sure.
Mesta: So, that started this big shift towards a rapid SDLC, the ICD artifact, the deployment of an artifact that could be built and shipped within seconds—minutes or seconds from pushing to a Git repository.
So, it kinda started with containers, and that helped us move faster and then—
Manico: When did that happen? When did Docker really start to take off and change development? Was that, like, 10 years ago, 5 years ago? When do you think Docker really took a foothold?
Mesta: I mean, in my observations, where I was working, and these are companies that were kind of experimenting with bleeding edge—air quotes, if you will—technologies, it was probably five to six years ago where people said, “Hey, Docker, I’m gonna buy into this. This is the future.” And now we’re seeing that catch up kinda happen in larger enterprise organizations, so.
Ashley: Yeah, Docker was all the jam about, what, 2013, ’14, somewhere around that time frame?
Mesta: Yeah.
Manico: Yeah.
Ashley: It still is.
Manico: Is there pushback against Docker? Is Docker, like, not—was the promise too much? Are people pulling away from it? What’s the downside to the Docker revolution, if there is any?
Mesta: Well, of course, in those days, there was pushback, because anything new is scary and more work and security wasn’t positive if this was a good thing, right? We’re sharing the kernel of a host operating system versus building carved out virtual machines where we get our warm and fuzzy safe place that is a virtual machine with our own kernel.
So, there’s for sure still pushback, but I think that the ecosystem has evolved so much, including the security of Docker. It’s been hammered over the past five, six years and CDEs, we’ve seen them sprout out, and we’re still going to see that.
But we’re seeing the pushback is less and less now. I mean, even at some of the largest kind of ancient companies, if you will, that move pretty slow, Docker is—you know, you get who uses Docker here in the beginning of class when we’re out teaching and it’s always 50 percent or more of the room in some capacity.
So, I think the pushback has faded, and now we’re looking for ways to take a Docker image or a running container and orchestrate it, right? That’s where Kubernetes comes into the picture.
Ashley: Yeah. I think we owe a lot to Docker. I mean, Docker, Chef—all of that was really kinda what started this, some of the tools behind DevOps.
So, I want to get back to your premise that AppSec, that that’s the future of security is Cloud Native. Tell me why AppSec is the future. Why is that changing security for all of our, how we construct contemporary applications?
Mesta: Yeah, so, I’ll take a stab at that and I’ll ask Jim a couple questions, too, because he’s been doing AppSec and been in front of developers for a long time, now. So, for me, it’s a few things. I actually have a couple notes, here.
Number one, it starts with the infrastructure. The place that your application source code is running is vastly different now. Cloud Native technologies don’t look like our traditional virtual machine or kind of bare metal infrastructure that we used to have. So, if you don’t—
Ashley: Although serverless architecture is different, right?
Mesta: – serverless, Docker, queueing systems—Cloud Native is much more than, you know, serverless is just a small component of it.
Ashley: Piece of it, yeah.
Mesta: It’s service meshes, it’s tracing, it’s debugging, it’s how do we handle—you know, how does our application handle retries or how does it handle auto scaling?
So, the way we architect our applications is different, right? We don’t have this just, you know, big, beefy box that, you know, application just runs on, and when we need more compute power, we just buy another box. If you use Cloud Native technologies, it just really changes how your systems look. And if you ignore where your application’s running or how you’re going to secure it, you’re either going to run into vulnerability kinda security issues—and we can talk about that, too—or you’re just not utilizing or taking advantage of some of these really promising features, auto scaling type features, serverless systems, build systems, et cetera. So, yeah.
Manico: From a developer point of view, the thing that’s changed the most for my world is that I used to build apps to a certain standard, and I would kick it over the fence to infrastructure and they would deploy it and all would be well.
Ashley: Mm-hmm.
Manico: As the shift goes to, really, these bleeding edge Cloud Native technologies, my code is now infrastructure. I’m no longer depending on a third entity to deploy my code, to handle setting up my infrastructure. I am now, as a developer, I am the infrastructure now. My code that I check into GitHub, my YAML files—that’s my entire infrastructure now, which is, frankly, that’s daunting. That’s a lot of responsibility put into my hands as a developer. And, you know, every line of configuration I build directly impacts the security of the systems that I’m building.
I think that culture change is dramatic and still ongoing within the world of software engineering. We’re still trying to get it right.
Ashley: Mm-hmm. Let me ask you—so, any time there’s a new generation or a turnover of technology and we construct and build things differently, one of the issues you often run into is trying to build things the old way, but in the new tools. How do you avoid doing that so that you’re truly taking advantage of Docker and containers or Kubernetes and all the infrastructure that goes with it instead of trying to write old stateful applications that assume—make certain assumptions around the infrastructure?
Manico: Let me take a swing at this one and I’ll pass it to Jimmy, but I love being a software engineer. I’ve been writing code professionally almost 30 years, now. And the thing that I laugh about is that I got hired to build an accounting system in mainframe, and I got hired to build the same, exact piece of software in client server, in web 1.0, in web 2.0, in mobile. Now, I have to secure it all. It’s like, I’m writing the same software in every major technology stack—what a great job.
Now, that aside—wow, what was the question? Let’s go back to the actual question. What was the question? The question was?
Ashley: The question was, how do you kinda redesign your thinking so you’re building apps in the new contemporary way using the tools of this generation and not trying to rebuild it the old way with just new tools?
Manico: A few quick bits on that is that, when you’re building to, say, Kubernetes and you’re building to the pod standard, you have to completely redesign your code. You can’t just take a monolithic, old school, service based API and slam it into Kubernetes. You’re defeating the whole point of why you use Kubernetes.
So, instead, you want to break down your application into basically what people call microservices and understand and have those communicate together well, have those different services, scale completely differently within the same infrastructure, and it’s an absolutely different way of writing code.
So, when people are taking these old monoliths or old web apps and migrating them to newer technologies—yeah, I guess some of that code can be rescued and reused. But you really have to rethink development. You really have to write applications to an entirely different infrastructure standard, which is usually a rewrite, as you migrate to new technologies, if you want to get the benefit of moving to something like Kubernetes.
Ashley: Okay, good. Thanks, Jim. Let’s paint the scenario now of, let’s say I’m a security person, security engineer, I’m the leader of a security organization, and I’m going to this talk at RSA and I’m hearing the future of AppSec is Cloud Native.
How do you convince a security person—not another developer, but someone from a security group that this is how it works, this is why it’s secure, here’s what you need to look at so you’re gonna feel comfortable about it, understanding some of these things maybe even the security team doesn’t directly control themselves any more?
Mesta: Yeah, I’ll take a stab at that. So, I think you mentioned how do you convince somebody that this is secure—hopefully, by the end of the talk, that’s not the takeaway. I’ve found just over the years, you could build a cloud infrastructure that is free of major vulnerabilities, or at least on the scale of being secure or not, it is hardened, you’ve done the right things.
But I think, as I’ve learned over the past couple years, seeing these transformations happen, is that it’s easy to mess this stuff up, and it’s really a lot to learn. So, for me, just understanding the intricacies and the details around Kubernetes defaults without doing anything, right, with just—I want a cluster, here’s my options, I spin one up, here’s my Docker container and I’m running it now in this containerized environment. So, what do I have to worry about from a security perspective? And it’s completely different than just basic, you know, operating system security.
Ashley: Mm-hmm, mm-hmm.
Mesta: It’s a lot of different moving pieces that are not fully understood, even by the best of the best, right? So, you add that plus the complexities of public cloud offerings, right? Identity and access management in general is not easy, right?
And also, a lot of the security tooling that has existed in the past that we’ve leaned pretty heavily on in AppSec and as well as infrastructure and network security, it doesn’t—you can’t just point it at a highly containerized, modern environment and hope that you’re gonna get the same results that you did in the past. Kubernetes has dozens of ports open by default that just have to be there.
Ashley: Mm-hmm.
Mesta: So, how do we educate people on, “Well, so this one’s okay. This one’s okay when it has authentication. And this one’s really bad. You never want it open to the Internet, right?” So, there’s all these things that Kubernetes kind of abstracts out of the way as well as other Cloud Native technologies that will come back to haunt you later unless you educate yourself early.
So, that would be my message to any decision maker or kinda security professional attending is—hey, there’s a certain amount of leveling up we need to do in our entire industry to get on board with this stuff and make sure that we are adapting our tool sets or building new tools or just generally hiring people who have this intimate knowledge of cloud environments, or you’re gonna be left in the dust.
Ashley: Mm-hmm.
Manico: When you asked that question, I was like, I immediately took a step back away from Jimmy, because I knew that he can—I thought his head was gonna pop.
Ashley: [Laughter]
Manico: I know he can answer that question for like 20 hours nonstop and still answer it, so—
Ashley: You know it’s coming, you know it’s coming at some point, right? [Laughter] Well, you know what I like about your answer, Jimmy, is there’s so much to be learned but also so much to be shared from what people are doing in Cloud Native applications and these modern architectures.
And part of what I heard you say is, the developer community, it’s super important that it is a community, that they’re working together, sharing, learning from each other, et cetera is kinda the state of the art, it’s being advanced. It’s not just downloading the latest tool, it’s learning from the community, and the corollary to that is, you know, from my perspective, both from software but also from security—security’s often been from the outside in when it comes to applications and security. Yes, you have authentication, we do vulnerability scanning on the services, we monitor the network, what parts do you have open?
But you really think to—I think you’re saying you really start to need to learn some of the Cloud Native tools and technologies and be comfortable with that as a security person so that as you’re reviewing these architectures and what’s being built or has been built, it’s not just to sort of look at it from the hard exterior shell of the m&m, you kinda have to look at the chocolate, too, and understand how it’s working.
Is that what you’re saying? Am I getting it?
Mesta: Exactly, yeah. And a concrete example of that would be talking about Istio, Envoy proxy. So, Istio is a service mesh that people often like to explore implementing inside a Kubernetes, because it has a lot of great security promises. There’s all sorts of kinda policy based rules that you can build into it, and it’s great, right? It checks a lot of boxes for security teams, but it is over 5,000 lines of YAML that you need to install this thing, right?
Ashley: Wow.
Mesta: And it’s not a straightforward task. If you just read the headlines about a service mesh architecture, you’re like, “This is great, you know? Mutually authenticated microservices inside of my cluster and I get all this certificate management, all this stuff out of the box.” But as we were seeing, there’s not a lot of eyes on this from the security perspective, because a lot of people don’t understand it.
So, recently, there was a path traversal bug inside of Envoy proxy. It’s CVE-9901, and it’s basically like a dot dot slash path traversal bug that we figured out a really long time ago in the security world, right? That’s not something that, Nginx has it figured out 15 years ago, but this is the kinda stuff we’re gonna start seeing surfacing inside of Kubernetes, inside of service meshes, serverless—you name it, because it’s new and we’re just putting all our eggs in this basket and ignoring the underlying plumbing.
So, that’s kinda what the talk gets at, right? It’s like—yes, embrace this new world. It’s gonna help you in the long run, but let’s not forget our roots and our basic senses as we start just picking and choosing off this menu, this little a la carte menu now of AWS or GCP—it doesn’t matter, we can just click a few things and we’re running this really complex system. And, you know, I just wanna make sure people just take a step back, that’s all.
Ashley: Well, gentlemen, I think you’ve done a very thorough job of at least whetting our appetite to come to this talk. I’m certainly very interested and want to hear more. Wish you the best of luck. Just a reminder to our listeners, the talk is, “The Future of AppSec is Cloud Native.” This is at RSA in Singapore on July 17thh at 4:30 p.m.
I’d like to thank Jim Manico, founder of Manicode Security, and Jimmy Mesta, CEO and co-founder of KSOC and CTO of Manicode Security. Gentlemen, thank you for being on the podcast.
Manico: Thank you so much, Mitchell.
Mesta: Thanks so much. Hope to see you in Singapore.
Ashley: And I’d like to thank you—you, our listeners—for joining us, as always. This is Mitch Ashley with DevOps.com. You’ve listened to another DevOps Chat. Thank you for joining us. Be careful out there.
