Security Boulevard Chats: Top DevSecOps Lessons with Zane Lackey

In this Security Boulevard Chat we catch up with Signal Sciences co-founder Zane Lackey. Zane was the first CISO at Etsy before founding Signal Sciences. As part of this experience, he has assembled a great list of top DevSecOps lessons.

Zane shares some terrific knowledge with us and this chat is well worth the listen if you are looking for the top DevSecOps lessons you can learn now. As usual, the streaming media file is immediately below and the transcript of our conversation is below that.

Audio

Transcript

 Zane Lackey

Zane Lackey is the Co-Founder/CSO at Signal Sciences and serves on the Advisory Boards of the Internet Bug Bounty Program and the US State Department-backed Open Technology Fund.

Prior to Signal Sciences, Zane was the Director of Security Engineering at Etsy and a Senior Security Consultant at iSEC Partners. He has been featured in notable media outlets such as the BBC, Associated Press, Forbes, Wired, CNET, Network World, and SC Magazine.

A frequent speaker at top industry conferences, he has presented at BlackHat, RSA, USENIX, Velocity, Microsoft BlueHat, SANS, OWASP, QCon, and has given invited lectures at Facebook, Goldman Sachs, IBM, and the Federal Trade Commission. He is a contributing author of Mobile Application Security (McGraw-Hill), a co-author of Hacking Exposed: Web 2.0 (McGraw-Hill), and a contributing author/technical editor of Hacking VoIP (No Starch Press).

He holds a Bachelor of Arts in Economics with a minor in Computer Science from the University of California, Davis.

Alan Shimel: Hey, everyone, this is Alan Shimel of DevOps.com and Security Boulevard, and we’re here talking to Zane Lackey of Signal Sciences. Hi, Zane, welcome.

Zane Lackey: Hey, Alan, thanks for having me on! Really looking forward to the conversation.

Shimel: Cool. So, Zane, not everyone listening or reading this may be familiar with who you are. Why don’t you give us a little background?

Lackey: Yeah, absolutely! I would certainly say—yeah, definitely don’t feel bad if you don’t know who I am. I am certainly not that famous. [Laughter] My background is as a few different sides of the security industry. So, I actually started out as a security consultant and pen tester with iSEC Partners, which later became NCC group.

And after the acquisition by NCC, I got made an offer I couldn’t refuse to go over to Etsy and become their first CSO and really build and run the security program there. And that was particularly interesting at the time, because it was really—at the time, it was Etsy on the East Coast and Netflix on the West Coast that were pioneering what we’re now calling DevOps and this kind of shift to cloud and shift to agile and everything here.

So then, flash forward a few years from that, and myself and my two cofounders spun out of Etsy to really take our lessons learned from that and turn that into Signal Sciences. And so, we’ve been going for about three and a half years now, been growing like crazy, and it’s been a lot of fun.

Shimel: Absolutely, it certainly has been. But you know what? That’s why, I think—you know, why do we do what we do, Zane? We wouldn’t do it if it wasn’t fun, right? When it becomes work—it’s funny, I was listening to a commercial for a business broker down here, and he was like, you know, if you forgot why you started your business or what made it fun, it may be time to sell. [Laughter]

Lackey: [Laughter] Yeah, absolutely.

Shimel: It’s somewhat true there, absolutely. So anyway, though—Zane, I wanted to talk today about some of the lessons you’ve learned at Etsy and other places and bringing this all together into sort of practical tips for defending web apps in the age of DevOps. And, you know, who better than you, maybe, right, to share with us some of your lessons learned that—you know, you’ve got the scars and the T-shirts for.

Lackey: [Laughter]

Shimel: For our audience out here—and keep in mind, Zane, our audience, right, is both security people who are having to live this every day as well as DevOps developers and ops folks who are realizing that more and more, you know, security responsibility lies with them as well.

Lackey: Oh, yeah. Absolutely.

Shimel: Yep, and—

Lackey: Yeah, I think—

Shimel: Go ahead.

Lackey: Oh, I was gonna say, I’ve certainly lived in all of those shoes. I think the perfect example of that was my last pen test that I was on with iSEC and NCC was with a U.S. health care company that deployed once every 18 months. And I left there on a Friday and I started at Etsy as their head of security on a Monday, and that Monday morning, they set me down and said, “We deploy to production 20 times a day. Figure out security in this new world.”

And so—yeah, it was a very jarring jump, that I certainly have, as you said, one thing you have painful scars and T-shirts from along the way, so I’m more than happy to share lessons learned around that.

Shimel: Yep. So, go ahead, now we’ve broached it—Zane, let’s click off a few things. What would you say is maybe the most important lesson learned, the biggest thing people need to understand?

Lackey: Yeah, I think the biggest kind of existential shift that security is going through right now is that security has always historically functioned as this kind of gatekeeper, if we want to be optimistic about it, or blocker, really, if we want to be more pessimistic about it. And I think that’s the biggest change that’s happening is, security is really shifting to think about how does it enable the business by default? That its role shifts from being this blocker or gatekeeper to actually thinking about, how do I enable the rest of the business to move faster—whether that’s the development team, whether that’s the DevOps teams—whatever side of the business they’re interfacing with, the real shift becomes, how do we enable them to move faster?

Now, even if you paid me, I don’t think that I could make a more cliché sounding statement, but that doesn’t actually make it untrue, right—which is that we really need to fundamentally shift our approaches there. And how that starts to change is, it changes kind of both sides of security. It changes the cultural perspective and how we interact with other teams, and it also really changes the technical aspect, right, and how we think about our tooling and our techniques and how we think about building and deploying secure infrastructure and secure applications.

Shimel: Exactly. So, you know, Zane, we hear a lot about the term shifting left.

Lackey: Yeah.

Shimel: And people tend to think of—well, that’s a security guy’s problem.

Lackey: Right. [Laughter]

Shimel: But, to me, shifting left means shifting the whole bubble of security, if you will, the whole thing of security left. Meaning, earlier in the decision process, earlier into the people who are on that left side, if you will—developers, architects, all of them.

Lackey: Yeah.

Shimel: The issue that—so, people get that, Zane, but the issue then becomes, you know, how do you start that? How do you—what do you do? I mean, do you just say, “Hey, dude, I want you to do more security” or what do you do?

Lackey: [Laughter] Yeah, I think there’s a couple things, here. And I actually like to take the shift left mentality—which I agree with—and actually make it something even bigger. Which is that, a lot of times, when I hear shift left described, it’s as you said, you know, kind of trying to get security involved earlier, which I think is spot on.

I think the missing piece—and this is really what shift left means to me is, security needs to be directly consumable by the development teams and the DevOps teams. It can’t be—and this is something that we’ve really seen over the last 20 years of security is that the tools and techniques and approaches have really been by security people for security people. And the challenge that, as you’re going through the shift to cloud or the shift to DevOps is that everything is increasing in velocity and, regardless of whether it’s a particular tool or a technique that you’re using or an overall approach, if it can only be consumed by the security team, you’re never gonna keep up, right? You just fall over in this hail to DevOps and cloud.

And so, I think that what shift left really becomes meaningful to me for is, it’s about empowering those development teams and those DevOps teams with the resources they need to be able to be security self-sufficient for their services. And so, that might be getting involved earlier in the cycle. It might actually be in, as things are running in production, right? How do we give them feedback and visibility that isn’t just consumable by a security expert.

And so I think that was the first real big lesson learned for me at Etsy is that the, kind of just trying to adapt the way that we’ve done security for the last 20 years and trying to shoehorn it into fitting DevOps or cloud—often, that really just doesn’t make the jump. It’s the first thing that people try, of course, and some of that does, but for the most part, you start to think about—okay, we’ve organizationally got this greenfield opportunity for the first time, really, in a generation, as we’re going to DevOps and cloud. How do we think about security approaches that actually empower those teams directly? And let’s really focus on that, whether we’re calling it shift left, whether we’re calling it security self-sufficiency, whatever. I think this overall movement that I’m so excited about is that, for the first time, we can really empower those teams to be security self-sufficient directly.

Shimel: Yep. So, Zane, you actually just described exactly why I got into DevOps. You know, I—

Lackey: Awesome! [Laughter]

Shimel: – that was exactly it. I read a manuscript of “The Phoenix Project” that Gene and shown me, and I thought, “This”—

Lackey: Yeah!

Shimel: – “is the greatest, last, best chance for security, to try to get things right.”

Lackey: I completely agree. I think we’re at that shift—you know, we’re kind of at this tipping point, like, you know, sort of like IT … about seven, eight years ago with the start of the rise of cloud. Which is that, when you were a tech lead or an architect or a DevOps lead or something like that—or, back then, it would be a SysOps lead—and you needed to spin up a new service or some new infrastructure or a new application and you went to that procurement group and they said, “Oh, it’ll take us 12 months to get you some new servers to put in the data center”—why we saw the explosion of cloud is, people could just say, “That’s great, but I’m gonna just put down a credit card and get a dozen servers in five minutes in the cloud.”

And security is kind of at that same inflection point where, if we can’t really help enable this shift to DevOps or cloud, we just get routed around. And so that really fundamentally changes the way we think about our security programs. And it doesn’t mean we stop doing what we’ve been doing, it means that we have this real opportunity to adapt and to change right now.

Shimel: Yep. Agreed. Let me bring up another thing, though, Zane—and I’ve unfortunately seen it firsthand. Because like you, you know, we come from the security world. My grandmother always told me it takes two to tango, and—

Lackey: Right.

Shimel: – as much as we talk about evangelizing and changing developers and DevOps folks and non-security people and building security in and all of that stuff, I think there’s fundamentally, within the security industry, a cultural change that needs to take place, too. You referenced it when you said we have to stop being blockers and become enablers or something like that, right?

Lackey: Yeah, I—

Shimel: You know, [Cross talk] the people who say no.

Lackey: Yeah, I completely agree, there. I think there’s always been this trope from the security side that, “Oh, developers don’t care about security” or, “DevOps folks don’t care about security.” And I gotta say, over the course of my career, I just—I have not found that to be true. I think that people really see security correctly, which is as a piece of good engineering. And when you talk to a DevOps person or a development engineer or anything like that, they want to take pride in their work and do good engineering work here.

And so, I think the real challenge from the security perspective is in, in a lot of ways, a lot of the security tools or vendors or anything like that have really failed us in that regard of, they’ve caused us more problems than they’ve actually solved, and so you see developers or DevOps folks get—you know, kinda wince when they hear a new security tool coming or something because they’ve had negative experiences in the past.

But I really fundamentally believe that folks want to—really want to embrace security and really think about that, and this is where I think it’s on us as security professionals to really think about, how can we be thinking about new tooling or new techniques that actually help those teams become more secure? And that, you know, the vendors that have caused us more pain than they’ve solved there—those are not the answers.

Shimel: I agree. I agree with you 100 percent. But enough about us, Zane. [Laughter]

Lackey: [Laughter]

Shimel: Give us some other practical tips, if you don’t mind.

Lackey: Yeah, absolutely. I think the other side of things here, too, and to kind of drill down on what enablement actually means and everything like that—I’d say that it certainly means things on the cultural side, here. I’ll stick to the technical side at first, which is—one of my biggest lessons learned on that is, security tools, like I was saying before a little bit, security tools that can only be used by security experts? Those just—they don’t survive the jump to DevOps and cloud.

But really taking it one step further is that, even if it could be used by a developer or a DevOps engineer—if someone, if you’re showing up with new security tools where it’s, the responsibility of those teams now go, “Check another dashboard once a week” or something like that? No one has time for that.

And so, when I think about enablement of enabling those teams with security resources directly, it’s about plugging into what they’re already doing, and really thinking about security as a piece of the DevOps tool chain that folks are already thinking about, right? So, it might be integrating with Slack to get a message when there’s something security relevant there. It might be integrating with pager duty to make sure that someone gets a very actionable alert when you need them to get it. It might be integrating with JIRA for tickets or Datadog for certain monitoring or anything like that. It’s really about thinking about security not as this isolated bucket, but really as something that plugs into the broader DevOps toolchain. And that’s really from the technical perspective how we enable those teams.

I mean, I’ll tell you firsthand, one of the biggest lessons we learned there was, we started building a lot of visibility and everything in house because there was nothing on the market that actually helped us out there. And at first, we just kind of put those dashboards and everything up around in the security area. What we really, of our HQ, what we really learned was, the more that we brought those dashboards to the general engineering teams and DevOps teams, the more they could actually—the more they, first of all, became interested, and second of all, the more they could really start to self-serve and use that data to say, “Oh, it looks like someone’s actually abusing my service in this way. I can make a change to defend it myself.”

And so that’s really how we survive the scale to DevOps and cloud, because, for all of us that are on security teams, right, we can always have a million open head count, because we can’t possibly hire fast enough. And yet, at the same time, the development teams and DevOps teams are moving quicker, and so the only way we can keep up with this scale and this velocity is really bringing them tooling that plugs into the rest of what they’re already doing. And that really, actually—I mean, we got so fed up with the vendors in that space that that actually became what we spun out with Signal Sciences was building a product around exactly that. Because we had to build so much stuff in house during this shift.

Shimel: Sure, sure. Excellent. So, Zane, we spoke a little bit about Signal Sciences, but for those of our community members from the DevOps side of things, they may not be as familiar. Can you give us a little bit more?

Lackey: Yeah, totally. And I’ll kinda keep it from the practitioner perspective, because I don’t think anyone, myself included, loves to hear a vendor pitch or anything like that.

So, for us, the challenges were really that, you know, when we were going through the shift to DevOps and cloud, we needed a modern, effective way to defend our web applications, our APIs, our micro services, right? Everything at the web layer that we were exposing. And the only thing that we’d ever historically seen in the past year were things like legacy lapse, which really caused us more pain than they actually solved.

And so, we really experimented with a ton of things internally, and when we found a few things that were successful, the more that we talked to our peers, the more that we saw folks going through the same shift. And so eventually, we spun that out as Signal Sciences which—now, today, we make a SaaS product that you can drop in to defend your web applications, your APIs, your micro services. And I think the really crazy thing that—it’s kind of funny. No one believes me when I say this until they actually get to try it. I mean, the crazy thing that we actually see with our customers is, you can drop us in to get that protection and that defense with genuinely no learning and no tuning, no having to go adjust false positives or rules or anything there.

I think that’s been one of the most shocking things for our customers is that over 95 percent of them are in full blocking mode for defending their applications for their production traffic at scale. And there’s no kind of star on the end of that of, you know, only for one or two rules or one or two services. I mean what I’m saying there of at scale for the production traffic, full blocking mode for defense of those applications.

And I think the other side that they really get excited about is—you know, a modern architecture that’s not getting in the way of their DevOps or development choices, and that really plugs into their DevOps toolchain and provides them with that visibility and that enablement that I’ve been talking about.

Shimel: Got it. So, you know, Zane, the flip side of all this is that DevSecOps, if we can call it that, or you know, DevOps and Security—let’s leave it at that, rugged DevOps [Cross talk].

Lackey: Sure. [Laughter]

Shimel: Right? Whatever you want to call it, as long as we’re not late for dinner—it’s one of the most sought after, hot areas within this greater DevOps jungle, which is pretty hot itself. Do you think it’s just a fad that we’re going through? Do you think it’s a realization that, jeez, security really is important? Where do you think that goes in the future?

Lackey: Yeah, I certainly don’t think it’s a fad. I think that what I’ve really seen, you know that—I’m really fortunate that, every day, I get to talk to a different CIO or CTO or CSO that’s going through this shift. And I think what we’ve all seen over the last couple of years is that people have really recognized this as a strategic shift for their business. That they’re going to cloud, whatever that means, whether it’s public cloud or private cloud or one cloud provider or multiple cloud providers—you know, whatever the actual details mean for your business. But people are going to cloud. At the same time, they’re embracing DevOps.

And really, why—I’d say the fundamental reason why I don’t view it as a fad as far as its overall shift is that, once you’re able to move quicker, there’s no business on the planet that says, “Oh, actually, what we wanna do is move slower. We want to shift features slower than our competition. We want to be able to spit up new applications or infrastructures slower than we were doing,” right? Once you start to move faster, the business really recognizes the value of that.

Now, I certainly won’t say there won’t be some shiny new marketing term for this space or something in a few years. This is clearly why I’m not in marketing. But I think the underlying shift of what we’re all kind of calling DevOps and cloud right now is very much here to stay, and in a lot of cases will actually only increase in velocity.

Shimel: Excellent. I agree. I do agree. Zane, we are—we’re probably way over the time we were supposed to do.

Lackey: [Laughter]

Shimel: It was too much fun—I didn’t want to stop.

Lackey: Exactly!

Shimel: What have you got coming up? Anything new on the horizon exciting that you want to share that you can share?

Lackey: Yeah, I think, I’m just excited about this space and that more folks are really—you know, as we’re seeing this shift to DevOps and cloud, I think that it really gives us a chance to bring in some fundamentally new approaches to security that can really help enable the development teams and DevOps teams.

And, you know, on a personal note, that’s something I’m just really excited about. I think that security as this kind of outsourced blocker—I don’t think it really did us as much good as we had hoped at the start of that years ago, and I think that it gives us this really exciting chance to really be better at security and make team security self-sufficient, which is really how we scale.

So, I’m excited on that perspective. Signal Sciences’ perspective and everything there, it’s absolutely been a wild ride so far and it’s only been three and a half years. I mean, the sort of massive name brand customers that we work with that we’re really excited to help protect has been extremely exciting. And I think the other thing that’s been exciting on that side is just the fact that we’ve been able to really work with them across the spectrum of threats that they actually face at the web layer, right? Not just something pigeonholed for … OWASP injection issues or something like that, but the real threats that we all face as defenders, right? It’s the misuse of our applications and our APIs. It’s the account takeovers, it’s the application DDoS and bots as well as the OWASP injection issues and things like that.

So, from a company perspective, that one has been really exciting that we’ve been able to help people get coverage across that full spectrum of threats that they actually face. But, overall, I think whether you’re on the development side or the DevOps side or the security side, I think what’s so exciting to me is that all of those are really merging together into the ability to move quicker and the ability to become security self-sufficient.

Shimel: Yep. I agree with you. Zane, we’re gonna call it a wrap. You know what, Zane, we’d love to have you on more, though, and we’ll schedule maybe another time to have you on for a further chat, and of course—

Lackey: Oh, fantastic.

Shimel: – as with RSA and everything else, it’ll be great.

Lackey: No, this was absolutely fantastic. I really enjoyed the conversation. And yeah, I didn’t notice that we went over, either, because I was having so much fun, so thank you for having me on.

Shimel: Yeah, we try, but you know, it’s all good, Zane. Hey, Zane Lackey, Signal Sciences—thanks for being our guest on this chat, and this is Alan Shimel for DevOps.com and Security Boulevard. Hope to see you soon on another chat. Bye bye, everyone.

Alan Shimel

Alan Shimel

Throughout his career spanning over 25 years in the IT industry, Alan Shimel has been at the forefront of leading technology change. From hosting and infrastructure, to security and now DevOps, Shimel is an industry leader whose opinions and views are widely sought after.

Alan’s entrepreneurial ventures have seen him found or co-found several technology related companies including TriStar Web, StillSecure, The CISO Group, MediaOps, Inc., DevOps.com and the DevOps Institute. He has also helped several companies grow from startup to public entities and beyond. He has held a variety of executive roles around Business and Corporate Development, Sales, Marketing, Product and Strategy.

Alan is also the founder of the Security Bloggers Network, the Security Bloggers Meetups and awards which run at various Security conferences and Security Boulevard.

Most recently Shimel saw the impact that DevOps and related technologies were going to have on the Software Development Lifecycle and the entire IT stack. He founded DevOps.com and then the DevOps Institute. DevOps.com is the leading destination for all things DevOps, as well as the producers of multiple DevOps events called DevOps Connect. DevOps Connect produces DevSecOps and Rugged DevOps tracks and events at leading security conferences such as RSA Conference, InfoSec Europe and InfoSec World. The DevOps Institute is the leading provider of DevOps education, training and certification.

Alan has a BA in Government and Politics from St Johns University, a JD from New York Law School and a lifetime of business experience. His legal education, long experience in the field, and New York street smarts combine to form a unique personality that is always in demand to appear at conferences and events.

alan has 7 posts and counting.See all posts by alan