Alan and Chris discuss the findings from SoSS (State of Software Security) 12 report and its implications. The video and a transcript of the conversation are below. 

Announcer:             This is Digital Anarchist.

Alan Shimel:            Hey everyone, welcome to another Techstrong TV segment. I’m really happy to be joined by my friend, Chris Eng of Veracode. Chris has been coming on here for as long as we’re doing Techstrong TV, but I’ve actually been interviewing Chris about the State of Security report that Veracode does, oh I don’t know, ten years, maybe more, I don’t even remember.

Chris Eng:            You might be one of the longest standing coverage of everybody I talk to.

Shimel:            Absolutely Chris.

Eng:            You know?

Shimel:            Absolutely, it goes back, I know to probably when I was doing InfoWorld stuff, right?

Eng:            Oh yeah, sure.

Shimel:            And even before the network. Anyway, Chris Eng it’s good to have you on, I hope all is well. How are you, man?

Eng:            I’m good, it’s good to be here, it’s nice to see you.

Shimel:            It’s nice to see you, and for those who in our audience are unfamiliar, Chris, well, is it VP research now?

Eng:            Yeah, I’m the Chief Research Officer here at Veracode.

Shimel:            Chief Research Officer.

Eng:            Yeah.

Shimel:            That’s grown over the years too, right?

Eng:            It’s hard to keep track of, but yeah, the team’s been growing and we handle everything form applied research to product security to incubating new products now; so a lot of stuff going on.

Shimel:            Very cool stuff as well, and for those who don’t know, Chris is a very modest guy. He’s not gonna tell you, but he’s really one of the sharpest guys in our industry and always a pleasure to talk with. Chris let’s give… I kind of mentioned already we’re talking about the State of Security, but let’s give the folks a little background on the report; this isn’t your first time through this.

Eng:            This is not; this is the 12th year that we’ve been doing it, and actually probably like the 14th or 15th report, although it is called number 12. We’ve done a few little interim ones in the middle to kinda focus on different areas. But yeah this is something that we are uniquely positioned to do as a cloud provider. Because Veracode customers scan their software through our platform in the cloud, we have insights in to what they’re doing, how often they’re scanning, what they’re finding, how quickly they’re fixing things, and we can report on those things over time.

So unlike some of the reports that you might read that are survey based, that are asking people, “Hey what are you doing? What do you plan to do?” These are actually… there’s no hiding from the truth on these; this is what you’re actually doing, what we’re actually finding, and how things are getting better or worse over time. So we can slice the data however we want, and we like to put that out in the public every year at least, and the data set has just grown and grown and grown over time.

This time we’re talking about half a million applications. The first time we did this we were talking about about a thousand applications; so it’s come a long way. So it’s real data.

Shimel:            It really has. It is real data.

Eng:            You can see what’s happening.

Shimel:            And that is… look, we have Techstrong research here, as part of the team, and they do a fair amount of surveys and everything. And a lot of the reports we see out here are based on surveys, and the problem with surveys with me is a lot of times it’s like Facebook. On Facebook everyone’s life is either wonderful, like, “Oh my God, I wish my life was that easy,” right?

Eng:            Yeah.

Shimel:            Or you get the other… not the other half, but the other subset of people who are just always woe is me, this terrible thing happened to me, this one died, that one’s sick, the dog and cat are this, and there’s no in between of just real life; it’s either that or that that’s polar. And it’s the same thing with surveys, people… are they really gonna tell you where their warts are?

Eng:            Right, even if it’s anonymized.   

Shimel:            Even if it’s anonymized; no one wants to put on paper, or click a radio button that says, “Where’s the _____;” insecure here or something.

Eng:            Yeah or, like, “I plan to do these things in the coming here,” and everybody has great plans but then things change.

Shimel:            Exactly.

Eng:            So it’s nice because… and those do tell you a lot, they tell you about intention, and they tell you all different… but it’s just a different perspective; to see what’s actually happening. So we like that we can do that –

Shimel:            Yeah, no that’s great stuff.

Eng:            – and I think we’re the… probably by far the biggest dataset out there for software security anywhere. So we like to take advantage of that, and ask some different questions each time, and try and see if we can find some nuggets of interest in there each time to share.

Shimel:            Absolutely, all right so let’s dig in to this year’s report. Well you mentioned it was a huge dataset in terms of the amount of applications, right?

Eng:            Yes.

Shimel:            But let’s dig in a little bit more.

Eng:            Half a million applications over their entire lifetime of scanning with Veracode. So if it was active in the platform year 2021, we looked back at its entire history, going back to sometimes 2016, 2015 and before, because customers have been scanning these same applications as they’ve evolved over time. It’s a different approach than we’ve taken in previous reports, where we just took a slice, we took a window of, let’s say, 12 months and we said, “Show us everything that’s happened in that window, and kinda forget everything that’s happened before.”

And so this has given us some kind of longer term trending of some of the items that we covered; so I’ll tell you what some of those are. One of the things I think was the most interesting, especially as we’ve seen the focus on Open Source recently, _____ and everything else surrounding software billing materials and the Open Source ecosystem. The amount of time that it’s taking organizations to patch their vulnerable Open Source is dropping dramatically; it’s getting a lot faster. Now, it’s still not fast by any stretch, but I’ll give you a couple data points:  in 2017 it took about 3 years for an organization to fix half of their known Open Source vulnerabilities. So half… just to get to the halfway point, and there’s a long tail after that.

But today, or in 2021 at least, it’s taking organizations just over a year to get to that same halfway point. And that is… that’s a three X improvement; we’ve shaved two years off that median time. And clearly there’s a long way to go, if you look at what the log4j response time had to be last month, or actually in December. There’s no waiting a year for those things; you had to do those… people were working over the weekend to get that done in a couple of days, or…. and the long tail was a couple weeks.

So we have a long way to go, but that’s a dramatic improvement, and I think that really speaks to the increased visibility that we’re seeing on Open Source security these days; so that was a good one.

Shimel:            Yeah, I have two kinda points I’d like to discuss with you on that, Chris. The first one is why, but let’s hold it off for a second. I think the log4j; the whole log4 thing is a great example of why sometimes just looking at the average isn’t really a good indicator, or a good –

Eng:            Yeah.

Shimel:            – metric to follow, right? Because when you do get a critical, super critical, whatever you wanna call three alarm blaze like log4 was, that timeframe is compressed, as you mentioned, drastically. I think when we look at something like well we went from three years to a year; that includes probably a lot of just not as critical vulnerabilities, not as mission critical, not as… and so I’d love to see slices of that based upon severity. And I bet you we’ve probably made even more progress.

Eng:            Yeah, and that’s partly why we like to talk about the median, to get to that 50 percent point, because we know there’s a lot after that point that’s not gonna get fixed maybe ever. And we like to show it in the form of a curve, so you can imagine sort of the asymptotic long tail, but something that starts with kinda this steeper drop off that shows that well in the first… let’s say the first few months, you cut a lot bigger chunk of that vulnerable population, and then it kinda flattens out after that. So there’s some charts in the report to show what that looks like, not just for Open Source, but for first party vulnerabilities, for static and dynamic.

And Open Source is actually the slowest of the bunch, probably –

Shimel:            Really?

Eng:            Yeah, probably because it’s a little bit newer of a problem to some development teams, I think. But there’s… so there’s good progress in every area, but yeah you’re right, averages are always difficult to extract the real truth from; but we do what we can.

Shimel:            So what do you attribute the improvement though? Still, three X improvement is nothing to sneeze at. What do you attribute that to?

Eng:            Well I think, and this is interesting, because in our previous report we tried to isolate the different factors that actually affected fixed time. So we said, “We’ll look at all the organizations that are doing… that have a faster… that are scanning more frequently versus frequency; one standard deviation above versus one standard deviation below. How does this group’s fixed times look compared to this other group’s fixed time?”

And we did that for a dozen different factors, just isolating in them like that and pairing. And then we actually found that organizations that scan more frequently and more regularly, so let’s say every week instead of a bunch of times at the end of the month, both of those factors actually correlated with faster fixed times. And so more frequent scanning is another trend that we have seen over the years. The average application is scanned 20 times more often today than it was 10 years ago. I think it’s static, three times a week or more; I think 90 percent of applications or more are being scanned three times a week. Which is… would have been unheard of in 2010 or even 2015.

So you look at that scan frequency and regularity going up, and you cross reference that with the previous finding that we had last year that showed that scan frequency improves fixed time, and you can see why… it makes sense why we’re seeing the things that we are. And there are of course other factors, right, there are cultural factors; there are probably some processing technology factors also. But… and I think the reason that affects the fixed time is because it’s just in your face more often; you see it… you’re scanning three times a week, maybe every day, that _____ ticket is getting refreshed every time, it’s there. There’s just a little bit more visibility than there may have been in the past.

Shimel:            But here’s another thing that I’d like to bring up and get your opinion on, scanning… that’s fantastic, by the way, frequency of scanning, going back to my vulnerability scanning days in 2005 when we couldn’t talk people in to scanning once a year. But scanning gets done by the security team, or in many times it’s automated, but set up by the security team. Fixing the vulnerabilities though, especially _____ vulnerabilities, a lot of times is getting done by developers.

Eng:            Oh always, always being done by developers.

Shimel:            Right, and so –

Eng:            But the thing is… yeah that’s shifted too though.

Shimel:            Yes, that’s the point is hey, we are now at a place… not only are we scanning more frequently, but the results of those scans, maybe it’s because of tools like GitLab or whatever. But the results of those scans are getting to the developers who are responsible for the fixes. And that… that might be a bigger story than the frequency of scanning, quite frankly, that we have developers fixing three X more.

Eng:            Yeah, well it’s all tied together; why are they able to scan more frequently? Well it’s because the tooling is better; it’s because the API’s are better. It’s that there are integrations being built for GitLab, and Jenkins, and whatever CICD… you’re following dev ops way more closely than I have and you see all that happening. All the tools are building integrations, and so if it’s happening in the pipeline, that’s way better than if it’s happening kind of out of the process, and then a security person’s coming with a PDF and handing it to you, right? So everything is being moved –

Shimel:            It’s kind of _____ _____.

Eng:            Everything’s being moved in to the pipeline, and as early as possible, sometimes as far left as in the developers’ IBE. And then that actually ties in really well with the other finding that I wanted to make sure we covered, which was that the tie-in to security training, which happens before anybody even writes a line of code, or can… it kinda influence your thinking. And we looked at companies that are using our latest security training product versus those that are not, and found that the ones that are using it fixed 35 percent faster.

Shimel:            Wow.

Eng:            And so again, thinking about that half-life, how long does it take you to fix half of your known flaws; that’s 110 days versus 170. And the reason I think that is is because this is not normal training; this is not the slideware that you are accustomed to clicking through and reading about the _____ _____ Top 10, and just reading some words on a slide. These are live environments that as you’re a student, you start up a lab on SQL injection. Well an AWS _____ gets spun up with a vulnerable application on it, we have some steps that walk you through it, and you’re running on a live server and able to practice exploiting those things. Understanding what the code looks like when it’s vulnerable, fixing it, proving it… proving to yourself that you have fixed it, looking up the logs, messing around with whatever you want.

There’s no guardrails other than the fact that you can’t touch other thing in the AWS account. But there’s no guardrails on the server, so you’re learning by doing and you’re learning by experimenting, and I think it has… I just think it’s a better way to learn, it’s more interactive.

Shimel:            So let’s talk about… this is a Veracode training product?

Eng:            Yeah it is, actually it’s something that we acquired a few years ago, and then have continued to build lessons on, because we just thought the approach was so great, and it’s just so differentiated from what else was out there. You have some lab style training products out there, but they’re simulations, it’s like you’re… it’s as if you were interacting with the product. But if you try to do a command that it wasn’t ready for, it’ll just not recognize it.

Shimel:            Right.

Eng:            Yeah, so we’ve been doing this… had this rolled out now for a few years, and the momentum was good enough that we were able to get a good, solid amount of data from it. And just kinda see what’s that correlation to fix time. And it could be… yeah.

Shimel:            People who are interested in that training, Chris, where… what’s it called, where do they go?

Eng:            Yeah it’s called Veracode Security Labs, and so it’s on Veracode.com like everything else, and there’s actually I think even, I don’t wanna lie here, but I think there’s still sort of a free trial thing for developers where you can just try it out for a couple weeks and see if you like it. But it’s a very cool way to learn, I think, and I think the impact is even probably better than 35 percent, because what we’ve looked at there was organizations that are using it versus not. So that could be 1 developer out of 1,000 using it versus… it’d be great to look at, “What is the impact on an individual developer or individual team?”

But unfortunately we don’t have that granularity; we don’t know who’s running what so.

Shimel:            You don’t look at the org level.

Eng:            Yeah, but it’s _____.

Shimel:            But it proves something, Chris, that I’ve always believed, which is, “Hey it’s great to make better tools, it’s great to automate more, but at the end of the day, if you don’t get buy in at the people level, whether it’s the security team, the developer team, the dev ops team, the test team, the biz team, the exec team. If you don’t get buy in at the people level, all the… we can’t fix this with tools, smart tools and automation alone.”

Eng:            Yeah, and it’s not –

Shimel:            We need people.

Eng:            It’s the buy in, but it’s also the habit forming, and that’s right. You can buy in to… I can say, “Oh that sounds like a great idea,” and then nobody does anything for a year. But if you say, “That sounds like a great idea, and I’m gonna integrate it in to my tooling, and now everyone’s gonna have to see it every week and report on it.” Now you’re forming a habit; now you’re… psychologically it’s just becoming part of the way that you do work. And I think that… all of those things are contributing here.

So there’s a lot of vulnerabilities still out there, but it’s nice to be able to kinda come back and show that some trends are going in the right direction.

Shimel:            Absolutely. What else from the report? I have another question on Open Source, but I know you got a lot to cover.

Eng:            No, no go ahead. What else you got? I’ll give you one more; I’ll give you one more.

Shimel:            Go ahead.

Eng:            This is… ’cause I think you’ll like this one, especially with how much you’ve paid attention to development over the past few years, and the rise of microservices, and the changing architecture of software. And between 2018 and ’21 we noticed that there was a drop in… so if we looked at applications that were uploaded to us that contained multiple languages in the same kind of tarball or .zip file, whatever’s uploaded to us. It was 20 percent in 2018; it is 5 percent today, and we think… and one of the theories there was that, “Well applications are getting smaller and more simplified because of this architectural move to microservices.”

We overlaid the Google trend, Google search trend for the word, ‘microservices,’ over the… our dataset showing the percentage of applications with multiple languages and you can kinda see the curve go like that, and they kind of overlap each other. Probably other reasons too, but I thought that was one that you would find interesting.

Shimel:            Yeah I have to… well we’ve seen numerous surveys… new applications there’s little doubt that 75 percent or more of Greenfield applications are utilizing _____ _____ microservices, that type of architecture.

Eng:            Yeah.

Shimel:            Of course, there’s a lot more old applications than there are Greenfield applications.

Eng:            Yes.

Shimel:            And how many of those are being converted, or updated to, or… just still functioning the way they were? And that, of course, is –

Eng:            Plenty of the monolithic ones that are never being re-written. But the number of apps that are being, I think, just developed from… I think there are a lot of Greenfield, right? The number of apps that we’re seeing in the system has just tripled over the past… yeah.

Shimel:            Well I think the reason, Chris, is people look, and I’ve spoken to _____, we launched this site, DigitalCXO.com, it aims at digital CXO’s of all things. And here’s an interesting thing, clearly a lot of organizations, rather than updating that monolithic app, or converting it to microservices, think, “The heck with that.” They’ll just go start from scratch, make it clean, nice, and then when the times comes they switch it over, and they… that’s it.

Eng:            Sometimes it’s easier… yeah.

Shimel:            _____ not set that update _____.

Eng:            If you have the luxury –

Shimel:            They start over.

Eng:            Right, if you can do that; I can see doing that on a 5 year old product, or even a 10 year old… the older it gets, the harder it is because you’ve got a 20 year old app, well what’s happened over the years though? People have… things have broken, and then people have put little patches in there to handle these edge _____.

Shimel:            Right, it’s very… it’s a mish mosh.

Eng:            Nothing’s documented.

Shimel:            But not _____ _____ we’re doing a dinner down here in Miami, a digital CXO event, and we’ve got some folks from the gaming industry.

Eng:            Mm-hmm.

Shimel:            Right? So the gaming… the Caesar’s, and the MGM’s and these people for years have had apps, whether you’re a high roller, or you’re whatever. And with the advent of online gaming, they could have built that on to the existing.

Eng:            Yeah.

Shimel:            No, they said, “We’re done with this.” They started fresh with a modern architecture, and that’s what allows them to scale and handle billions of dollars in bets and everything.

Eng:            And update quicker, and add new games and add new features, whatever the case is, yeah, without being tied down to the other… yeah.

Shimel:            And it makes the whole program better.

Eng:            Yeah.

Shimel:            So… and that’s one example, but you can go industry by industry with this stuff. Part of the digital… it’s such an overused word, but part of that digital transformation is, “Hey I’m not gonna try to take your Father’s Ford and make it an electric F-150 Lightning. I’m just gonna build a whole new platform for the electric Lightning.”

Eng:            Right.

Shimel:            And you can keep your old gas powered Lightning until it dies, but you want the electric one and that’s the deal. But it’s interesting… hey, we’re way over time as usual whenever I talk to you, I apologize.

Eng:            It always happens.

Shimel:            For people who wanna really dig in to the survey, and guys and gals out here, this is a great survey; it’s one of the best surveys in security if you really wanna dig in and get at the data. Chris, where can they go?

Eng:            Yeah so it’s on our website www.veracode.com/soss, which stands for State of Software Security. So the PDF is there; there’s also an interactive that you can kinda play with different languages and _____ types that have appeared over time, and there’s different artifacts there. But yeah, it’s a good read; we’ve tried to shorten the report every time, it’s still 40 pages, but there’s a lot of pictures, so hopefully that makes up for it.

Shimel:            People like pictures.

Eng:            Yeah.

Shimel:            Chris Eng, it’s great seeing you, man; I hope to see you in June at RSA, if not before.

Eng:            Me too, me too; that would be great.

Shimel:            Yeah, keep our fingers crossed.

Eng:            Yeah.

Shimel:            All right, good work man and thanks very much.

Eng:            You bet, good seeing you, bye-bye.

Shimel:            Chris Eng, Chief Research Officer at Veracode on Techstrong TV. We’re gonna take a break and we’ve got another guest coming up.

[Music]

[End of Audio]

Recent Articles By Author

• Closing the Gap Between AI-Scale Attacks and Enterprise Remediation
• When AI Agents Inherit Your Identity Dark Matter
• What If Students Graduated With Experience?

More from Alan Shimel