SANS Institute: Preparing for Cyberattacks

Security incident response and forensics are at the top of every CISO’s capability list. But how well are the executives, department and organization prepared? Rob Lee, SANS Institute chief curriculum director and faculty lead, discusses the skills and new levels of preparedness security teams and businesses must have to deal with the onslaught of supply chain, vendor and nation-state breaches occurring on a near-daily basis with Mitch Ashley. The video is below followed by a transcript of the conversation.

Mitch Ashley: I have the pleasure of being joined today by someone who I’m excited to talk about, because my relationship with SANS goes way back almost 15, 17, 18 years. I’m joined by Rob Lee. Rob, welcome. It’s good to be talking to you today.

Rob Lee: Hey, thanks for having me. I’m really excited to be here. Thank you.

Mitch Ashley: Yeah, I’m excited to talk to you because you’ve been doing things with SANS Institute for a long time, and an expert in some really topical areas, which we’ll get into. You’re chief curriculum director and faculty lead with SANS Institute. Tell us a little bit about you and just a little bit about SANS for the one person in the world who might not know what SANS Institute is. Tell us a bit about that.

Rob Lee: I’ll start with SANS and why I love working with SANS. I like being able to engage and train individuals, to teach them key skills. My background is incident response and digital forensics, and being able to help them out to really engage and to do it right, and not just spin their wheels when an actual incident happens. My background is I’ve been working in the cybersecurity industry since 1996. I’m an Air Force Academy grad. I landed at the 609th Information Warfare Squadron, which was one of the first cyber units that was ever created, and then quickly transitioned over to an investigative unit at the Air Force Office of Special Investigations, in which it was in 1998 that I started working on advanced persistent threat cases of attacks coming from Russia.

Pretty much my entire career from that point forward working as a government contractor, Mandiant, various other startups during that point in time, really focused on incident response, dealing with nation state attacks ‒ China, Russia, Iran, North Korea, criminal groups ‒ and so forth, all kind of centered around and fed this really deep experience. The reason SANS does what it does is, you know, “Hey, were you working full time for SANS?” No. SANS has our practitioners as instructors, so I was basically moonlighting as an instructor and writing courseware overnight. All of that experience basically fed into the courseware. My courses were the advanced incident response course and the Windows forensic analysis course. In the past year and a half, SANS has asked me to move from my curriculum lead role, which is digital forensics, to an overall role for running all the curricula, offensive operations, cloud, management, blue team operations, cyber defense, and of course digital forensics.

SANS has over 80 courses and we have over 150 certified instructors that are all practitioners out there in the field doing this. It’s one of those things that when you see people come back to you after they’re working a case, especially in forensics, a lot of the stuff you see on the TV, I’ll get an e-mail from an FBI agent or someone in law enforcement, Interpol, people working the actual cases, Solar Winds and stuff like that. It’s like, “Hey, just so you know, all those skills you taught me actually made a huge difference,” and that’s this passion of mine, to make sure we’re doing it right, and we’re educating and helping people be able to move that line forward.

Mitch Ashley: Gratifying. Talk about changing people’s lives, helping them in their career ‒

Rob Lee: Yeah, it’s important. This stuff is so fast-moving. It is so hard to keep up with it, and we’re constantly updating our content with the latest information because it’s constantly moving science, too, that you’re trying to understand what are the best defensive practices, what are we recommending, all the cloud stuff. I mean, the interfaces and the abilities change and APIs change on a daily basis, so keeping up with that and business e-mail compromises, all the way down to everything sitting in the cloud, for organizations, how do we defend it? How do we analyze it? How do we potentially deal with it in an incident response? It’s hard. But, at the same time, it’s important because if we’re not doing it, who is?

Mitch Ashley: Exactly. I don’t know that we need to go through every attack, because by the time this airs, you know, it could be 30 minutes from now, there will be some new thing to talk about ‒

Rob Lee: _____. Yeah, exactly.

Mitch Ashley: ‒ and maybe that’s the point, is it seems to me we’ve reached or we’re past the tipping point. We’re all being attacked all the time and that kind of thing, but we’re now into an era where so many of these attacks we still have to be concerned even if it’s not against us because it’s happening to our suppliers, our partners, software that we’re using. A lot of it gets labeled supply chain, and sometimes it really is and sometimes it’s not. The fact of it is we have to defend against everything we can, but I think, and not just because you’re on and we’re talking, but I think the response is the most important element of it now, so it’s not just, “Will we be attacked? Yes, we will and we’ll have to deal with it.” I think our effectiveness at response, effectiveness at forensics, and ability to, not just in the technical role but all through the organization, the whole response plan ‒

Rob Lee: Yeah.

Mitch Ashley: ‒ _____, it’s vital. That has got to be the number one skill, in addition to defending ourselves. I’m guessing, given your demeanor, you probably agree with me and I may be preaching to the choir, but I really do believe that’s the era we’re in now. Forget about, “This is an occasional thing.” It’s a regular thing.

Rob Lee: It’s a regular thing, and there is always a balance. Some of the things that we end up focusing in on are the things that you’re doing before the attack help prep during the attacks and the response of those, and being able to be prepared. This is where it’s not just having a really crucial capability to do response, it is the red team, even tabletop exercise through how management reacts, how the tactical, technical staff reacts, how they deal with certain servers being taken offline. The thing about incident response, and I always joke about this in the class, incident response is a game. The game is actually called, “Which least worst decision am I about to make?” There is no winning in response. It is, “Okay, so I have to have three choices, and you have to evaluate all three choices, and all three of them end up with you losing.”

The question is which one of those losses is the least worst, and that mindset is what I’m really trying to get into management, because everyone on the ground, the tactical, technical folks, they kind of understand this, but it’s the management that says, “What do you mean we can’t just get back to where we were?” I’m like, “No, there’s going to be an impact. It’s how bad do you want that impact to be, and your choices that you make, especially early on, could exaggerate it or minimize it.” The question is how do you navigate that properly? It’s not just having a bunch of smart people and throw them at the problem, but it’s both management and the smart people, technical people working side by side, and even PR and marketing, like if you come out too soon, “Oh, there are only a few systems that were affected.”

Well, if you miss that, then the trust of the organization, especially some of these organizations that are impacted, are hit heavily. It’s like, “How do we do our communications plan? Who is in charge of it? Why are they writing it? Who is going to be evaluating that?” All of these things feed into that because you have to inform people. This is where even publicly one of the things in response I always get at is it’s okay to say what you don’t know. “I don’t know, we don’t know. We only know what we’re at here.” But, if you try to pontificate, “It’s only limited to …” and you start stating facts too early, that could run you into trouble later if your story changes.

Mitch Ashley: We try to minimize it, like we have it under control when we don’t. I think that’s another thing that we’re starting to see the shift in. We just had a distributed denial-of-service attack a couple of weeks ago against some of our servers, and we were just public about it. We were like, “It’s happening, it’s going on right now, and we’re working on it. We’re trying to figure it out. We don’t know all the answers yet, but it’s impacting our service. We want you to know that we know, we want you to know we don’t know all the answers, but we’ll communicate as we get along into it,” as opposed to, “Everything is fine. It’s just only impacting us a little bit.” I think some of that old mentality, or maybe that’s still mostly how we think, of those three options, all of them seem equally bad to the either uninformed or unpracticed, and they’re not all equally bad, right? They all just sound terrible, so we just cover it up or we hide it. But, is that part of what you have to educate people on, is really understanding what the impact of those options are?

Rob Lee: Well, yeah. I mean, it’s like a shark attack analogy, is that you hear about shark attacks, and yeah, it sounds horrible, but, in reality, it is this panic button and it makes good news. It generates a lot of eyeballs, depending on how things go down. But, in some cases, trying to explain it to say, “Well, these things occur all the time. There are a ton of sharks that don’t attack anyone.” But, it sounds awful and it generates a very emotional response for the uninformed, and that’s one of the reasons it is good news for the information, and they’re surprised when you don’t know things as fast. It’s like, “Isn’t there a fire burning all around you?” It’s the meme that’s like, “This is fine,” as there is fire burning all around you. It’s like, “Wait. What? You just said this is fine, and there is fire everywhere. Are you sure that that’s what is going on?” It’s like, “Yeah, we have incidents all the time, outages. It’s just no different. We just have to restore from backup. We’re going to be up in a few days.”

But, to the average person who sees that, it’s like that meme. You’re seeing the dog say, “This is fine,” with the fire around them. That doesn’t look okay, and so the reaction to that, it’s very judgmental. But, because no one is used to it or the cadence that’s involved in trying to analyze it and not just start doing things for the sake of doing things, which is what we have to train people to not do. It’s like in the military, you need a soldier and airmen to follow their orders, but not just react emotionally to a situation they’re in that’s potentially dangerous. They need to rely on their training, their cadence, who is telling them what to do, even though all this stuff that potentially created emotional responses there, a lot of organizations are just like, “Flip all the switches, yank all the cables, take the chainsaw through the server rack,” all of these things. They’re not really doing that, but there’s a movie out there ‒

Mitch Ashley: Cut all the cables, you know, like there were the days where we did that, but it’s almost like the response in a situation for a lack of planning or readiness is not just running in any direction, because ‒

Rob Lee: Yeah.

Mitch Ashley: ‒ _____ fire might be in multiple of those directions, right?

Rob Lee: Exactly.

Mitch Ashley: So, it’s be thoughtful. I mean, yeah, we’re all concerned about loss and that kind of thing. Obviously, people are wondering where we are and how we deal with all these things, because the line between nation state attacks, against other nation state assets, you know, maybe it never was really as _____ [inaudible due to audio distortion] but it certainly is blurred now. Whether it’s _____ or anybody else, it is backed by some government entity, we can all speculate or maybe some of us know more than others, the fact of the matter is that’s happening. It’s a fact of life, and we’re going to have to respond to it. How do you work in a world where the attack may not be against you but it is one of your suppliers? We’re in this shared risk model. It is not outsourcing risk, because if you’re using someone else’s service as a software, it’s shared risk. How do you live in that kind of a world, especially where we see so many more attack vectors coming in that way?

Rob Lee: It’s an interesting question, but it’s the same one, you know, like you buy lettuce at the grocery store and there is always a chance that ‒

Mitch Ashley: It might be tainted.

Rob Lee: Tainted, yeah, the meat and so forth. The downside of all this is that I think it’s eye opening in the fact that most people are now realizing, “Oh, everything may be an attack vector,” and it doesn’t matter how ‒ the good thing is from a management perspective, and my days at Mandiant all the way through now, it’s like you’re always trying to tell people, I say, “Listen, this is not World War I, where you just build massive fortifications and saying you’re going to be able to keep people out.” There are way too many ways into your organization. We know spear-phishing, and we’ve kind of thought about spear-phishing and said, “Okay, how do we prevent spear-phishing?” We try to educate the users, but you and I will still be susceptible to spear-phishing attacks, especially very clever ones, and everyone has seen them at this point. We’re now seeing additional vectors, like any software. We’re using stream or casting software, we’re using software that’s doing monitoring, software that’s managing the network.

All of these locations are potential avenues into your network. That’s where they say the supply chain, something that is an auto-updating feature, and that’s some things where, “Do we want to update everything automatically so it gets infected, because that’s a provocation mechanism, or is that now something that we need to make sure that we analyze every single one of the vendors’ patches before they come in?” You’re always going to be concerned, and this is what led to many years, people never upgraded anything because they were saying, “Well, if I upgrade, it could take everything offline.” Now they got to the fact where they’re doing automatic patch updates, and that has now led to a level of sophisticated attacks.

Mitch Ashley: Sometimes you don’t have a choice because ‒

Rob Lee: Yeah.

Mitch Ashley: ‒ your suppliers, your SaaS services, whatever, they’re doing their own update, regardless of what you want them to do.

Rob Lee: The main idea, though, is I think we’re finally reaching that level of understanding from the CISOs and the organizations that it doesn’t matter what your perimeter defense looks like. It matters being able to say, “Well, we should be able to start controlling what is on everyone’s system, understanding where risk factors are, that it could happen at any one of these locations, and we need to shift what our defenses are.” It goes back into more monitoring, detection, and then response. That’s where earlier you could detect an anomaly, or it would be threat hunting or cyber threat intelligence, it may be a target, I mean any of these vendors out there, SolarWinds and the latest ones, you take a look at them and it’s like they should be thinking about, “Hey, our stuff is installed on a lot of people’s systems. We might be a candidate victim for this, so how do we make sure we’re not that?” That’s where ‒

[Crosstalk]

‒ where would you invest the money? It’s a detection monitoring, auditing, threat hunting mindset, and it’s like, “Well, how do you do that?” I say, “Well, it’s hard. It’s difficult. It requires a SOC, it requires cyber threat intelligence capability, it requires smart people, and you have to sit there and think about, ‘How do we do it?'” This is where you go back to some of the, “How did they detect it? How did Mandiant/FireEye detect it before they made their big announcement?” On the other side of that coin, I sit there and kind of laugh at this, we were talking about the nation states, can you imagine what’s going on in that room? Someone said, “Do you know who we should hit? Let’s go after those guys,” or someone just did it.

They had all of their malware installed everywhere, no one knew about it, and they were able to infect as much as they can. Of course, once it’s detected, the big announcement was made, “Everyone look for this. We’re now on all incident response mode dealing with this.” But, you can imagine around that room it was like, “Wait a minute. We actually had access to how many organizations, and someone decided to go poke the biggest bear because of what reason?” If you’re the Russians at that point, it was like, “Where did that person go after that decision was made?” I mean, that’s a tabletop conversation I would be dying to witness at some point.

Mitch Ashley: I think there is a counterpoint to that, and it is sort of while the big thing is going off over here, guess what’s happening over here? We’re all looking in this direction. That’s what is happening now, and we’re all not aware of it. There is that possibility, too.

Rob Lee: I just think that as a whole, people are saying, it’s just like, “Hey, no matter how much you try to bubble-wrap your kids in school, they are going to come home sick,” and then it comes down to, “Well, we’re just going to do our best, detect it early enough and then decide once my child is infected, I’m going to keep them at home so they don’t infect other kids.” It’s no different. It’s just the mentality of, “Well, what do you mean my kid got a cold? Aren’t you cleaning the area and the school?” Once we get beyond the shock of, “Oh wow, there is a lot of ways that that’s going to happen,” you go back into, “All right. We’ll do our best, practice good hygiene, and then if we do detect it, we will then keep someone home,” and the same thing with COVID and a lot of other things.

I’m using that analogy here, but that mentality is directly applicable to the cybersecurity industry, and it goes into just don’t put all your eggs in the automatic detection bucket, in the high-priority defense, let’s-encrypt-everything bucket. The vendors are going to sell you that their magic jellybean is the magic jellybean that is going to be used. You’re only as good as your people who are using them, and the detection, monitoring, and threat hunting, cyber threat intelligence, has started to become the key to getting in front of it earlier. As you said, how you respond to it technically is also key, as well.

Mitch Ashley: I would add to what you said, Rob, which is all fantastic, is rewrite the assumptions in your plan and change it to, “We will be attacked, we will be compromised. First, do no harm. Let’s not shoot ourselves in the foot,” and then it’s, “Let’s be prepared and then also learn from that experience.” The things that you mentioned about _____ I’ll have the detection, I’ll have the forensics on the back end, whether that’s internally or with some partners. It’s a different kind of plan, knowing you’re going to get compromised. It’s a fact of life. You will not be able to block everything or stop every attack, period. It’s just not going to happen.

Rob Lee: A lot of organizations, honestly, they get a lot of focus on, “Were we intentionally attacked, or was it just propagated to us?” and that’s where they think about, “Well, at this point it doesn’t matter. Early on in the process, don’t worry about it. Don’t worry about, ‘Is this Russia, China, Iran?’ Who cares. The bad guy, whoever that might be, just hit us. Why they hit us, I don’t know. We have to deal with what’s in front of us now. We can do analysis later.” Sometimes the focus in those early discussions is, “Why did they hit us? Why is this going on? Why us? Why are we being bullied?” It doesn’t matter. Deal with what’s in front of you, and then we can do the long-term analysis later.

That’s where tabletop exercises are helpful because you need to really get them to focus early on mitigating the incident, getting back to square one, and then we can talk all day long about, “Why us? Were we intentionally targeted? Was it a propagation from another location that was hit? Are we the person that they’re passing through to hit their intended target?” All these things could be answered, but don’t worry about it right now. Worry about that in a week once we’ve potentially gotten to stabilization. But, again, it’s like everyone is fascinated with why, like stop. We need to deal with the what initially.

Mitch Ashley: Yeah, you don’t think, “Why did I get a virus, or why did I get ‒

Rob Lee: No, you don’t.

[Laughter]

You kind of take it personally. It’s like, “The virus has it out for me. I don’t know.”

Mitch Ashley: Well, this has been awesome. We really enjoyed talking with you. We would love to have you back and we’ll explore it some more. Obviously, with the forensics, the incident responses, and other great, great courses folks can take, SANS.org, remember, is the site.

Rob Lee: Yeah, SANS.org. For a lot of people out there, SANS.org also has a ton of free resources. We put out webcasts, cheat sheets, live streams all the time. If you’re interested in seeing what we’re providing, SANS.org/free. We have a bevy of resources in there to really help train people on a daily basis through our capabilities.

Mitch Ashley: Excellent. Well, I’m glad you’re here and doing what you’re doing. Continue to do that, and glad to see you’ve elevated your role in the SANS Institute beyond instructor and courseware. You’re helping others deliver those materials, so thanks for being with us, Rob.

Rob Lee: All right. Thank you.

Mitch Ashley: Rob Lee, with SANS Institute.

[End of Audio]