Overcoming Ransomware Attacks – Techstrong TV
When crucial data is infected by ransomware, most companies try to keep it from public exposure. However, data storage company Spectra Logic has chosen to be completely transparent with its ransomware attack. Tony discusses his first-hand experience of this attack, including the signs of ransomware attacks, data storage defense tactics and best practices to overcome an attack. The video and a transcript of the conversation are below.
Recorded Voice: This is Digital Anarchist.
Charlene O’Hanlon: Hey everybody, welcome back to Techstrong TV. I’m Charlene O’Hanlon and I am here now with Tony Mendoza, who is a senior director of IT over at Spectra Logic. Tony, thank you so much for taking a couple minutes and talking with me today. Do appreciate it.
Tony Mendoza: Absolutely. Thanks for having me.
Charlene O’Hanlon: So I’m interested in hearing, you guys recently actually went through a ransomware attack and wanna hear about your experiences with that, and also why you guys decided to actually take the information public. But first I wonder if you can let the audience know about Spectra Logic in case there are some folks out there who don’t know what you guys do.
Tony Mendoza: Yeah, absolutely. So Spectra Logic is a company we that’s been around for about 45 years. I’ve been at the company for 20 years and been through the entire IT organization, and now part of the business leadership team here. Spectra Logic, we like to call ourselves experts in storage. We manufacture hardware storage products, such as tape automation and disc. We also have recently become involved in storage software.
Charlene O’Hanlon: Okay, great, great. So definitely a company that has been around the block a couple times and I’m sure you’ve seen a lot of trends happening in the IT space in general. Tell me about your ransomware attack. What happened there?
Tony Mendoza: Yeah, absolutely. So this is I think back to early part of 2020 when we were being infected with COVID. We had to start thinking about what we were gonna do with our workforce and how to keep our workforce safe. And we decided to move our traditionally on-premise workforce to a remote workforce. And this is pre-hybrid, so this is purely remote to keep people safe.
And we did it very quickly. And so we went from a, like I said, traditionally on-premise workforce. IT was moving everyone remote and building the tools that they could use to connect. And security was part of that discussion, but this is also the time where ransomware was taking off because it was an opportunity for them to start attacking people. So we got attacked.
And this is, like I said, right in the middle of us moving remote. So our IT group still was on site. And our remote workforce got attacked. We are here on site and we saw the systems failing, and they failed very quickly.
Charlene O’Hanlon: Wow. So I imagine that that was a bit of a shocker for you guys, especially since we had heard about ransomware happening here and there. But it really was not until to your point, everybody got sent home, everybody was working remotely that cybercriminals really took advantage of things. So, how did you guys know that you were up against a ransomware attack and how did you guys get through it?
Tony Mendoza: Yeah, it’s interesting. We started seeing minor system failures and we started to investigate them. We started getting calls from end users saying that they’re seeing some weird anomalies. And we kind of took our time and thought, okay, there’s something going on here with our infrastructure or there’s a failure here.
Until we saw a directory where there would be thousands of files, there was now one file with a note and it was a very simple text note that said, “Your files have been encrypted. Here’s the ransom. You have five days to pay it.” And as soon as we saw the first one, we started looking at other systems and found out that it was propagating itself very quickly.
Charlene O’Hanlon: Huh.
Tony Mendoza: And literally this is no joke. We ran to the data center and started pulling plugs.
Charlene O’Hanlon: Wow. Wow. So, how much of your company was actually affected by it? Were you guys shut down for a while to get through it? Or what was your response?
Tony Mendoza: Yeah. So jumping ahead to the end, after we did our analysis of it, we figured that about 40 percent of our systems were infected by this. The problem was the majority of those were our tier one systems that we need for business to function. So our business was shut down for about five to seven days before we got people back up and working, not our entire infrastructure recovered.
But taking a step back, we shut down the entire infrastructure and started looking at our disaster recovery plan and our co-op plan and started contacting people saying we’re down and we don’t know what the extent of the damage is. We don’t know when we’ll be back up. So the first few days were really my team working 24 hours a day, trying to literally stop it and assess it.
Charlene O’Hanlon: Wow. I can’t imagine having to go through that on top of everything else that that was going on in the world. And kudos to you guys for being able to get up as quickly as you did. I’ve heard stories of companies that are down for weeks and even months trying to reconstruct their files and get back online.
Tony Mendoza: Yeah. And let me be honest with you, we took about five days before we got to a point where we were comfortable with our decision to recover our systems. And honestly, it was 30 days of getting our systems back up and running, obviously a priority was to get our workforce working again, and so we slowly got them working again.
But it was about 30 days to get our infrastructure back up to where we thought it was performing at about 80 percent. And it took probably six months to finish cleaning up the mass.
Charlene O’Hanlon: All right, all right. All right. We do hear about companies undergoing a ransomware attack, but not a lot of them are very public about the fact that they’ve done it. And it’s really only kind of, they’re being forced into disclosing the fact that their files were compromised and their systems were compromised.
You guys, you’re being very open and honest about it. So why did you guys decide to take this route and just let folks know that, “Hey, we were attacked, we lost files. We had to go through everything to get back up online. Why are you guys being so open and transparent about it?
Tony Mendoza: You said something that a lot of companies have to go public because they’re forced to go public. I’m in IT and I don’t like to go public about what we consider a breach or a failure. But one thing that happened with us in this attack is, when we got to that day five, which is their deadline to pay this 3.6 million ransom, which for a company like ours is a substantial amount of money, we got to this point where we saw that our backups on tape were immutable, they were untouched.
And we had disc snapshots that we call virtual air gaps that were also untouched. And we made a decision at that point that, “Hey, look, we’re protected here. We better be. We’re in the storage industry. We call ourselves experts.” But we had a formula, and the formula worked and we also realized that we had no data that left our company.
So we had two things going for us. We had a formula for recovery, and we were confident that no data left our company. So we were protected there. And we sat down with leadership and decided, “Hey, we should share this formula with people.” It’s not about paying the ransom. And sometimes when you pay that ransom, you’re not gonna get the recovery keys from the threat actors.
And if you do, it’s gonna take you months to get your data back to a usable point, if you can even recover it. And so we thought this is something that we ought to share and help people protect themselves, and at least give everyone a fighting chance with it.
And another thing it did is, with this formula, we sat down with our product teams and said, “Hey, we can develop something here that helps people out that provides markets this formula to people. And so that’s where our attack hardened products came out of the picture, is we said, “Look, we –” I don’t wanna say we figured it out. And I don’t wanna say we’re 100 percent, but we have something that could give people, ease their minds a bit about it.
Charlene O’Hanlon: That’s really interesting. You guys turned something as, could be potentially devastating as a ransomware attack, actually turned it into a business opportunity, which kudos to you guys. That’s pretty awesome that you guys –
Tony Mendoza: Yeah, yeah. That worked out well for us. And it’s funny, I speak about it quite a bit and I know, to a certain extent, I’m putting a target on Spectra’s back by talking about these things, but I’m also a believer that it’s gonna happen again and you can’t block 100 percent of it. And you just have to have at least the tools to arm yourself with decision-making power of, “I can at least recover my data.” It’s painful, but you can do it.
Charlene O’Hanlon: Well, a lot of things in life are painful, unfortunately. So what is the plan that then that you guys are selling, I guess for lack of a better word? Are you guys advocating for more on-premises type data backup and protection, or are you looking at more of a cloud-based solution, or is it some sort of hybrid of the two or? In simple terms, what are you guys advocating for?
Tony Mendoza: It’s really hybrid. What we realize is tape is still the silver bullet. We keep tapes offsite, those are untouchable, those are pure air gap systems that now come into play with this. They used to be solely for disaster recovery, what we consider disaster recovery, which is a physical disaster.
We realized they work well in this kind of this cyber-attack and in this business continuity recovery. So that’s one of those things. If you have that silver bullet, you can start now making decisions on how you can protect your data in other ways that’s maybe quicker to recover or more near line to your production network.
So what we’re advocating is still obviously that air gap, you can’t get away from that, but we have this, what we call virtual air gap with our attacked harden products that you can take snapshots throughout the day. And we do it every 15 minutes now. And you can protect them. You can snapshot them, you can replicate them and you can keep them protected.
And that goes for your cloud data too. And I hate to really blanket all cloud data, but it’s up to you to protect it. It’s not up to your providers to protect that data or to back it up. So we built that into our systems that you can now get snapshots of your cloud data also.
Charlene O’Hanlon: Oh, that’s very, very cool. I like the idea of a hybrid solution, because it’s always good to hedge your bets whenever you can. Even if some of it can’t be recovered, at least you’ve got part of it that you’re not starting from ground zero, basically.
Tony Mendoza: Yeah. And really, one of the things we really advocate is to limit your attack surface area, your blast radius. So, if you have multiple systems and you have multiple snapshots and you have things protected, you have options.
Charlene O’Hanlon: Yeah. And considering the amount of stuff that has gotten up into the cloud, especially since the onset of the pandemic, it just makes sense. Tony, what a great story. Thank you so much for walking me through it. And I’m so glad to hear that you guys recovered nicely and have actually made it into a business opportunity. I’m amazed at that. I think it’s a great story and I’m glad you shared it with us. I do appreciate it.
Tony Mendoza: Great. Thank you. I appreciate it.
Charlene O’Hanlon: All right. All right, everybody, please stick around. We’ve got lots more Techstrong TV coming up. So stay tuned.
[End of Audio]
