The Challenge of Secrets Leakage & Preventative Steps – Techstrong TV
Charlene O’Hanlon and Yury Koldobanov tackle the challenge of secrets leakage in Git repos and what organizations can do to prevent it.
Voiceover: This is Digital Anarchist.
Charlene O’Hanlon: Hey, everybody, welcome back to Techstrong TV. I’m Charlene O’Hanlon and I am here now with Yury Koldobanov, who is the Director of IT and Acting CISO over at Mirantis. Yury, thank you so much for being here with me, and I sure hope that I got your name pronounced correctly. Did I or did I mess it up?
Yury Koldobanov: Thanks, Charlene. Thank you for having me. Yeah, you’ve done it right, no worries.
Charlene O’Hanlon: Yay, all right. That’s half my day right there. If I can pronounce your name correctly, then I’m in good shape.
I wanted to talk to you a little bit about some of the things that you guys are doing over at Mirantis in the way of secrets management, and how you guys are kind of controlling secrets leakage using the public repositories that so many organizations are using these days. I know your organization uses GitHub extensively, right?
Yury Koldobanov: Yeah, that’s correct.
Charlene O’Hanlon: Great. So what prompted you guys to think about secrets leakage when you’re dealing with all of your developers and other folks in your organizations, using the GitHub repo to store code and other stuff? Did something happen that you guys thought, “Oh my gosh. We really need to focus on this,” or was this just something that you started kind of noodling around and decided that something had to be done?
Yury Koldobanov: Yeah. It was like a few things happening at some point of time. Actually, maybe I’ll start with a few words about the company I work for. So I am Director of IT with Mirantis. Mirantis is a software company, so my internal clients are engineers, software engineers, developers, great customer support, professional services and so on.
Historically, we were doing software products like _____ based OpenStack, and Mirantis was one of the top contributors to OpenStack despite our size. We are a not so big company. We’re not RedHat, but still, we were in the top three, if I’m not mistaken.
Then we switched more to Kubernetes containers and that type of infrastructure, and we provide developers a way to ship their code faster. Importantly, as our software developers were working on the development of those products, there was some shift from private repositories, like we were using to – we used to host internal _____ repositories, for example. Then there was some shift toward GitHub and public repositories because of different reasons.
At some point, we’ve seen the growth of users of public repositories like GitHub, and the growth of the concept of infrastructure service, which means that you have your infrastructure described as code in GitHub repository, for example, in Git usually. If it’s your infrastructure described as code, then naturally you need to have some secrets, right. Your code needs to go to Amazon or to OpenStack with Kubernetes and use some credentials to start some clusters or some EMs or whatever is needed to be done.
This leads to an approach of storing those credentials somehow in the repository, which is the natural way. I have my code there and I put my secrets there.
Then I’ve heard about some cases when this approach leads to some leakage. So someone made a mistake and made some repository public that’s supposed to be private. We’re all just human; that could happen, right.
Charlene O’Hanlon: Right.
Yury Koldobanov: At some point, I got in touch with a couple of vendors that were promoting a solution to solve this problem. At some point, I saw that it made sense. We, together with my boss, we met with a couple of companies. Then GitGuardian, the company we are still working, they got in touch and I talked to them, and I found out that – I mean trust, this is _____ to the problem, so it’s not just something you see in use or in media, and second is that we can help with that.
It’s impossible to do something with that without a tool. I can’t go physically through hundreds of repositories and find the secrets there. So I need some kind of solution or a tool for that.
Charlene O’Hanlon: So how long have you been using this technology then to help protect your secrets?
Yury Koldobanov: I would say two and a half years or so. Before that, I spent some time talking to them, trying to better understand how they work. I was talking to at least one other company in that field. I should say that back then, like two and a half years ago or maybe almost three years ago, I couldn’t say that there were a lot of players in this field. Maybe this is different now. I’ve heard a couple of new names, but at that point, there really were just maybe a couple of options.
Actually, I like to work, our corporation with this company, because they are growing. They’re a kind of startup, and it’s great to hear their recent feedback. So you talk to them and say, “Okay, here is my problem. Here’s how it works for us,” and I hear that they recently added a few features based on our input.
Charlene O’Hanlon: Oh, nice.
Yury Koldobanov: Yeah. That’s really great.
Charlene O’Hanlon: That’s great. You mentioned that the company itself really hadn’t dealt with any sort of repercussions from having secrets up on GitHub. You guys were, I guess, pretty lucky because it seems like with this increase in data breaches that’s happening, a lot of that information is actually being found on GitHub or the Git repos and being exploited in that way.
So what are some of the best practices that you guys have implemented to prevent, I mean beyond getting the GitGuardian technology, but have you guys implemented any particular rules and regulations or best practices to make sure that the secrets that your organization holds remain secret?
Yury Koldobanov: Well, I can speak mostly for my team. I’m not a developer. I’m kind of corporate IT. I am kind of keeping close to that, but I’m not really doing the code or doing the product. So product teams, they have their own approach. They have their own best practices. We are not dictating those development practices, if you will.
At the same time, we mostly rely on GitGuardian and process was built around that. So we have approaches how we receive alerts from them, how we reach out to developers who have kind of _____ system and – [indiscernible]. I have security engineers in the _____ who are _____.
We’ve kind of evolved over time. So we started from just simple _____, like, “Okay. Here’s an alert, Mr. Developer. Please let us know anything about this thing.” Then we formalized like a standard questionnaire, like five questions or so. Like if this is production, if this is something you use for a test, either those credentials are valid or expired, that type of stuff.
Then we started to use GitGuardian. They implemented the new mechanism that basically embeds those four into some kind of Web interface. So instead of going back and forth in e-mail, we just ask people to fill out this questionnaire online, and then we see it on our dashboard on GitGuardian. It’s kind of our, I don’t know, radar. So we see, okay, “How many incidents did we have? What’s false positive? What’s wrong here? What’s good there? What reaction did we get from developers?”
Sometimes we need to interact with developers or directly with the services, because I think that most critical thinkers, when you have public cloud tokens, of course it could be immediately put to use. In such cases, we can directly go and revoke the tokens to prevent malicious actors from using them.
One of the key features of why we started to use it, that was one of the main reasons, they sent alerts. I can’t recall any false positives. So in many cases, this is just some test credential, so if you are pushing some dump credentials, say like passwords or something. But real false positives, I can’t recall we show anyone.
Also, they send in alerts immediately and automatically. So we have integrations with Slack. We have e-mail alerts. We’re integrated now with _____, so my guys on call also get such types of alerts. And we’re trying to use this immediate _____ to act properly and act fast.
This is a real help. So I can’t imagine how else you can do something like that, except for building your own tool, which is apparently not the easiest thing.
Charlene O’Hanlon: That’s great. It sounds like secrets management is well in hand at Mirantis. You guys have the technologies to make sure that what gets put into Git stays locked down. So that’s great. Thanks for sharing your story with me. I really do appreciate it.
Yury Koldobanov: Yeah. Actually, I probably could add a couple of points to that. Also, recently different companies created secret storage or secret management solutions. I will not go too much into details. I think it’s kind of sensitive, and I don’t think you are interested in too many details or your audience.
So among those tools that we are using are GitHub secrets. Basically, you can use their own embedded mechanism to store secrets. Instead of storing it in code, you can store it secretly in an encrypted way, so that your automation just pulls it only when it’s needed.
The second thing, we are also using this HashiCorp Vault, which is also a secrets storage solution. So those things could help you to avoid the risk of storing those secrets in code.
But still, I know from talks with our folks that it’s not working for all scenarios. Like GitHub secrets works only if you are using – it makes sense to use if your GitHub says you are _____. It’s not so beneficial if you are not using it.
Vault is also maybe not applicable for some cases. But we’re trying to find a solution that would work for different scenarios, so you combine those products.
Charlene O’Hanlon: Yeah. I think there is no one-size-fits-all solution for every company in every scenario. So I think a united effort, if you will, to that approach to secrets management. And cybersecurity in general I think is definitely the best approach for any organization to take.
Yury, thank you so much for having the conversation with me. It’s always fascinating to hear how organizations are approaching cybersecurity, especially in today’s – it seems like every day there’s another headline with a data breach or vulnerability or ransomware attack or something like that. So it’s always great to hear this story. So thank you again for your time and your expertise. I appreciate it.
Yury Koldobanov: Thank you. The pleasure was mine. Thank you. Thanks. Bye.
Charlene O’Hanlon: All right, everybody, please stick around. We’ve got lots more TechStrong TV coming up, so stay tuned.
[Music]
[End of Audio]
