Hot Topics
Cloud Security Security Bloggers Network 
SBN

Protect Your AWS Environment Beyond Patching Log4j

by Lior Zatlavi on December 29, 2021

Unless you’ve been under a rock or otherwise off the cybersecurity grid, you probably heard of the serious vulnerability in the Apache Log4j library. In a nutshell, for versions Log4j2 2.0-beta9 through 2.12.1, and 2.13.0 through 2.15.0, this vulnerability creates a situation in which, according to the NVD, “an attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.”

It’s important to understand this vulnerability because the Log4j library is so ubiquitous the event has been referred to as a realization of the August 2020 xkcd comic:

"Dependency," from xkcd
Source: https://xkcd.com/2347/

Widespread use of the Log4j library has made software vulnerable on a massive scale – with subsequent massive impact on a wide variety of vendors.

The technical aspects of the vulnerability have been thoroughly analyzed so we won’t take the time to do so here. For an excellent technical review of the vulnerability, check out the Log4j post by our colleagues at Mitiga and the relevant list of technical resources they have compiled.

We do feel the need to discuss the general, and probably more strategic, lessons to learn from this incident. Even though a very serious cybersecurity incident, the Log4j vulnerability is not the only one that will be discovered and likely not even the only one placing infrastructure around the world at grave risk as we speak.

The approach many may take to this incident – “I need to upgrade to Log4j 2.17.0 and everything will be fine” – can be visually described as:

"This is Fine"
From “Gunshow” by K.C. Green

You may have patched up the issue at hand but are ignoring other threats that put your infrastructure at risk. If you recognize that such an event will probably happen again, you can use it to improve your cybersecurity strategy and be better positioned next time one creeps up on the industry.

Log4j and Your AWS Security Strategy

Other than the AWS services affected by this finding (which you can keep track of through AWS bulletins like this one), the Log4j issue puts a spotlight on other aspects of AWS security strategy that are on your side of the Shared Responsibility Model. Among these are the permanent credentials and third party access to your AWS resources.

Permanent AWS Credentials – What Are They Good For?

AWS credentials can be stored in environment variables and/or local files on computing resources and workloads. These storage locations have not eluded criminals designing malware, such as the TNT team exploits we reported on earlier in 2021. Just the other day, Christophe Tafani-Dereeper determined in his analysis of 2021 cloud security breaches and vulnerabilities that “static credentials remain the major initial access vector” for breaching cloud environments.

As described by Greg Linares, the weaponization of this specific vulnerability for the purpose of harvesting AWS credentials from such locations didn’t take long:

And this use of it was apparently rather popular:

Identity impersonation is especially dangerous when harvested credentials are permanent, such as IAM user access keys. For this reason, the Log4j vulnerability discovery is a timely reminder for security practitioners to ask themselves: “What are permanent credentials even good for?” The answer — typically very little.

Of course, it’s convenient to use IAM users and access keys since setting up an effective replacement can be a bit troublesome. That “bit” makes it hard enough for most organizations to stop using permanent credentials as they can’t see the benefit of removing them. We argue, though, that the potential fallout from being hacked due to a vulnerability such as permanent credentials far outweighs the slight inconvenience of replacing them.

This aspect of credentials management is also worth your time because many good solutions exist today for just about every use of IAM users – and specifically access keys. Here are a few:

  • Are you using access keys for CLI? Then move to using AWS SSO and, if not possible, consider adopting saml2aws and using identities managed elsewhere to generate temporary credentials for use as needed. If you must, use aws-vault to store credentials in the OS’s secure keystore.
  • Are you allowing users to access the AWS console with IAM users? Switch to federating identities from your identity provider or adopt AWS SSO for easy multi-account access management.
  • Do you provide third party organizations with permanent IAM user access keys to access resources in your environment? STOP EVERYTHING YOU’RE DOING and switch to a service that you can give access to using a third party role with a properly set up external ID – and then disable those permanent access keys.

Aidan Steele recently posted his thoughts on exactly this topic:

Aidan goes on to explain that even for the obscure case of running a Raspberry Pi in a closet, he can suggest a good replacement (and provides open source to assist in implementing it).

Bottom line: It may take some work to fully eliminate use of IAM user access keys or IAM users in your environment. But if you have the slightest fear of AWS credentials being harvested – and the Log4j incident is solid proof – eliminating permanent ones is a good investment of your time and effort. On a more general note, avoid using any kind of static credentials as much as possible.

Third Party Access – Do You Have It Under Control?

Supply chain security is an extremely important part of your overall security posture – and doesn’t always get center stage. Typically, only a handful of legal and technical controls are available for ensuring that access delegated to third parties is not compromised or mismanaged. So it is crucial that you properly manage and monitor such access since it could be a clear attack vector through which a malicious actor can gain unfettered access to your environment.

As mentioned, there is a proper way to give external organizations access to your AWS environment. That said, even if you grant access legitimately and correctly, once the third party is hacked, the access to your environment can be abused. The Log4j library event showed that such threats are no longer theoretical. It’s very likely that some vendors that have legitimate access to your cloud resources were exposed to the reported vulnerability – potentially allowing attackers to use their permissions to your environment. How do you keep things under control?

First, keep track of vendor release notices relating to their current status regarding this vulnerability. If you can’t find such a notice, don’t be embarrassed: contact each vendor and ask point blank if they were exposed and how they are actively mitigating the exposure. Second, the entire event should be an alarming reminder for you to place all third party access under close scrutiny and continuous monitoring. The obvious next question to ask yourself is whether you are aware of all third parties that can currently access your AWS environment. Do you know exactly what actions they are permitted to perform and is that access really necessary for them to perform the services they provide? In fact, are they even active – or is the access a relic of an old project that no one took the time to remove?

Where Does Ermetic Fit In?

As a cloud security vendor, we work extremely hard to provide proper controls for such strategic issues. Using our SaaS technology we constantly and continuously review your environment’s configurations, permissions and logs – allowing you to easily gain insights on the threats arising from such configurations. Let’s see how.

Detection of Static AWS Credentials

Leveraging the Ermetic platform’s unique view into identity intelligence, you gain immediate visibility into the existence of static credentials in the form of IAM user access keys:

Ermetic's identity intelligence filtered to show IAM Users with active access keys

More specifically, the platform delivers immediate results on which identities have been inactive for a significant period of time. Such findings are likely a “quick-win” to discard.

Ermetic's excessive permissions dashboard showing inactive IAM users with active access keys

Permissions Rightsizing

Having access to each identity’s permissions and the logs that indicate what actions the identity is actually performing, and on which resource, allows the platform to automatically generate a per-identity least privilege policy recommendation to perform relevant business functions.

Proactively rightsizing the permissions of the identities with access to your cloud infrastructure is extremely effective in reducing the blast radius from any type of compromise. This is also true with regard to exposure to the Log4j vulnerability.

For example, let’s say one of your workloads runs on an EC2. While you know that this EC2 requires access to S3 buckets you don’t quite know which, and for what purpose. As a result, you decide to attach the AmazonS3FullAccess or even S3ReadOnlyAccess policy to the IAM role that the EC2 assumes. Albeit a very bad practice, we all know this happens more often than we care to admit.

If it runs software with a vulnerable version of the Log4j library, the EC2 could have vast access to all S3 buckets within the environment – with potentially catastrophic implications for the availability of mission critical data and/or access to confidential or sensitive information. Kat Traxler did a great job of articulating how Log4j can be used to exfiltrate credentials from EC2 instances – allowing attackers to impersonate the EC2 and use its permissions in the attacked environment. By proactively limiting the EC2 permitted actions to the bare business functions, the potential fallout from such a breach would be minimal.

Of course, proactively limiting permissions continuously and at scale is easier said than done. Ermetic fully supports this process by automatically generating right sized policies and allowing AWS environment administrators to apply them instead of over permissive ones.

Ermetic automatically generating a suggestion for a rightsized policy based on CloudTrail activity log

Anomalous Behavior Alerts

Analyzing CloudTrail logs allows Ermetic to detect malicious anomalous behavior in your environment.

In a recent webinar, we used the Pacu exploitation framework to demonstrate various elements of the cyber “Kill Chain” that an adversary would use to wage a campaign on a victim’s environment. During the session we showcased multiple behaviors that a threat actor would be inclined (if not compelled) to perform in the attacked environment, and how the accurate analysis of logs allows the person managing the breached environment to detect malicious activity and nip the campaign in the bud.

For example, an attacker could be inclined to perform privilege escalation on itself by attaching an over permissive policy on a role that it is allowed to assume. If this operation has not occurred before, Ermetic will flag it as anomalous and consequently immediately raise an alert.

Ermetic detecting anomalous behavior of an IAM Role escalating its privileges

Keeping 3rd Parties Under Control

Right-sizing policies and keeping tabs on identities is crucial. As described, doing so in the case of third party identities is ultra-crucial because, if these external entities are compromised, the repercussions can be far reaching – and beyond your immediate control.

Typically, permissions provided to third parties are as per their request. Unfortunately, administrators rarely question the vendor request and actual need. For example, as part of our work with many vendors, we received a CloudFormation template that provisioned a role with an inline policy that allowed the vendor to use “iam:AttachRolePolicy” on “*”. This specific permission may seem harmless – but it actually allows the subject to privilege escalate their permissions to admin. During our analysis of actual usage by the third party, we were astounded to discover that the permission was excessive and not necessary for the regular business function the third party served – so could have been removed.

Inline IAM policy generated for 3rd party including the iam:AttachRolePolicy action

Ermetic also helps you easily find long forgotten roles allowing third party access that haven’t been used for a significant period of time.

Ermetic's excessive permissions dashboard showing inactive IAM Roles allowing access to 3rd parties

Summary and a New Year’s Resolution

The Log4j issue was a serious, high-profile incident that required immediate attention by affected parties. We encourage you to use any tool at your disposal to detect and remediate exposure to the Log4j vulnerability by removing unnecessary use of the vulnerable components or upgrading to a safe version. However keep in mind that such actions are merely tactical. Let Log4j, which is by no means a one-off event, remind you of the bigger picture. As Haroon Meer ironically puts it:

Other severe and ubiquitous vulnerabilities are surely lurking in the shadows – and will surface one day. Let Log4j spur you to strategic action: take a close, deep look at your security initiatives and evaluate how – and, perhaps more importantly, how effectively – you are managing and controlling your AWS environment security, and take resolute steps to make it better.

The post Protect Your AWS Environment Beyond Patching Log4j appeared first on Ermetic.

*** This is a Security Bloggers Network syndicated blog from Ermetic authored by Lior Zatlavi. Read the original post at: https://ermetic.com/blog/aws/protect-your-aws-environment-beyond-patching-log4j/

Lior Zatlavi