Impact Analysis: CVE-2022-29218, Allows Unauthorized Takeover of New Gem Versions via Cache Poisoning

It’s been a bad month for RubyGems vulnerabilities. Critical CVE-2022-29176 was issued May 8, 2022, and another critical CVE-2022-29218 was discovered less than a week later, on May 11. This new vulnerability would allow for a takeover of new versions of some platform-specific gems under certain circumstances.

Nature of the vulnerability

In my opinion, this vulnerability is much more interesting than CVE-2022-29176. It could deliver malicious versions only to certain parts of the world while ensuring that companies analyzing open source software packages only pick up on versions that don’t contain malicious code.

But how is it even possible?

Let’s start by explaining how RubyGems work. Each time you download a new package version, it is fetched from S3 and then cached using the Fastly CDN platform. In order to make downloads fast and smooth, Fastly uses scattered points of presence to ship packages from the most optimal locations. This ensures that your bundle install process is as fast as possible.

Since versions are immutable, the cache TTL can be set to a really long time–or forever–and all should be good. After a cache miss during first fetch from RubyGems, Fastly will cache the RubyGems response and use it.

How can it be exploited?

The security researcher who reported the vulnerability elevated two issues in the gems serving process:

• Incomplete data structure validation within the metadata file that is being used to read gem data upon version release to trigger an error
• Ordering mistake in the code that accepts gem uploads that would store the package version despite the release process being halted with an exception

 Using the two issues together allows the attacker to save “not yet released” versions in the RubyGems S3 bucket. So far, so good. Since there is no release as an exception was raised, where’s the problem? Bundler will not see it and no one will be able to download it. Until…

Until the correct release happens. If you “force” RubyGems to store your package file, it will be overwritten by a proper release, but until then it can be fetched and cached by Fastly. This means that a smart actor can run a cache warmup. Once the cache warmup happens, the file will be shipped via Fastly without being aware that there is a new, “correct” version.

What makes it even worse is the fact that such a cache warmup could be done per point of presence, allowing the attacker to target companies based on their location while staying hidden from security companies that use different POPs to fetch packages. It also makes the security assessment from the end-user perspective much harder, as in theory each dev, staging, and production machine states should be checked independently.

Brilliant idea indeed!

In short, this is quite similar to creating and sending an invitation email prior to the user being saved into the database. It may turn out that there is an exception, but despite that the email has been sent and there is no way to roll it back.

What about package versions immutability?

RubyGems would issue a proper checksum for a proper and legit version and would be unaware of the fact Fastly was serving something else (chance cache poisoning).

What about Bundler checksum verification?

Not many know that Bundler does perform checksums verification, and it is turned on by default. Starting with Bundler version 1.1.14, it has access to checksums for every .gem file. Bundler actively validates those checksums against downloaded .gem files before installing them. Unfortunately there are still cases where Bundler fallbacks to API endpoints do not give checksum insights, effectively skipping this step. That is why, while it could stop the install process in some cases, we decided not to fully rely on this capability when running the assessment.

Wouldn’t having better checksum verification in Bundler help?

Assuming Bundler correlates with local checksums once served via the API and against the cached/downloaded data: yes. It should detect cases of this nature. On the other hand, if Bundler had relied on the local checksums, this checksum might have been incorrectly computed during the new packages install.

Impact assessment and incident analysis

Note: this research was performed by me, Andre Anko, David Radcliffe, and other RubyGems security team members.

In order for an attack like this to be successful, the main requirement is that the original gem platform has to end with a number. Since we’re yet again working with the assumption that malicious actors might already exploit this vulnerability, we’ve started with a lookup of how many platform-specific versions are in the registry.

It totaled 15,452 versions from 554 packages, targeting a total of 120 platforms. Luckily for us, this exploit required triggering an application error. All of the errors of that nature are stored in RubyGems bug tracker. Thanks to that, we were able to narrow the timeframe quite significantly.

Our 554 packages were further reduced down to five, out of which only one was of a non-research nature: sorbet-static. We did not find any evidence of it being poisoned, though we cannot eliminate this possibility fully at the moment.

Assuming it was compromised, the compromised version was available for only a couple hours, since all of the Fastly caches were completely purged right after the patch was released.

What about other potential malicious files targeting the “not yet released versions”?

Although a look at the error log tells us this is highly unlikely, we cannot eliminate the possibility that there were other files stored to S3 that could poison the cache for future releases. That is why we analyzed all of the RubyGems S3 data, which includes 1,233,237 files. This data was correlated with available releases of packages, pointing us towards 1,949 versions. After taking the platform constraint into the requirements, we ended up with 47 potentially affected versions. All of those files were at least five years old and corresponded to no-longer existing package versions.

How can I check I was not affected?

Alongside fixing this vulnerability in a few hours, releasing the patched version, and purging Fastly caches, due to the nature of this incident the team created a tool called bundler-checksums. You can use it to validate all the cached packages against their correct checksums served by the RubyGems versions API.

You can also use it to generate correct checksums for all of the dependencies of your application to validate production infrastructure without having to install this in production:

Summary

Our initial assumption was that the impact of this vulnerability was huge. Even after we were able to narrow it down to only a few gems, we did not stop investigating until we concluded that the RubyGems ecosystem was not affected.

While we cannot fully exclude the possibility that malicious/unwanted versions of the packages in question were served, given Bundler checksum verification and our investigation, we have concluded that the RubyGems ecosystem was not exploited beyond the work of this amazing security researcher.

Mend’s automated malware detection platform, Supply Chain Defender, checks to make sure you’re only using verified package sources and prevents you from importing any malicious package into your organization or personal machine. Mend Supply Chain Defender is free to use. Sign up here >>

Meet The Author

Maciej Mensfeld

Maciej Mensfeld is the creator of the Mend Supply Chain Defender platform and a Senior Product Manager. He writes about Supply Chain Security and Open Source Software in general.