Ultralytics PyPI Package Compromised Through GitHub Actions Cache Poisoning

Over the weekend, the popular Ultralytics PyPI package was compromised in a supply chain attack that was detected following reports of a discrepancy between the library’s code on GitHub and the code that was published to PyPI for v8.3.41. Another user logged a GitHub issue for 100% CPU spikes after updating to 8.3.41. Later in the weekend, two more versions delivered malware via previously compromised API tokens.

The first compromised versions, v8.3.41 and v8.3.42 injected malicious code for cryptocurrency mining and both were removed from PyPI while the Ultralytics team conducted an investigation. They immediately stopped automatic deployments and attributed the attack to a compromise in the build environment for the package.

PyPI administrator Ee Durbin published a timeline from the registry’s internal audit logs, showing the events relevant to the first two malicious releases (times in UTC):

• 2024-12-04 20:51:12.278525 ultralytics 8.3.41 release created
• 2024-12-04 20:51:15.187966 ultralytics-8.3.41.tar.gz uploaded
• 2024-12-04 20:51:12.278525 ultralytics-8.3.41-py3-none-any.whl uploaded
• 2024-12-05 09:15:06.355361 ultralytics 8.3.41 release removed
• 2024-12-05 12:47:29.385127 ultralytics 8.3.42 release created
• 2024-12-05 12:47:29.385127 ultralytics-8.3.42-py3-none-any.whl uploaded
• 2024-12-05 12:47:32.363731 ultralytics-8.3.42.tar.gz uploaded
• 2024-12-05 13:47:30.142165 ultralytics 8.3.42 release removed

The first malicious release was live for approximately 12 hours before being removed, and the second release was removed an hour after being uploaded. Ultralytics’ logs tracked the unauthorized access to a user in Hong Kong, @openimbot, who seems to be active in other open source projects.

Ultralytics’ GitHub Actions Exploited via Cache Poisoning#

Security researcher Adnan Khan was able to determine that the payload for the first two compromised versions was delivered through cache poisoning that exploited the GitHub Action workflow used to publish to PyPI.

This is a technique that Khan had tested and written about extensively in a post titled, “The Monsters in Your Build Cache – GitHub Actions Cache Poisoning.” He demonstrated how effective attackers can be using cache poisoning as part of a supply chain attack, tampering with build artifacts that will still produce signed provenance without leaving a single trace.

Khan tested the security of various repositories and projects belonging to major tech companies, and was able to compromise the production deployment secrets for many of them using cache poisoning, including the following:

angular/dev

angular.io

angular/components

mdn/content

hyperledger/besu

google/temporian

chainguard-dev/terraform-provider-cosign

The key takeaway from his research, which has now been demonstrated in the wild with the latest Ultralytics supply chain attack:

Never run untrusted code within the context of the main branch if any other workflows use GitHub Actions caching.

This also applies to dependencies. If an attacker compromises a package that is used in a locked down workflow that runs in a repository’s main branch, then that poisoned dependency can steal the GitHub cache credentials and use it to move laterally to other workflows within victim repositories that use caching.

This is a major blindspot for organizations that rely on GitHub Actions without fully understanding the risks associated with caching and dependency management. Attackers can exploit this oversight to escalate privileges, compromise sensitive credentials, and tamper with critical workflows, leading to potentially catastrophic supply chain attacks.

More Malicious Versions Released via Compromised API Tokens#

A mere 36 hours later, the Ultralytics PyPI package was compromised again, with subsequent malicious releases v8.3.45 and v8.3.46. The attacker delivered more unauthorized code to new releases on PyPI that downloaded and executed cryptocurrency mining software, but this time they didn’t use the CI in the Ultralytics repo.

William Woodruff, engineering director at Trail of Bits, the company that helped build Trusted Publishing + attestations for PyPI, commented that the subsequent releases were made using an API token from the maintainer’s account. Ultralytics failed to delete the compromised tokens when registering a Trusted Publisher.

“Ultralytics should assume that every secret they have is compromised and rotate all of them,” Khan commented in the GitHub thread for the incident. “I’m shocked that wasn't the immediate reaction.

“The attacker must have had access to a credential belonging to the UltralyticsAssistant user since they made the workflow changes using that account. That means they had the ability to modify workflows and therefore steal secrets. Even from other repos if the credential was a classic access token.”

Sviatoslav Sydorenko, Ansible engineer and author of the pypi-publish action, also made several recommendations for publishing packages more securely, including moving it to a dedicated workflow with scoped permissions.

Ultralytics Publishes Security Advisory#

Ultralytics CEO Glenn Jocher published a security advisory following the removal of the four compromised releases from PyPI, disclosing that the crypto mining software was injected into the PyPI release artifacts and was not present in the public GitHub repository.

“This incident appears to be a sophisticated supply chain attack that bypassed PyPI provenance signing,” Jocher said. “We are conducting a thorough investigation and implementing enhanced security measures to prevent similar incidents.”

Since the comment was locked, Woodruff addressed the wording of the advisory in the original GitHub issue where the supply chain attack was being investigated. He called out the advisory as inaccurately characterizing the incident as an attempt that was sophisticated enough to bypass PyPI’s digital attestations:

1. The provenance step was not bypassed: your publishing workflow produced valid attestations, which PyPI successfully verified. This ended up being useful, since it gave us a signed timestamp and exact source reference to corroborate the cache poisoning theory. You can find these signed timestamps and their linked attestations above, in this thread, as well as in the write up I've linked above.

2. This attack was not sophisticated in most senses: the attacker used a pre-weaponized script for a publicly known weakness, and their entrance point was an insecure GitHub workflow target + template injection vulnerability of the sort that's been publicly documented since at least 2021. To add to this, the template injection vulnerability was in a custom action that had a previous injection vulnerability earlier in the year of the exact same type, suggesting that no checks were added after the previous vulnerability.

Woodruff said Ultralytics was “very lucky” that the attacker only injected a crypto miner both times, especially in the second attack where they leveraged an API token that wasn’t rotated following a previous attack.

"This could have been a lot worse (and could still be, since it's been less than a day since the last indicator of compromise), largely because of poor security practices within this repository's CI/CD, not PyPI or any other component of your supply chain,” Woodruff said.

The Limitations of Provenance#

One person commented, “BTW this is why we need PEP740,” referencing Python’s standard for defining cryptographically verifiable attestations hosted by indices like PyPI. When PyPI introduced digital attestations in November, roughly 5% of the 360 most-downloaded packages had attestations uploaded. That figure is now at 9% of the top packages offering attestations for their latest release.

There are several significant lessons to learn from this particular supply chain attack, not the least of which is the limitation of attestation in scenarios where build artifacts can be tampered with through cache poisoning.

“Attestation only creates a tamper-proof guarantee between an artifact and a build job + commit,” Khan said in his research on GitHub Actions cache poisoning.“Common build pitfalls like referencing un-trusted third party reusable actions can lead to build integrity compromise.

“Cache poisoning is unique in that it is very hard to detect for someone consuming an artifact. Provenance also links the artifact to the workflow file definition. A consumer can see that the workflow that built an artifact uses insecure actions and choose not to trust the binary. With cache poisoning this is harder if not impossible, because there is no way to inspect the contents of build caches as a consumer of a software package.”

Nevertheless, the valid attestations proved useful in this particular case as researchers needed the timestamps to confidently identify cache poisoning as the vector here. As Woodruff noted in his detailed timeline of the events, these releases were all uploaded to PyPI with a Trusted Publisher and valid attestations for each distribution, matching the ultralytics/ultralytics Trusted Publisher identity.

“Don’t get me wrong, GitHub Actions build attestation and provenance in general is a big step forward for supply chain security, but it is important not to fall into the pattern of trust simply because something checks a box,” Khan said.

While attestation may improve security, it cannot force maintainers to secure their workflows or rotate compromised credentials. Using open source software still comes with a healthy amount of risk where even the largest software organizations can have blindspots that translate into catastrophic supply chain failures.

After this high-profile event, it will be trivial for attackers to replicate these cache poisoning techniques against other vulnerable GitHub Actions workflows. This supply chain attack is a stark reminder for organizations to secure their CI/CD pipelines, as threat actors become more aggressive about exploiting GitHub Actions.