PyPI on Ultralytics Supply Chain Attack: Poor CI/CD Practices to Blame, No Security Flaws in PyPI Exploited

The Python Package Index (PyPI) has officially responded to the Ultralytics supply chain attack that happened last weekend, when the widely used package had four versions compromised. Given the days of intense media scrutiny and the significant impact of this incident, it was important for PyPI to address Ultralytics’ unfounded claims that this supply chain attack bypassed PyPI provenance signing.

To summarize the attack, four versions of the Ultralytics package—v8.3.41, v8.3.42, v8.3.45, and v8.3.46—were compromised with malicious code for cryptocurrency mining. The first two compromises exploited GitHub Actions via cache poisoning, which enabled the threat actor to tamper with the build artifacts while still producing signed provenance. The second set of malicious packages were published using a previously compromised API token.

In a security advisory on the incident Ultralytics CEO Glenn Jocher claimed that it “appeared to be a sophisticated supply chain attack that bypassed PyPI provenance signing.” At the time, William Woodruff, engineering director at Trail of Bits, the company that helped build Trusted Publishing + attestations for PyPI, addressed this claim, citing the valid attestations the compromised packages produced. He attributed the incident to poor security practices within this repository's CI/CD, not PyPI’s infrastructure, as the root cause for the supply chain attack.

PyPI: “No security flaw in PyPI was used to execute this attack.”#

PyPI is officially refuting the claims this week in a post on the registry’s blog, confirming to the Python community that no bypass of PyPI occurred in this incident:

Last week, the Python project “ultralytics” suffered a supply-chain attack through a compromise of the projects’ GitHub Actions workflows and subsequently its PyPI API token. No security flaw in PyPI was used to execute this attack.

PSF Security Developer-in-Residence Seth Larson highlighted how attestations and Trusted Publishers provided visibility in this incident, giving the community a better understanding of the attack and its timeline.

“Despite the success of the attack, many things went right from PyPI’s perspective, especially the ability to audit the attack while it was occurring and after the fact,” Larson said.

Because the second set of malicious packages were published using an unrevoked PyPI API token, which staff presume was available prior to Ultralytics adopting Trusted Publishers, there were no publish attestations for those two releases.

“Once tools begin utilizing these publish attestations to record the ‘expected’ provenance of software, this type of attack will be less effective as the lack of provenance information will be more apparent and verifiable at install time,” Larson said.

He also urged publishers to audit their existing workflows for insecure configurations and lock or pin dependencies to explicit versions with checksums wherever possible, along with a list of other security recommendations.

PyPI to Implement Security Improvements for Publishing#

As a result of this incident, PyPI has initiated two new discussions in its Warehouse repository, where contributors are exploring ways to better protect the publishing process from hijacking:

• Trusted Publishing: revoke manual API tokens once a publisher is used #17260
• Warning or error for Trusted Publisher users when GitHub Environment claim is included but not checked #17241

The consideration of revoking manual API tokens for those using Trusted Publishers is tricky, as the method isn’t supported by all platforms. Contributors are discussing adding in some extra conditions before revoking, including the project having used Trusted Publishers recently and a configured API token not having been used in a set amount of time. The specifics there are still under discussion.

The second improvement they are considering is warning users when they are not using the recommended check for a specified GitHub Environment, an optional feature of the Trusted Publisher process:

Creating a Trusted Publisher for GitHub optionally allows adding a "GitHub Environment" to be checked during the OIDC->API token exchange. In the documentation it's recommended to use a GitHub Environment, but users may not know they need to update the Trusted Publisher in addition to the workflow definition to get the benefits of a GitHub Environment requirement.

Contributors are exploring adding either an error or an email warning that indicates a GitHub Environment claim is included but not checked, and perhaps offer a magic link that helps the them resolve it.

These efforts are aimed at keeping publishers safer from opportunistic threats and nudging them towards better security practices for distributing packages on PyPI. The platform is also working on reducing the time that malware is available to be installed. Any projects publishing packages to PyPI should review the critical lessons from the Ultralytics supply chain attack and follow PyPI’s security recommendations to prevent and mitigate future attacks.