PyPI Slashes Malware Response Time: 90% of Issues Resolved in Under 24 Hours

PyPI (Python Package Index) has significantly bolstered its security measures over the past year, with a big jump in 2FA adoption, according to a new report from the Python Software Foundation’s Safety & Security Engineer Mike Fiedler.

Since the very first version of PyPI was opened for public use in 2002, the registry has grown to include over 850,000 accounts in the system. Through a phased process, PyPI now has more than 140,000 users who have enabled one or more forms of 2FA. Now that 2FA is mandatory for publishing/performing any actions on PyPI, Fiedler reports that ~80% of all users that have logged in since January 2024 have 2FA enabled.

PyPI's Improved Malware Response: From Months to Minutes#

Prior to Fiedler’s 2023 initiatives to address malware, which he said “is a constant threat to the Python ecosystem,” some of these packages were live on PyPI for months. The only way to contact admins was via email, but this protocol has now been updated to use a shared inbox system, where the team has the ability to automate responses with workflows, tag with metadata, and collect reports.

Fiedler also extended the PyPI admin UI to include strategic views and quick response buttons that help the team make faster evaluations and take action. This has led to astonishingly fast response times on malware:

• Since August 2023, over 90% of issues were resolved in less than 24 hours.
• Over 95% of those issues had a handle time of under one minute.
• Approximately 900 projects were removed since mid-March 2024.
• Around 1,300 distinct reports were generated based on qualified malware notifications.
• These reports are currently available as a private beta for interested parties.

Fiedler reported a markedly decreased time of malware being live on PyPI. Malware that used to sit there for months is now removed in just hours and oftentimes minutes from publication.

PyPI Introduces New “Quarantine” Status for Suspicious Packages#

During the time when suspicious packages are being investigated, PyPI admins now have the option to put them in “quarantine,” a new status for projects that is aimed at protecting users.

“This state allows PyPI Admins to place a project in a non-destructive quarantine for further analysis, removing the ability for end-users to easily install the project until it's been further investigated,” Fiedler said.

This new status option is crucial as client installation tools access the PyPI service to obtain packages at a rate of ~35,000 times per second, according to the PSF’s official comments on CISA’s request for information (RFI) on open source software security. The new “quarantine” status has the potential to mitigate the damaging effects of a supply chain attack. On the flip side, it also protects legitimate projects from premature or reactionary actions in the case of false positives.

“Since project removal from PyPI is a destructive action, creating a quarantine state allows for restoring a project if deemed a false positive report without destroying any of the project's history or metadata,” Fiedler said.

PSF’s Critical Role in Protecting the Ecosystem from Supply Chain Threats#

It’s impossible to overstate the importance of PyPI to the open source ecosystem. In the PSF’s comments on CISA’s RFI, the foundation noted that PyPI has grown to host 490,000 projects with more than 5 million releases that are downloaded over a billion times per day. PyPI manages a constant flow of software updates and serves a staggering number of requests per day:

• 12,000-15,000 new releases per day uploaded by maintainers.
• 3.7 to 4 million unique visitors to the PyPI website per month.
• 3 billion web requests per day for projects hosted on PyPI, totaling 90 billion per month.

The PSF made specific recommendations to CISA based on the unsurprising fact that the vast majority of Python projects are maintained by a single individual:

Out of 490,000 projects on PyPI, 91% of projects have a single account with the maintainer role. For example, maintainers using a username/password for publishing packages may receive an email directing them to adopt API Tokens, but still places the burden of adoption on the maintainer. Providing maintainers with tools to easily adopt the newer practice could accelerate better security practices, reducing risk exposure across the ecosystem.

In addition to the ongoing security projects that Fiedler has been leading, the PSF has received funding from Alpha-Omega to extend Seth Larson’s work in the Security Developer-in-Residence role through the end of 2024. While he has initially been focused on working with the Python core developer team, Larson outlined a few new areas of focus on his blog that will extend to the wider community of projects using Python:

• Enabling Build Provenance and Software Bill-of-Materials for Python packages.
• Adoption of security features, hardening, and best practices for Python packages.
• Special focus on Python packaging tools and workflows.

“We believe it is critical to prioritize strengthening the software supply chain,” the PSF commented on CISA's RFI. “An enormous and diverse set of users—from hobbyists to scientists to multinational enterprises to U.S. government agencies—rely on PyPI and the packages it hosts. Currently, there is no standard way for those users to verify the source and provenance of these packages. This means PyPI users don’t have a guarantee that the software they download is what they are expecting, creating a vulnerability that can be exploited by malicious actors.

“Supply chain attacks targeting open source projects remain a constant and costly threat. Being able to verifiably link a package back to its source code and its build process protects against some of the most potent of these attacks.”

PyPI's recent security enhancements are a major step forward towards reducing malware risks and vulnerabilities. The rapid response times and new quarantine status will ensure a more proactive defense against threats. These advancements are critical for keeping pace with the growing volume of packages and users. The PSF continues to advocate for federal funding to provide better tools and support for maintainers, especially those working solo who sustain the bulwark of the Python ecosystem’s infrastructure.