.. readme-intro
.. |repo-status| image:: https://www.repostatus.org/badges/latest/active.svg :target: https://www.repostatus.org/#active :alt: Project Status: Active .. |pytest| image:: https://github.com/catalyst-cooperative/pudl/workflows/pytest/badge.svg :target: https://github.com/catalyst-cooperative/pudl/actions?query=workflow%3Apytest :alt: PyTest Status .. |codecov| image:: https://img.shields.io/codecov/c/github/catalyst-cooperative/pudl?style=flat&logo=codecov :target: https://codecov.io/gh/catalyst-cooperative/pudl :alt: Codecov Test Coverage .. |rtd| image:: https://img.shields.io/readthedocs/catalystcoop-pudl?style=flat&logo=readthedocs :target: https://catalystcoop-pudl.readthedocs.io/en/nightly/ :alt: Read the Docs Build Status .. |ruff| image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json :target: https://github.com/astral-sh/ruff .. |pre-commit-ci| image:: https://results.pre-commit.ci/badge/github/catalyst-cooperative/pudl/main.svg :target: https://results.pre-commit.ci/latest/github/catalyst-cooperative/pudl/main :alt: pre-commit CI .. |zenodo-doi| image:: https://zenodo.org/badge/80646423.svg :target: https://zenodo.org/badge/latestdoi/80646423 :alt: Zenodo DOI .. |office-hours| image:: https://img.shields.io/badge/calend.ly-officehours-darkgreen :target: https://calend.ly/catalyst-cooperative/pudl-office-hours :alt: Schedule a 1-on-1 chat with us about PUDL. .. |mastodon| image:: https://img.shields.io/mastodon/follow/110855618428885893?domain=https%3A%2F%2Fmastodon.energy&style=social&color=%23000000&link=https%3A%2F%2Fmastodon.energy%2F%40catalystcoop :target: https://mastodon.energy/@catalystcoop :alt: Follow Catalyst Cooperative on Mastodon .. |linkedin| image:: https://img.shields.io/badge/LinkedIn-0077B5?style=flat&logo=linkedin&logoColor=white :target: https://linkedin.com/company/catalyst-cooperative/ :alt: Follow Catalyst Cooperative on LinkedIn .. |bluesky| image:: https://img.shields.io/badge/Bluesky-0285FF?logo=bluesky&logoColor=fff&style=flat :target: https://bsky.app/profile/catalyst.coop :alt: Follow @catalyst.coop on BlueSky .. |kaggle| image:: https://img.shields.io/badge/Kaggle-20BEFF?style=flat&logo=Kaggle&logoColor=white :target: https://www.kaggle.com/datasets/catalystcooperative/pudl-project :alt: The PUDL Dataset on Kaggle .. |aws| image:: https://img.shields.io/badge/Amazon_AWS-FF9900?style=flat&logo=amazonaws&logoColor=white :target: https://registry.opendata.aws/catalyst-cooperative-pudl/ :alt: PUDL in the AWS Open Data Registry
|repo-status| |pytest| |codecov| |rtd| |ruff| |pre-commit-ci| |zenodo-doi| |office-hours| |mastodon| |linkedin| |bluesky| |kaggle| |aws|
The PUDL <https://catalyst.coop/pudl/>__ Project (pronounced puddle) is an open source
data processing pipeline that makes US energy data easier to access and use
programmatically.
Hundreds of gigabytes of valuable data are published by US government agencies, but it's often difficult to work with. PUDL takes the original spreadsheets, CSV files, and databases and turns them into a unified resource. This allows users to spend more time on novel analysis and less time on data preparation.
The project is focused on serving researchers, activists, journalists, policy makers, and small businesses that might not otherwise be able to afford access to this data from commercial sources and who may not have the time or expertise to do all the data processing themselves from scratch.
We want to make this data accessible and easy to work with for as wide an audience as possible: anyone from a grassroots youth climate organizers working with Google sheets to university researchers with access to scalable cloud computing resources and everyone in between!
PUDL is comprised of three core components:
Raw Data Archives
^^^^^^^^^^^^^^^^^
PUDL archives <https://github.com/catalyst-cooperative/pudl-archiver>__ all our raw
inputs on Zenodo <https://zenodo.org/communities/catalyst-cooperative/?page=1&size=20>__ to ensure
permanent, versioned access to the data. In the event that an agency changes how it
publishes data or deletes old files, the data processing pipeline will still have access
to the original inputs. Each of the data inputs may have several different versions
archived, and all are assigned a unique DOI (digital object identifier) and made
available through Zenodo's REST API. You can read more about the Raw Data Archives in
the docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/#raw-data-archives>__.
Data Pipeline
^^^^^^^^^^^^^
The data pipeline (this repo) ingests raw data from the archives, cleans and integrates
it, and writes the resulting tables to SQLite <https://sqlite.org>__ and Apache Parquet <https://parquet.apache.org/>__ files, with some accompanying metadata stored as
JSON. Each release of the PUDL software contains a set of DOIs indicating which
versions of the raw inputs it processes. This helps ensure that the outputs are
replicable. You can read more about our ETL (extract, transform, load) process in the
PUDL documentation <https://catalystcoop-pudl.readthedocs.io/en/nightly/#the-etl-process>__.
Data Warehouse
^^^^^^^^^^^^^^
The SQLite, Parquet, and JSON outputs from the data pipeline, sometimes called "PUDL
outputs", are updated each night by an automated build process, and periodically
archived so that users can access the data without having to install and run our data
processing system. These outputs contain hundreds of tables and comprise a small
file-based data warehouse that can be used for a variety of energy system analyses.
Learn more about how to access the PUDL data <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html>__.
PUDL currently integrates data from:
Source Docs <https://www.eia.gov/electricity/data/eia860/>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/eia860.html>__Source Docs <https://www.eia.gov/electricity/data/eia860m/>__Source Docs <https://www.eia.gov/electricity/data/eia861/>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/eia861.html>__Source Docs <https://www.eia.gov/electricity/data/eia923/>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/eia923.html>__Source Docs <https://www.eia.gov/electricity/gridmonitor/>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/eia930.html>__Source Docs <https://www.eia.gov/outlooks/aeo/>__Source Docs <https://campd.epa.gov/>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/epacems.html>__Source Docs <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-1-electric-utility-annual>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/ferc1.html>__Source Docs <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electric/data>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/ferc714.html>__Source Docs <https://www.ferc.gov/industries-data/natural-gas/industry-forms/form-2-2a-3-q-gas-historical-vfp-data>__Source Docs <https://www.ferc.gov/general-information-1/oil-industry-forms/form-6-6q-historical-vfp-data>__Source Docs <https://www.ferc.gov/form-60-annual-report-centralized-service-companies>__Source Docs <https://atb.nrel.gov/electricity/2024/data>__Source Docs <https://gridlab.org/gridpathratoolkit/>__PUDL Docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_sources/gridpathratoolkit.html>__Source Docs <https://www.census.gov/geographies/mapping-files/2010/geo/tiger-data.html>__If you're interested in any of these datasets, we'd love to integrate them into PUDL. Get in touch!
PHMSA Natural Gas Annual Report <https://www.phmsa.dot.gov/data-and-statistics/pipeline/gas-distribution-gas-gathering-gas-transmission-hazardous-liquids>__EIA Form 176 <https://www.eia.gov/dnav/ng/TblDefs/NG_DataSources.html#s176>__
(The Annual Report of Natural Gas Supply and Disposition)FERC Electric Quarterly Reports (EQR) <https://www.ferc.gov/industries-data/electric/power-sales-and-markets/electric-quarterly-reports-eqr>__EIA Thermoelectric Water Usage <https://www.eia.gov/electricity/data/water/>__FERC Form 2 <https://www.ferc.gov/industries-data/natural-gas/overview/general-information/natural-gas-industry-forms/form-22a-data>__
(Annual Report of Major Natural Gas Companies)MHSA Mines <https://arlweb.msha.gov/OpenGovernmentData/OGIMSHA.asp>__For details on how to access PUDL data, see the data access documentation <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html>__. A quick
summary:
Datasette <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#-access-datasette>__
provides browsable and queryable data from our nightly builds on the web:
https://data.catalyst.coopKaggle <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-kaggle>__
provides easy Jupyter notebook access to the PUDL data, updated weekly:
https://www.kaggle.com/datasets/catalystcooperative/pudl-projectZenodo <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-zenodo>__
provides stable long-term access to our versioned data releases with a citeable DOI:
https://doi.org/10.5281/zenodo.3653158Nightly Data Builds <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-nightly-builds>__
push their outputs to the AWS Open Data Registry:
https://registry.opendata.aws/catalyst-cooperative-pudl/
See the nightly build docs <https://catalystcoop-pudl.readthedocs.io/en/nightly/data_access.html#access-nightly-builds>__
for direct download links.The PUDL Development Environment <https://catalystcoop-pudl.readthedocs.io/en/nightly/dev/dev_setup.html>__
lets you run the PUDL data processing pipeline locally... _pudl_orgs_start:
This is a partial list of organizations that have used PUDL in their work.
If your organization uses PUDL we'd love to list you here! Please open a
pull request or email us at hello@catalyst.coop <mailto:hello@catalyst.coop>__!
RMI <https://rmi.org>__ via both their Utility Transition Hub <https://utilitytransitionhub.rmi.org/>__ and Optimus financial modeling tool <https://utilitytransitionhub.rmi.org/optimus/>__PowerGenome Project <https://github.com/PowerGenome/PowerGenome>__ out of Jesse Jenkins' <https://mae.princeton.edu/people/faculty/jenkins>__ ZERO Lab at Princeton UniversityEnergy Innovation <https://energyinnovation.org>__Singularity Energy <https://singularity.energy>__ via the Open Grid Emissions Initiative <https://singularity.energy/open-grid-emissions>__PyPSA-USA <https://github.com/PyPSA/pypsa-usa>__ capacity expansion model.Win Climate <https://climate.win>__The Deployment Gap Model Education Fund <https://www.deploymentgap.fund/>__.. _pudl_orgs_end:
Find PUDL useful? Want to help make it better? There are lots of ways to help!
contribution guide <https://catalystcoop-pudl.readthedocs.io/en/nightly/CONTRIBUTING.html>__
including our Code of Conduct <https://catalystcoop-pudl.readthedocs.io/en/nightly/code_of_conduct.html>__.Github issue tracker <https://github.com/catalyst-cooperative/pudl/issues>__.Make a recurring financial contribution <https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=PZBZDFNKBJW5E&source=url>__
to support our work liberating public energy data.Hire us to do some custom analysis <https://catalyst.coop/hire-catalyst/>__ and
allow us to integrate the resulting code into PUDL.In general, our code, data, and other work are permissively licensed for use by anybody, for any purpose, so long as you give us credit for the work we've done.
the MIT License <https://opensource.org/licenses/MIT>__.Creative Commons Attribution License v4.0 <https://creativecommons.org/licenses/by/4.0/>__
(CC-BY-4.0).GitHub Issue <https://github.com/catalyst-cooperative/pudl/issues>__.GitHub Discussions <https://github.com/catalyst-cooperative/pudl/discussions>__sign up for our email list <https://catalyst.coop/updates/>__.Office Hours <https://calend.ly/catalyst-cooperative/pudl-office-hours>__Follow us here on GitHub <https://github.com/catalyst-cooperative/>__@CatalystCoop@mastodon.energy <https://mastodon.energy/@CatalystCoop>__@catalyst.coop <https://bsky.app/profile/catalyst.coop>__Follow us on LinkedIn <https://www.linkedin.com/company/catalyst-cooperative/>__Follow us on HuggingFace <https://huggingface.co/catalystcooperative>__@CatalystCoop <https://twitter.com/CatalystCoop>__Follow us on Kaggle <https://www.kaggle.com/catalystcooperative/>__hello@catalyst.coop <mailto:hello@catalyst.coop>__Catalyst Cooperative <https://catalyst.coop>__ is a small group of data wranglers
and policy wonks organized as a worker-owned cooperative consultancy. Our goal is a
more just, livable, and sustainable world. We integrate public data and perform
custom analyses to inform public policy
(Hire us! <https://catalyst.coop/hire-catalyst>__). Our focus is primarily on
mitigating climate change and improving electric utility regulation in the United
States.
FAQs
An open data processing pipeline for US energy data
We found that catalystcoop.pudl demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Ransomware negotiators share how modern cybercriminals operate like corporations, using specialized teams, negotiation tactics, and reputation management.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.