A tiny, zero-dependency replacement for Python's zipfile.ZipFile for creating reproducible/deterministic ZIP archives.
A tiny, zero-dependency replacement for Python's zipfile.ZipFile library for creating reproducible/deterministic ZIP archives.
"Reproducible" or "deterministic" in this context means that the binary content of the ZIP archive is identical if you add files with identical binary content in the same order. It means you can reliably check equality of the contents of two ZIP archives by simply comparing checksums of the archive using a hash function like MD5 or SHA-256.
This Python package provides a ReproducibleZipFile class that works exactly like zipfile.ZipFile from the Python standard library, except that certain file metadata are set to fixed values. See "How does repro-zipfile work?" below for details.
You can also optionally install a command-line program, rpzip. See "rpzip command line program" below for more information.
repro-zipfile is available from PyPI. To install, run:
pip install repro-zipfile
It is also available from conda-forge. To install, run:
conda install repro-zipfile -c conda-forge
Simply import ReproducibleZipFile and use it in the same way you would use zipfile.ZipFile from the Python standard library.
from repro_zipfile import ReproducibleZipFile
with ReproducibleZipFile("archive.zip", "w") as zp:
# Use write to add a file to the archive
zp.write("examples/data.txt", arcname="data.txt")
# Or writestr to write data to the archive
zp.writestr("lore.txt", data="goodbye")
Note that files must be written to the archive in the same order to reproduce an identical archive. Be aware that functions that like os.listdir, os.glob, Path.iterdir, and Path.glob return files in a nondeterministic order—you should call sorted on their returned values first.
See examples/usage.py for an example script that you can run, and examples/demo_vs_zipfile.py for a demonstration in contrast with the standard library's zipfile module.
For more advanced usage, such as customizing the fixed metadata values, see the subsections under "How does repro-zipfile work?".
You can optionally install a lightweight command-line program, rpzip. This includes an additional dependency on the typer CLI framework. You can install it either directly or using the cli extra with repro-zipfile:
pip install rpzip
# or
pip install repro-zipfile[cli]
rpzip is designed to a partial drop-in replacement ubiquitous zip program. Use rpzip --help to see the documentation. Here are some usage examples:
# Archive a single file
rpzip archive.zip examples/data.txt
# Archive multiple files
rpzip archive.zip examples/data.txt README.md
# Archive multiple files with a shell glob
rpzip archive.zip examples/*.py
# Archive a directory recursively
rpzip -r archive.zip examples
In addition to the fixed file metadata done by repro-zipfile, rpzip will also always sort all paths being written.
ZIP archives are not normally reproducible even when containing files with identical content because of file metadata. In particular, the usual culprits are:
repro_zipfile.ReproducibleZipFile is a subclass of zipfile.ZipFile that overrides the write, writestr, and mkdir methods with versions that set the above metadata to fixed values. Note that repro-zipfile does not modify the original files—only the metadata written to the archive.
You can effectively reproduce what ReproducibleZipFile does with something like this:
from zipfile import ZipFile
with ZipFile("archive.zip", "w") as zp:
# Use write to add a file to the archive
zp.write("examples/data.txt", arcname="data.txt")
zinfo = zp.getinfo("data.txt")
zinfo.date_time = (1980, 1, 1, 0, 0, 0)
zinfo.external_attr = 0o644 << 16
# Or writestr to write data to the archive
zp.writestr("lore.txt", data="goodbye")
zinfo = zp.getinfo("lore.txt")
zinfo.date_time = (1980, 1, 1, 0, 0, 0)
zinfo.external_attr = 0o644 << 16
It's not hard to do, but we believe ReproducibleZipFile is sufficiently more convenient to justify a small package!
See the next two sections for more details about the replacement metadata values and how to customize them.
ZIP archives store the last-modified timestamps of files and directories. ReproducibleZipFile will set this to a fixed value. By default, the fixed value is 1980-01-01 00:00 UTC, which is the earliest timestamp that is supported by the ZIP format specifications.
You can customize this value with the SOURCE_DATE_EPOCH environment variable. If set, it will be used as the fixed value instead. This should be an integer corresponding to the Unix epoch time of the timestamp you want to set, e.g., 1704067230 for 2024-01-01 00:00:00 UTC. SOURCE_DATE_EPOCH is a standard created by the Reproducible Builds project for software distributions.
ZIP archives store the file-system permissions of files and directories. The default permissions set for new files or directories often can be different across different systems or users without any intentional choices being made. (These default permissions are controlled by something called umask.) ReproducibleZipFile will set these to fixed values. By default, the fixed values are 0o644 (rw-r--r--) for files and 0o755 (rwxr-xr-x) for directories, which matches the common default umask of 0o022 for root users on Unix systems. (The 0o prefix is how you can write an octal—i.e., base 8—integer literal in Python.)
You can customize these values using the environment variables REPRO_ZIPFILE_FILE_MODE and REPRO_ZIPFILE_DIR_MODE. They should be in three-digit octal Unix numeric notation, e.g., 644 for rw-r--r--.
ZIP archives are often useful when dealing with a set of multiple files, especially if the files are large and can be compressed. Creating reproducible ZIP archives is often useful for:
FAQs
A tiny, zero-dependency replacement for Python's zipfile.ZipFile for creating reproducible/deterministic ZIP archives.
We found that repro-zipfile demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.
Security News
Research
Socket researchers found a malicious Maven package impersonating the legitimate ‘XZ for Java’ library, introducing a backdoor for remote code execution.