pypi-simple - npm Package Compare versions

Comparing version 1.2.0 to 1.3.0

docs/examples.rst

+Examples
+========
+
+Getting a Project's Dependencies
+--------------------------------
+
+`pypi_simple` can be used to fetch a project's dependencies (specifically, the
+dependencies for each of the project's distribution packages) as follows.
+
+Note that Warehouse only began storing the contents of package :file:`METADATA`
+files in May 2023.  Packages uploaded prior to that point are gradually having
+their metadata "backfilled" in; see
+<https://github.com/pypi/warehouse/issues/8254> for updates.
+
+.. code:: python
+
+    # Requirements:
+    #     Python 3.8+
+    #     packaging 23.1+
+    #     pypi_simple 1.3+
+
+    from packaging.metadata import parse_email
+    from pypi_simple import PyPISimple
+
+    with PyPISimple() as client:
+        page = client.get_project_page("pypi-simple")
+        for pkg in page.packages:
+            if pkg.has_metadata:
+                src = client.get_package_metadata(pkg)
+                md, _ = parse_email(src)
+                if deps := md.get("requires_dist"):
+                    print(f"Dependencies for {pkg.filename}:")
+                    for d in deps:
+                        print(f"    {d}")
+                else:
+                    print(f"Dependencies for {pkg.filename}: NONE")
+            else:
+                print(f"{pkg.filename}: No metadata available")
+            print()
+
+
+Downloading With a Rich Progress Bar
+------------------------------------
+
+The `PyPISimple.download_package()` method can be passed a callable for
+constructing a progress bar to use when downloading.  `pypi_simple` has
+built-in support for using a tqdm_ progress bar, but any progress bar can be
+used if you provide the right structure.
+
+Here is an example of using a progress bar from rich_.  The progress bar uses
+the default settings; adding customization is left as an exercise to the
+reader.
+
+.. _tqdm: https://tqdm.github.io
+.. _rich: https://github.com/Textualize/rich
+
+.. code:: python
+
+    from __future__ import annotations
+    from dataclasses import InitVar, dataclass, field
+    from types import TracebackType
+    from pypi_simple import PyPISimple
+    from rich.progress import Progress, TaskID
+
+    @dataclass
+    class RichProgress:
+        bar: Progress = field(init=False, default_factory=Progress)
+        task_id: TaskID = field(init=False)
+        size: InitVar[int | None]
+
+        def __post_init__(self, size: int | None) -> None:
+            self.task_id = self.bar.add_task("Downloading...", total=size)
+
+        def __enter__(self) -> RichProgress:
+            self.bar.start()
+            return self
+
+        def __exit__(
+            self,
+            _exc_type: type[BaseException] | None,
+            _exc_val: BaseException | None,
+            _exc_tb: TracebackType | None,
+        ) -> None:
+            self.bar.stop()
+
+        def update(self, increment: int) -> None:
+            self.bar.update(self.task_id, advance=increment)
+
+
+    with PyPISimple() as client:
+        page = client.get_project_page("numpy")
+        pkg = page.packages[-1]
+        client.download_package(pkg, path=pkg.filename, progress=RichProgress)

CHANGELOG.md

@@ -0,1 +1,8 @@
+v1.3.0 (2023-11-01)
+-------------------
+- Support Python 3.12
+- Update for PEP 714
+- Gave `PyPISimple` a `get_package_metadata()` method
+- Added an examples page to the documentation
+
 v1.2.0 (2023-09-23)
@@ -2,0 +9,0 @@ -------------------

docs/api.rst

@@ -60,2 +60,3 @@ .. currentmodule:: pypi_simple
     :show-inheritance:
+.. autoexception:: NoMetadataError()
 .. autoexception:: NoSuchProjectError()
@@ -74,4 +75,2 @@ .. autoexception:: UnsupportedContentTypeError()
 .. [#pep700] The ``versions``, ``size``, and ``upload_time`` fields are only
-   populated if the response was JSON from a server supporting :pep:`700`.  At
-   time of writing, Warehouse does not yet provide these fields; see
-   <https://github.com/pypi/warehouse/pull/12727> for details.
+   populated if the response was JSON from a server supporting :pep:`700`.

docs/changelog.rst

@@ -6,2 +6,10 @@ .. currentmodule:: pypi_simple
 
+v1.3.0 (2023-11-01)
+-------------------
+- Support Python 3.12
+- Update for PEP 714
+- Gave `PyPISimple` a `~PyPISimple.get_package_metadata()` method
+- Added an examples page to the documentation
+
+
 v1.2.0 (2023-09-23)
@@ -8,0 +16,0 @@ -------------------

docs/index.rst

@@ -17,2 +17,3 @@ .. module:: pypi_simple
     api
+    examples
     changelog
@@ -22,8 +23,9 @@
 specified in :pep:`503` and updated by :pep:`592`, :pep:`629`, :pep:`658`,
-:pep:`691`, and :pep:`700`.  With it, you can query `the Python Package Index
-(PyPI) <https://pypi.org>`_ and other `pip <https://pip.pypa.io>`_-compatible
-repositories for a list of their available projects and lists of each project's
-available package files.  The library also allows you to download package files
-and query them for their project version, package type, file digests,
-``requires_python`` string, PGP signature URL, and metadata URL.
+:pep:`691`, :pep:`700`, and :pep:`714`.  With it, you can query `the Python
+Package Index (PyPI) <https://pypi.org>`_ and other `pip
+<https://pip.pypa.io>`_-compatible repositories for a list of their available
+projects and lists of each project's available package files.  The library also
+allows you to download package files and query them for their project version,
+package type, file digests, ``requires_python`` string, PGP signature URL, and
+metadata URL.
 
@@ -33,4 +35,3 @@ Installation
 ``pypi-simple`` requires Python 3.7 or higher.  Just use `pip
-<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install
-``pypi-simple`` and its dependencies::
+<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install it::
 
@@ -37,0 +38,0 @@ python3 -m pip install pypi-simple

docs/requirements.txt

@@ -1,3 +0,3 @@
-Sphinx~=5.0
+Sphinx~=7.0
 sphinx-copybutton~=0.5.0
 sphinx_rtd_theme~=1.0

PKG-INFO

 Metadata-Version: 2.1
 Name: pypi-simple
-Version: 1.2.0
+Version: 1.3.0
 Summary: PyPI Simple Repository API client library
@@ -20,2 +20,3 @@ Home-page: https://github.com/jwodder/pypi-simple
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: Implementation :: CPython
@@ -67,8 +68,9 @@ Classifier: Programming Language :: Python :: Implementation :: PyPy
 specified in :pep:`503` and updated by :pep:`592`, :pep:`629`, :pep:`658`,
-:pep:`691`, and :pep:`700`.  With it, you can query `the Python Package Index
-(PyPI) <https://pypi.org>`_ and other `pip <https://pip.pypa.io>`_-compatible
-repositories for a list of their available projects and lists of each project's
-available package files.  The library also allows you to download package files
-and query them for their project version, package type, file digests,
-``requires_python`` string, PGP signature URL, and metadata URL.
+:pep:`691`, :pep:`700`, and :pep:`714`.  With it, you can query `the Python
+Package Index (PyPI) <https://pypi.org>`_ and other `pip
+<https://pip.pypa.io>`_-compatible repositories for a list of their available
+projects and lists of each project's available package files.  The library also
+allows you to download package files and query them for their project version,
+package type, file digests, ``requires_python`` string, PGP signature URL, and
+metadata URL.
 
@@ -82,4 +84,3 @@ See `the documentation <https://pypi-simple.readthedocs.io>`_ for more
 ``pypi-simple`` requires Python 3.7 or higher.  Just use `pip
-<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install
-``pypi-simple`` and its dependencies::
+<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install it::
 
@@ -86,0 +87,0 @@ python3 -m pip install pypi-simple

README.rst

@@ -28,8 +28,9 @@ .. image:: http://www.repostatus.org/badges/latest/active.svg
 specified in :pep:`503` and updated by :pep:`592`, :pep:`629`, :pep:`658`,
-:pep:`691`, and :pep:`700`.  With it, you can query `the Python Package Index
-(PyPI) <https://pypi.org>`_ and other `pip <https://pip.pypa.io>`_-compatible
-repositories for a list of their available projects and lists of each project's
-available package files.  The library also allows you to download package files
-and query them for their project version, package type, file digests,
-``requires_python`` string, PGP signature URL, and metadata URL.
+:pep:`691`, :pep:`700`, and :pep:`714`.  With it, you can query `the Python
+Package Index (PyPI) <https://pypi.org>`_ and other `pip
+<https://pip.pypa.io>`_-compatible repositories for a list of their available
+projects and lists of each project's available package files.  The library also
+allows you to download package files and query them for their project version,
+package type, file digests, ``requires_python`` string, PGP signature URL, and
+metadata URL.
 
@@ -43,4 +44,3 @@ See `the documentation <https://pypi-simple.readthedocs.io>`_ for more
 ``pypi-simple`` requires Python 3.7 or higher.  Just use `pip
-<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install
-``pypi-simple`` and its dependencies::
+<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install it::
 
@@ -47,0 +47,0 @@ python3 -m pip install pypi-simple

setup.cfg

@@ -27,2 +27,3 @@ [metadata]
 	Programming Language :: Python :: 3.11
+	Programming Language :: Python :: 3.12
 	Programming Language :: Python :: Implementation :: CPython
@@ -63,3 +64,3 @@ Programming Language :: Python :: Implementation :: PyPy
 allow_untyped_defs = False
-ignore_missing_imports = True
+ignore_missing_imports = False
 no_implicit_optional = True
@@ -66,0 +67,0 @@ implicit_reexport = False

src/pypi_simple.egg-info/PKG-INFO

 Metadata-Version: 2.1
 Name: pypi-simple
-Version: 1.2.0
+Version: 1.3.0
 Summary: PyPI Simple Repository API client library
@@ -20,2 +20,3 @@ Home-page: https://github.com/jwodder/pypi-simple
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: Implementation :: CPython
@@ -67,8 +68,9 @@ Classifier: Programming Language :: Python :: Implementation :: PyPy
 specified in :pep:`503` and updated by :pep:`592`, :pep:`629`, :pep:`658`,
-:pep:`691`, and :pep:`700`.  With it, you can query `the Python Package Index
-(PyPI) <https://pypi.org>`_ and other `pip <https://pip.pypa.io>`_-compatible
-repositories for a list of their available projects and lists of each project's
-available package files.  The library also allows you to download package files
-and query them for their project version, package type, file digests,
-``requires_python`` string, PGP signature URL, and metadata URL.
+:pep:`691`, :pep:`700`, and :pep:`714`.  With it, you can query `the Python
+Package Index (PyPI) <https://pypi.org>`_ and other `pip
+<https://pip.pypa.io>`_-compatible repositories for a list of their available
+projects and lists of each project's available package files.  The library also
+allows you to download package files and query them for their project version,
+package type, file digests, ``requires_python`` string, PGP signature URL, and
+metadata URL.
 
@@ -82,4 +84,3 @@ See `the documentation <https://pypi-simple.readthedocs.io>`_ for more
 ``pypi-simple`` requires Python 3.7 or higher.  Just use `pip
-<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install
-``pypi-simple`` and its dependencies::
+<https://pip.pypa.io>`_ for Python 3 (You have pip, right?) to install it::
 
@@ -86,0 +87,0 @@ python3 -m pip install pypi-simple

src/pypi_simple.egg-info/SOURCES.txt

@@ -11,2 +11,3 @@ CHANGELOG.md
 docs/conf.py
+docs/examples.rst
 docs/index.rst
@@ -13,0 +14,0 @@ docs/requirements.txt

src/pypi_simple/__init__.py

@@ -17,3 +17,3 @@ """
 
-__version__ = "1.2.0"
+__version__ = "1.3.0"
 __author__ = "John Thorvald Wodder II"
@@ -72,3 +72,3 @@ __author_email__ = "[e-mail]"
 from .classes import DistributionPackage, IndexPage, ProjectPage
-from .client import NoSuchProjectError, PyPISimple
+from .client import NoMetadataError, NoSuchProjectError, PyPISimple
 from .errors import (
@@ -93,2 +93,3 @@ DigestMismatchError,
     "NoDigestsError",
+    "NoMetadataError",
     "NoSuchProjectError",
@@ -95,0 +96,0 @@ "PYPI_SIMPLE_ENDPOINT",

src/pypi_simple/classes.py

@@ -144,3 +144,3 @@ from __future__ import annotations
             has_sig = None
-        mddigest = link.get_str_attrib("data-dist-info-metadata")
+        mddigest = link.get_str_attrib("data-core-metadata")
         metadata_digests: Optional[dict[str, str]]
@@ -147,0 +147,0 @@ if mddigest is not None:

src/pypi_simple/client.py

@@ -331,3 +331,60 @@ from __future__ import annotations
 
+    def get_package_metadata(
+        self,
+        pkg: DistributionPackage,
+        verify: bool = True,
+        timeout: float | tuple[float, float] | None = None,
+    ) -> str:
+        """
+        .. versionadded:: 1.3.0
 
+        Retrieve the `distribution metadata`_ for the given
+        `DistributionPackage`.  The metadata can then be parsed with, for
+        example, |the packaging package|_.
+
+        Not all packages have distribution metadata available for download; the
+        `DistributionPackage.has_metadata` attribute can be used to check
+        whether the repository reported the availability of the metadata.  This
+        method will always attempt to download metadata regardless of the value
+        of `~DistributionPackage.has_metadata`; if the server replies with a
+        404, a `NoMetadataError` is raised.
+
+        .. _distribution metadata:
+           https://packaging.python.org/en/latest/specifications/core-metadata/
+        .. |the packaging package| replace:: the ``packaging`` package
+        .. _the packaging package:
+           https://packaging.pypa.io/en/stable/metadata.html
+
+        :param DistributionPackage pkg:
+            the distribution package to retrieve the metadata of
+        :param bool verify:
+            whether to verify the metadata's digests against the retrieved data
+        :param timeout: optional timeout to pass to the ``requests`` call
+        :type timeout: float | tuple[float,float] | None
+
+        :raises NoMetadataError:
+            if the repository responds with a 404 error code
+        :raises requests.HTTPError: if the repository responds with an HTTP
+            error code other than 404
+        :raises NoDigestsError:
+            if ``verify`` is true and the given package's metadata does not
+            have any digests with known algorithms
+        :raises DigestMismatchError:
+            if ``verify`` is true and the digest of the downloaded data does
+            not match the expected value
+        """
+        digester: AbstractDigestChecker
+        if verify:
+            digester = DigestChecker(pkg.metadata_digests or {})
+        else:
+            digester = NullDigestChecker()
+        r = self.s.get(pkg.metadata_url, timeout=timeout)
+        if r.status_code == 404:
+            raise NoMetadataError(pkg.filename)
+        r.raise_for_status()
+        digester.update(r.content)
+        digester.finalize()
+        return r.text
+
+
 class NoSuchProjectError(Exception):
@@ -347,1 +404,15 @@ """
         return f"No details about project {self.project!r} available at {self.url}"
+
+
+class NoMetadataError(Exception):
+    """
+    Raised by `PyPISimple.get_package_metadata()` when a request for
+    distribution metadata fails with a 404 error code
+    """
+
+    def __init__(self, filename: str) -> None:
+        #: The filename of the package whose metadata was requested
+        self.filename = filename
+
+    def __str__(self) -> str:
+        return f"No distribution metadata found for {self.filename}"

src/pypi_simple/html.py

@@ -5,3 +5,3 @@ from __future__ import annotations
 from urllib.parse import urljoin
-from bs4 import BeautifulSoup
+from bs4 import BeautifulSoup, Tag
 from .util import basejoin, check_repo_version
@@ -52,6 +52,9 @@
         if base_tag is not None:
+            assert isinstance(base_tag, Tag)
+            href = base_tag["href"]
+            assert isinstance(href, str)
             if base_url is None:
-                base_url = base_tag["href"]
+                base_url = href
             else:
-                base_url = urljoin(base_url, base_tag["href"])
+                base_url = urljoin(base_url, href)
         pep629_meta = soup.find(
@@ -62,3 +65,6 @@ "meta",
         if pep629_meta is not None:
-            repository_version = pep629_meta["content"]
+            assert isinstance(pep629_meta, Tag)
+            content = pep629_meta["content"]
+            assert isinstance(content, str)
+            repository_version = content
             check_repo_version(repository_version)
@@ -65,0 +71,0 @@ else:

src/pypi_simple/pep691.py

@@ -29,3 +29,3 @@ from __future__ import annotations
     requires_python: Optional[str] = None
-    dist_info_metadata: Union[StrictBool, Dict[str, str], None] = None
+    core_metadata: Union[StrictBool, Dict[str, str], None] = None
     gpg_sig: Optional[StrictBool] = None
@@ -52,12 +52,12 @@ yanked: Union[StrictBool, str] = False
     def has_metadata(self) -> Optional[bool]:
-        if isinstance(self.dist_info_metadata, dict):
+        if isinstance(self.core_metadata, dict):
             return True
         else:
-            return self.dist_info_metadata
+            return self.core_metadata
 
     @property
     def metadata_digests(self) -> Optional[dict[str, str]]:
-        if isinstance(self.dist_info_metadata, dict):
-            return self.dist_info_metadata
-        elif self.dist_info_metadata is True:
+        if isinstance(self.core_metadata, dict):
+            return self.core_metadata
+        elif self.core_metadata is True:
             return {}
@@ -64,0 +64,0 @@ else:

src/pypi_simple/progress.py

@@ -5,3 +5,3 @@ from __future__ import annotations
 from types import TracebackType
-from typing import Any, Optional, TypeVar
+from typing import TYPE_CHECKING, Any, Optional
 
@@ -13,6 +13,6 @@ if sys.version_info[:2] >= (3, 8):
 
+if TYPE_CHECKING:
+    from typing_extensions import Self
 
-T = TypeVar("T", bound="ProgressTracker")
 
-
 @runtime_checkable
@@ -28,3 +28,3 @@ class ProgressTracker(Protocol):
 
-    def __enter__(self: T) -> T:
+    def __enter__(self) -> Self:
         ...
@@ -31,0 +31,0 @@

test/test_distrib_pkg.py

@@ -97,3 +97,3 @@ from __future__ import annotations
                     "data-gpg-sig": "true",
-                    "data-dist-info-metadata": "sha256=ae718719df4708f329d58ca4d5390c1206c4222ef7e62a3aa9844397c63de28b",
+                    "data-core-metadata": "sha256=ae718719df4708f329d58ca4d5390c1206c4222ef7e62a3aa9844397c63de28b",
                     "data-yanked": "Oopsy.",
@@ -127,3 +127,3 @@ },
                     "data-gpg-sig": "false",
-                    "data-dist-info-metadata": "sha256=true",
+                    "data-core-metadata": "sha256=true",
                 },
@@ -219,3 +219,3 @@ ),
 @pytest.mark.parametrize(
-    "dist_info_metadata,has_metadata,metadata_digests",
+    "core_metadata,has_metadata,metadata_digests",
     [
@@ -229,3 +229,3 @@ (False, False, None),
 def test_from_json_data_metadata(
-    dist_info_metadata: bool | dict[str, str],
+    core_metadata: bool | dict[str, str],
     has_metadata: bool,
@@ -243,3 +243,3 @@ metadata_digests: Optional[dict[str, str]],
             "yanked": False,
-            "dist-info-metadata": dist_info_metadata,
+            "core-metadata": core_metadata,
         }
@@ -246,0 +246,0 @@ )

tox.ini

 [tox]
-envlist = lint,typing,py37,py38,py39,py310,py311,pypy3
+envlist = lint,typing,py37,py38,py39,py310,py311,py312,pypy3
 skip_missing_interpreters = True
@@ -29,3 +29,5 @@ isolated_build = True
     mypy
+    {[testenv]deps}
     tqdm-stubs
+    types-beautifulsoup4
     types-requests
@@ -32,0 +34,0 @@ commands =

Alert delta unavailable

Currently unable to show alert delta for PyPI packages.

Improved metrics

Total package byte prevSize

270705

2.5%

Number of package files

55

1.85%

Lines of code

5391

1.3%