pathspec

PathSpec

pathspec is a utility library for pattern matching of file paths. So far this only includes Git's wildmatch pattern matching which itself is derived from Rsync's wildmatch. Git uses wildmatch for its gitignore_ files.

.. _gitignore: http://git-scm.com/docs/gitignore

Tutorial

Say you have a "Projects" directory and you want to back it up, but only certain files, and ignore others depending on certain conditions::

>>> import pathspec
>>> # The gitignore-style patterns for files to select, but we're including
>>> # instead of ignoring.
>>> spec_text = """
...
... # This is a comment because the line begins with a hash: "#"
...
... # Include several project directories (and all descendants) relative to
... # the current directory. To reference a directory you must end with a
... # slash: "/"
... /project-a/
... /project-b/
... /project-c/
...
... # Patterns can be negated by prefixing with exclamation mark: "!"
...
... # Ignore temporary files beginning or ending with "~" and ending with
... # ".swp".
... !~*
... !*~
... !*.swp
...
... # These are python projects so ignore compiled python files from
... # testing.
... !*.pyc
...
... # Ignore the build directories but only directly under the project
... # directories.
... !/*/build/
...
... """

We want to use the GitWildMatchPattern class to compile our patterns. The PathSpec class provides an interface around pattern implementations::

>>> spec = pathspec.PathSpec.from_lines(pathspec.patterns.GitWildMatchPattern, spec_text.splitlines())

That may be a mouthful but it allows for additional patterns to be implemented in the future without them having to deal with anything but matching the paths sent to them. GitWildMatchPattern is the implementation of the actual pattern which internally gets converted into a regular expression. PathSpec is a simple wrapper around a list of compiled patterns.

To make things simpler, we can use the registered name for a pattern class instead of always having to provide a reference to the class itself. The GitWildMatchPattern class is registered as gitwildmatch::

>>> spec = pathspec.PathSpec.from_lines('gitwildmatch', spec_text.splitlines())

If we wanted to manually compile the patterns we can just do the following::

>>> patterns = map(pathspec.patterns.GitWildMatchPattern, spec_text.splitlines())
>>> spec = PathSpec(patterns)

PathSpec.from_lines() is simply a class method which does just that.

If you want to load the patterns from file, you can pass the file instance directly as well::

>>> with open('patterns.list', 'r') as fh:
>>>     spec = pathspec.PathSpec.from_lines('gitwildmatch', fh)

You can perform matching on a whole directory tree with::

>>> matches = spec.match_tree('path/to/directory')

Or you can perform matching on a specific set of file paths with::

>>> matches = spec.match_files(file_paths)

Or check to see if an individual file matches::

>>> is_matched = spec.match_file(file_path)

There is a specialized class, pathspec.GitIgnoreSpec, which more closely implements the behavior of gitignore. This uses GitWildMatchPattern pattern by default and handles some edge cases differently from the generic PathSpec class. GitIgnoreSpec can be used without specifying the pattern factory::

>>> spec = pathspec.GitIgnoreSpec.from_lines(spec_text.splitlines())

License

pathspec is licensed under the Mozilla Public License Version 2.0. See LICENSE or the FAQ_ for more information.

In summary, you may use pathspec with any closed or open source project without affecting the license of the larger work so long as you:

• give credit where credit is due,

• and release any custom changes made to pathspec.

.. _Mozilla Public License Version 2.0: http://www.mozilla.org/MPL/2.0 .. _LICENSE: LICENSE .. _FAQ: http://www.mozilla.org/MPL/2.0/FAQ.html

Source

The source code for pathspec is available from the GitHub repo cpburnz/python-pathspec_.

.. _cpburnz/python-pathspec: https://github.com/cpburnz/python-pathspec

Installation

pathspec is available for install through PyPI_::

pip install pathspec

pathspec can also be built from source. The following packages will be required:

• build_ (>=0.6.0)

pathspec can then be built and installed with::

python -m build
pip install dist/pathspec-*-py3-none-any.whl

.. _PyPI: http://pypi.python.org/pypi/pathspec .. _build: https://pypi.org/project/build/

Documentation

Documentation for pathspec is available on Read the Docs_.

.. _Read the Docs: https://python-path-specification.readthedocs.io

Other Languages

The related project pathspec-ruby_ (by highb) provides a similar library as a Ruby gem_.

.. _pathspec-ruby: https://github.com/highb/pathspec-ruby .. _Ruby gem: https://rubygems.org/gems/pathspec

Change History

0.12.1 (2023-12-10)

Bug fixes:

• Issue #84_: PathSpec.match_file() returns None since 0.12.0.

.. _Issue #84: https://github.com/cpburnz/python-pathspec/issues/84

0.12.0 (2023-12-09)

Major changes:

• Dropped support of EOL Python 3.7. See Pull #82_.

API changes:

• Signature of protected method pathspec.pathspec.PathSpec._match_file() (with a leading underscore) has been changed from def _match_file(patterns: Iterable[Pattern], file: str) -> bool to def _match_file(patterns: Iterable[Tuple[int, Pattern]], file: str) -> Tuple[Optional[bool], Optional[int]].

New features:

• Added pathspec.pathspec.PathSpec.check_*() methods. These methods behave similarly to .match_*() but return additional information in the pathspec.util.CheckResult objects (e.g., CheckResult.index indicates the index of the last pattern that matched the file).
• Added pathspec.pattern.RegexPattern.pattern attribute which stores the original, uncompiled pattern.

Bug fixes:

• Issue #81_: GitIgnoreSpec behaviors differ from git.
• Pull #83_: Fix ReadTheDocs builds.

Improvements:

• Mark Python 3.12 as supported. See Pull #82_.
• Improve test debugging.
• Improve type hint on on_error parameter on pathspec.pathspec.PathSpec.match_tree_entries().
• Improve type hint on on_error parameter on pathspec.util.iter_tree_entries().

.. _Issue #81: https://github.com/cpburnz/python-pathspec/issues/81 .. _Pull #82: https://github.com/cpburnz/python-pathspec/pull/82 .. _Pull #83: https://github.com/cpburnz/python-pathspec/pull/83

0.11.2 (2023-07-28)

New features:

• Issue #80_: match_files with negated path spec. pathspec.PathSpec.match_*() now have a negate parameter to make using .gitignore logic easier and more efficient.

Bug fixes:

• Pull #76_: Add edge case: patterns that end with an escaped space
• Issue #77/Pull #78: Negate with caret symbol as with the exclamation mark.

.. _Pull #76: https://github.com/cpburnz/python-pathspec/pull/76 .. _Issue #77: https://github.com/cpburnz/python-pathspec/issues/77 .. _Pull #78: https://github.com/cpburnz/python-pathspec/pull/78/ .. _Issue #80: https://github.com/cpburnz/python-pathspec/issues/80

0.11.1 (2023-03-14)

Bug fixes:

• Issue #74_: Include directory should override exclude file.

Improvements:

• Pull #75_: Fix partially unknown PathLike type.
• Convert os.PathLike to a string properly using os.fspath.

.. _Issue #74: https://github.com/cpburnz/python-pathspec/issues/74 .. _Pull #75: https://github.com/cpburnz/python-pathspec/pull/75

0.11.0 (2023-01-24)

Major changes:

• Changed build backend to flit_core.buildapi_ from setuptools.build_meta. Building with setuptools through setup.py is still supported for distributions that need it. See Issue #72.

Improvements:

• Issue #72/Pull #73: Please consider switching the build-system to flit_core to ease setuptools bootstrap.

.. _flit_core.buildapi: https://flit.pypa.io/en/latest/index.html .. _Issue #72: https://github.com/cpburnz/python-pathspec/issues/72 .. _Pull #73: https://github.com/cpburnz/python-pathspec/pull/73

0.10.3 (2022-12-09)

New features:

• Added utility function pathspec.util.append_dir_sep() to aid in distinguishing between directories and files on the file-system. See Issue #65_.

Bug fixes:

• Issue #66/Pull #67: Package not marked as py.typed.
• Issue #68_: Exports are considered private.
• Issue #70/Pull #71: 'Self' string literal type is Unknown in pyright.

Improvements:

• Issue #65_: Checking directories via match_file() does not work on Path objects.

.. _Issue #65: https://github.com/cpburnz/python-pathspec/issues/65 .. _Issue #66: https://github.com/cpburnz/python-pathspec/issues/66 .. _Pull #67: https://github.com/cpburnz/python-pathspec/pull/67 .. _Issue #68: https://github.com/cpburnz/python-pathspec/issues/68 .. _Issue #70: https://github.com/cpburnz/python-pathspec/issues/70 .. _Pull #71: https://github.com/cpburnz/python-pathspec/pull/71

0.10.2 (2022-11-12)

Bug fixes:

• Fix failing tests on Windows.
• Type hint on root parameter on pathspec.pathspec.PathSpec.match_tree_entries().
• Type hint on root parameter on pathspec.pathspec.PathSpec.match_tree_files().
• Type hint on root parameter on pathspec.util.iter_tree_entries().
• Type hint on root parameter on pathspec.util.iter_tree_files().
• Issue #64_: IndexError with my .gitignore file when trying to build a Python package.

Improvements:

• Pull #58_: CI: add GitHub Actions test workflow.

.. _Pull #58: https://github.com/cpburnz/python-pathspec/pull/58 .. _Issue #64: https://github.com/cpburnz/python-pathspec/issues/64

0.10.1 (2022-09-02)

Bug fixes:

• Fix documentation on pathspec.pattern.RegexPattern.match_file().
• Pull #60_: Remove redundant wheel dep from pyproject.toml.
• Issue #61_: Dist failure for Fedora, CentOS, EPEL.
• Issue #62_: Since version 0.10.0 pure wildcard does not work in some cases.

Improvements:

• Restore support for legacy installations using setup.py. See Issue #61_.

.. _Pull #60: https://github.com/cpburnz/python-pathspec/pull/60 .. _Issue #61: https://github.com/cpburnz/python-pathspec/issues/61 .. _Issue #62: https://github.com/cpburnz/python-pathspec/issues/62

0.10.0 (2022-08-30)

Major changes:

• Dropped support of EOL Python 2.7, 3.5, 3.6. See Issue #47_.
• The gitwildmatch pattern dir/* is now handled the same as dir/. This means dir/* will now match all descendants rather than only direct children. See Issue #19_.
• Added pathspec.GitIgnoreSpec class (see new features).
• Changed build system to pyproject.toml_ and build backend to setuptools.build_meta_ which may have unforeseen consequences.
• Renamed GitHub project from python-path-specification_ to python-pathspec. See Issue #35.

API changes:

• Deprecated: pathspec.util.match_files() is an old function no longer used.
• Deprecated: pathspec.match_files() is an old function no longer used.
• Deprecated: pathspec.util.normalize_files() is no longer used.
• Deprecated: pathspec.util.iter_tree() is an alias for pathspec.util.iter_tree_files().
• Deprecated: pathspec.iter_tree() is an alias for pathspec.util.iter_tree_files().
• Deprecated: pathspec.pattern.Pattern.match() is no longer used. Use or implement pathspec.pattern.Pattern.match_file().

New features:

• Added class pathspec.gitignore.GitIgnoreSpec (with alias pathspec.GitIgnoreSpec) to implement gitignore behavior not possible with standard PathSpec class. The particular gitignore behavior implemented is prioritizing patterns matching the file directly over matching an ancestor directory.

Bug fixes:

• Issue #19_: Files inside an ignored sub-directory are not matched.
• Issue #41_: Incorrectly (?) matches files inside directories that do match.
• Pull #51_: Refactor deprecated unittest aliases for Python 3.11 compatibility.
• Issue #53_: Symlink pathspec_meta.py breaks Windows.
• Issue #54_: test_util.py uses os.symlink which can fail on Windows.
• Issue #55_: Backslashes at start of pattern not handled correctly.
• Pull #56_: pyproject.toml: include subpackages in setuptools config
• Issue #57_: ! doesn't exclude files in directories if the pattern doesn't have a trailing slash.

Improvements:

• Support Python 3.10, 3.11.
• Modernize code to Python 3.7.
• Issue #52: match_files() is not a pure generator function, and it impacts tree*() gravely.

.. _python-path-specification: https://github.com/cpburnz/python-path-specification .. _python-pathspec: https://github.com/cpburnz/python-pathspec .. _pyproject.toml: https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ .. _setuptools.build_meta: https://setuptools.pypa.io/en/latest/build_meta.html .. _Issue #19: https://github.com/cpburnz/python-pathspec/issues/19 .. _Issue #35: https://github.com/cpburnz/python-pathspec/issues/35 .. _Issue #41: https://github.com/cpburnz/python-pathspec/issues/41 .. _Issue #47: https://github.com/cpburnz/python-pathspec/issues/47 .. _Pull #51: https://github.com/cpburnz/python-pathspec/pull/51 .. _Issue #52: https://github.com/cpburnz/python-pathspec/issues/52 .. _Issue #53: https://github.com/cpburnz/python-pathspec/issues/53 .. _Issue #54: https://github.com/cpburnz/python-pathspec/issues/54 .. _Issue #55: https://github.com/cpburnz/python-pathspec/issues/55 .. _Pull #56: https://github.com/cpburnz/python-pathspec/pull/56 .. _Issue #57: https://github.com/cpburnz/python-pathspec/issues/57

0.9.0 (2021-07-17)

• Issue #44/Pull #50: Raise GitWildMatchPatternError for invalid git patterns.
• Pull #45_: Fix for duplicate leading double-asterisk, and edge cases.
• Issue #46_: Fix matching absolute paths.
• API change: util.normalize_files() now returns a Dict[str, List[pathlike]] instead of a Dict[str, pathlike].
• Added type hinting.

.. _Issue #44: https://github.com/cpburnz/python-pathspec/issues/44 .. _Pull #45: https://github.com/cpburnz/python-pathspec/pull/45 .. _Issue #46: https://github.com/cpburnz/python-pathspec/issues/46 .. _Pull #50: https://github.com/cpburnz/python-pathspec/pull/50

0.8.1 (2020-11-07)

• Pull #43_: Add support for addition operator.

.. _Pull #43: https://github.com/cpburnz/python-pathspec/pull/43

0.8.0 (2020-04-09)

• Issue #30_: Expose what patterns matched paths. Added util.detailed_match_files().
• Issue #31_: match_tree() doesn't return symlinks.
• Issue #34_: Support pathlib.Path\ s.
• Add PathSpec.match_tree_entries and util.iter_tree_entries() to support directories and symlinks.
• API change: match_tree() has been renamed to match_tree_files(). The old name match_tree() is still available as an alias.
• API change: match_tree_files() now returns symlinks. This is a bug fix but it will change the returned results.

.. _Issue #30: https://github.com/cpburnz/python-pathspec/issues/30 .. _Issue #31: https://github.com/cpburnz/python-pathspec/issues/31 .. _Issue #34: https://github.com/cpburnz/python-pathspec/issues/34

0.7.0 (2019-12-27)

• Pull #28_: Add support for Python 3.8, and drop Python 3.4.
• Pull #29_: Publish bdist wheel.

.. _Pull #28: https://github.com/cpburnz/python-pathspec/pull/28 .. _Pull #29: https://github.com/cpburnz/python-pathspec/pull/29

0.6.0 (2019-10-03)

• Pull #24_: Drop support for Python 2.6, 3.2, and 3.3.
• Pull #25_: Update README.rst.
• Pull #26_: Method to escape gitwildmatch.

.. _Pull #24: https://github.com/cpburnz/python-pathspec/pull/24 .. _Pull #25: https://github.com/cpburnz/python-pathspec/pull/25 .. _Pull #26: https://github.com/cpburnz/python-pathspec/pull/26

0.5.9 (2018-09-15)

• Fixed file system error handling.

0.5.8 (2018-09-15)

• Improved type checking.
• Created scripts to test Python 2.6 because Tox removed support for it.
• Improved byte string handling in Python 3.
• Issue #22_: Handle dangling symlinks.

.. _Issue #22: https://github.com/cpburnz/python-pathspec/issues/22

0.5.7 (2018-08-14)

• Issue #21_: Fix collections deprecation warning.

.. _Issue #21: https://github.com/cpburnz/python-pathspec/issues/21

0.5.6 (2018-04-06)

• Improved unit tests.
• Improved type checking.
• Issue #20_: Support current directory prefix.

.. _Issue #20: https://github.com/cpburnz/python-pathspec/issues/20

0.5.5 (2017-09-09)

• Add documentation link to README.

0.5.4 (2017-09-09)

• Pull #17_: Add link to Ruby implementation of pathspec.
• Add sphinx documentation.

.. _Pull #17: https://github.com/cpburnz/python-pathspec/pull/17

0.5.3 (2017-07-01)

• Issue #14_: Fix byte strings for Python 3.
• Pull #15_: Include "LICENSE" in source package.
• Issue #16_: Support Python 2.6.

.. _Issue #14: https://github.com/cpburnz/python-pathspec/issues/14 .. _Pull #15: https://github.com/cpburnz/python-pathspec/pull/15 .. _Issue #16: https://github.com/cpburnz/python-pathspec/issues/16

0.5.2 (2017-04-04)

• Fixed change log.

0.5.1 (2017-04-04)

• Pull #13_: Add equality methods to PathSpec and RegexPattern.

.. _Pull #13: https://github.com/cpburnz/python-pathspec/pull/13

0.5.0 (2016-08-22)

• Issue #12_: Add PathSpec.match_file().
• Renamed gitignore.GitIgnorePattern to patterns.gitwildmatch.GitWildMatchPattern.
• Deprecated gitignore.GitIgnorePattern.

.. _Issue #12: https://github.com/cpburnz/python-pathspec/issues/12

0.4.0 (2016-07-15)

• Issue #11_: Support converting patterns into regular expressions without compiling them.
• API change: Subclasses of RegexPattern should implement pattern_to_regex().

.. _Issue #11: https://github.com/cpburnz/python-pathspec/issues/11

0.3.4 (2015-08-24)

• Pull #7_: Fixed non-recursive links.
• Pull #8_: Fixed edge cases in gitignore patterns.
• Pull #9_: Fixed minor usage documentation.
• Fixed recursion detection.
• Fixed trivial incompatibility with Python 3.2.

.. _Pull #7: https://github.com/cpburnz/python-pathspec/pull/7 .. _Pull #8: https://github.com/cpburnz/python-pathspec/pull/8 .. _Pull #9: https://github.com/cpburnz/python-pathspec/pull/9

0.3.3 (2014-11-21)

• Improved documentation.

0.3.2 (2014-11-08)

• Pull #5_: Use tox for testing.
• Issue #6_: Fixed matching Windows paths.
• Improved documentation.
• API change: spec.match_tree() and spec.match_files() now return iterators instead of sets.

.. _Pull #5: https://github.com/cpburnz/python-pathspec/pull/5 .. _Issue #6: https://github.com/cpburnz/python-pathspec/issues/6

0.3.1 (2014-09-17)

• Updated README.

0.3.0 (2014-09-17)

• Pull #3_: Fixed trailing slash in gitignore patterns.
• Pull #4_: Fixed test for trailing slash in gitignore patterns.
• Added registered patterns.

.. _Pull #3: https://github.com/cpburnz/python-pathspec/pull/3 .. _Pull #4: https://github.com/cpburnz/python-pathspec/pull/4

0.2.2 (2013-12-17)

• Fixed setup.py.

0.2.1 (2013-12-17)

• Added tests.
• Fixed comment gitignore patterns.
• Fixed relative path gitignore patterns.

0.2.0 (2013-12-07)

• Initial release.