Pillory
A linter to scrutinize how you are using mocks in Python.
If you often hear or make the comment "patch the import not the definition" or
spend a lot of time explaining or helping people debug mocks, this linter could
help you.
Usage
Install from PyPI with pip:
pip install pillory
Pillory needs to be able to find your code and its dependencies as if it were
importing it (but it doesn’t actually import it). This means you need to install
any other dependencies and set up your environment as if you were going to run
your tests. How you do this can vary between projects, but could include making
a virtualenv, pip install -r requirements.txt
, and pip install -e .
. If you
don’t do this pillory will find fewer errors than it should, or maybe find
no errors at all, a misleading positive result.
Run pillory on the current directory:
python -m pillory
Or give a specific files and directories:
python -m pillory tests/ test_example.py
Rules
PM101 patched implementation
: You patched the implementation of a class or function instead of where it is
imported to in the module under test. e.g. "parsers.Parser" where Parser is
defined instead of "app.Parser" where it is used. This means you may not
have affected the module under test at all, or you have replaced the target in a
way which will affect other code that uses it, which is bad for test isolation
(making sure tests don't affect other tests, each test tests only what it
intends to, and doesn’t rely on how other parts of the code work). There is a
warning about not affecting the right module in the Python Standard Library
docs, but there is an even better article by Ned
Batchelder explaining how it works and the additional problems with
test isolation.
PM102 patched is not a top level module attribute
: You patched something like a method on a class. Because class methods can't be
imported by themselves, this means all uses of the class will be affected, not
just the module under test.
PM103 patched builtins
: You patched the builtins module instead of the built-in function in the module
under test. Built-ins are actually added to every module and that's where they
should be patched, to avoid similar issues to patching the implementation. There
is a CPython detail that means the builtins module may be added to the lookup of
each module, so patching the builtins module can work, but it's not guaranteed
and it still has problems with test isolation.
Known issues
- No console script entry point (pillory command), have to use with python -m.
- Does not have a non-zero exit code when errors found.
- Only tested with Python 3.10.
- No config file support.
- No comments to ignore rules.
- Not fast.
- No further explanations for the errors.
- No pretty error handling, just tracebacks.
- Will error when mocking something in the module under test, which is arguably
"OK".
- No pre-commit integration.
What's with the name?
I thought it was funny that mock can also mean "make fun of" as well as the
meaning of "mimic" that we use in testing. I imagined the linter cruelly calling
out how you are using mocks incorrectly. Except I couldn't call it "mock", or
"mock mocker", that would be confusing! So I picked a name with a similar
meaning, and starting with a P for that Python feeling.
Contributing
Thank you for your interest in making a contribution.
Please talk to the maintainer before making a pull request to make sure what you
are adding is wanted.
This project uses the Apache License 2.0. You will be credited in the git
history, but for ease of maintenance copyright stays with the maintainer.
There are linters, formatters, and tests as part of the CI. You can check them
yourself locally too. To set up the linters as a pre-commit hook:
# Install the dev dependencies if not done already
pip install -r dev-requirements.txt
# Install the pre-commit hooks
pre-commit install
You can run the linters on command with:
pre-commit run -a
The unit tests are supposed to be fast tests for testing logic in a test driven
developement style. They shouldn't have mocking or accounting for side effects.
You can run them with:
python test_pillory.py
The example tests have a more complicated setup to include the file system and
command line args. Try to limit the number of example tests. The tests use
doctest to easily check the outputs. You can run them with:
python test_example.py -v
To run all the different tests and see the coverage given you can use:
make
The default make target is the coverage report in HTML format. You can look at
the file in a browser, or if you don’t want to leave the terminal but still want
a line by line coverage report, you can use browsh as your browser.
You can run the tests, start a HTTP server, and view the report in browsh with
this oneliner:
( make && cd build/htmlcov && python -m http.server 8081 &>/dev/null & browsh http://localhost:8081 ; kill $! )