ptr - Python Test Runner 🏃♂️
Python Test Runner (ptr) was born to run tests in an opinionated way, within arbitrary code repositories.
ptr supports many Python projects with unit tests defined in their setup.(cfg|py) files per repository.
ptr allows developers to test multiple projects/modules in one Python environment through the use of a single test virtual environment.
ptr requires >= python 3.7ptr itself uses ptr to run its tests 👌🏼ptr is supported and tested on Linux, MacOS + Windows Operating SystemsBy adding ptr configuration to your either of your pyproject.toml, setup.cfg or setup.py you can have ptr perform the following,
per test suite, in parallel:
ptr into you virtualenv
pip install ptrpython3 test.py (possibly using unittest.main())ptr_params to setup.py (see example below), run:cd repo
ptr
ptr perform this magic? 🎩I'm glad you ask. Under the covers ptr performs:
setup.(cfg|py) files from BASE_DIR (defaults to your "current working directory" (CWD))
setup.py test requirementspyproject.toml or setup.cfg exists, load via configparser/tomli and prefer if a [ptr] section existsATONCE tests suites in parallel (i.e. per setup.(cfg|ptr))To use ptr all you need to do is cd to your project or set the base dir via -b and execute:
$ ptr [-dk] [-b some/path] [--venv /tmp/existing_venv]
For faster runs when testing, it is recommended to reuse a Virtual Environment:
-k - To keep the virtualenv created by ptr.--venv VENV_PATH to reuse to an existing virtualenv created by the user.usage: ptr.py [-h] [-a ATONCE] [-b BASE_DIR] [-d] [-e] [-k] [-m MIRROR]
[--print-cov] [--print-non-configured]
[--progress-interval PROGRESS_INTERVAL] [--run-disabled]
[--stats-file STATS_FILE] [--system-site-packages] [--venv VENV]
[--venv-timeout VENV_TIMEOUT]
optional arguments:
-h, --help show this help message and exit
-a ATONCE, --atonce ATONCE
How many tests to run at once [Default: 6]
-b BASE_DIR, --base-dir BASE_DIR
Path to recursively look for setup.py files [Default:
/Users/cooper/repos/ptr]
-d, --debug Verbose debug output
-e, --error-on-warnings
Have Python warnings raise DeprecationWarning on tests
run
-k, --keep-venv Do not remove created venv
-m MIRROR, --mirror MIRROR
URL for pip to use for Simple API [Default:
https://pypi.org/simple/]
--print-cov Print modules coverage report
--print-non-configured
Print modules not configured to run ptr
--progress-interval PROGRESS_INTERVAL
Seconds between status update on test running
[Default: Disabled]
--run-disabled Force any disabled tests suites to run
--stats-file STATS_FILE
JSON statistics file [Default: /var/folders/tc/hbwxh76
j1hn6gqjd2n2sjn4j9k1glp/T/ptr_stats_12510]
--system-site-packages
Give the virtual environment access to the system
site-packages dir
--venv VENV Path to venv to reuse
--venv-timeout VENV_TIMEOUT
Timeout in seconds for venv creation + deps install
[Default: 120]
ptr is configured by placing directives in one or more of the following files. .ptrconfig provides
base configuration and default values for all projects in the repository, while each setup.(cfg|py)
overrides the base configuration for the respective packages they define.
.ptrconfigptr supports a general config in ini (ConfigParser) format.
A .ptrconfig file can be placed at the root of any repository or in any directory within your repository.
The first .ptrconfig file found via a recursive walk to the root ("/" in POSIX systems) will be used.
Please refer to ptrconfig.sample for the options available.
setup.pyThis is per project in your repository. A simple example, based on ptr itself:
# Specific Python Test Runner (ptr) params for Unit Testing Enforcement
ptr_params = {
# Where mypy will run to type check your program
"entry_point_module": "ptr",
# Base Unittest file
"test_suite": "ptr_tests",
"test_suite_timeout": 300,
# Relative path from setup.py to module (e.g. ptr == ptr.py)
"required_coverage": {"ptr.py": 99, "TOTAL": 99},
# Run `black --check` or not
"run_black": False,
# Run mypy or not
"run_mypy": True,
}
pyproject.tomlThis is per project in your repository and if exists is preferred over setup.py and setup.cfg.
Please refer to pyproject.toml
for the options available + format.
setup.cfgThis is per project in your repository and if exists is preferred over setup.py.
Please refer to setup.cfg.sample
for the options available + format.
When enabled, (in setup.(cfg|py)) mypy can support using a custom mypy.ini for each setup.py (module) defined.
To have ptr run mypy using you config:
mypy.ini in the same directory as your setup.pysetup.cfgmypy Configuration Documentation can be found here
setup.cfg can be seen here.Here are some example runs.
ptr Run:Here is what you want to see in your CI logs!
[2019-02-06 21:51:45,442] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:51:59,471] INFO: Successfully created venv @ /var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_24397 to run tests (14s) (ptr.py:547)
[2019-02-06 21:51:59,472] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:52:00,726] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
[2019-02-06 21:52:04,153] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-02-06 21:52:04,368] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:09,915] INFO: Running flake8 for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:10,422] INFO: Running pylint for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:14,020] INFO: Running pyre for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-02-06 21:52:07,733] INFO: /Users/cooper/repos/ptr/setup.py has passed all configured tests (ptr.py:509)
-- Summary (total time 22s):
✅ PASS: 1
❌ FAIL: 0
⌛️ TIMEOUT: 0
💩 TOTAL: 1
-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running
ptr Run Examples:Here are some examples of runs failing. Any "step" can fail. All output is predominately the underlying tool.
[2019-02-06 21:53:58,121] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:53:58,143] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:53:59,698] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
-- Summary (total time 5s):
✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1
-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running
-- Failure Output --
/Users/cooper/repos/ptr/setup.py (failed 'tests_run' step):
...F....................
======================================================================
FAIL: test_config (__main__.TestPtr)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/cooper/repos/ptr/ptr_tests.py", line 125, in test_config
self.assertEqual(len(sc["ptr"]["venv_pkgs"].split()), 4)
AssertionError: 5 != 4
----------------------------------------------------------------------
Ran 24 tests in 3.221s
FAILED (failures=1)
[2019-02-06 21:55:42,947] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:55:42,969] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:55:44,920] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
[2019-02-06 21:55:49,628] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
-- Summary (total time 7s):
✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1
-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running
-- Failure Output --
/Users/cooper/repos/ptr/setup.py (failed 'analyze_coverage' step):
The following files did not meet coverage requirements:
ptr.py: 84 < 99 - Missing: 146-147, 175, 209, 245, 269, 288-291, 334-336, 414-415, 425-446, 466, 497, 506, 541-543, 562, 611-614, 639-688
[2019-02-06 22:34:20,029] INFO: Starting ptr.py (ptr.py:804)
[2019-02-06 22:34:20,060] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:430)
[2019-02-06 22:34:21,614] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:430)
[2019-02-06 22:34:25,208] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
[2019-02-06 22:34:25,450] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
[2019-02-06 22:34:26,422] INFO: Running black for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
-- Summary (total time 7s):
✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1
-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running
-- Failure Output --
/Users/cooper/repos/ptr/setup.py (failed 'black_run' step):
would reformat /Users/cooper/repos/ptr/ptr.py
All done! 💥 💔 💥
1 file would be reformatted, 4 files would be left unchanged.
[2019-02-06 22:35:39,480] INFO: Starting ptr.py (ptr.py:802)
[2019-02-06 22:35:39,531] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:428)
[2019-02-06 22:35:41,203] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:428)
[2019-02-06 22:35:45,156] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:428)
[2019-02-06 22:35:45,413] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:428)
-- Summary (total time 6s):
✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1
-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running
-- Failure Output --
/Users/cooper/repos/ptr/setup.py (failed 'mypy_run' step):
/Users/cooper/repos/ptr/ptr.py: note: In function "_write_stats_file":
/Users/cooper/repos/ptr/ptr.py:179: error: Argument 1 to "open" has incompatible type "Path"; expected "Union[str, bytes, int]"
/Users/cooper/repos/ptr/ptr.py: note: In function "run_tests":
/Users/cooper/repos/ptr/ptr.py:700: error: Argument 1 to "_write_stats_file" has incompatible type "str"; expected "Path"
cooper-mbp1:ptr cooper$ /tmp/tp/bin/ptr --venv /var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_49117
[2019-05-03 14:51:43,623] INFO: Starting /tmp/tp/bin/ptr (ptr.py:1023)
[2019-05-03 14:51:43,657] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:565)
[2019-05-03 14:51:44,840] INFO: Running ptr_tests tests via coverage (ptr.py:565)
[2019-05-03 14:51:47,361] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,559] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,827] INFO: Running black for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,996] INFO: Running flake8 for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:48,566] INFO: Running pylint for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:52,301] INFO: Running pyre for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:54,983] INFO: /Users/cooper/repos/ptr/setup.py has passed all configured tests (ptr.py:668)
-- Summary (total time 11s):
✅ PASS: 1
❌ FAIL: 0
⌛️ TIMEOUT: 0
💩 TOTAL: 1
-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running
-- Failure Output --
/Users/cooper/repos/ptr/setup.py (failed 'pyre_run' step):
2019-05-03 14:54:14,173 INFO No binary specified, looking for `pyre.bin` in PATH
2019-05-03 14:54:14,174 INFO Found: `/var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_49117/bin/pyre.bin`
... *(truncated)* ...
ptr.py:602:25 Undefined name [18]: Global name `stdout` is not defined, or there is at least one control flow path that doesn't define `stdout`.
If your imports are not making usort happy it would look like this:
[2021-05-29 09:30:56,044] INFO: Starting /tmp/tp/bin/ptr (ptr.py:1129)
[2021-05-29 09:30:56,051] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:637)
[2021-05-29 09:30:56,587] INFO: Running ptr_tests tests via coverage (ptr.py:637)
[2021-05-29 09:30:58,238] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
[2021-05-29 09:30:58,341] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
[2021-05-29 09:30:58,436] INFO: Running usort for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
-- Summary (total time 2s):
✅ PASS: 0
❌ FAIL: 1
️⌛ TIMEOUT: 0
🔒 DISABLED: 0
💩 TOTAL: 1
-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running
-- Failure Output --
/Users/cooper/repos/ptr/setup.py (failed 'usort_run' step):
Would sort /Users/cooper/repos/ptr/setup.py
ptr developers recommend that if you want output, please cause a test to fail
raise ZeroDivisionErrorsetup.py test using a ptr created venv:
cd to/my/code/tmp/venv/bin/python setup.py testrequirements.txt to pre-install before running ptrpip PEP 440 version specifiers are supportedptr attempts to update from a PyPI compatible mirror (PEP 381) or PyPI itself-k or --venv option to no recreate a virtualenv each run when debugging your tests!pyre on Windows?pyre (pyre-check on PyPI) does not ship a Windows wheel with the ocaml pyre.bincoverage 5.0 introduced using sqlite and we don't want to have a mix of 4.x and 5.x for ptrTo chat in real time, hit us up on IRC. Otherwise, GitHub issues are always welcome!
IRC: #pythontestrunner on FreeNode
See the CONTRIBUTING file for how to help out.
ptr is MIT licensed, as found in the LICENSE file.
Copyright © Meta Platforms, Inc. and affiliates
ptr Change HistoryEach release to PyPI I'm going to give a codename as to where I am or was in the world 🌏.
Codename: Seattle 🦑
Release named in honoUr of my visit to Seattle last week ...
pyproject.toml user config support - PR #120pyproject.toml basic PEP517 building support - PR #117Codename: Richie 2 for 22 🏏
Release named in honoUr of the Late Great Richie Benaud
asyncio.run for starting asynciopylint in CI
3.11 CI is failing due to depedencies. Will watch and fix asap.
Codename: A 🅰️
Release to support 'Three dot Ten' and A == 10 in hexidecimal
Codename: Memorial Day 🇺
Release on Memorial Day long weekend
Codename: Wildwood 🪵
First release from my South Lake Tahoe residence
required_coverage - PR #108--print-cov to work when required_coverage does not exist in ptr config - PR #109Codename: Month after Straya Day 🇦🇺🇺
1 month since @cooperlees was in AU for Australia day!
Codename: Vernon Hills, IL 🎅
@cooperlees visiting girlfriend's family in IL
Codename: College Park, MD 🐢
@omikader studied at the University of Maryland, College Park
.ptrconfig files - PR: #83Codename: Sapporo, Japan 🇯🇵
@omikader is going skiing there in February
--system-site-packages during venv creation - PR: #80Codename: Santa Clara, CA
@thatch lives there!*
Codename: Russian River, CA 🇺🇸
@cooperlees is going to the Russian River tomorrow
test optionCodename: Mumbai, India 🇮🇳
@spurav is from Mumbai, India
Codename: Cairo, Egypt 🇪🇬
@adhaamehab who first time contributed is from there! Thanks!
Codename: Stanford, CA
@cooperlees tailgated there yesterday 🚐🏈
Codename: Clowntown 🤡
--print-cov + --error-on-warningsCodename: Rathbone Square
@cooperlees is releasing this release from FB London Office
Codename: Ellicott City
@omikader who reported and tested the fix for AST parsing is from there. Ellicott City is also home to one of the oldest surviving train stations in the US!
id attribute - Issue: #54black by default on 3.7 now that it runs on > 3.7.2 - Issue: #41Codename: California City
An over developed desert with infrastructure and little people
--print-non-configured to find non ptr modules in repos - Issue: #50Codename: Concord
🛫 @cooperlees is playing Aussie Football @ Concord, CA today 🏈
disable test suites and add a --run-disabled option to force the run - Issue: #46Codename: Cleveland
🇺🇸 @cooperlees + @jreese are at PyCon US in Cleveland, OH 🦅
pyre Type Checking step support - Issue: #38 + #40Known Bug: black.exe does not run in Windows 3.7 - disabled by default on Python 3.7 on Windows
Codename: Jaipur
🇮🇳 @cooperlees releasing whilst in Jaipur, India for a wedding 💒
Known Bug: black.exe does not run in Windows 3.7 - disabled by default on Python 3.7 on Windows
Codename: Forbes
Forbes, NSW, Australia is the home of @aijayadams 👨🏻🦰🇦🇺
Codename: Carnival
🇧🇷 @cooperlees was in Rio de Janeiro, Brazil for Carnival 3 years ago today 🇧🇷
setup.cfg support for ptr_params - Issue: #1ci.py - Issue: #7setup.py URL to ptr GitHubsetup.py fixes - e.g. Classifiers + License informationCodename: Snowbird
Recent shredding of Snowbird, UT, USA took place 🏂 🇺🇸
FAQs
Parallel asyncio Python setup.(cfg|py) Test Runner
We found that ptr demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket’s threat research team has detected six malicious npm packages typosquatting popular libraries to insert SSH backdoors.
Security News
MITRE's 2024 CWE Top 25 highlights critical software vulnerabilities like XSS, SQL Injection, and CSRF, reflecting shifts due to a refined ranking methodology.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.