Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
.. image:: https://img.shields.io/pypi/v/zope.testrunner.svg :target: https://pypi.org/project/zope.testrunner/ :alt: Latest release
.. image:: https://img.shields.io/pypi/pyversions/zope.testrunner.svg :target: https://pypi.org/project/zope.testrunner/ :alt: Supported Python versions
.. image:: https://github.com/zopefoundation/zope.testrunner/actions/workflows/tests.yml/badge.svg :target: https://github.com/zopefoundation/zope.testrunner/actions/workflows/tests.yml
.. image:: https://coveralls.io/repos/github/zopefoundation/zope.testrunner/badge.svg?branch=master :target: https://coveralls.io/github/zopefoundation/zope.testrunner?branch=master
.. image:: https://readthedocs.org/projects/zopetestrunner/badge/?version=latest :target: https://zopetestrunner.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status
.. contents::
This package provides a flexible test runner with layer support.
Detailed documentation is hosted at https://zopetestrunner.readthedocs.io
.. IMPORTANT: This document is included in the long_description for rendering on PyPI, so it cannot use Sphinx-only features.
zope.testrunner is often used for projects that use buildout_::
[buildout]
develop = .
parts = ... test ...
[test]
recipe = zc.recipe.testrunner
eggs = mypackage
The usual buildout process ::
python bootstrap.py
bin/buildout
creates a bin/test
script that will run the tests for mypackage.
.. tip::
zc.recipe.testrunner_ takes care to specify the right
``--test-path`` option in the generated script. You can add
other options (such as ``--tests-pattern``) too; check
zc.recipe.testrunner_'s documentation for details.
pip install zope.testrunner
and you'll get a zope-testrunner
script. Run your tests with ::
zope-testrunner --test-path=path/to/your/source/tree
Your source code needs to be available for the testrunner to import,
so you need to run python setup.py install
or pip install -e .
into the same virtualenv_.
-p show a progress indicator -v increase verbosity -c colorize the output -t test specify test names (one or more regexes) -m module specify test modules (one or more regexes) -s package specify test packages (one or more regexes) --list-tests show names of tests instead of running them -x stop on first error or failure -D, --pdb enable post-mortem debugging of test failures --xml path generate XML reports to be written at the given path --help show all command-line options (there are many more!)
For example ::
bin/test -pvc -m test_foo -t TestBar
runs all TestBar tests from a module called test_foo.py.
zope.testrunner
expects to find your tests inside your package
directory, in a subpackage or module named tests
. Test modules
in a test subpackage should be named test*.py
.
.. tip::
You can change these assumptions with ``--tests-pattern`` and
``--test-file-pattern`` test runner options.
Tests themselves should be classes inheriting from
unittest.TestCase
, and if you wish to use doctests, please tell
the test runner where to find them and what options to use for them
in by supplying a function named test_suite
.
Example::
import unittest
import doctest
class TestArithmetic(unittest.TestCase):
def test_two_plus_two(self):
self.assertEqual(2 + 2, 4)
def doctest_string_formatting():
"""Test Python string formatting
>>> print('{} + {}'.format(2, 2))
2 + 2
"""
def test_suite():
return unittest.TestSuite([
unittest.defaultTestLoader.loadTestsFromName(__name__),
doctest.DocTestSuite(),
doctest.DocFileSuite('../README.txt',
optionflags=doctest.ELLIPSIS),
])
In addition to per-package and per-module filtering, zope.testrunner has other mechanisms for grouping tests:
layers allow you to have shared setup/teardown code to be used
by a group of tests, that is executed only once, and not for each
test. Layers are orthogonal to the usual package/module structure
and are specified by setting the layer
attribute on test
suites.
levels allow you to group slow-running tests and not run them
by default. They're specified by setting the level
attribute
on test suites to an int.
zope.testrunner can profile your tests, measure test coverage, check for memory leaks, integrate with subunit_, shuffle the test execution order, and run multiple tests in parallel.
.. _buildout: https://buildout.readthedocs.io .. _virtualenv: https://virtualenv.pypa.io/ .. _zc.recipe.testrunner: https://pypi.python.org/pypi/zc.recipe.testrunner .. _subunit: https://pypi.python.org/pypi/python-subunit
Make tests compatible with Python 3.13 + add support for that version.
(#181 <https://github.com/zopefoundation/zope.testrunner/pull/181>
_)
Drop support for Python 3.7.
Remove setuptools fossils.
unittest.TestCase.subTest
support
(#91 <https://github.com/zopefoundation/zope.testrunner/issues/91>
_).
remove support for setup.py
's test
command.
Support for this command has been dropped by modern
setuptools
versions and correspondingly has been removed from
most zopefoundation
packages; zope.testrunner
now follows.
setup.py
's ftest
command is now only supported
when the used setuptools
version still supports test
.
#160 <https://github.com/zopefoundation/zope.testrunner/issues/160>
_)#163 <https://github.com/zopefoundation/zope.testrunner/issues/163>
_)Exit cleanly when using the test runner --version
argument.
(#102 <https://github.com/zopefoundation/zope.testrunner/issues/102>
_)
Add new --xml <path>
option to write JUnit-like XML reports.
Code comes from collective.xmltestreport
, but be aware that here --xml
is not a boolean, but expects a path!
(#148 <https://github.com/zopefoundation/zope.testrunner/issues/148>
_).
Add support for Python 3.13a3.
startTest
for skipped tests
(#157 <https://github.com/zopefoundation/zope.testrunner/issues/157>
_).Add support for Python 3.12.
Update code and tests to python-subunit >= 1.4.3
thus requiring at least
this version.
#149 <https://github.com/zopefoundation/zope.testrunner/issues/149>
_)Add support for Python 3.11.
Inline a small part of random.Random.shuffle
which was deprecated in
Python 3.9 and removed in 3.11 (#119 <https://github.com/zopefoundation/zope.testrunner/issues/119>
_).
Don't trigger post mortem debugger for skipped tests. ( #141 <https://github.com/zopefoundation/zope.testrunner/issues/141>
_).
--at-level=level
with level <= 0
run the tests
at all levels (rather than at no level)
#138 <https://github.com/zopefoundation/zope.testrunner/issues/138>
_.Use sys._current_frames
(rather than threading.enumerate
)
as base for new thread detection, fixes
#130 <https://github.com/zopefoundation/zope.testrunner/issues/130>
_.
New option --gc-after-test
. It calls for a garbage collection
after each test and can be used to track down ResourceWarning
s
and cyclic garbage.
With rv = gc.collect()
, !
is output on verbosity level 1 when
rv
is non zero (i.e. when cyclic structures have been released),
[
rv]
on higher verbosity levels and
a detailed cyclic garbage analysis on verbosity level 4+.
For details, see
#133 <https://github.com/zopefoundation/zope.testrunner/pull/133>
_.
Allow the filename for the logging configuration to be specified
via the envvar ZOPE_TESTRUNNER_LOG_INI
.
If not defined, the configuration continues to be locked for
in file log.ini
of the current working directory.
Remember the logging configuration file in envvar
ZOPE_TESTRUNNER_LOG_INI
to allow spawned child processes
to recreate the logging configuration.
For details, see
#134 <https://github.com/zopefoundation/zope.testrunner/pull/134>
_.
Improve --help
documentation for --package-path
option
(#121 <https://github.com/zopefoundation/zope.testrunner/pull/121>
_).
Do not disable existing loggers during logsupport initialization
(#120 <https://github.com/zopefoundation/zope.testrunner/pull/120>
_).
Fix tests with testtools >= 2.5.0 (#125 <https://github.com/zopefoundation/zope.testrunner/issues/125>
_).
Add support for Python 3.10.
Add support for Python 3.9.
Fix package init file missing
warning
(#112 <https://github.com/zopefoundation/zope.testrunner/pull/112>
_).
Make standard streams provide a buffer
attribute on Python 3 when using
--buffer
or testing under subunit.
Add support for Python 3.8.
When a layer is run in a subprocess, read its stderr in a thread to avoid
a deadlock if its stderr output (containing failing and erroring test IDs)
overflows the capacity of a pipe (#105 <https://github.com/zopefoundation/zope.testrunner/issues/105>
_).
Recover more gracefully when layer setUp or tearDown fails, producing useful subunit output.
Prevent a spurious warning from the --require-unique
option if the
--module
option was not used.
Add optional buffering of standard output and standard error during tests,
requested via the --buffer
option or enabled by default for subunit.
Fix incorrect failure counts in per-layer summary output, broken in 4.0.1.
Fix test failures and deprecation warnings occurring when using Python 3.8a1.
(#89 <https://github.com/zopefoundation/zope.testrunner/pull/89>
_)
Drop support for Python 3.4.
TypeError: a bytes-like object is required, not 'str'
running tests in parallel on Python 3. See issue 80 <https://github.com/zopefoundation/zope.testrunner/issues/80>
_.#81 <https://github.com/zopefoundation/zope.testrunner/issues/81>
).Drop support for Python 3.3.
Add support for Python 3.7.
Enable test coverage reporting on coveralls.io and in tox.ini.
Host documentation at https://zopetestrunner.readthedocs.io
Remove untested support for the --pychecker
option. See
issue 63 <https://github.com/zopefoundation/zope.testrunner/issues/63>
_.
Update the command line interface to use argparse
instead of
optparse
. See issue 61 <https://github.com/zopefoundation/zope.testrunner/issues/61>
_.
Use ipdb instead of pdb for post-mortem debugging if available
(#10 <https://github.com/zopefoundation/zope.testrunner/issues/10>
_).
Add a --require-unique option to check for duplicate test IDs. See
LP #682771 <https://bugs.launchpad.net/launchpad/+bug/682771>
_.
Reintroduce optional support for subunit
, now with support for both
version 1 and version 2 of its protocol.
Handle string in exception values when formatting chained exceptions.
(#74 <https://github.com/zopefoundation/zope.testrunner/pull/74>
_)
DeprecationWarning
earlier, when discovering test
modules. This lets warnings that are raised on import (such as those
produced by zope.deprecation.moved
) be reported. See issue 57 <https://github.com/zopefoundation/zope.testrunner/issues/57>
_.DeprecationWarning
when running tests. This
is recommended by the Python core developers and matches the
behaviour of the unittest
module. This can be overridden with
Python command-line options (-W
) or environment variables
(PYTHONWARNINGS
). See issue 54 <https://github.com/zopefoundation/zope.testrunner/issues/54>
_.subunit
.Make the subunit
support purely optional: applications which have
been getting the dependencies via zope.testrunner
should either add
zope.testrunner[subunit]
to their install_requires
or else
depend directly on python-subunit
.
New option --ignore-new-thread=<regexp>
to suppress "New thread(s)"
warnings.
Support Python 3.6.
Fixed: Using the -j
option to run tests in multiple processes
caused tests that used the multiprocessing
package to hang
(because the testrunner replaced sys.stdin
with an unclosable
object).
Drop conditional dependency on unittest2
(redundant after dropping
support for Python 2.6).
Stop tests for all layers when test fails/errors when started with
-x/--stop-on-error
(#37 <https://github.com/zopefoundation/zope.testrunner/pull/37>
_).
Drop support for Python 2.6 and 3.2.
Add support for Python 3.5
(#31 <https://github.com/zopefoundation/zope.testrunner/pull/31>
_).
Insert extra paths (from --path
) to the front of sys.argv
(#32 <https://github.com/zopefoundation/zope.testrunner/issues/32>
_).
-j
, parallelize all the tests, including the first test layer
(#28 <https://github.com/zopefoundation/zope.testrunner/issues/28>
_).Support skipped tests in subunit output
(#25 <https://github.com/zopefoundation/zope.testrunner/pull/25>
_).
More efficient test filtering
(#26 <https://github.com/zopefoundation/zope.testrunner/pull/26>
_).
#24 <https://github.com/zopefoundation/zope.testrunner/issues/24>
_).#20 <https://github.com/zopefoundation/zope.testrunner/pull/20>
_).Sort related layers close to each other to reduce the number of unnecessary
teardowns (fixes #14 <https://github.com/zopefoundation/zope.testrunner/issues/14>
_).
Run the unit test layer first (fixes LP #497871 <https://bugs.launchpad.net/zope.testrunner/+bug/497871>
__).
Drop support for Python 3.1.
Fix post-mortem debugging when a non-printable exception happens (https://github.com/zopefoundation/zope.testrunner/issues/8).
Updated boostrap.py
to version 2.2.
Fix nondeterministic test failures on Python 3.3
Tear down layers after post_mortem
debugging is finished.
Fix tests that write to source directory, it might be read-only.
Fix tests selection when the negative "!" pattern is used several times (LP #1160965)
Moved tests into a 'tests' subpackage.
Made python -m zope.testrunner
work again.
Support 'skip' feature of unittest2 (which became the new unittest in Python 2.7).
Better diagnostics when communication with subprocess fails (https://github.com/zopefoundation/zope.testrunner/issues/5).
Do not break subprocess execution when the test suite changes the working directory (https://github.com/zopefoundation/zope.testrunner/issues/6).
Count test module import errors as errors (LP #1026576).
python setup.py ftest
since it tried to run setup.py with all the
command line options. It now detects setup.py
runs and we run the test
runner directly.SkipLayers
class in cases where the distribution specifies a
test_suite
value.Fixed a bug in the ftest
command and added a test.
Fixed a trivial test failure with Python 3 of the previous release.
Expose ftest
distutils command via an entry point.
Added tests for zope.testrunner.eggsupport
.
six
_.Dropped use of zope.fixers (LP: #1118877).
Fixed tox test error reporting; fixed tests on Pythons 2.6, 3.1, 3.2, 3.3 and PyPy 1.9.
Fix --shuffle ordering on Python 3.2 to be the same as it was on older Python versions.
Fix --shuffle nondeterminism when multiple test layers are present. Note: this will likely change the order of tests for the same --shuffle-seed.
New option: --profile-directory. Use it in the test suite so that tests executed by detox in parallel don't conflict.
Use a temporary coverage directory in the test suite so that tests executed by detox in parallel don't conflict.
Fix --post-mortem (aka -D, --pdb) when a test module cannot be imported or is invalid (LP #1119363).
Replaced deprecated zope.interface.implements
usage with equivalent
zope.interface.implementer
decorator.
Dropped support for Python 2.4 and 2.5.
Made StartUpFailure compatible with unittest.TextTestRunner() (LP #1118344).
Added back Python 3 support which was broken in 4.0.1.
Fixed Unexpected success
_ support by implementing the whole concept.
Added support for the new pycache directories in Python 3.2.
Unexpected success
_ (concept introduced in Python 2.7) is
no longer handled as success but as failure. This is a workaround. The
whole unexpected success concept might be implemented later... _Unexpected success
: http://www.voidspace.org.uk/python/articles/unittest2.shtml#more-skipping
Update fix for LP #221151 to a spelling compatible with Python 2.4.
Timestamps are now always included in subunit output (r114849).
LP #591309: fix a crash when subunit reports test failures containing UTF8-encoded data.
zope.testrunner
itself).LP #221151: keep unittest.TestCase.shortDescription
happy by supplying
a _testMethodDoc
attribute.
LP #595052: keep the distribution installable under Python 2.4: its
distutils appears to munge the empty __init__.py
file in the
foo.bar
egg used for testing into a directory.
LP #580083: fix the bin/test
script to run only tests from
zope.testrunner
.
LP #579019: When layers were run in parallel, their tearDown was not called. Additionally, the first layer which was run in the main thread did not have its tearDown called either.
Having 'sampletests' in the MANIFEST.in gave warnings, but doesn't actually seem to include any more files, so I removed it.
Moved zope.testing.exceptions to zope.testrunner.exceptions. Now zope.testrunner no longer requires zope.testing except for when running its own tests.
.. _six: http://pypi.python.org/pypi/six
FAQs
Zope testrunner script.
We found that zope.testrunner demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 31 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.