pytest-rerunfailures
Readme
.. contents::
pytest-rerunfailures is a plugin for pytest <https://pytest.org>_ that
re-runs tests to eliminate intermittent failures.
.. image:: https://img.shields.io/badge/license-MPL%202.0-blue.svg :target: https://github.com/pytest-dev/pytest-rerunfailures/blob/master/LICENSE :alt: License .. image:: https://img.shields.io/pypi/v/pytest-rerunfailures.svg :target: https://pypi.python.org/pypi/pytest-rerunfailures/ :alt: PyPI .. image:: https://github.com/pytest-dev/pytest-rerunfailures/workflows/Test/badge.svg :target: https://github.com/pytest-dev/pytest-rerunfailures/actions :alt: GitHub Actions
You will need the following prerequisites in order to use pytest-rerunfailures:
This plugin can recover from a hard crash with the following optional prerequisites:
This package is currently tested against the last 5 minor pytest releases. In case you work with an older version of pytest you should consider updating or use one of the earlier versions of this package.
To install pytest-rerunfailures:
.. code-block:: bash
$ pip install pytest-rerunfailures
If one or more tests trigger a hard crash (for example: segfault), this plugin
will ordinarily be unable to rerun the test. However, if a compatible version of
pytest-xdist is installed, and the tests are run within pytest-xdist using the -n
flag, this plugin will be able to rerun crashed tests, assuming the workers and
controller are on the same LAN (this assumption is valid for almost all cases
because most of the time the workers and controller are on the same computer).
If this assumption is not the case, then this functionality may not operate.
To re-run all test failures, use the --reruns command line option with the
maximum number of times you'd like the tests to run:
.. code-block:: bash
$ pytest --reruns 5
Failed fixture or setup_class will also be re-executed.
To add a delay time between re-runs use the --reruns-delay command line
option with the amount of seconds that you would like wait before the next
test re-run is launched:
.. code-block:: bash
$ pytest --reruns 5 --reruns-delay 1
To re-run only those failures that match a certain list of expressions, use the
--only-rerun flag and pass it a regular expression. For example,
the following would only rerun those errors that match AssertionError:
.. code-block:: bash
$ pytest --reruns 5 --only-rerun AssertionError
Passing the flag multiple times accumulates the arguments, so the following
would only rerun those errors that match AssertionError or ValueError:
.. code-block:: bash
$ pytest --reruns 5 --only-rerun AssertionError --only-rerun ValueError
To re-run only those failures that do not match a certain list of expressions, use the
--rerun-except flag and pass it a regular expression. For example,
the following would only rerun errors other than that match AssertionError:
.. code-block:: bash
$ pytest --reruns 5 --rerun-except AssertionError
Passing the flag multiple times accumulates the arguments, so the following
would only rerun those errors that does not match with AssertionError or OSError:
.. code-block:: bash
$ pytest --reruns 5 --rerun-except AssertionError --rerun-except OSError
.. note::
When the AssertionError comes from the use of the assert keyword,
use --rerun-except assert instead::
$ pytest --reruns 5 --rerun-except assert
To mark individual tests as flaky, and have them automatically re-run when they
fail, add the flaky mark with the maximum number of times you'd like the
test to run:
.. code-block:: python
@pytest.mark.flaky(reruns=5) def test_example(): import random assert random.choice([True, False])
Note that when teardown fails, two reports are generated for the case, one for the test case and the other for the teardown error.
You can also specify the re-run delay time in the marker:
.. code-block:: python
@pytest.mark.flaky(reruns=5, reruns_delay=2) def test_example(): import random assert random.choice([True, False])
You can also specify an optional condition in the re-run marker:
.. code-block:: python
@pytest.mark.flaky(reruns=5, condition=sys.platform.startswith("win32")) def test_example(): import random assert random.choice([True, False])
Exception filtering can be accomplished by specifying regular expressions for
only_rerun and rerun_except. They override the --only-rerun and
--rerun-except command line arguments, respectively.
Arguments can be a single string:
.. code-block:: python
@pytest.mark.flaky(rerun_except="AssertionError") def test_example(): raise AssertionError()
Or a list of strings:
.. code-block:: python
@pytest.mark.flaky(only_rerun=["AssertionError", "ValueError"]) def test_example(): raise AssertionError()
You can use @pytest.mark.flaky(condition) similarly as @pytest.mark.skipif(condition), see pytest-mark-skipif <https://docs.pytest.org/en/6.2.x/reference.html#pytest-mark-skipif>_
.. code-block:: python
@pytest.mark.flaky(reruns=2,condition="sys.platform.startswith('win32')")
def test_example():
import random
assert random.choice([True, False])
# totally same as the above
@pytest.mark.flaky(reruns=2,condition=sys.platform.startswith("win32"))
def test_example():
import random
assert random.choice([True, False])
Note that the test will re-run for any condition that is truthy.
Here's an example of the output provided by the plugin when run with
--reruns 2 and -r aR::
test_report.py RRF
================================== FAILURES ================================== __________________________________ test_fail _________________________________
def test_fail():
assert False
E assert False
test_report.py:9: AssertionError ============================ rerun test summary info ========================= RERUN test_report.py::test_fail RERUN test_report.py::test_fail ============================ short test summary info ========================= FAIL test_report.py::test_fail ======================= 1 failed, 2 rerun in 0.02 seconds ====================
Note that output will show all re-runs. Tests that fail on all the re-runs will be marked as failed.
flaky <https://pypi.org/project/flaky/>_, you can only have
pytest-rerunfailures or flaky but not both.Issue Tracker <https://github.com/pytest-dev/pytest-rerunfailures/issues>_Code <https://github.com/pytest-dev/pytest-rerunfailures/>_Test execution count can be retrieved from the execution_count attribute
in test item's object. Example:
.. code-block:: python
@hookimpl(tryfirst=True) def pytest_runtest_makereport(item, call): print(item.execution_count)
Bug fixes +++++++++
#234 <https://github.com/pytest-dev/pytest-rerunfailures/issues/234>)
and (#241 <https://github.com/pytest-dev/pytest-rerunfailures/issues/241>)Breaking changes ++++++++++++++++
Drop support for Python 3.7.
Drop support for pytest < 7.2.
Features ++++++++
Breaking changes ++++++++++++++++
Features ++++++++
Bug fixes +++++++++
Fix crashitem names mismatch between client and server.
(#172 <https://github.com/pytest-dev/pytest-rerunfailures/issues/172>_)
Fix crash when setup fails with --rerun-except flag.
(#230 <https://github.com/pytest-dev/pytest-rerunfailures/issues/230>_)
Breaking changes ++++++++++++++++
Features ++++++++
Add only_rerun and rerun_except arguments to @pytest.mark.flaky marker.
Add support for pytest 7.3, 7.4.
Bug fixes +++++++++
--only-rerun
pattern (if given) and none of the --rerun-except patterns. Previously,
using both --only-rerun and --rerun-except together could cause
failures to be rerun even if they did not match any --only-rerun
pattern, and when using multiple --rerun-except patterns, all failures
would be rerun unless they matched every pattern.
(#225 <https://github.com/pytest-dev/pytest-rerunfailures/issues/225>_)Bug fixes +++++++++
Bug fixes +++++++++
Fix crash during teardown when runtest protocol hook is overwritten by another plugin.
Fix crash during teardown when TestCase class is used as base class.
Bug fixes +++++++++
Features ++++++++
reruns and reruns_delay through pytest.ini file.Breaking changes ++++++++++++++++
Drop support for Python 3.6.
Drop support for pytest < 6.
Bug fixes +++++++++
Fix crash when pytest-xdist is installed but disabled.
(Thanks to @mgorny <https://github.com/mgorny>_ for the PR.)
Fix crash when xfail(strict=True) mark is used with --rerun-only flag.
Features ++++++++
Added option --rerun-except to rerun failed tests those are other than the mentioned Error.
Add support for Python 3.11.
Add support for pytest 7.0, 7.1, 7.2.
Features ++++++++
@hugovk <https://github.com/hugovk>_ for the PR.)Features ++++++++
str as condition for
@pytest.mark.flaky(condition)
which gets evaluated dynamically similarly to
@pytest.mark.skipif(condition).
(#162 <https://github.com/pytest-dev/pytest-rerunfailures/pull/162>_
provided by @15klli <https://github.com/15klli>_)Backwards incompatible changes ++++++++++++++++++++++++++++++
Drop support for Python 3.5.
Drop support for pytest < 5.3.
Features ++++++++
Add condition keyword argument to the re-run marker.
(Thanks to @BeyondEvil_ for the PR.)
Add support for Python 3.9.
(Thanks to @digitronik_ for the PR.)
Add support for pytest 6.3.
(Thanks to @bluetech_ for the PR.)
Add compatibility with pytest-xdist >= 2.0.
(Thanks to @bluetech_ for the PR.)
Other changes +++++++++++++
.. _@BeyondEvil: https://github.com/BeyondEvil .. _@digitronik: https://github.com/digitronik .. _@bluetech: https://github.com/bluetech
Compatibility fix. ++++++++++++++++++
Ignore --result-log command line option when used together with ``pytest
= 6.1.0``, as it was removed there. This is a quick fix, use an older version of pytest, if you want to keep this feature for now. (Thanks to
@ntessore_ for the PR)
Support up to pytest 6.1.0.
.. _@ntessore: https://github.com/ntessore
Features ++++++++
--only-rerun to allow for users to rerun only certain
errors.Other changes +++++++++++++
Drop dependency on mock.
Add support for pre-commit and add a linting tox target.
(#117 <https://github.com/pytest-dev/pytest-rerunfailures/pull/117>)
(PR from @gnikonorov)
.. _@gnikonorov: https://github.com/gnikonorov
Backwards incompatible changes ++++++++++++++++++++++++++++++
Drop support for pytest version 4.4, 4.5 and 4.6.
Drop support for Python 2.7.
Features ++++++++
Add support for pytest 5.4.
Add support for Python 3.8.
Backwards incompatible changes ++++++++++++++++++++++++++++++
Drop support for pytest version 3.10, 4.0, 4.1, 4.2 and 4.3
Drop support for Python 3.4.
Features ++++++++
Bug fixes +++++++++
#98 <https://github.com/pytest-dev/pytest-rerunfailures/pull/98>)
(PR from @Eric-Arellano).. _@Eric-Arellano: https://github.com/Eric-Arellano
Backwards incompatible changes ++++++++++++++++++++++++++++++
Features ++++++++
Bug fixes +++++++++
pytest_runtest_logfinish hooks.
(#83 <https://github.com/pytest-dev/pytest-rerunfailures/issues/83>)
(PR from @KillAChicken).. _@KillAChicken: https://github.com/KillAChicken
Backwards incompatible changes ++++++++++++++++++++++++++++++
Features ++++++++
Bug fixes +++++++++
rerun
attribute on the test report.
(#77 <https://github.com/pytest-dev/pytest-rerunfailures/issues/77>)
(Thanks to @RibeiroAna for the PR)... _@RibeiroAna: https://github.com/RibeiroAna
Drop support for pytest versions < 3.6 to reduce the maintenance burden.
Add support up to pytest version 3.10. Thus supporting the newest 5 pytest releases.
Add support for Python 3.7.
Fix issue can occur when used together with pytest-flake8
(#73 <https://github.com/pytest-dev/pytest-rerunfailures/issues/73>_)
Fixed #64 issue related to setup_class and fixture executions on
rerun (Thanks to @OlegKuzovkov_ for the PR).
Added new execution_count attribute to reflect the number of test case
executions according to #67 issue. (Thanks to @OlegKuzovkov_ for the PR).
.. _@OlegKuzovkov: https://github.com/OlegKuzovkov
Node.get_closest_marker() (Thanks to
@The-Compiler_ for the PR)... _@The-Compiler: https://github.com/The-Compiler
Added option to add a delay time between test re-runs (Thanks to @Kanguros_
for the PR).
Added support for pytest >= 3.3.
Drop support for pytest < 2.8.7.
.. _@Kanguros: https://github.com/Kanguros
@davehunt_ for the PR).. _@davehunt: https://github.com/davehunt
Add support for Python 3.6.
Add support for pytest 2.9 up to 3.2
Drop support for Python 2.6 and 3.3.
Drop support for pytest < 2.7.
--rerun=0. (Thanks to @sublee_ for the PR).. _@sublee: https://github.com/sublee
Add default value of reruns=1 if pytest.mark.flaky() is called
without arguments.
Also offer a distribution as universal wheel. (Thanks to @tltx_ for the PR)
.. _@tltx: https://github.com/tltx
Prepare CLI options to pytest 3.0, to avoid a deprecation warning.
Fix error due to missing CHANGES.rst when creating the source distribution by adding a MANIFEST.in.
--resultlog option by parsing reruns accordingly. (#28)Rewrite to use newer API of pytest >= 2.3.0
Improve support for pytest-xdist by only logging the final result. (Logging intermediate results will finish the test rather rerunning it.)
FAQs
pytest plugin to re-run tests to eliminate flaky failures
We found that pytest-rerunfailures demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 7 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
Security News
As cyber threats become more autonomous, AI-powered defenses are crucial for businesses to stay ahead of attackers who can exploit software vulnerabilities at scale.
Security News
UnitedHealth Group disclosed that the ransomware attack on Change Healthcare compromised protected health information for millions in the U.S., with estimated costs to the company expected to reach $1 billion.