Python syntax highlighted Markdown doctest
Command line program and Python library to test Python syntax highlighted code examples in Markdown.
Website | Docs | Repos | pytest | Codecov | License
Introduction | Installation | Sample usage | Sample Usage with HTML comment directives | CI usage | --report | Identifying blocks | Directives | skip | label on code and sessions | label on any fenced code block | pytest skip | pytest skipif | setup | teardown | share-names | clear-names | pytest mark decorator | label skip and mark example | setup and teardown example | share-names clear-names example | Configuration | Inline annotations | skipping blocks with --skip | --skip | short form of --skip | --fail-nocode | --setup | --teardown | Setup example | Setup for sessions | Execution context | Send outfile to stdout | Usage | Run as a Python module | Python API | pytest fixtures | Simulate command line | Hints | Directive hints | Related projects
Changes | Contributions | About
It is advisable to install in a virtual environment.
python -m pip install phmdoctest
Given the Markdown file example1.md shown in raw form here...
# This is Markdown file example1.md
## Interactive Python session (doctest)
```py
>>> print("Hello World!")
Hello World!
```
## Source Code and terminal output
Code:
```python
from enum import Enum
class Floats(Enum):
APPLES = 1
CIDER = 2
CHERRIES = 3
ADUCK = 4
for floater in Floats:
print(floater)
```
sample output:
```
Floats.APPLES
Floats.CIDER
Floats.CHERRIES
Floats.ADUCK
```
the command...
phmdoctest doc/example1.md --outfile test_example1.py
creates the python source code file test_example1.py shown here...
"""pytest file built from doc/example1.md"""
from phmdoctest.functions import _phm_compare_exact
def session_00001_line_6():
r"""
>>> print("Hello World!")
Hello World!
"""
def test_code_14_output_28(capsys):
from enum import Enum
class Floats(Enum):
APPLES = 1
CIDER = 2
CHERRIES = 3
ADUCK = 4
for floater in Floats:
print(floater)
_phm_expected_str = """\
Floats.APPLES
Floats.CIDER
Floats.CHERRIES
Floats.ADUCK
"""
_phm_compare_exact(a=_phm_expected_str, b=capsys.readouterr().out)
Then run a pytest command something like this in your terminal to test the Markdown session, code, and expected output blocks.
pytest --doctest-modules
Or these two commands:
pytest
python -m doctest test_example1.py
The line_6 in the function name session_00001_line_6 is the
line number in example1.md of the first line
of the interactive session. 00001 is a sequence number to
order the doctests.
The 14 in the function name test_code_14_output_28 is the
line number of the first line
of python code. 28 shows the line number of the expected
terminal output.
One test case function gets generated for each:
The --report option below shows the blocks discovered and
how they are tested.
Given the Markdown file shown in raw form here...
<!--phmdoctest-mark.skip-->
<!--phmdoctest-label test_example-->
```python
print("Hello World!")
```
```
incorrect expected output
```
the command...
phmdoctest tests/one_mark_skip.md --outfile test_one_mark_skip.py
creates the python source code file shown here...
"""pytest file built from tests/one_mark_skip.md"""
import pytest
from phmdoctest.functions import _phm_compare_exact
@pytest.mark.skip()
def test_example(capsys):
print("Hello World!")
_phm_expected_str = """\
incorrect expected output
"""
_phm_compare_exact(a=_phm_expected_str, b=capsys.readouterr().out)
Run the --outfile with pytest...
$ pytest -vv test_one_mark_skip.py
test_one_mark_skip.py::test_example SKIPPED
Test Python examples in README.md in Continuous Integration scripts. In this snippet for Linux the pytest test suite is in the tests folder.
mkdir tests/tmp
phmdoctest README.md --report --outfile tests/tmp/test_readme.py
pytest --doctest-modules -vv tests
This console shows testing Python examples in project.md. Look for the tmp tests at the bottom. Windows Usage on Appveyor.
See this excerpt from ci.yml Actions usage example. It runs on Windows, Linux, and macOS. Please find the phmdoctest command at the bottom.
No changes to README.md are needed here, look in the last job log.
To see the GFM fenced code blocks in the MARKDOWN_FILE use the
--report option like this:
phmdoctest doc/example2.md --report
which lists the fenced code blocks it found in
the file example2.md.
The test role column shows how each fenced code block gets tested.
doc/example2.md fenced blocks
------------------------------------------------
block line test TEXT or directive
type number role quoted and one per line
------------------------------------------------
python 9 code
14 output
python 20 code
26 output
31 --
python 37 code
python 44 code
51 output
yaml 59 --
text 67 --
py 75 session
python 87 code
94 output
py 102 session
------------------------------------------------
7 test cases.
1 code blocks with no output block.
The PYPI commonmark project provides code to extract fenced code blocks from Markdown. Specification CommonMark Spec and website CommonMark.
Python code, expected output, and Python interactive sessions get extracted.
Only GFM fenced code blocks are considered.
A block is a session block if the info_string starts with py
and the first line of the block starts with the
session prompt: '>>> '.
To be treated as Python code the opening fence should start with one of these:
```python
```python3
```py3
plus the block contents can't start with '>>> '.
The examples use the info_strings python for code and py for sessions
since they render with coloring on GitHub, readthedocs, GitHub Pages,
and Python package index.
project.md has more examples of code and session blocks.
It is ok if the info string is laden with additional text, it will be ignored. The entire info string will be shown in the block type column of the report.
An output block is a fenced code block that immediately follows a Python block and starts with an opening fence like this which has an empty info string.
```
A Python code block has no output if it is followed by any of:
Test code gets generated for it, but there will be no assertion statement.
Directives are HTML comments containing test generation commands.
They are edited into the Markdown file immediately before a fenced
code block. It is OK if other HTML comments are present.
See the <!--phmdoctest-skip--> directive in the
raw Markdown below.
With the skip directive no test code will be
generated from the fenced code block.
<!--phmdoctest-skip-->
<!--Another HTML comment-->
```python
print("Hello World!")
```
Expected Output
```
Hello World!
```
List of Directives
Directive HTML comment | Use on blocks
---------------------------------- | ---------------------
<!--phmdoctest-skip--> | code, session, output
<!--phmdoctest-label IDENTIFIER--> | code, session
<!--phmdoctest-label TEXT--> | any
<!--phmdoctest-mark.skip--> | code
<!--phmdoctest-mark.skipif<3.N--> | code
<!--phmdoctest-setup--> | code
<!--phmdoctest-teardown--> | code
<!--phmdoctest-share-names--> | code
<!--phmdoctest-clear-names--> | code
<!--phmdoctest-mark.ATTRIBUTE--> | code
The skip directive or --skip TEXT command line option
prevents code generation for the code or session block.
The skip directive can be placed on an expected output block.
There it prevents checking expected against actual output.
Example.
When used on a Python code block or session the label directive changes the name of the generated test function. Example. Two generated tests, the first without a label, shown in pytest -v terminal output:
test_readme.py::test_code_93 FAILED
test_readme.py::test_beta_feature FAILED
On any fenced code block, the label directive identifies the block
for later retrieval by the class phmdoctest.tool.FCBChooser().
The FCBChooser is used separately from phmdoctest in
a different pytest file. This allows the test developer to write
additional test cases for fenced code blocks that are not handled by
phmdoctest. The directive value can be any string.
# This is file doc/my_markdown_file.md
<!--phmdoctest-label my-fenced-code-block-->
```
The label directive can be placed on any fenced code block.
```
Here is Python code to fetch it:
import phmdoctest.tool
chooser = phmdoctest.tool.FCBChooser("doc/my_markdown_file.md")
contents = chooser.contents(label="my-fenced-code-block")
print(contents)
Output:
The label directive can be placed on any fenced code block.
The <!--phmdoctest-mark.skip--> directive generates a test
case with a @pytest.mark.skip() decorator.
Example.
The <!--phmdoctest-mark.skipif<3.N--> directive generates
a test case with the pytest decorator
@pytest.mark.skipif(sys.version_info < (3, N), reason="requires >=py3.N").
N is a Python minor version number.
Example.
A single Python code block can assign names visible to other code blocks by adding a setup directive or using the --setup command line option.
Names assigned by the setup block get copied to the test module's global namespace after the setup block runs.
Here is an example setup block from setup.md:
import math
mylist = [1, 2, 3]
a, b = 10, 11
def doubler(x):
return x * 2
Using setup modifies the execution context of the
Python code blocks in the Markdown file.
The names math, mylist, a, b, and doubler are visible
to the other Python code blocks. The objects can be modified.
Example.
Selects a single Python code block that runs at test module teardown time. A teardown block can also be designated using the --teardown command line option. Example.
Names assigned by the Python code block get copied to the test module as globals after the test code runs. This happens at run time. These names are now visible to subsequent test cases generated for Python code blocks in the Markdown file. share-names modifies the execution context as described for the setup directive above. The share-names directive can be used on more than one code block. Example.
This directive effectively joins its Python code block to the following Python code blocks in the Markdown file.
After the test case generated for the Python code block with the clear-names directive runs, all names that were created by one or more preceding share-names directives get deleted. The names that were shared are no longer visible. This directive also deletes the names assigned by setup. Example.
The <!--phmdoctest-mark.ATTRIBUTE--> directive adds
a @pytest.mark.ATTRIBUTE decorator to the
generated test function. ATTRIBUTE is a valid Python attribute
identifier. This defines a marker to pytest that is used to
select and deselect tests. See the pytest documentation section
"Working with custom markers".
The file mark_example.md contains
example usage of the user defined marker "slow". It generates
test_mark_example.py
The file directive1.md contains
example usage of label, skip, and mark directives.
The command below generates
test_directive1.py.
phmdoctest doc/directive1.md --report
produces this
report.
phmdoctest doc/directive1.md --outfile test_directive1.py
The file directive2.md contains
example usage of label, skip, and mark directives.
The command below generates
test_directive2.py.
phmdoctest doc/directive2.md --report
produces this
report.
phmdoctest doc/directive2.md --outfile test_directive2.py
The file directive3.md contains
example usage of share-names and clear-names directives.
The command below generates
test_directive3.py.
phmdoctest doc/directive3.md --report
produces this
report.
phmdoctest doc/directive3.md --outfile test_directive3.py
Supply a .ini, .cfg, or .toml configuration file in place of the Markdown file. Configuration features:
Place a [tool.phmdoctest] section in the configuration file.
How to configure.
Inline annotations comment out sections of code. They can be added to the end of lines in Python code blocks. They should be in a comment.
phmdoctest:omit comments out a section of code. The line it is on,
plus following lines at greater indent get commented out.phmdoctest:pass comments out one line of code and prepends the pass statement.Here is a snippet showing how to place phmdoctest:pass in the code.
The second block shows the code that is generated. Note there is no #
immediately before phmdoctest:pass. It is not required.
import time
def takes_too_long():
time.sleep(100) # delay for awhile. phmdoctest:pass
takes_too_long()
import time
def takes_too_long():
pass # time.sleep(100) # delay for awhile. phmdoctest:pass
takes_too_long()
Use phmdoctest:omit on single or multi-line statements. Note the two
commented out time.sleep(99). They follow and are indented more
that the if condition:line with phmdoctest:omit.
import time # phmdoctest:omit
condition = True
if condition: # phmdoctest:omit
time.sleep(99)
time.sleep(99)
# import time # phmdoctest:omit
condition = True
# if condition: # phmdoctest:omit
# time.sleep(99)
# time.sleep(99)
Inline annotation processing counts the number of commented
out sections and adds the count as the suffix
_N to the name of the pytest function in the
generated test file.
Inline annotations are similar, but less powerful
than the Python standard library doctest directive #doctest+SKIP.
Improper use of phmdoctest:omit can cause Python syntax errors.
The examples above are snippets that illustrate how to use inline annotations. Here is an example that produces a pytest file from Markdown. The command below takes inline_example.md and generates test_inline_example.py.
phmdoctest doc/inline_example.md --outfile test_inline_example.py
If you don't want to generate test cases for Python
blocks precede the block with a skip directive or
use the --skip TEXT option. More than one skip directive
or--skip TEXTis allowed.
The following describes using --skip TEXT.
The code in each Python block gets searched
for the substring TEXT. Zero, one or more blocks will contain
the substring. These blocks will not generate test cases in the
output file.
The report shows which Python blocks get skipped in the test role column, and the Python blocks that matched each --skip TEXT in the skips section.
This option makes it very easy to inadvertently exclude
Python blocks from the test cases. In the event no test cases get
generated, the option --fail-nocode described below is useful.
Three special --skip TEXT strings work a little differently.
They select one of the first, second, or last of the Python blocks.
Only Python blocks get counted.
--skip FIRST skips the first Python block.--skip SECOND skips the second Python block.--skip LAST skips the final Python block.This command using --skip:
phmdoctest doc/example2.md --skip "Python 3.7" --skip LAST --report --outfile test_example2.py
Produces the report
doc/example2.md fenced blocks
-----------------------------------------------------
block line test TEXT or directive
type number role quoted and one per line
-----------------------------------------------------
python 9 code
14 output
python 20 skip-code "Python 3.7"
26 skip-output
31 --
python 37 code
python 44 code
51 output
yaml 59 --
text 67 --
py 75 session
python 87 code
94 output
py 102 skip-session "LAST"
-----------------------------------------------------
5 test cases.
1 skipped code blocks.
1 skipped interactive session blocks.
1 code blocks with no output block.
skip pattern matches (blank means no match)
------------------------------------------------
skip pattern matching code block line number(s)
------------------------------------------------
Python 3.7 20
LAST 102
------------------------------------------------
creates the output file test_example2.py
This is the same command as above using the short -s form of the --skip option
in two places.
It produces the same report and outfile.
phmdoctest doc/example2.md -s "Python 3.7" -sLAST --report --outfile test_example2.py
The --fail-nocode option produces a pytest file that will always
fail when no Python code or session blocks get found.
Evem if no Python code or session blocks exist in the
Markdown file a pytest file gets generated.
This also happens when --skip eliminates all the
Python code blocks.
The generated pytest file will have the function
def test_nothing_passes().
If the option --fail-nocode is passed the
function is def test_nothing_fails() which raises an
assertion.
A single Python code block can assign names visible to
other code blocks by giving the --setup TEXT option.
Please see the setup directive above.
The rules for TEXT are the same as for --skip TEXT plus...
TEXT.TEXT.A single Python code block can supply code run by the pytest
teardown_module() fixture. Use the --teardown TEXT option.
Please see the teardown directive above.
The rules for TEXT are the same as for --setup above except
TEXT won't match a setup block.
For the Markdown file setup.md run this command to see how the blocks get tested.
phmdoctest doc/setup.md --setup FIRST --teardown LAST --report
doc/setup.md fenced blocks
-------------------------------------------------
block line test TEXT or directive
type number role quoted and one per line
-------------------------------------------------
python 9 setup "FIRST"
python 20 code
27 output
python 37 code
42 output
python 47 code
51 output
python 58 teardown "LAST"
-------------------------------------------------
3 test cases.
This command
phmdoctest doc/setup.md --setup FIRST --teardown LAST --outfile test_setup.py
creates the test file test_setup.py
The pytest option --doctest-modules is required to
run doctest on sessions. pytest runs doctests in
a separate context.
For more on this see Execution context below.
To allow sessions to see the variables assigned by the --setup
code block, add the option --setup-doctest
Here is an example with setup code and sessions setup_doctest.md. The first part of this file is a copy of setup.md.
This command uses the short form of setup and teardown. -u for setup and -d for teardown.
phmdoctest doc/setup_doctest.md -u FIRST -d LAST --setup-doctest --outfile test_setup_doctest.py
It creates the test file test_setup_doctest.py
When run without --setup
--doctest-modules:
--setup--doctest-modules:
are not visible to the sessions.share-names--doctest-modules.clear-names
--setup and --setup-doctestSame as the setup section plus:
The live logging demos reveals pytest execution contexts. pytest Live Logs show the execution order of setup_module(), test cases, sessions, and teardown_module(). There are 2 demo invocations in the workflow action called pytest Live Log Demo. GitHub login required.
To redirect the above outfile to the standard output stream use one of these two commands.
Be sure to leave out --report when sending --outfile to standard output.
phmdoctest doc/example2.md -s "Python 3.7" -sLAST --outfile -
or
phmdoctest doc/example2.md -s "Python 3.7" -sLAST --outfile=-
phmdoctest --help
Usage: phmdoctest [OPTIONS] MARKDOWN_FILE
MARKDOWN_FILE may also be .toml, .cfg, or .ini configuration file.
Options:
--outfile TEXT Write generated test case file to path TEXT. "-" writes
to stdout.
-s, --skip TEXT Any Python code or interactive session block that
contains the substring TEXT is not tested. More than
one --skip TEXT is ok. Double quote if TEXT contains
spaces. For example --skip="python 3.7" will skip every
Python block that contains the substring "python 3.7".
If TEXT is one of the 3 capitalized strings FIRST
SECOND LAST the first, second, or last Python code or
session block in the Markdown file is skipped.
--report Show how the Markdown fenced code blocks are used.
--fail-nocode This option sets behavior when the Markdown file has no
Python fenced code blocks or interactive session blocks
or if all such blocks are skipped. When this option is
present the generated pytest file has a test function
called test_nothing_fails() that will raise an
assertion. If this option is not present the generated
pytest file has test_nothing_passes() which will never
fail.
-u, --setup TEXT The Python code block that contains the substring TEXT
is run at test module setup time. Variables assigned at
the outer level are visible as globals to the other
Python code blocks. TEXT should match exactly one code
block. If TEXT is one of the 3 capitalized strings
FIRST SECOND LAST the first, second, or last Python
code or session block in the Markdown file is matched.
A block will not match --setup if it matches --skip, or
if it is a session block. Use --setup-doctest below to
grant Python sessions access to the globals.
-d, --teardown TEXT The Python code block that contains the substring TEXT
is run at test module teardown time. TEXT should match
exactly one code block. If TEXT is one of the 3
capitalized strings FIRST SECOND LAST the first,
second, or last Python code or session block in the
Markdown file is matched. A block will not match
--teardown if it matches either --skip or --setup, or
if it is a session block.
--setup-doctest Make globals created by the --setup Python code block
or setup directive visible to session blocks and only
when they are tested with the pytest --doctest-modules
option. Please note that pytest runs doctests in a
separate context that only runs doctests. This option
is ignored if there is no --setup option.
--version Show the version and exit.
--help Show this message and exit.
To run phmdoctest from the command line:
python -m phmdoctest doc/example2.md --report
Call main.testfile() to generate a pytest file in memory. Please see the Python API here. The example generates a pytest file from doc/setup.md and compares the result to doc/test_setup.py.
from pathlib import Path
import phmdoctest.main
generated_testfile = phmdoctest.main.testfile(
"doc/setup.md",
setup="FIRST",
teardown="LAST",
)
expected = Path("doc/test_setup.py").read_text(encoding="utf-8")
assert expected == generated_testfile
Use fixture testfile_creator to generate a test file in memory.
Pass the test file to fixture testfile_tester to run
the test file in the pytester environment.
Fixture API | Example.
See more uses in tests/test_examples.py, tests/test_details.py, and
tests/test_many_markdown.py.
The fixtures run pytest much faster than run_and_pytest()
below since there is no subprocess call.
In the readthedocs documentation see the section Development tools API 1.4.0.
pytest's pytester is suitable for pytest plugin development.
To simulate a command line call to phmdoctest from
within a Python script phmdoctest.simulator offers the
function run_and_pytest().
Please see the Latest Development tools API section or
the docstring of the function run_and_pytest() in the file simulator.py.
Pass pytest_options as a list of strings as shown below.
import phmdoctest.simulator
command = "phmdoctest doc/example1.md --report --outfile temporary.py"
simulator_status = phmdoctest.simulator.run_and_pytest(
well_formed_command=command, pytest_options=["--doctest-modules", "-v"]
)
assert simulator_status.runner_status.exit_code == 0
assert simulator_status.pytest_exit_code == 0
To read the Markdown file from the standard input stream.
Use - for MARKDOWN_FILE.
Write the test file to a temporary directory so that it is always up to date.
In CI scripts the following shell command will create the temporary directory tmp in the tests folder on Windows, Linux, and macOS.
python -c "from pathlib import Path; d = Path('tests') / 'tmp'; d.mkdir(mode=0o700)"
It is easy to use --output by mistake instead of --outfile.
If Python code block has no output, put assert statements in the code.
Use pytest option --doctest-modules to test the sessions.
Markdown indented code blocks (Spec section 4.4) are ignored.
simulator_status.runner_status.exit_code == 2 is the click command line usage error.
Since phmdoctest generates code, the input file should be from a trusted source.
An empty code block gets given the role del-code. It is not tested.
Use special TEXT values FIRST, SECOND, LAST for the command
line options --setup and --teardown since they only match one block.
The variable names managenamespace, doctest_namespace,
capsys, and _phm_expected_str should not be used in
Markdown Python code blocks since they may be used in generated code.
Setup and teardown code blocks cannot have expected output.
To have pytest collect a code block with the label directive
start the value with test_.
With the --setup-doctest option, names assigned by the setup code
block are globally visible to the entire test suite.
This is due to the scope of the pytest doctest_namespace
fixture. Try using a separate pytest command to test
just the phmdoctest test.
The module phmdoctest.fixture is imported at pytest time to support setup, teardown, share-names, and clear-names features.
The phmdoctest Markdown parser finds fenced code blocks enclosed by
html <details> and </details> tags.
The tags may require a preceding and trailing blank line
to render correctly. See example in tests/test_details.py.
Try redirecting phmdoctest standard output into PYPI Pygments to colorize the generated test file.
python -m phmdoctest project.md --outfile - | pygmentize
If the --outfile is written into a folder that pre-exists in the repository, consider adding the outfile name to .gitignore. If the outfile name later changes, the change will be needed in .gitignore too.
# Reserved for generated test file.
tests/test_readme.py
--report start with a dash like
this: -label test_i_ratio.scope="module" level.-->. Running with --report
will expose that problem.--log-cli-level=DEBUG --color=yes<!--phmdoctest-mark.xfail--> might be useful as
an alternative to <!--phmdoctest-mark.skip--> for failing examples.<!--phmdoctest-mark.ATTRIBUTE--> will not be
effective when used with <!--phmdoctest-setup--> or
<!--phmdoctest-teardown--> because pytest marks can only
be applied to tests. They have no effect on fixtures.
Setup and teardown use fixtures.Copyright (c) 2021 Mark Taylor
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
FAQs
Python syntax highlighted Markdown doctest.
We found that phmdoctest demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.