pe

Parsing Expressions

---

pe is a library for parsing expressions, including parsing expression grammars (PEGs). It aims to join the expressive power of parsing expressions with the familiarity of regular expressions. For example:

>>> import pe
>>> pe.match(r'"-"? [0-9]+', '-38')  # match an integer
<Match object; span=(0, 3), match='-38'>

A grammar can be used for more complicated or recursive patterns:

>>> float_parser = pe.compile(r'''
...   Start    <- INTEGER FRACTION? EXPONENT?
...   INTEGER  <- "-"? ("0" / [1-9] [0-9]*)
...   FRACTION <- "." [0-9]+
...   EXPONENT <- [Ee] [-+]? [0-9]+
... ''')
>>> float_parser.match('6.02e23')
<Match object; span=(0, 7), match='6.02e23'>

Quick Links

• Documentation
  • Specification
  • Guides
  • API Documentation
  • FAQ
• Example Parsers

Features and Goals

• Grammar notation is backward-compatible with standard PEG with few extensions
• A specification describes the semantic effect of parsing (e.g., for mapping expressions to function calls)
• Parsers are often faster than other parsing libraries, sometimes by a lot; see the benchmarks
• The API is intuitive and familiar; it's modeled on the standard API's re module
• Grammar definitions and parser implementations are separate
  • Optimizations target the abstract grammar definitions
  • Multiple parsers are available (currently packrat for recursive descent and machine for an iterative "parsing machine" as from Medeiros and Ierusalimschy, 2008 and implemented in LPeg).

Syntax Overview

pe is backward compatible with standard PEG syntax and it is conservative with extensions.

# terminals
.            # any single character
"abc"        # string literal
'abc'        # string literal
[abc]        # character class

# repeating expressions
e            # exactly one
e?           # zero or one (optional)
e*           # zero or more
e+           # one or more
e{5}         # exactly 5
e{3,5}       # three to five

# combining expressions
e1 e2        # sequence of e1 and e2
e1 / e2      # ordered choice of e1 and e2
(e)          # subexpression

# lookahead
&e           # positive lookahead
!e           # negative lookahead

# (extension) capture substring
~e           # result of e is matched substring

# (extension) binding
name:e       # bind result of e to 'name'

# grammars
Name <- ...  # define a rule named 'Name'
... <- Name  # refer to rule named 'Name'

# (extension) auto-ignore
X <  e1 e2   # define a rule 'X' with auto-ignore

Matching Inputs with Parsing Expressions

When a parsing expression matches an input, it returns a Match object, which is similar to those of Python's re module for regular expressions. By default, nothing is captured, but the capture operator (~) emits the substring of the matching expression, similar to regular expression's capturing groups:

>>> e = pe.compile(r'[0-9] [.] [0-9]')
>>> m = e.match('1.4')
>>> m.group()
'1.4'
>>> m.groups()
()
>>> e = pe.compile(r'~([0-9] [.] [0-9])')
>>> m = e.match('1.4')
>>> m.group()
'1.4'
>>> m.groups()
('1.4',)

Value Bindings

A value binding extracts the emitted values of a match and associates it with a name that is made available in the Match.groupdict() dictionary. This is similar to named-capture groups in regular expressions, except that it extracts the emitted values and not the substring of the bound expression.

>>> e = pe.compile(r'~[0-9] x:(~[.]) ~[0-9]')
>>> m = e.match('1.4')
>>> m.groups()
('1', '4')
>>> m.groupdict()
{'x': '.'}

Actions

Actions (also called "semantic actions") are callables that transform parse results. When an arbitrary function is given, it is called as follows:

func(*match.groups(), **match.groupdict())

The result of this function call becomes the only emitted value going forward and all bound values are cleared.

For more control, pe provides the Action class and a number of subclasses for various use-cases. These actions have access to more information about a parse result and more control over the match. For example, the Pack class takes a function and calls it with the emitted values packed into a list:

func(match.groups())

And the Join class joins all emitted strings with a separator:

func(sep.join(match.groups()), **match.groupdict())

Auto-ignore

The grammar can be defined such that some rules ignore occurrences of a pattern between sequence items. Most commonly, this is used to ignore whitespace, so the default ignore pattern is simple whitespace.

>>> pe.match("X <- 'a' 'b'", "a b")  # regular rule does not match
>>> pe.match("X <  'a' 'b'", "a b")  # auto-ignore rule matches
<Match object; span=(0, 3), match='a b'>

This feature can help to make grammars more readable.

Example

Here is one way to parse a list of comma-separated integers:

>>> from pe.actions import Pack
>>> p = pe.compile(
...   r'''
...     Start  <- "[" Values? "]"
...     Values <- Int ("," Int)*
...     Int    <  ~( "-"? ("0" / [1-9] [0-9]*) )
...   ''',
...   actions={'Values': Pack(list), 'Int': int})
>>> m = p.match('[5, 10, -15]')
>>> m.value()
[5, 10, -15]

Similar Projects

• Lark (Python)
• nom (Rust)
• Parsimonious (Python)
• Rosie (Multiple bindings)
• TatSu (Python)
• PEG.js (Javascript)
• Pegged (D)
• pegen (Python / C)
• LPeg (Lua)