datarules

DataRules

Goal and motivation

The idea of this project is to define rules to validate and correct datasets. Whenever possible, it does this in a vectorized way, which makes this library fast.

Reasons to make this:

• Implement an alternative to https://github.com/data-cleaning/ based on python and pandas.
• Implement both validation and correction. Most existing packages provide validation only.
• Support a rule based way of data processing. The rules can be maintained in a separate file (python or yaml) if required.
• Apply vectorization to make processing fast.

Usage

This package provides two operations on data:

• checks (if data is correct). Also knows as validations.
• corrections (how to fix incorrect data)

Checks

In checks.py

from datarules import check


@check(tags=["P1"])
def check_almost_square(width, height):
    return (width - height).abs() <= 4


@check(tags=["P3", "completeness"])
def check_not_too_deep(depth):
    return depth <= 2

In your main code:

import pandas as pd
from datarules import CheckList

df = pd.DataFrame([
    {"width": 3, "height": 7},
    {"width": 3, "height": 5, "depth": 1},
    {"width": 3, "height": 8},
    {"width": 3, "height": 3},
    {"width": 3, "height": -2, "depth": 4},
])

checks = CheckList.from_file('checks.py')
report = checks.run(df)
print(report)

Output:

                  name                           condition  items  passes  fails  NAs error  warnings
0  check_almost_square  check_almost_square(width, height)      5       3      2    0  None         0
1   check_not_too_deep           check_not_too_deep(depth)      5       1      4    0  None         0

Corrections

In corrections.py

from datarules import correction
from checks import check_almost_square


@correction(condition=check_almost_square.fails)
def make_square(width, height):
    return {"height": height + (width - height) / 2}

In your main code:

from datarules import CorrectionList

corrections = CorrectionList.from_file('corrections.py')
report = corrections.run(df)
print(report)

Output:

          name                                 condition                      action  applied error  warnings
0  make_square  check_almost_square.fails(width, height)  make_square(width, height)        2  None         0

Similar work (python)

These work on pandas, but only do validation:

• Pandera - Like us, their checks are also vectorized.
• Pandantic - Combination of validation and parsing based on pydantic.

The following offer validation only, but none of them seem to be vectorized or support pandas directly.

• Great Expectations - An overengineered library for validation that has confusing documentation.
• contessa - Meant to be used against databases.
• validator
• python-valid8
• pyruler - Dead project that is rule-based.
• pyrules - Dead project that supports rule based corrections (but no validation).

Similar work (R)

This project is inspired by https://github.com/data-cleaning/. Similar functionality can be found in the following R packages:

• validate - Checking data (implemented)
• dcmodify - Correcting data (implemented)
• errorlocate - Identifying and removing errors (not yet implemented)
• deductive - Deductivate correction based on checks (not yet implemented)

Features found in one of the packages above but not implemented here, might eventually make it into this package too.