validata-table

Validata Table

Validata Table is python package used as a tabular data validation service.

It includes four subpackages, where you can find their respective documentations :

• validata_core
• validata_ui
• validata_api
• tests used for testing the project (in development environment or in continuous integration)

It offers also a command line tool validata used to validate tabular data. See validata_core/README.md for more details.

Using validata-table package

You can use locally this package validata-table, doing:

pip install validata-table 

This allows you to use validata command line tool to validate tabular data:

validata --help

Development

This project is based on Docker to use a local developement environement.

This project includes a Makefile, which allows you to run predefined actions by running specific commands.

Dependency management tool used : Poetry version 1.6.1

Requirements

First install Docker Engine and docker-compose >= version 2 on your machine if not already done.

Then you may clone source code in your development directory:

git clone https://gitlab.com/validata-table/validata-table.git
cd validata-table

Run on a local development environment

Configuration is done by editing environment variables in .env, (see .env.example file to set .env file)

Warning: Docker env files do not support using quotes around variable values!

Launch the local development environment, thanks to the makefile command:

# in validata-table/
make serve_dev

This launches two docker containers:

• validata-api-dev
• validata-ui-dev

Validata Table API (using validata-table-api-dev docker container)

To access to the API of Validata Table click on http://localhost:5000/

Try Validata Table API

Validata Table UI (using validata-table-ui-dev docker container)

To access to the API of Validata Table click on http://localhost:5001/

Validata Table command line tool (using validata-table-api-dev docker container)

To use validata command line tool in the docker development environment, run:

docker exec -it validata-api-dev bash
validata --help

Test

To launch tests in the development environment, run:

make test

Linting

Some code linting tools are configured for this project:

• black: to format code, run make black

• isort: to format import, run make isort

• flake8: to enforce style coding, run make flake8

• pyright: to check static types. pyright will be executed in local virtual environment with poetry:

  First you need to create a virtual environment .venv at the root of the project and configure it:

# At /validata-table/ root 
python3.9 -m venv .venv  # install virtual environement locally
poetry config virtualenvs.in-project true
poetry config --list # Check if correctly configured
>>>
...
virtualenvs.in-project = true
...

poetry install  # install project dependencies

Then execute locally pyright with poetry:

poetry run pyright .

Continuous Integration

The continuous integration is configured in .gitlab-ci.yml file

Release a new version

On master branch :

• Update version in pyproject.toml and CHANGELOG.md files
• Update version Docker images used in docker-compose.yml file:
  • registry.gitlab.com/validata-table/validata-table/validata-table-api:vX.X.X
  • registry.gitlab.com/validata-table/validata-table/validata-table-ui:vX.X.X
• Update CHANGELOG.md
• Commit changes using Release as commit message
• Create git tag (starting with "v" for the release) git tag -a
• Git push: git push && git push --tags
• Check that pypi package is created and container images for validata_ui and validata_api are well-built (validata-table pipelines)

Creating and pushing a new release will trigger the pipeline in order to automatically:

• publish a new version of validata-table package on PyPI
• build a new tag of the Docker image validata-table-ui, based on the new version just created of the package validata-table, and publish it on the gitlab container registry validata-table-ui, used to run user interface Validata
• build a new tag of the Docker image validata-table-api, based on the new version just created of the package validata-table and publish it on the gitlab container registry validata-table-api, used to run the API of Validata

This pipeline runs when a new tag under the format 'vX.X.X' is pushed.

Deploy to production

You can deploy this project on your own production server by using Docker.

Production environment is based on Docker images validata-table-uiand validata-table-api hosted on gitlab container registries validata-table-ui and validata-table-api

To deploy in production, you can follow these steps described below.

First you may clone source code in your deployment directory:

git clone https://gitlab.com/validata-table/validata-table.git
cd validata-table

Configuration is done by editing environment variables in .env, (see .env.example file to set .env file).

Warning: Docker env files do not support using quotes around variable values !

Launch the docker production environment with makefile:

make serve_prod

OR launch the docker production environment with docker compose command:

docker compose -f docker-compose.yml up --build -d

Then:

• To access to the API of Validata Table click on http://localhost:<PORT_NUMBER_API>/ (replacing PORT_NUMBER_API with the value you choose)
• To access to the UI of Validata Table click on http://localhost:<PORT_NUMBER_UI>/ (replacing PORT_NUMBER_UI with the value you choose)
• To access to the validata command lines tool:

docker exec -it validata-table-api bash
validata --help

History

To keep track of the project's history, Validata Table comes from the merge of four gitlab repositories :

• Validata core
• Validata UI
• Validata API
• Validata Docker