QALITA Command Line Interface (CLI) is a tool intended to be used by Data Engineers who setup's QALITA Platform's agents, sources and assets.
It gives easy to use command to help them make an up & running qalita platform's environment in no time.
As simple as :
pip install qalita
or there is a Docker Image available at :
If you want to have more detailed and contextual help, type
qalita COMMAND -h
Usage: qalita [OPTIONS] COMMAND [ARGS]...
QALITA Command Line Interface
This CLI command communicates with the QALITA Platform API backend.
There are several layers of configuration depending of your needs :
The agent will help you identify it in the frontend interface, there are no restrictions on the name.
The mode of the agent :
Job : In job mode, when you use the command qalita agent run, it will immediately try to run a job on the local current context.
Worker : In worker mode, when you use the command qalita agent run it will wait for the backend to gives him jobs to run. It is simmilar to a scheduler.
Note that the command
qalita agent runneeds more configuration to run correctly, it will displays error otherwise.
Example : http://localhost:3080/api/v1
The agent url endpoint gives the ability for the agent to communicate with the qalita's platform endpoints, it enables :
* Listing packs
* Running Jobs
* Publishing sources
* Publishing packs
The token is provided while doing the quickstart steps in the frontend app. It is associated with your user and your role.
Note that you need to have at least the [Data Engineer] role to use the QALITA CLI
You can alternatively make an .env file and export the values to your environment.
.env-local
QALITA_AGENT_NAME=<agent_name>
QALITA_AGENT_MODE=<job/worker>
QALITA_AGENT_ENDPOINT=https://api.company.com/api/v1
QALITA_AGENT_TOKEN=<api_token>
QALITA_PACK_NAME=<pack_name>
Then export the values of the file to ENV values with :
export $(xargs < .env-local)
The qalita agent command allow you to :
Parameters :
qalita agent login registers your local agent to the platform, it enables you to run jobs, or create routines (schedules) to run pack programmaticaly.
You need to have configured your agent with :
* QALITA_AGENT_ENDPOINT=<backend_api_url>
* QALITA_AGENT_TOKEN=<api_token>
You can get your token from the frontend or with an OAUTH2 API call to the /users/signin backend's endpoint
More info on your frontend documentation, and on the Connected config of the doc
Parameters :
Specific parameters in job mode :
qalita agent run
runs in different mode :
The agent will run given configuration
-p : a pack_id given with the qalita pack list, note that your pack needs to be pushed to the platform in order to have an id.-s : a source_id given with the qalita source list, note that your source needs to be pushed to the platform in order to have an id.The agent will wait until it receives an order from the frontend, it will then worke as same as in job mode.
Note that this mode will run indefinitely
Parameters :
List jobs from the platform backend.
Parameters :
Get infos about your local agent configuration.
The qalita pack command allow you to :
Parameters :
Initialize a new pack, you need to have set a name, it will create a new folder with the name of the pack.
You can set your name by passing a new parameters to the commandline or setting a new environment variable : QALITA_PACK_NAME=<my-super-pack>.
Here is the arborescence created :
./<pack-name>_pack/
/run.sh # Entrypoint file that will be run with qalita agent run
/README.md # Documentation file
/properties.yaml # Properties file that contains properties about the pack
/main.py # (pack specific) The main script (you can run your pack with whatever langage you choose)
/config.json # (pack specific) The config file of your pack, you can use it to set any configurations you like.
/requirements.txt # (pack specific) The requirements file that is run inside the run.sh
Parameters :
qalita agent loginList all the packs that are accessible to you with the Qalita Platform.
Parameters :
Run your locally configured pack
Parameters :
Validate your locally configured pack
Parameters :
Push your locally configured pack
The qalita source command allow you to :
Note , by default the
qalita-conf.ymlfile is stored to~/.qalita/qalita-conf.yml, setQALITA_HOMEenv to customize the default path.
This function will help you add a new source to your configuration file qalita-conf.yaml
This command doesn't have parameters, you need to follow command prompts.
file : Source pathdatabase : host / port / username / password / databaseAt the end of the prompt, the cli will check reachability of the source depending of the configuration and type, this step is called validate_source
to complete the process of registering a new source to the platform, you need to push your source with the command : qalita source push
Helper function to help you add a new source to your configuration file qalita-conf.yaml
Parameters :
You need to have a qalita-conf.yaml file that contains your sources configuration.
Exemple :
version: 1
sources:
- config:
path: /home/user/data_dir
description: Folder containing csv files
name: my_csv_files
owner: user
reference: false
visibility: private
sensitive: false
type: file
In this exemple we have :
General keys
| Key | Type | Description |
|---|---|---|
| version | int | The version of the configuration |
| sources | list | The list of sources |
Source keys
| Key | Type | Description |
|---|---|---|
| name | string | The name of the source |
| description | string | The description of the source |
| owner | string | The owner of the source |
| type | string | The type of the source |
| config | dict | The configuration of the source |
| visibility | string | The visibility of the source <private/internal/public> |
| reference | bool | Is the source a reference source |
| sensitive | bool | Is the source containing sensitive data |
Registers your sources to the platform
Note: If you want to run a pack on your source, you will first need to push your source to the platform. It will give you a source_id with which you can run your pack.
A pack is an entity run by the agent, it can be created by anyone.
It's purpose is to process the source and retrieve usefull informations about it to feed back into the platform.
To create the base pack, see : qalita pack init.
The entrypoint of the pack is the run.sh file that is located at the root path of the temp local folder created by the agent.
run.sh Example :
#/bin/bash
python -m pip install --quiet -r requirements.txt
python main.py
The pack is feed by a source_conf.json file containing the source's config: data. This file is located alongside the run.sh entrypoint.
source_conf.json Example :
{
"config": {
"path": "/home/lucas/desktop"
},
"description": "Desktop files",
"id": 1,
"name": "local_data",
"owner": "lucas",
"type": "file",
"reference": false,
"sensitive": false,
"visibility": "private",
"validate": "valid"
}
Note : The pack is responsible for managing itself it's
source typecompatibility by checking the source type in thesource_conf.jsonfile.
A the end of the pack run, the agent searchs for :
logs.txt : File uploaded to give feedback logs to the platform in the frontend.logs.txt Example :
2023-07-21 11:51:12,688 - qalita.commands.pack - INFO - ------------- Pack Run -------------
2023-07-21 11:51:15,087 - qalita.commands.pack - INFO - CSV files found:
2023-07-21 11:51:15,222 - qalita.commands.pack - ERROR - Summarize dataset: 0%| | 0/5 [00:00<?, ?it/s]
...
recommendations.jsonRecommendations file contains the recommendations given by the pack about the source.
recommendations.json Example :
{
[
{
"type":"<type of recommendation>",
"scope":"<scope : is a json>",
"content":"<any content>"
},
{
...
}
...
]
}
metrics.jsonMetrics file contains the metrics given by the pack about the source.
metrics.json Example :
{
[
{
"scope":"<scope : is a json>",
"key":"<metric key>",
"value":"<metric value>"
},
{
...
}
...
]
}
Metrics & recommendations are pushed to the platform and are then available to the source's pack run view.
FAQs
Qalita Command Line Interface
We found that qalita demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.