fts3

FTS3-REST-Flask

FTS3-REST-Flask is the RESTful API for the FTS3 Service.

This is the front-facing component to communicate with the FTS3 Service, allowing submission of transfers, querying transfer updates, as well as configuration management.

The project consists of an Apache module running Python3 + Flask. It is also a direct continuation of the former fts-rest, where the migration decision and evaluation can be seen here.

The officially supported platform is Alma9. The server part is provided via RPMs for the official platform. The client is provided via EPEL for multiple platforms, as well as via PyPi/fts3.

Installation

The project should be installed via RPMs, provided via the FTS repository.

Installing the client

Make sure to have the FTS repositories enabled:

• production repository: official RPMs deemed ready for production
• development repository: RPMs built on the latest develop branch commit

Installing the client:

$ dnf install -y fts-rest-client

Installing the server

The project relies on mod_wsgi built for Python 3.9. Some of the dependencies require the crb (CodeReady Linux Builder) repository to be enabled. All requirements are specified and should be brought-in via the spec file.

$ dnf config-manager --enable crb
$ dnf install -y fts-rest-server

Configuring the server is done via the following two configuration files:

• /etc/fts3/fts3restconfig
• /etc/htpd/conf.d/fts3rest.conf

Development

The project is tracked using JIRA, under the FTS Project (requires CERN log-in). Soon the project will be mirrored on GitHub, where issues and pull requests are also accepted (and encouraged!).

For development purposes, using a virtual environment is encouraged.

Git Workflow

This project follows a simplified GitFlow model. The main branch is the develop branch, with master being reserved only for tagged released.

Feature branches are developed separately and merged into the develop branch. When preparing a release, either develop is merged directly into master or a release branch is created. Production hotfixes are allowed only in case of necessity. Both release and hotfix branch changes must be merged back into develop.

Development workflow

When taking on a task, the following workflow model should be followed:

1. Create a JIRA ticket for the identified problem
2. Estimate the magnitude of the task:
   • Small changes can go directly on the develop branch
   • Larger changes must go into a separate feature branch
3. Implement and test your changes
4. Merge them into the develop branch:
   • Merge large changes via merge requests and request code review
   • Changes affecting the database must also be code reviewed!
5. Close the JIRA ticket and mark the appropriate fixVersion

Continuous Integration

The project uses Gitlab-CI for CI/CD. The pipeline runs for every push, in every branch:

• black: checks code must be formatted with black
• pylint: checks for syntax errors (runs for every supported Python3 version)
  • If you are sure that pylint is mistaken, add # pylint: skip-file at the beginning of the relevant file
• functional tests: run for every supported Python3 version
• bandit: detects potential security issues in the code
  • Allowed to fail as it flags many false positives.
  • To ignore a false positive, append # nosec to the offending line
• build: RPM for the client and server, plus sdist and wheel for the client
• deploy: upload client and server RPM to the FTS testing repository

Merge requests will proceed only if the pipeline succeeds.
In case of emergency the pipeline can be skipped.

The pipeline runs in a container from the image tagged as ci. The dockerfile is in the .gitlab-ci directory and the image is hosted in the container registry of this project. The image contains the pre-installed Python environment in order to speed up CI execution. When new environment is desired (such as new or upgraded dependencies), a new image must be created via the .gitlab-ci/docker/create-ci-image.sh script.

During development

Developers may add the .gitlab-ci/pre-commit hook to their local repository, enabling the following checks before every commit:

• Run black to format the changed files
• Run pylint only on the changed files
  • Note: As pylint works better when it is run on the full project, some rules have been disabled
• Run radon and bandit only on the changed files
  • In case of false positives, the hook can be skipped with the commit option --no-verify

A convenience script is provided to install the pre-commit hook:

$ cd .gitlab-ci/
$ ./precommit_install.sh

Python dependencies

The project uses pip-tools to manage dependencies:

• requirements.in: list of dependencies for the production app
• dev-requirements.in: extra list of packages used for development (e.g. static code analysis)

Setting up the development environment

The pipcompile.sh script is provided to help set up a development environment:

$ virtualenv venv
$ source venv/bin/activate
(venv) $ pip install --upgrade pip
(venv) $ pip install pip-tools
(venv) $ ./pipcompile.sh --sync

Running pipcompile.sh will generate requirements.txt. Running the command with --sync will also synchronize the virtual environment with the requirements.

Additionally, you will need to configure the HTTPd config file. A template version for development is offered at src/fts3rest/fts3rest_dev_conf.in. To fill it with the right paths and install it, run the install-httpd-dev.sh script:

$ cd src/fts3rest/
$ ./install-httpd-dev.sh

Functional tests

The functional tests reside under src/fts3rest/fts3rest/tests/functional.

In oder to run the tests, a fully functional MySQL-compatible database is required. The database must have the appropriate FTS schema and the connection details configured in the FTS3 REST config file.

Example on Centos 7 with locally installed mariadb:

$ wget https://gitlab.cern.ch/fts/fts3/-/raw/v3.11.0/src/db/schema/mysql/fts-schema-7.0.0.sql
$ mysql -u root --execute "CREATE DATABASE ftsflask;"
$ mysql -u root --execute "CREATE USER 'fts3'@'%';"
$ mysql -u root --execute "GRANT ALL PRIVILEGES ON ftsflask.* TO 'fts3'@'%' IDENTIFIED BY 'ftsflaskpass';"
$ mysql -u fts3 --password=ftsflaskpass ftsflask
MariaDB [ftsflask]> source fts-schema-7.0.0.sql

Before running the tests, make sure to include the fts3rest project in the PYTHONPATH.
Additionally, the FTS3TESTCONFIG environment variable can also be set to use a different config file during testing:

$ export PYTHONPATH="${PROJECT_DIR}/src:${PROJECT_DIR}/src/fts3rest"
$ export FTS3TESTCONFIG="${PROJECT_DIR}/src/fts3rest/fts3rest/tests/fts3testconfig"

For ease-of-use, the runtests.sh script is provided:

$ cd src/fts3rest
$ ./runtests.sh

Note: OIDC tests are not run in the CI as the container would need a client registered and this has not yet been set up.
To run OIDC tests in a development environment, the environment variables xdc_ClientId and xdc_ClientSecret must be set.

Building packages

In order to build RPM packages, two utility scripts are provided for the client and server: fts-rest-client.sh and fts-rest-server.sh.

The following example shows how to build server packages:

$ ./packaging/server/fts-rest-server.sh
$ tree "build/server/RPMS"

Useful links

• Documentation: https://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/index.html
• FTS Repository: https://fts-repo.web.cern.ch/fts-repo/
• Apache License, version 2.0: https://www.apache.org/licenses/LICENSE-2.0
• Support channels: [e-mail]
• Development team contact: [e-mail]