funding-service-design-utils
Advanced tools
Utils serves as a shared library for funding service design applications. These utilities are meant to be reused across the project, streamlining our coding process and ensuring the reliability of our funding service design applications.
This library can be installed into other python repos and the packages used by those repos.
This service depends on:
In order to run the unit tests, setup a virtual env and install requirements Developer setup guide
Install pre-commit hook: https://github.com/communitiesuk/funding-service-design-workflows/blob/main/readmes/python-repos-ide-setup.md
If you add any packages needed by services that consume fsd_utils, add them into pyproject.yaml.
Tests in this repository run using pytest, as per our standardise Testing in Python repos practice.
However, because this is a shared utils library that can be installed under different conditions (mainly Python or Flask versions), we matrix test the combinations using tox.
To setup tox for use with uv:
uv tool install tox~=4.0 --with tox-uv
tox --version # check that you are using a tox with the tox-uv plugin installed
To run tests via tox, tox. To run suites in parallel, run eg tox -p 8.
To create a new release of funding-service-design-utils (note no longer need to manually update the version):
main..github/workflows/test-and-tag.yml will create a new tag and release, named for the version in pyproject.toml. This is triggered automatically on a push to main.Publish to PyPI step with the following: - name: Publish to PyPI
id: publish-to-pypi
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_API_TOKEN_TEST }}
repository_url: https://test.pypi.org/legacy/
Either of the following options will install the funding-service-design-utils into your python project. The package fsd_utils can then be imported.
To install a particular version, use uv add funding-service-design-utils==0.0.1 (update the version number as appropriate).
To reference the latest commit from a particular branch:
uv add git+https://github.com/communitiesuk/funding-service-design-utils --branch <branch_name>
Build docker with following commands
docker compose build <service_name> --no-cache
or
docker compose build --no-cache
When working and testing locally, you can also install the fsd_utils package from your local filesystem:
uv remove funding-service-design-utils
uv add /path/to/your/working/directory/funding-service-design/utils
Note: When testing locally using the docker runner, docker might use the cached version of fsd_utils. To avoid this and pick up your intended changes, run docker compose build <service_name> --no-cache first before running docker compose up. Docker will not be able to use your local checkout of utils.
Currently the configclass allows for pretty print debugging of config keys and the class from which they are created. This allows devs to quickly diagnoise problems arrising from incorrectly set config. To activate this functionality, one must decorate each config class with the @configclass decorator.
This defines config values that are common across different services, eg. url patterns. Usage example:
from fsd_utils import CommonConfig
@configclass
class DefaultConfig:
SECRET_KEY = CommonConfig.SECRET_KEY
The gunicorn utility allows consistent configuration of gunicorn across microservices.
To use it, First - add run/gunicorn directory in your application with config scripts that import the appropriate config from this util's choice of config files eg.
# in run/gunicorn/local.py
from fsd_utils.gunicorn.config.local import * # noqa
Then - add a wsgi.py script to the root of your application eg.
# in wsgi.py
from app import app
if __name__ == "__main__":
app.run()
Finally - To run with gunicorn, call the gunicorn command in your manifest.yml's or in the terminal, passing the wsgi application and the appropriate configuration file as arguments eg.
in the terminal:
gunicorn wsgi:app -c run/gunicorn/dev.py
in a manifest file:
# manifest-dev.yml
---
applications:
...
command: gunicorn wsgi:app -c run/gunicorn/devtest.py
The logging utility formats logs to a consistent format (text in development and json in other environments). The utility also supports logging from gunicorn
To use it add the following to your flask create_app factory
from fsd_utils.logging import logging
def create_app():
...
# Initialise logging
logging.init_app(flask_app)
and in your env config classes, specify the log level you want to report eg.:
@configclass
class DefaultConfig(object):
...
# Logging
FSD_LOG_LEVEL = logging.WARNING
The authentication utility provides a consistent authentication functions including a @login_required decorator that can be used to restrict routes that should only be accessible to authenticated users.
To use the utility: First - ensure that the following environment variables are set to the appropriate values eg.:
FSD_USER_TOKEN_COOKIE_NAME = "fsd_user_token"
AUTHENTICATOR_HOST = "https://funding-service-design-authenticator-dev.london.cloudapps.digital"
RSA256_PUBLIC_KEY = "{RSA PUBLIC KEY}"
NOTE: These values (and keys) need to be shared/common across all microservices that use each common authenticator host. If any of the environment keys for each of these attributes needs to be modified these can be reconfigured in fsd_utils/authentication/config.py.
To generate new keys in your cwd, you use the following commands:
openssl genrsa -out private_key.pem 2048
openssl rsa -pubout -in private_key.pem -out public_key.pem
If you're changing keys, you'll need to change them in GitHub secrets across repos. Please also add them to BitWarden and let the team know.
Then - to use the @login_required decorator just add it to routes you need to protect eg:
# in ...routes.py
from fsd_utils.authentication.decorators import login_required
@login_required
def example_route(account_id):
#...account_id will be available here if the user is authenticated
#...if not logged in the user will be redirected to re-authenticate
Adds the route /healthcheck to an application. On hitting this endpoint, a customisable set of checks are run to confirm the application is functioning as expected and a JSON response is returned.
Response codes:
Example usage:
from fsd_utils.healthchecks.healthcheck import Healthcheck
from fsd_utils.healthchecks.checkers import DbChecker, FlaskRunningChecker
health = Healthcheck(flask_app)
health.add_check(FlaskRunningChecker())
health.add_check(DbChecker(db))
The above will initialise the /healthcheck url and adds 2 checks.
The following 2 checks are provided in fsd_utils in checkers.py:
FlaskRunningChecker: Checks whether a current_app is available from flask, returns True if it is, False if not.DbChecker: Runs a simple query against the DB to confirm database availability.Custom checks can be created as subclasses of CheckerInterface and should contain a method with the following signature:
def check(self) -> Tuple[bool, str]:
Where
bool is a True or False whether the check was successfulstr is a message to display in the result JSON, typically OK or FailMultiple language support is provided by flask-babel. Docs here: https://python-babel.github.io/flask-babel/#
The python setup relies on 2 methods and an initialisation. In create_app we initialise babel on our app:
from fsd_utils.locale_selector.get_lang import get_lang
babel = Babel(flask_app)
babel.locale_selector_func = get_lang
LanguageSelector(flask_app)
The get_lang() function reads the user-selected language from a cookie (if set by LanguageSelector - see below), or if that is not present uses a built-in function of babel to negotiate the language based on the request headers and the supported languages.
LanguageSelector creates an additional route /language/<language> that sets the user's selected language in a cookie. Used the cookie rather than the session so it can be shared across the microservices.
Set COOKIE_DOMAIN on the app to the domain you want to set the cookie on.
Add trans tags around items in your jinja html file that you want to translate. What's contained in the trans tag should be the english version of this text. eg:
A greeting is: {% trans %}Good Morning{% endtrans %}
Generate a new translations template file (messages.pot)
pybabel extract -F babel.cfg -o messages.pot .
If initialising a new set of translations for a new repo or adding a new language, use init. This will override any changes already made in the translations directory!
# Initialise a welsh translations file in app/translations
pybabel init -i messages.pot -d app/translations -l cy
The directory supplied to -d (eg. app/translations) must sit at the same level as the templates folder, in our case within the app directory. This command generates a new messages.po file in translations/cy/LC_MESSAGES/ Where cy is the language code. (cy = Welsh/Cymraeg)
If you've added new strings to an existing template, or added new templates after running init, use update:
pybabel update -i messages.pot -d app/translations
This will append new strings to the existing messages.po file, preserving any translations you already have in there.
To add translations for strings, edit the messages.po file.
#: app/templates/index.html:10
msgid "Good Morning"
msgstr "Bore da"
where msgid is the english version of the string from the original template file, and msgstr is the translation of the string.
Once the translations are ready, use compile to generate the binary for use at runtime:
pybabel compile -d app/translations
## Sentry Enables Sentry integration.
from fsd_utils import init_sentry
init_sentry()
SENTRY_DSN environment variable on the appFolder to hold miscellaneous simple utilities.
Date comparison functions that accept an ISO Format string, for use in the frontend determining display logic based on dates.
Contains shared fixtures that can be used for unit tests in other repos. To include a fixture from fsd_utils in your project, add the following to your conftest.py:
pytest_plugins = ["fsd_test_utils.fixtures.db_fixtures"]
Where the part inside [] is the paths to the python files you want to load as fixtures.
Individual fixtures are explained below, but this is the general expected usage. In the conftest.py of your repo, define a fixture that will seed the database with the test data you require and make those records avaialble to tests. That fixture should request clear_test_data and enable_preserve_test_data to prevent test pollution and ensure a suitable database is available for tests. The tests themselves should only update data that is inserted by that fixture. Examples:
Basic Usage
In conftest.py
@pytest.fixture(scope="function")
def create_test_data(clear_test_data, enable_preserve_test_data):
# Logic to insert some test data
created_data = [all_created_data]
yield created_data
# No teardown logic required as this is covered in clear_test_data
In your test file
def test_get_data(create_test_data):
id = create_test_data[0].id
record = get_record_using_api_under_test(id)
assert record.id == id
If your tests need to directly manipulate the database, use the _db fixture:
def test_after_update(create_test_data, _db):
id = create_test_data[0].id
record = get_record(id)
record.name = "new name"
_db.session.add(record)
_db.session.commit()
# Do some further tests post update
If you want to be able to inspect the database after running a test:
@pytest.mark.preserve_test_data(True)
def test_get_data(create_test_data):
id = create_test_data[0].id
record = get_record_using_api_under_test(id)
assert record.id == id
After the test runs, any created data will still be there for manual inspection. It will then be removed at the start of the next test run.
This fixture is module scoped so at the end of each test file all data is removed from the database (providing preserve_test_data is not set - see below.) Helps to prevent test pollution.
This fixture looks for the marker pytest.mark.preserve_test_data(True) and if found, test data is not cleared at the end of the test session, allowing you to manually inspect the database to aid debugging.
Note: Not intended to be used as a default for tests as it can lead to test pollution, so only use to debug and then remove again.
Provides direct access to the database for tests. Useful if your test needs to explicitly insert/update records to prepare test data.
This fixture reads the pytest cache to determine whether the DB can be reused for this test run and then the db is recreated accordingly. Updates the cache value to enable reuse of the DB for future test runs. This is session scoped so the DB is not recreated for each test in a run.
This is requested by clear_test_data so you do not need to include it separately in your project if using that fixture.
The mapping feature serves the purpose of mapping and formatting the questions and answers of the application into a text file. This functionality is utilised in the following scenarios:
To accomplish the mapping of the application's questions and answers, import
from fsd_utils import extract_questions_and_answers
To format the questions and answers of the application into a text file, import
from fsd_utils import generate_text_of_application
FAQs
Utilities used by the MHCLG Funding Service Team
We found that funding-service-design-utils demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.
Security News
Research
Socket researchers found a malicious Maven package impersonating the legitimate ‘XZ for Java’ library, introducing a backdoor for remote code execution.