skald

Skáld

📟 a simple and efficient experiment logger for Python 🐍

Skáld - An Old Norse word for a poet, usually applied to a Norwegian or Icelandic court poet or bard of the period from the 9th century to the 13th. Skaldic verse is marked by its elaborate patterns of metre, rhyme, and alliteration, and by its use of kennings.

📃 Table of Contents

• 💡 Motivation
• 👀 Concepts
  • 📂 Logging Structure
• 📦 Installation
• 🚀 Usage

💡 Motivation

During my PhD, I tried different Experiment/Metrics loggers including:

• TensorBoard
• MLFlow
• Weights and Biases
• DVCLive
• other logging solutions from deep learning and continual learning frameworks (mostly stdout and csv loggers)

While those are quite mature logging solutions, which often offer beautiful dashboards, I was looking for something light-weight, local and file-based as DVCLive, but with a more ergonomic and tidy ¹ structure of the logs for simpler consumption and analysis.

The latest of my workflows before Skáld involved using a mixture of DVCLive and a custom logger from FACIL ² in combination with log crawling CLI scripts and analysis and visualizations performed in jupyter notebooks 🤨

Another problem I faced with those solutions is that all of them offered a single "step" identifier for each logged metric, which is not sufficient for deep learning use-cases outside the conventional epoch-based training loops.

Because I like building python packages and I felt the need to tidy my experiment logs, I created skald as a small side project. I hope that some people find some enjoyment in using the package too ❤️

👀 Concepts

Skáld is an experiment logger, that offers a standardized logging structure and interface to log different aspects of an experiment including:

• parameters|arguments - meta information and variables, that don't change during the experiment
• metrics|scalars - single-valued, numeric variables that change during the experiment
• artifacts - additional result files like images or plots

Each metric has a unique name and a user-defined set of id variables, that identify the value of the metric at a certain step.

While a stateful version of Skáld is planned, that updates an id/step variables through some manual call (often in a Callback). The first version is very explicit and requires every id to be logged in each call.

📂 Logging Structure

Logs of metrics will be represented by tidy dataframes that are stored as readable metrics.csv or more space efficient metrics.parquet files.

To save space, parameters will not be included in these dataframes, but in a separate file (params.yaml) by default.

Artifacts will be stored in a separate sub-directory (artifacts/ by default).

Skáld has its own loguru logger instance and exposes the logging functions. The logs will stored in a console.log file. If you use the logger as a context manager, stdout (~ print statements) will also be saved in console.log.

📦 Installation

The package can be installed with:

pip install skald

🧑‍💻 to install a development environment (which you need if you want to work on the package, instead of just using the package), cd into the project's root directory and call:

poetry install --sync --compile

🚀 Usage

The API of Skáld is very similar to DVCLive and other loggers. It also exposes loguru's logging api, so you can also use a Skáld logger as your terminal logger.

A basic example:

from skald import Logger

# get experiment parameters from CLI arguments, parameter files ...
params: dict = { ... }

# instanciate a logger with a certain run name
with Logger("test-run-1") as logger:
    logger.log_params(params)
    # experiment logic
    metric: float = evaluate(model)
    logger.log_metric("accuracy", metric)
    logger.success("finished experiment!")

To launch the experiment viewer TUI after the run is completed, use tui=True in the constructor of the Logger.

You can also launch the experiment viewer manually to inspect your logs:

$skald view <experiment_run_dir>

🖼️ ©️ Banner Artwork Attribution

The art in the banner of this README is licensed under a Creative Commons Attribution-NonCommercial-No Derivatives Works 3.0 License. It was made by th3dutchzombi3. Check out his beautiful artwork ❤️

📄 References

Footnotes

• H. Wickham, “Tidy Data,” Journal of Statistical Software, vol. 59, pp. 1–23, Sep. 2014, doi: 10.18637/jss.v059.i10. ↩

• M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer, “Class-incremental learning: survey and performance evaluation on image classification.” arXiv, Oct. 11, 2022. doi: 10.48550/arXiv.2010.15277. ↩