hydraflow

Hydraflow

Overview

Hydraflow is a library designed to seamlessly integrate Hydra and MLflow, making it easier to manage and track machine learning experiments. By combining the flexibility of Hydra's configuration management with the robust experiment tracking capabilities of MLflow, Hydraflow provides a comprehensive solution for managing complex machine learning workflows.

Key Features

• Configuration Management: Utilize Hydra's advanced configuration management to handle complex parameter sweeps and experiment setups.
• Experiment Tracking: Leverage MLflow's tracking capabilities to log parameters, metrics, and artifacts for each run.
• Artifact Management: Automatically log and manage artifacts, such as model checkpoints and configuration files, with MLflow.
• Seamless Integration: Easily integrate Hydra and MLflow in your machine learning projects with minimal setup.
• Rich CLI Interface: Command-line tools for managing experiments and viewing results.
• Cross-Platform Support: Works consistently across different operating systems.

Installation

You can install Hydraflow via pip:

pip install hydraflow

Quick Start

Here is a simple example to get you started with Hydraflow:

from __future__ import annotations

from dataclasses import dataclass
from typing import TYPE_CHECKING

import hydraflow
import mlflow

if TYPE_CHECKING:
    from mlflow.entities import Run


@dataclass
class Config:
    """Configuration for the ML training experiment."""
    # Training hyperparameters
    learning_rate: float = 0.001
    batch_size: int = 32
    epochs: int = 10

    # Model architecture parameters
    hidden_size: int = 128
    dropout: float = 0.1

    # Dataset parameters
    train_size: float = 0.8
    random_seed: int = 42


@hydraflow.main(Config)
def app(run: Run, cfg: Config):
    """Train a model with the given configuration.

    This example demonstrates how to:

    1. Define a configuration using dataclasses
    2. Use Hydraflow to integrate with MLflow
    3. Track metrics and parameters automatically

    Args:
        run: MLflow run for the experiment corresponding to the Hydra app.
            This `Run` instance is automatically created by Hydraflow.
        cfg: Configuration for the experiment's run.
            This `Config` instance is originally defined by Hydra, and then
            automatically passed to the app by Hydraflow.
    """
    # Training loop
    for epoch in range(cfg.epochs):
        # Simulate training and validation
        train_loss = 1.0 / (epoch + 1)
        val_loss = 1.1 / (epoch + 1)

        # Log metrics to MLflow
        mlflow.log_metrics({
            "train_loss": train_loss,
            "val_loss": val_loss
        }, step=epoch)

        print(f"Epoch {epoch}: train_loss={train_loss:.4f}, val_loss={val_loss:.4f}")


if __name__ == "__main__":
    app()

This example demonstrates:

• Configuration management with Hydra
• Automatic experiment tracking with MLflow
• Parameter logging and metric tracking
• Type-safe configuration with dataclasses

Documentation

For detailed documentation, including advanced usage examples and API reference, visit our documentation site.

Contributing

We welcome contributions! Please see our contributing guide for details.

License

This project is licensed under the MIT License - see the LICENSE file for details.