prefect-dbt-flow

prefect-dbt-flow

prefect-dbt-flow is a Python library that enables Prefect to convert dbt workflows into independent tasks within a Prefect flow. This integration simplifies the orchestration and execution of dbt models and tests using Prefect, allowing you to build robust data pipelines and monitor your dbt projects efficiently.

dbt is an immensely popular tool for building and testing data transformation models, and Prefect is a versatile workflow management system. This integration brings together the best of both worlds, empowering data engineers and analysts to create robust data pipelines.

Key features:

• Simplified Orchestration: Define and manage your dbt projects and models as Prefect tasks, creating a seamless pipeline for data transformation.
• Monitoring and Error Handling: Gain deep insights into the execution of your dbt workflows and take immediate action in case of issues.
• Workflow Consistency: Ensure your dbt workflows run consistently by managing them through Prefect. This consistency is crucial for maintaining data quality and reliability.
• Advanced Configuration: Customize your dbt workflow by adjusting the dbt project, profile, and DAG options. You can also use Prefect features like scheduling, notifications, and task retries to monitor and manage your dbt flows effectively.

To get started, check out our getting started guide.

Active Development Notice: prefect-dbt-flow is actively under development and may not be ready for production use. We advise users to be aware of potential breaking changes as the library evolves. Please check the changelog for updates.

How to Install

You can install prefect-dbt-flow via pip:

pip install prefect-dbt-flow

Note: prefect-dbt-flow does not come with dbt as a dependency. You will need to install dbt or a dbt-adapter separately.

Basic Usage

Here's an example of how to use prefect-dbt-flow to create a Prefect flow for your dbt project:

from prefect_dbt_flow import dbt_flow
from prefect_dbt_flow.dbt import DbtProfile, DbtProject

my_flow = dbt_flow(
    project=DbtProject(
        name="jaffle_shop",
        project_dir="path_to/jaffle_shop",
        profiles_dir="path_to/jaffle_shop",
    ),
    profile=DbtProfile(
        target="dev",
        overrides={
            "type": "duckdb",
            "path": "path_to/duckdb.db",
        },
    ),
)

if __name__ == "__main__":
    my_flow()

For more information consult the docs

Inspiration

prefect-dbt-flow draws inspiration from various projects in the data engineering and workflow orchestration space, including:

• astronomer-cosmos
• dbt + Dagster
• prefect-dataplatform - Anna Geller

License

This project is licensed under the MIT License. You are free to use, modify, and distribute this software as per the terms of the license. If you find this project helpful, please consider giving it a star on GitHub.