dbt-column-lineage

dbt-column-lineage

This is a tool to visualize the colulmn level lineage of dbt models. It uses the manifest.json and catalog.json files generated by dbt to create a graph of the lineage of the models. It is a web application that uses a Flask backend and a Next.js frontend.

quickstart

Install dbt-column-lineage using pip:

pip install dbt-column-lineage

Run the following command:

# go to your dbt project directory
cd your-dbt-project/

# edit your model file
vi models/test.sql

# generate the manifest.json and catalog.json files
dbt docs generate 

# set the environment variable for the dialect you are using
export SQLGLOT_DIALECT=snowflake

# Launch dbt-column-lineage with test.sql as the initial model
dbt-coloumn-lineage run-params

development

To develop the application, you will need to run the backend and frontend separately.

git clone [e-mail]:Oisix/dbt-column-lineage.git
cd dbt-column-lineage

for backend

activate venv and run the following commands:

python3 -m venv venv
source venv/bin/activate

pip install --upgrade pip
pip install -r requirements.txt

uvicorn --app-dir src dbt_column_lineage.main:app --port=5000 --reload

for frontend

run the following commands:

npm install
npm run dev

after the frontend is running, Let's access http://localhost:3000

for Looker integration (optional)

If you want to integrate with Looker, you can use the following commands:

# set the environment variables
export LOOKERSDK_CLIENT_ID=(your client id)
export LOOKERSDK_CLIENT_SECRET=(your client secret)
export LOOKERSDK_BASE_URL=(your looker base url)
export LOOKER_IGNORE_FOLDERS=(comma separated list of folders to ignore)
export LOOKER_IGNORE_ELEMENTS=(comma separated list of dashboard elements to ignore)

# it analyze the looker models, target/dashboad_analysis.json will be created
python tools/looker_analyzer.py

# rerun the backend
uvicorn --app-dir src dbt_column_lineage.main:app --port=5000 --reload

for Google OAuth login test (optional)

If you want to test the OAuth login, you can use the following commands:

export GOOGLE_CLIENT_ID=(your client id)
export GOOGLE_CLIENT_SECRET=(your client secret)
docker build -t test .
docker run -p 5000:5000 -e USE_OAUTH=true -e GOOGLE_CLIENT_ID=$GOOGLE_CLIENT_ID -e GOOGLE_CLIENT_SECRET=$GOOGLE_CLIENT_SECRET -e DEBUG_MODE=true test