Reflekt
/rษหflek(t)/ - to embody or represent in a faithful or appropriate way
Reflekt helps teams define, govern, and model event data for warehouse-first product analytics. It integrates with Customer Data Platforms, Schema Registries, Data Warehouses, and dbt. Events are defined using JSON schema, version controlled, and updated by pull requests (PRs), enabling:
- Branches for parallel development and testing.
- Reviews and discussion amongst teams and stakeholders.
- CI/CD suites to:
reflekt lint
schemas against naming and metadata rules.reflekt push
schemas to deploy them to a schema registry for event data validation.
Stop writing SQL & YAML. reflekt build
a dbt package to model, document, and test events in the warehouse.
To learn more about reflekt
, checkout:
Getting Started
Install
Reflekt is available on PyPI. Install it with pip
or package manager, preferably in a virtual environment:
โฏ source /path/to/venv/bin/activate
โฏ pip install reflekt
โฏ reflekt --version
Reflekt CLI Version: 0.6.0
Create a Reflekt Project
To create a new Reflekt project, make a directory, initialize a Git repo, and run reflekt init
.
โฏ mkdir ~/Repos/my-reflekt-project
โฏ cd ~/Repos/my-reflekt-project
โฏ git init
โฏ reflekt init
You now have a Reflekt project with the structure:
my-reflekt-project
โโโ .logs/
โโโ .reflekt_cache/
โโโ artifacts/
โโโ schemas/
โโโ .gitignore
โโโ README.md
โโโ reflekt_project.yml
Configure a Reflekt Project
Reflekt uses 3 files to define and configure a Reflekt project.
reflekt_project.yml
Defines project settings, event and metadata conventions, data artifact generation, and optional registry config (Avo only).
Example: reflekt_project.yml
(click to expand)
version: 1.0
name: reflekt_demo
vendor: com.company_name
default_profile: dev_reflekt
profiles_path: ~/.reflekt/reflekt_profiles.yml
schemas:
conventions:
event:
casing: title
numbers: false
reserved: []
property:
casing: snake
numbers: false
reserved: []
data_types: [
string, integer, number, boolean, object, array, any, 'null'
]
registry:
avo:
branches:
staging: AbC12dEfG
main: main
artifacts:
dbt:
sources:
prefix: __src_
models:
prefix: stg_
docs:
prefix: _stg_
in_folder: false
tests:
id: [unique, not_null]
reflekt_profiles.yml
This file defines connections to schema registries and data warehouse connections.
Example: reflekt_profiles.yml
(click to expand)
version: 1.0
dev_reflekt:
registry:
- type: segment
api_token: segment_api_token
- type: avo
workspace_id: avo_workspace_id
service_account_name: avo_service_account_name
service_account_secret: avo_service_account_secret
source:
- id: snowflake
type: snowflake
account: abc12345
database: raw
warehouse: transforming
role: transformer
user: reflekt_user
password: reflekt_user_password
- id: redshift
type: redshift
host: example-redshift-cluster-1.abc123.us-west-1.redshift.amazonaws.com
database: analytics
port: 5439
user: reflekt_user
password: reflekt_user_password
- id: bigquery
type: bigquery
project: raw-data
dataset: jaffle_shop_segment
keyfile_json:
type: "service_account"
project_id: "foo-bar-123456"
private_key_id: "abc123def456ghi789"
private_key: "-----BEGIN PRIVATE KEY-----\nmy-very-long-private-keyF\n\n-----END PRIVATE KEY-----\n"
client_email: "reflekt-user@foo-bar-123456.iam.gserviceaccount.com"
client_id: "123456789101112131415161718"
auth_uri: "https://accounts.google.com/o/oauth2/auth"
token_uri: "https://oauth2.googleapis.com/token"
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/reflekt-user%40foo-bar-123456.iam.gserviceaccount.com"
schemas/.reflekt/meta/1-0.json
A meta-schema used by reflekt lint
to ensure all events in schemas/
follow the Reflekt format. Can also be used to define gloablly required metadata for all event schemas.
Example: schemas/.reflekt/meta/1-0.json
(click to expand)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": ".reflekt/meta/1-0.json",
"description": "Meta-schema for all Reflekt events",
"self": {
"vendor": "reflekt",
"name": "meta",
"format": "jsonschema",
"version": "1-0"
},
"type": "object",
"allOf": [
{
"$ref": "http://json-schema.org/draft-07/schema#"
},
{
"properties": {
"self": {
"type": "object",
"properties": {
"vendor": {
"type": "string",
"description": "The company, application, team, or system that authored the schema (e.g., com.company, com.company.android, com.company.marketing)"
},
"name": {
"type": "string",
"description": "The schema name. Describes what the schema is meant to capture (e.g., pageViewed, clickedLink)"
},
"format": {
"type": "string",
"description": "The format of the schema",
"const": "jsonschema"
},
"version": {
"type": "string",
"description": "The schema version, in MODEL-ADDITION format (e.g., 1-0, 1-1, 2-3, etc.)",
"pattern": "^[1-9][0-9]*-(0|[1-9][0-9]*)$"
},
"metadata": {
"type": "object",
"description": "Required metadata for all event schemas",
"properties": {
"code_owner": {"type": "string"},
"product_owner": {"type": "string"}
},
"required": ["code_owner", "product_owner"],
"additionalProperties": false
},
},
"required": ["vendor", "name", "format", "version"],
"additionalProperties": false
},
"properties": {},
"tests": {},
},
"required": ["self", "metadata", "properties"]
}
]
}
Defining Event Schemas
Events in a Reflekt project are defined using the JSON schema specification and are stored in the schemas/
directory of the project. Click to expand the Order Completed
example below.
Example: my-reflekt-project/schemas/jaffle_shop/Order_Completed/1-0.json
(click to expand)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "jaffle_shop/Order_Completed/1-0.json",
"description": "User completed an order.",
"self": {
"vendor": "com.thejaffleshop",
"name": "Order Completed",
"format": "jsonschema",
"version": "1-0",
"metadata": {
"code_owner": "@the-jaffle-shop/frontend-guild",
"product_owner": "pmanager@thejaffleshop.com",
}
},
"type": "object",
"properties": {
"coupon": {
"description": "Coupon code used for the order.",
"type": [
"string",
"null"
]
},
"currency": {
"description": "Currency for the order.",
"type": "string"
},
"discount": {
"description": "Total discount for the order.",
"type": "number"
},
"order_id": {
"description": "Unique identifier for the order.",
"type": "string"
},
"products": {
"description": "List of products in the cart.",
"type": "array",
"items": {
"type": "object",
"properties": {
"category": {
"description": "Category of the product.",
"type": "string"
},
"name": {
"description": "Name of the product.",
"type": "string"
},
"price": {
"description": "Price of the product.",
"type": "number"
},
"product_id": {
"description": "Unique identifier for the product.",
"type": "string"
},
"quantity": {
"description": "Quantity of the product in the cart.",
"type": "integer"
},
"sku": {
"description": "Stock keeping unit for the product.",
"type": "string"
}
},
"required": [
"product_id",
"sku",
"category",
"name",
"price",
"quantity"
],
"additionalProperties": false,
}
},
"revenue": {
"description": "Total revenue for the order.",
"type": "number"
},
"session_id": {
"description": "Unique identifier for the session.",
"type": "string"
},
"shipping": {
"description": "Shipping cost for the order.",
"type": "number"
},
"subtotal": {
"description": "Subtotal for the order (revenue - discount).",
"type": "number"
},
"tax": {
"description": "Tax for the order.",
"type": "number"
},
"total": {
"description": "Total cost for the order (revenue - discount + shipping + tax = subtotal + shipping + tax).",
"type": "number"
}
},
"required": [
"session_id",
"order_id",
"revenue",
"coupon",
"discount",
"subtotal",
"shipping",
"tax",
"total",
"currency",
"products"
],
"additionalProperties": false
}
Schema $id
and version
Schemas in a Reflekt project are identified and --select
ed by their $id
, which is their path relative to the schemas/
directory. For example:
File Path to Schema | Schema $id |
---|
~/repos/my-reflekt-project/schemas/jaffle_shop/Cart_Viewed/1-0.json | jaffle_shop/Cart_Viewed/1-0.json |
~/repos/my-reflekt-project/schemas/jaffle_shop/Cart_Viewed/2-1.json | jaffle_shop/Cart_Viewed/2-1.json |
Each schema has a version
(e.g., 1-0
, 2-1
), used to indicate changes to data collection requirements. New event schemas start at 1-0
and follow a MAJOR-MINOR
version spec, as shown in the table below.
Type | Description | Example | Use Case |
---|
MAJOR | Breaking change incompatible with previous data. | 1-0 , 2-0 (ends in `-0) | - Add/remove/rename a required property - Change a property from optional to required - Change a property's type |
MINOR | Non-breaking change compatible with previous data. | 1-1 , 2-3 | - Add/remove/rename an optional property - Change a property from required to optional |
[!NOTE]
For MINOR
schema versions (non-breaking changes), you can either:
- Update the existing schema and increment the MINOR version number.
- Create a new
.json
file with the updated schema and increment the MINOR version number.
For MAJOR
schema versions (breaking changes), you MUST:
- Create a new
.json
file with the updated schema. This way, an application/product/feature can begin using the new schema while others continue to use the old schema (migrating later).
Linting Event Schemas
Schemas can be linted to test if they follow the naming conventions in your [reflekt_project.yml
] and metadata conventions in schemas/.reflekt/meta/1-0.json
.
โฏ reflekt lint --select schemas/jaffle_shop
[18:31:12] INFO Running with reflekt=0.5.0
[18:31:12] INFO Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop
[18:31:12] INFO Found 9 schema(s) to lint
[18:31:12] INFO 1 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json
[18:31:19] INFO 2 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json
[18:31:20] INFO 3 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json
[18:31:24] INFO 4 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json
[18:31:26] INFO 5 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json
[18:31:32] INFO 6 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json
[18:31:37] INFO 7 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json
[18:31:40] INFO 8 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json
[18:31:44] INFO 9 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json
[18:31:51] INFO Completed successfully
Sending Event Schemas to a Schema Registries
In order to validate events as they flow from Application -> Registry -> Customer Data Platform (CDP) -> Data Warehouse, we need to send a copy of our event schemas to a schema registry (see supported registries). This is done with the reflekt push
command.
โฏ reflekt push --registry segment --select schemas/jaffle_shop
[18:41:05] INFO Running with reflekt=0.5.0
[18:41:06] INFO Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop
[18:41:06] INFO Found 9 schemas to push
[18:41:06] INFO 1 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json
[18:41:06] INFO 2 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json
[18:41:06] INFO 3 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json
[18:41:06] INFO 4 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json
[18:41:06] INFO 5 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json
[18:41:06] INFO 6 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json
[18:41:06] INFO 7 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json
[18:41:06] INFO 8 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json
[18:41:06] INFO 9 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json
[18:41:08] INFO Completed successfully
Building dbt
Packages to Model Event Data
Modeling event data in dbt
is a lot of work. Everyone wants staging models that are clean, documented, and tested. But who wants to write and maintain SQL and YAML for hundreds of events?
You don't have to choose. Put reflekt build
to work for you - staging models, documentation, even tests - all in a dbt package ready for you to use in your dbt project.
โฏ reflekt build --artifact dbt --select schemas/jaffle_shop --source snowflake.raw.jaffle_shop_segment --sdk segment
[18:56:25] INFO Running with reflekt=0.5.0
[18:56:26] INFO Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop
[18:56:26] INFO Found 9 schemas to build
[18:56:27] INFO Building dbt package:
name: jaffle_shop
dir: /Users/gclunies/Repos/reflekt/artifacts/dbt/jaffle_shop
--select: jaffle_shop
--sdk_arg: segment
--source: snowflake.raw.jaffle_shop_segment
[18:56:27] INFO Building dbt source 'jaffle_shop_segment'
[18:56:27] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json
[18:56:28] INFO Building dbt table 'order_completed' in source 'jaffle_shop_segment'
[18:56:28] INFO Building staging model 'stg_jaffle_shop_segment__order_completed.sql'
[18:56:28] INFO Building dbt documentation '_stg_jaffle_shop_segment__order_completed.yml'
[18:56:28] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json
[18:56:29] INFO Building dbt table 'identifies' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__identifies.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__identifies.yml'
[18:56:29] INFO Building dbt artifacts for schema: Segment 'users' table
[18:56:29] INFO Building dbt table 'users' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__users.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__users.yml'
[18:56:29] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json
[18:56:29] INFO Building dbt table 'product_clicked' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__product_clicked.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__product_clicked.yml'
[18:56:29] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json
[18:56:29] INFO Building dbt table 'cart_viewed' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__cart_viewed.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__cart_viewed.yml'
[18:56:29] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json
[18:56:30] INFO Building dbt table 'product_removed' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__product_removed.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__product_removed.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json
[18:56:30] INFO Building dbt table 'product_added' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__product_added.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__product_added.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json
[18:56:30] INFO Building dbt table 'checkout_step_viewed' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__checkout_step_viewed.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__checkout_step_viewed.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json
[18:56:30] INFO Building dbt table 'checkout_step_completed' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__checkout_step_completed.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__checkout_step_completed.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json
[18:56:30] INFO Building dbt table 'pages' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__pages.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__pages.yml'
[18:56:30] INFO Building dbt artifacts for schema: Segment 'tracks' table
[18:56:31] INFO Building dbt table 'tracks' in source 'jaffle_shop_segment'
[18:56:31] INFO Building staging model 'stg_jaffle_shop_segment__tracks.sql'
[18:56:31] INFO Building dbt documentation '_stg_jaffle_shop_segment__tracks.yml'
[18:56:31] INFO Copying dbt package from temporary path /Users/gclunies/Repos/reflekt/.reflekt_cache/artifacts/dbt/jaffle_shop to /Users/gclunies/Repos/reflekt/artifacts/dbt/jaffle_shop
[18:56:31] INFO Successfully built dbt package
CLI Commands
A description of commands can be seen by running reflekt --help
. The help page for each CLI command is shown below.
reflekt init
โฏ reflekt init --help
[11:17:16] INFO Running with reflekt=0.6.0
Usage: reflekt init [OPTIONS]
Initialize a Reflekt project.
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ --dir TEXT [default: .] โ
โ --verbose --no-verbose [default: no-verbose] โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
reflekt debug
โฏ reflekt debug --help
[11:18:07] INFO Running with reflekt=0.6.0
Usage: reflekt debug [OPTIONS]
Check Reflekt project configuration.
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
reflekt lint
โฏ reflekt lint --help
[11:20:29] INFO Running with reflekt=0.6.0
Usage: reflekt lint [OPTIONS]
Lint schema(s) to test for naming and metadata conventions.
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * --select -s TEXT Schema(s) to lint. Starting with 'schemas/' is optional. [default: None] [required] โ
โ --verbose -v Verbose logging. โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
reflekt push
โฏ reflekt push --help
[11:22:37] INFO Running with reflekt=0.6.0
Usage: reflekt push [OPTIONS]
Push schema(s) to a schema registry.
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * --registry -r [avo|segment] Schema registry to push to. [default: None] [required] โ
โ * --select -s TEXT Schema(s) to push to schema registry. Starting with 'schemas/' is optional. [default: None] โ
โ --delete -D Delete schema(s) from schema registry. Prompts for confirmation โ
โ --force -F Force command to run without confirmation. โ
โ --profile -p TEXT Profile in reflekt_profiles.yml to use for schema registry connection. โ
โ --verbose -v Verbose logging. โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
reflekt pull
โฏ reflekt pull --help
[11:25:18] INFO Running with reflekt=0.6.0
Usage: reflekt pull [OPTIONS]
Pull schema(s) from a schema registry.
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * --registry -r [avo|segment] Schema registry to pull from. [default: None] [required] โ
โ * --select -s TEXT Schema(s) to pull from schema registry. If registry uses tracking plans, starting with the plan name. [default: None] [required] โ
โ --profile -p TEXT Profile in reflekt_profiles.yml to use for schema registry connection. โ
โ --verbose -v Verbose logging. โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
reflekt build
โฏ reflekt build --help
[11:31:36] INFO Running with reflekt=0.6.0
Usage: reflekt build [OPTIONS]
Build data artifacts based on schemas.
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * --artifact -a [dbt] Type of data artifact to build. [default: None] [required] โ
โ * --select -s TEXT Schema(s) to build data artifacts for. Starting with 'schemas/' is optional. [default: None] [required] โ
โ * --sdk [segment] The SDK used to collect the event data. [default: None] [required] โ
โ * --source TEXT The <source_id>.<database>.<schema> storing raw event data. <source_id> must be a data warehouse source defined in reflekt_profiles.yml [default: None] [required] โ
โ --profile -p TEXT Profile in reflekt_profiles.yml to look for the data source specified by the --source option. Defaults to default_profile in reflekt_project.yml โ
โ --verbose -v Verbose logging. โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
reflekt report
โฏ reflekt report --help
[08:20:09] INFO Running with reflekt=0.6.0
Usage: reflekt report [OPTIONS]
Generate Markdown report(s) for schema(s).
โญโ Options โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ * --select -s TEXT Schema(s) to generate Markdown report(s) for. Starting with โ
โ 'schemas/' is optional. โ
โ [default: None] โ
โ [required] โ
โ --to-file -f Write report(s) to file instead of terminal. โ
โ --verbose -v Verbose logging. โ
โ --help Show this message and exit. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Integrations
Customer Data Platform (CDP)
Reflekt understands how Customer Data Platforms (CDPs) collect event data and load them into data warehouses, allowing it to:
- Parse the schemas in a Reflekt project.
- Find matching tables for events and columns for properties in the data warehouse.
- Build a
dbt
package with sources, staging models, and documentation for event data.
Schema Registry
Schema registries store and serve schemas. When a schema is registered in a regsitry, it can be used to validate events as they flow through your data collection infrastructure. Reflekt works with schema registries from CDPs, SaaS vendors, and open-source projects, letting teams to decide between managed and self-hosted solutions.
Registry | Open Source | Schema Versions | Recommended Workflow |
---|
Segment Protocols | โ | MAJOR only | Manage schemas in Reflekt. reflekt push to Protocols for event validation. reflekt build --artifact dbt to build dbt package. |
Avo | โ | MAJOR only | Manage schemas in Avo. reflekt pull to get schemas. reflekt build --artifact dbt to build dbt package. |
reflekt-registry ๐ง Coming Soon ๐ง | โ
| MAJOR & MINOR | Manage schemas in Reflekt. reflekt push to reflekt-registry. reflekt build --artifact dbt to build dbt package. |
Data Warehouse
In order to build dbt packages, Reflekt needs to connect to a cloud data warehouse where raw event data is stored.
Reflekt NEVER copies, moves, or modifies events in the data warehouse. It ONLY reads table and column names for templating.
dbt
dbt enables anyone that knows SQL to transform data in a cloud data warehouse. When modeling in dbt, it is best practice to:
But as the number of events intsrumented in your product grow or change as the product evolves, mainting this best practice can be burdensome and boring. This is where reflekt build
steps in.
Contribute
License
This project is licensed under the Apache-2.0 License.