You're Invited:Meet the Socket Team at BlackHat and DEF CON in Las Vegas, Aug 7-8.RSVP
Socket
Socket
Sign inDemoInstall

rudderstack-airflow-provider

Package Overview
Dependencies
Maintainers
1
Alerts
File Explorer

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

Install

rudderstack-airflow-provider

airflow provider for rudderstack


Maintainers
1

Readme

The Customer Data Platform for Developers

Website · Documentation · Slack Community


RudderStack Airflow Provider

The RudderStack Airflow Provider lets you programmatically schedule and trigger your Reverse ETL syncs from outside RudderStack and integrate them with your existing Airflow workflows.

For more information on using the Airflow Provider utility, refer to the documentation.

Installation

pip install rudderstack-airflow-provider

Usage

RudderstackOperator

[!NOTE]
Use RudderstackRETLOperator for reverse ETL connections

A simple DAG for triggering syncs for a RudderStack source:

with DAG(
    'rudderstack-sample',
    default_args=default_args,
    description='A simple tutorial DAG',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['rs']
) as dag:
    rs_operator = RudderstackOperator(
        source_id='<source-id>',
        task_id='<any-task-id>',
        connection_id='rudderstack_conn'
    )

For the complete code, refer to this example.

Operator Parameters
ParameterDescriptionTypeDefault
source_idValid RudderStack source IDStringNone
task_idA unique task ID within a DAGStringNone
wait_for_completionIf True, the task will wait for sync to complete.BooleanFalse
connection_idThe Airflow connection to use for connecting to the Rudderstack API.Stringrudderstack_default

The RudderStack operator also supports all the parameters supported by the Airflow base operator.

For details on how to run the DAG in Airflow, refer to the documentation.

RudderstackRETLOperator

Trigger syncs for RETL connections

with DAG('rudderstack-sample',
    default_args=default_args,
    description='A simple tutorial DAG',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['rs']) as dag:
    rs_operator = RudderstackRETLOperator(
        retl_connection_id='2aiDQzMqP6LNuUokWstmaubcZOP',
        task_id='retl-test-sync',
        connection_id='rudder_yeshwanth_dev',
        sync_type='full',
        wait_for_completion=True
    )
Operator parameters
ParameterDescriptionTypeDefault
retl_connection_idValid RudderStack RETL connection IDString (templatable)None
task_idA unique task ID within a DAGStringNone
wait_for_completionIf True, the task will wait for sync to complete.BooleanFalse
connection_idThe Airflow connection to use for connecting to the Rudderstack API.Stringrudderstack_default
sync_typeType of sync to triggerincremental or full (templatable)incremental

For details on how to run the DAG in Airflow, refer to the documentation.

Contribute

We would love to see you contribute to this project. Get more information on how to contribute here.

License

The RudderStack Airflow Provider is released under the MIT License.

Contact Us

For more information or queries on this feature, you can contact us or start a conversation in our Slack community.

Keywords

FAQs


Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc