Research
Security News
Malicious npm Package Targets Solana Developers and Hijacks Funds
A malicious npm package targets Solana developers, rerouting funds in 2% of transactions to a hardcoded address.
astronomer-providers
Advanced tools
Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer
.. image:: https://badge.fury.io/py/astronomer-providers.svg :target: https://badge.fury.io/py/astronomer-providers :alt: PyPI Version .. image:: https://img.shields.io/pypi/pyversions/astronomer-providers :target: https://img.shields.io/pypi/pyversions/astronomer-providers :alt: PyPI - Python Version .. image:: https://img.shields.io/pypi/l/astronomer-providers?color=blue :target: https://img.shields.io/pypi/l/astronomer-providers?color=blue :alt: PyPI - License .. image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/psf/black :alt: Code style: black .. image:: https://codecov.io/gh/astronomer/astronomer-providers/branch/main/graph/badge.svg?token=LPHFRC3CB3 :target: https://codecov.io/gh/astronomer/astronomer-providers :alt: CodeCov .. image:: https://readthedocs.org/projects/astronomer-providers/badge/?version=latest :target: https://astronomer-providers.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status .. image:: https://img.shields.io/badge/security-bandit-green.svg :target: https://github.com/PyCQA/bandit :alt: Security: bandit
.. warning::
The majority of operators and sensors within this repository have been deprecated and will not receive further updates.
Read more about the deprecation in the Deprecation Notice
section below.
With the release 1.19.0
of the astronomer-providers package, most of the operators and sensors are deprecated and will
no longer receive updates. We recommend migrating to the official Apache Airflow Providers for the latest features and support.
For the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Providers
is as simple as changing the import path from
.. code-block::
from astronomer.providers.*.*.operator_module import SomeOperatorAsync
to
.. code-block::
from airflow.providers.*.*.operator_module import SomeOperator
and setting the deferrable
argument to True
while using the operator or sensor in your DAG.
Setting the deferrable
argument to True
will ensure that the operator or sensor is using the async version
of the operator or sensor from the official Apache Airflow Providers.
For example, to migrate from
astronomer.providers.amazon.aws.operators.batch.BatchOperatorAsync
to
airflow.providers.amazon.aws.operators.s3.BatchOperator
, simply change the import path and pass
the deferrable argument:
.. code-block:: python
BatchOperator(
task_id="copy_object",
your_arguments,
your_keyword_arguments,
deferrable=True,
)
For more information on using the deferrable operators and sensors from the official Apache Airflow Providers, visit the following links:
.. note::
Although the default value for the deferrable
argument is False
, it's possible to configure the default value for the deferrable
argument across
your deployment by setting the default_deferrable
flag in the operators
sections of your Airflow configuration. Once you set the default_deferrable
flag to True
,
you can remove the deferrable
argument from your operators and sensors and they will use the async version of the operator or sensor from the official Apache Airflow Providers
if it exists.
See more at: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#default-deferrable
For troubleshooting of issues with migrations, you are suggested to open up a GitHub discussion <https://github.com/astronomer/astronomer-providers/discussions>
_
Install and update using pip <https://pip.pypa.io/en/stable/getting-started/>
_:
.. code-block:: bash
pip install astronomer-providers
This only installs dependencies for core provider. To install all dependencies, run:
.. code-block:: bash
pip install 'astronomer-providers[all]'
To only install the dependencies for a specific provider, specify the integration name as extra argument, example to install Kubernetes provider dependencies, run:
.. code-block:: bash
pip install 'astronomer-providers[cncf.kubernetes]'
Extras ^^^^^^
.. EXTRA_DOC_START
.. list-table:: :header-rows: 1
all
pip install 'astronomer-providers[all]'
amazon
pip install 'astronomer-providers[amazon]'
apache.hive
pip install 'astronomer-providers[apache.hive]'
apache.livy
pip install 'astronomer-providers[apache.livy]'
cncf.kubernetes
pip install 'astronomer-providers[cncf.kubernetes]'
databricks
pip install 'astronomer-providers[databricks]'
dbt.cloud
pip install 'astronomer-providers[dbt.cloud]'
google
pip install 'astronomer-providers[google]'
http
pip install 'astronomer-providers[http]'
microsoft.azure
pip install 'astronomer-providers[microsoft.azure]'
openlineage
pip install 'astronomer-providers[openlineage]'
sftp
pip install 'astronomer-providers[sftp]'
snowflake
pip install 'astronomer-providers[snowflake]'
.. EXTRA_DOC_END
This repo is structured same as the Apache Airflow's source code, so for example if you want to import Async operators, you can import it as follows:
.. code-block:: python
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor
waiting_for_s3_key = S3KeySensor(
task_id="waiting_for_s3_key",
bucket_key="sample_key.txt",
wildcard_match=False,
bucket_name="sample-bucket",
)
Example DAGs for each provider is within the respective provider's folder. For example,
the Kubernetes provider's DAGs are within the
astronomer/providers/cncf/kubernetes/example_dags <https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/cncf/kubernetes/example_dags>
_
folder.
We will only create Async operators for the "sync-version" of operators that do some level of polling (take more than a few seconds to complete).
For example, we won’t create an async Operator for a BigQueryCreateEmptyTableOperator
but will create one
for BigQueryInsertJobOperator
that actually runs queries and can take hours in the worst case for task completion.
To create async operators, we need to inherit from the corresponding airflow sync operators.
If sync version isn't available, then inherit from airflow BaseOperator
.
To create async sensors, we need to inherit from the corresponding sync sensors.
If sync version isn't available, then inherit from airflow BaseSensorOperator
.
We follow Semantic Versioning <https://semver.org/>
_ for releases.
Check CHANGELOG.rst <https://github.com/astronomer/astronomer-providers/blob/main/CHANGELOG.rst>
_
for the latest changes.
All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
A detailed overview on how to contribute can be found in the
Contributing Guide <https://github.com/astronomer/astronomer-providers/blob/main/CONTRIBUTING.rst>
_.
As contributors and maintainers to this project, you are expected to abide by the
Contributor Code of Conduct <https://github.com/astronomer/astronomer-providers/blob/main/CODE_OF_CONDUCT.md>
_.
apache-airflow
package and we would like to avoid
further bloating the Airflow repomode
which can be poke
and reschedule
.
In async sensors, this param has no usage since tasks gets deferred to Triggerer.Apache License 2.0 <https://github.com/astronomer/astronomer-providers/blob/main/LICENSE>
_
FAQs
Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer
We found that astronomer-providers demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
A malicious npm package targets Solana developers, rerouting funds in 2% of transactions to a hardcoded address.
Security News
Research
Socket researchers have discovered malicious npm packages targeting crypto developers, stealing credentials and wallet data using spyware delivered through typosquats of popular cryptographic libraries.
Security News
Socket's package search now displays weekly downloads for npm packages, helping developers quickly assess popularity and make more informed decisions.