Security News
Research
Data Theft Repackaged: A Case Study in Malicious Wrapper Packages on npm
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
import asyncio
from celery import Celery
# celery_pool_asyncio importing is optional
# It imports when you run worker or beat if you define pool or scheduler
# but it does not imports when you open REPL or when you run web application.
# If you want to apply monkey patches anyway to make identical environment
# when you use REPL or run web application app it's good idea to import
# celery_pool_asyncio module
import celery_pool_asyncio # noqa
# Sometimes noqa does not disable linter (Spyder IDE)
celery_pool_asyncio.__package__
app = Celery()
@app.task(
bind=True,
soft_time_limit=42, # raises celery.exceptions.SoftTimeLimitExceeded inside the coroutine
time_limit=300, # breaks coroutine execution
)
async def my_task(self, *args, **kwargs):
await asyncio.sleep(5)
@app.task
async def my_simple_task(*args, **kwargs):
await asyncio.sleep(5)
Then run celery:
$ celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPool
There are many monkey patches should be applied to make application working, and
some of them should be applied as early as possible. You are able to switch off
any of them by setting environment variable CPA_MONKEY_DENY
. Remember you
should have a great reason to do it.
Except critical for work features it allows:
# await data sending to broker
async_result = await my_simple_task.delay()
# await wainting for AsyncResult
result = await async_result.get()
You can manually disable any of them by enumerating it comma separated:
$ env CPA_MONKEY_DENY=CELERY.SEND_TASK,ALL_BACKENDS celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPool
Disabling is available for:
SIGNAL.SEND
CELERY.SEND_TASK
WORKCONTROLLER.USE_EVENTLOOP
WORKER.CPU_COUNT
BASERESULTCONSUMER.WAIT_FOR_PENDING
BASERESULTCONSUMER.DRAIN_EVENTS_UNTIL
ASYNCBACKENDMIXIN.WAIT_FOR_PENDING
ALL_BACKENDS
BEAT.SERVICE.START
BEAT.SERVICE.STOP
BUILD_TRACER
KOMBU.UTILS.COMPAT
RPC.RESULTCONSUMER.DRAIN_EVENTS
AMQPBACKEND.DRAIN_EVENTS
AMQPBACKEND.GET_MANY
AMQP_BACKEND
RPC_BACKEND
Default scheduler doesn't work. PersistentScheduler
is subclass of default
celery scheduler.
Running celery with scheduler:
$ celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPool --scheduler celery_pool_asyncio:PersistentScheduler
$ celery beat -A hello_async_celery.app --scheduler celery_pool_asyncio:PersistentScheduler
Embeding also supported:
$ celery worker -A hello_async_celery.app -P celery_pool_asyncio:TaskPool --scheduler celery_pool_asyncio:PersistentScheduler -B
WARNING: embeded scheduler startup is not stable. It starts correctly in ~50% of cases. It looks like race condition. But after correct startup it works well. That's why it's good idea to run scheduler in separated process.
from celery.signals import worker_init, worker_shutting_down
@worker_init.connect
async def do_startup_async(sender, **kwargs):
# Coroutine functions are available after pool initialized
await MyClass.init_async()
@worker_init.connect
def do_startup(sender, **kwargs):
# regular functions are available too
pass
@worker_shutting_down.connect
async def do_shutdown(sender=None, **kwargs):
await MyClass.shutdown()
There is an example project uses celery-pool-asyncio.
celery.signals
-c, --concurency
)soft_time_limit
revoke
CPA_MONKEY_DENY
await AsyncResult.get()
patch_send_task
to own functionpatch_result_get
to await AsyncResult.get
AsyncResult.get
FAQs
Celery pool to run coroutine tasks
We found that celery-pool-asyncio demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.