Luigi Monitor
Send summary messages of your Luigi jobs to Slack.
Overview
Luigi is a great tool for making
job pipelines, but it's hard to know the status of a run. Luigi has
support for error emails, but this requires configuring your machine
to send email, which is a hassle for short-lived EMR clusters. Further,
it sends an email for every failure, which can quickly swamp your inbox.
By contrast, this tool gathers all your failures and missing dependencies
and sends a summary Slack message when the job is finished.
Usage
With default app username:
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, max_print=10):
luigi.run(main_task_cls=MainClass)
With dynamic app username:
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, max_print=10, username="FooBar Monitor"):
luigi.run(main_task_cls=MainClass)
Monitoring and notifying on various events:
Currently supports: SUCCESS
, DEPENDENCY_MISSING
, and FAILURE
By default, all three of the above are monitored and notified on. If, SUCCESS
event is monitored and
all tasks succeed then the notification text is "Job ran successfully" instead of listing all
successful tasks.
import luigi
from luigi_monitor import monitor
...
if __name__ == "__main__":
with monitor(slack_url=<your_slack_url>, events=['DEPENDENCY_MISSING', 'FAILURE']):
luigi.run(main_task_cls=MainClass)
Alternatively:
luigi-monitor --module path.to.module TaskName
NB: if you plan to use luigi-monitor from the command line, set options using luigi.cfg
:
[luigi-monitor]
slack_url=<slack_hook>
max_print=<int>
username=<string>
This is a work in progress. Particularly, note that:
- It only sends notifications via Slack