Build Status <https://circleci.com/gh/iopipe/iopipe-python>__ Code Coverage <https://codecov.io/gh/iopipe/iopipe-python>__ PyPI Version <https://badge.fury.io/py/iopipe>__ Apache 2.0 <https://github.com/iopipe/iopipe-python/blob/master/LICENSE>__
Slack <https://iopipe.now.sh>__
This package provides analytics and distributed tracing for event-driven applications running on AWS Lambda.
Installation <https://github.com/iopipe/iopipe-python#installation>__
Usage <https://github.com/iopipe/iopipe-python#usage>__
Configuration <https://github.com/iopipe/iopipe-python#configuration>__Reporting Exceptions <https://github.com/iopipe/iopipe-python#reporting-exceptions>__Custom Metrics <https://github.com/iopipe/iopipe-python#custom-metrics>__Labels <https://github.com/iopipe/iopipe-python#labels>__Core Agent <https://github.com/iopipe/iopipe-python#core-agent>__Disabling Reporting <https://github.com/iopipe/iopipe-python#disabling-reporting>__Step Functions <https://github.com/iopipe/iopipe-python#step-functions>__Plugins <https://github.com/iopipe/iopipe-python#plugins>__
Event Info Plugin <https://github.com/iopipe/iopipe-python#event-info-plugin>__
Logger Plugin <https://github.com/iopipe/iopipe-python#logger-plugin>__
Profiler Plugin <https://github.com/iopipe/iopipe-python#profiler-plugin>__
Trace Plugin <https://github.com/iopipe/iopipe-python#trace-plugin>__
Auto DB Tracing <https://github.com/iopipe/iopipe-python#auto-db-tracing>__Auto HTTP Tracing <https://github.com/iopipe/iopipe-python#auto-http-tracing>__Creating Plugins <https://github.com/iopipe/iopipe-python#creating-plugins>__
Supported Python Versions <https://github.com/iopipe/iopipe-python#supported-python-versions>__
Lambda Layers <https://github.com/iopipe/iopipe-python#lambda-layers>__
Framework Integration <https://github.com/iopipe/iopipe-python#framework-integration>__
Chalice <https://github.com/iopipe/iopipe-python#chalice>__Serverless <https://github.com/iopipe/iopipe-python#serverless>__Zappa <https://github.com/iopipe/iopipe-python#zappa>__Accessing Context <https://github.com/iopipe/iopipe-python#accessing-context>__Contributing <https://github.com/iopipe/iopipe-python#contributing>__
Running Tests <https://github.com/iopipe/iopipe-python#running-tests>__
License <https://github.com/iopipe/iopipe-python#license>__
We expect you will import this library into an existing (or new) Python
project intended to be run on AWS Lambda. On Lambda, functions are
expected to include module dependencies within their project paths, thus
we use -t $PWD. Users building projects with a requirements.txt file
may simply add iopipe to their dependencies.
From your project directory:
.. code:: bash
$ pip install iopipe -t .
$ pip install iopipe[local] -t .
Your folder structure for the function should look similar to:
::
index.py # contains your lambda handler /iopipe - init.py - iopipe.py /requests - init.py - api.py - ...
Installation of the requests library is necessary for local dev/test, but not when running on AWS Lambda as this library is part of the default environment via the botocore library.
More details about lambda deployments are available in the AWS documentation <https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html>__.
Simply use our decorator to report metrics:
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe('your project token here')
@iopipe def handler(event, context): pass
The agent comes preloaded with the Event Info <https://github.com/iopipe/iopipe-python#event-info-plugin>,
Profiler <https://github.com/iopipe/iopipe-python#profiler-plugin>
and Trace <https://github.com/iopipe/iopipe-python#trace-plugin>__
plugins. See the relevant plugin sections for usage.
Configuration
The following may be set as kwargs to the IOpipe class initializer:
``token`` (string: required)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Your IOpipe project token. If not supplied, the environment variable
``$IOPIPE_TOKEN`` will be used if present. `Find your project
token <https://dashboard.iopipe.com/install>`__
``debug`` (bool: optional = False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Debug mode will log all data sent to IOpipe servers. This is also a good
way to evaluate the sort of data that IOpipe is receiving from your
application. If not supplied, the environment variable ``IOPIPE_DEBUG``
will be used if present.
``enabled`` (bool: optional = True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Conditionally enable/disable the agent. For example, you will likely
want to disabled the agent during development. The environment variable
``$IOPIPE_ENABLED`` will also be checked.
``network_timeout`` (int: optional = 5000)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The number of milliseconds IOpipe will wait while sending a report
before timing out. If not supplied, the environment variable
``$IOPIPE_NETWORK_TIMEOUT`` will be used if present.
``timeout_window`` (int: optional = 150)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
By default, IOpipe will capture timeouts by exiting your function 150
milliseconds early from the AWS configured timeout, to allow time for
reporting. You can disable this feature by setting ``timeout_window`` to
``0`` in your configuration. If not supplied, the environment variable
``$IOPIPE_TIMEOUT_WINDOW`` will be used if present.
Reporting Exceptions
The IOpipe decorator will automatically catch, trace and reraise any
uncaught exceptions in your function. If you want to trace exceptions
raised in your case, you can use the .error(exception) method. This
will add the exception to the current report.
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
@iopipe def handler(event, context): raise Exception('This exception will be added to the IOpipe report automatically')
@iopipe def handler(event, context): try: raise Exception('This exception is being caught by your function') except Exception as e: # Makes sure the exception is added to the report context.iopipe.error(e)
It is important to note that a report is sent to IOpipe when error()
is called. So you should only record exceptions this way for failure
states. For caught exceptions that are not a failure state, it is
recommended to use custom metrics (see below).
Custom Metrics
You can log custom values in the data sent upstream to IOpipe using the
following syntax:
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
@iopipe
def handler(event, context):
# the name of the metric must be a string
# numerical (int, long, float) and string types supported for values
context.iopipe.metric('my_metric', 42)
Metric key names are limited to 128 characters, and string values are
limited to 1024 characters.
Labels
~~~~~~
Label invocations sent to IOpipe by calling the ``label`` method with a
string (limit of 128 characters):
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
@iopipe
def handler(event, context):
# the name of the label must be a string
context.iopipe.label('this-invocation-is-special')
Core Agent
~~~~~~~~~~
By default, the IOpipe agent comes pre-loaded with all the bundled
plugins in ``iopipe.contrib.*``. If you prefer to run the agent without
plugins or configure which plugins are used, you can use ``IOpipeCore``:
.. code:: python
from iopipe import IOpipeCore
from iopipe.contrib.trace import TracePlugin
# Load IOpipe with only the trace plugin
iopipe = IOpipeCore(plugins=[TracePlugin()])
@iopipe
def handler(event, context):
pass
Disabling Reporting
You can programmatically disable IOpipe reporting for a single
invocation using the disable method:
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
@iopipe def handler(event, context): if some_condition: context.iopipe.disable()
Reporting will be re-enabled on the next invocation.
Step Functions
IOpipe is compatible with AWS Lambda step functions. To enable step
function tracing:
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
@iopipe.step
def handler(event, context):
pass
The ``@iopipe.step`` decorator will enable step function mode, which
will collect additional meta data about your step functions.
Plugins
-------
IOpipe’s functionality can be extended through plugins. Plugins hook
into the agent lifecycle to allow you to perform additional analytics.
Event Info Plugin
The IOpipe agent comes bundled with an event info plugin that
automatically extracts useful information from the event object and
creates custom metrics for them.
Here’s an example of how to use the event info plugin:
.. code:: python
from iopipe import IOpipe from iopipe.contrib.eventinfo import EventInfoPlugin
iopipe = IOpipe(plugins=[EventInfoPlugin()])
@iopipe def handler(event, context): # do something here
When this plugin is installed, custom metrics will be created automatically for the following event source data:
Now in your IOpipe invocation view you will see useful event information.
Logger Plugin
**Note:** This plugin is in beta. Want to give it a try? Find us on
`Slack <https://iopipe.now.sh>`__.
The IOpipe agent comes bundled with a logger plugin that allows you to
attach IOpipe to the ``logging`` module so that you can see your log
messages in the IOpipe dashboard.
Here’s an example of how to use the logger plugin:
.. code:: python
from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin
iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True)])
@iopipe
def handler(event, context):
context.iopipe.log.info('Handler has started execution')
Since this plugin adds a handler to the ``logging`` module, you can use
``logging`` directly as well:
.. code:: python
import logging
from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin
iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True)])
logger = logging.getLogger()
@iopipe
def handler(event, context):
logger.error('Uh oh')
You can also specify a log name, such as if you only wanted to log
messages for ``mymodule``:
.. code:: python
from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin
iopipe = IOpipe(plugins=[LoggerPlugin('mymodule', enabled=True)])
This would be equivalent to ``logging.getLogger('mymodule')``.
By default the logger plugin is disabled. You must explicitly set
``enabled=True`` when instantiating or use the ``IOPIPE_LOGGER_ENABLED``
environment variable to enable it.
The default logger plugin log level is ``logging.INFO``, but it can be
set like this:
.. code:: python
import logging
from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin
iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True, level=logging.DEBUG)])
Putting IOpipe into ``debug`` mode also sets the log level to
``logging.DEBUG``.
The logger plugin also redirects stdout by default, so you can do the
following:
.. code:: python
from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin
iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True)])
@iopipe
def handler(event, context):
print('I will be logged')
Now in your IOpipe invocation view you will see log messages for that
invocation.
If you prefer your print statements not to be logged, you can disable
this by setting ``redirect_stdout`` to ``False``:
.. code:: python
iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True, redirect_stdout=False)])
**Note:** Due to a change to the way the python3.7 runtime configures
logging, stdout redirection is disabled for this runtime. Use
``context.iopipe.log.*`` instead.
By default the logger plugin will log messages to an in-memory buffer.
If you prefer to log messages to your Lambda function’s ``/tmp``
directory:
.. code:: python
iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True, use_tmp=True)])
With ``use_tmp`` enabled, the plugin will automatically delete log files
written to ``/tmp`` after each invocation.
Profiler Plugin
The IOpipe agent comes bundled with a profiler plugin that allows you to
profile your functions with
cProfile <https://docs.python.org/3/library/profile.html#module-cProfile>__.
Here’s an example of how to use the profiler plugin:
.. code:: python
from iopipe import IOpipe from iopipe.contrib.profiler import ProfilerPlugin
iopipe = IOpipe(plugins=[ProfilerPlugin()])
@iopipe def handler(event, context): # do something here
By default the plugin will be disabled and can be enabled at runtime by
setting the IOPIPE_PROFILER_ENABLED environment variable to
true/True.
If you want to enable the plugin for all invocations:
.. code:: python
iopipe = IOpipe(plugins=[ProfilerPlugin(enabled=True)])
Now in your IOpipe invocation view you will see a “Profiling” section where you can download your profiling report.
Once you download the report you can open it using pstat’s interactive browser with this command:
.. code:: bash
python -m pstats
Within the pstats browser you can sort and restrict the report in a
number of ways, enter the help command for details. Refer to the
pstats Documentation <https://docs.python.org/3/library/profile.html#module-pstats>__.
Trace Plugin
The IOpipe agent comes bundled with a trace plugin that allows you to
perform tracing.
Here’s an example of how to use the trace plugin:
.. code:: python
from iopipe import IOpipe
from iopipe.contrib.trace import TracePlugin
iopipe = IOpipe(plugins=[TracePlugin()])
@iopipe
def handler(event, context):
context.iopipe.mark.start('expensive operation')
# do something here
context.iopipe.mark.end('expensive operation')
Or you can use it as a context manager:
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
@iopipe
def handler(event, context):
with context.iopipe.mark('expensive operation'):
# do something here
Or you can use it as a decorator:
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
@iopipe
def handler(event, context):
@context.iopipe.mark.decorator('expensive operation'):
def expensive_operation():
# do something here
expensive_operation()
Any block of code wrapped with ``start`` and ``end`` or using the
context manager or decorator will be traced and the data collected will
be available on your IOpipe dashboard.
By default, the trace plugin will auto-measure any trace you make. But
you can disable this by setting ``auto_measure`` to ``False``:
.. code:: python
from iopipe import IOpipe
from iopipe.contrib.trace import TracePlugin
iopipe = IOpipe(plugins=[TracePlugin(auto_measure=False)])
@iopipe
def handler(event, context):
with context.iopipe.mark('expensive operation'):
# do something here
# Manually measure the trace
context.iopipe.mark.measure('expensive operation')
Auto DB Tracing
^^^^^^^^^^^^^^^
The trace plugin can trace your database requests automatically. To
enable this feature, set ``auto_db`` to ``True`` or set the
``IOPIPE_TRACE_AUTO_DB_ENABLED`` environment variable. For example:
.. code:: python
iopipe = IOpipe(plugins=[TracePlugin(auto_db=True)])
With ``auto_db`` enabled, you will see traces for any database requests
you make within your function on your IOpipe dashboard. Currently this
feature supports
`MongoDB <https://github.com/mongodb/mongo-python-driver>`__,
`MySQL <https://github.com/PyMySQL>`__,
`PostgreSQL <https://github.com/psycopg/psycopg2>`__ and
`Redis <https://github.com/andymccurdy/redis-py>`__.
MongoDB example:
.. code:: python
from iopipe import IOpipe
from pymongo import MongoClient
iopipe = IOpipe(plugins=[TracePlugin(auto_db=True)])
@iopipe
def handler(event, context):
client = MongoClient("your-host-here", 27017)
db = client.test
db.my_collection.insert_one({"x": 10})
db.my_collection.find_one()
Redis example:
.. code:: python
from iopipe import IOpipe
from redis import Redis
iopipe = IOpipe(plugins=[TracePlugin(auto_db=True)])
@iopipe
def handler(event, context):
r = redis.Redis(host="your-host-here", port=6379, db=0)
r.set("foo", "bar")
r.get("foo")
Auto HTTP Tracing
^^^^^^^^^^^^^^^^^
The trace plugin can trace your HTTP/HTTPS requests automatically. To
enable this feature, set ``auto_http`` to ``True`` or set the
``IOPIPE_TRACE_AUTO_HTTP_ENABLED`` environment variable. For example:
.. code:: python
iopipe = IOpipe(plugins=[TracePlugin(auto_http=True)])
With ``auto_http`` enabled, you will see traces for any HTTP/HTTPS
requests you make within your function on your IOpipe dashboard.
Currently this feature only supports the ``requests`` library, including
``boto3`` and ``botocore`` support.
To filter which HTTP requests are traced use ``http_filter``:
.. code:: python
def http_filter(request, response):
if request['url'].startswith('https://www.iopipe.com'):
# Exceptions raised will delete the trace
raise Exception(Do not trace this URL')
# You can also remove data from the trace
response['headers'].pop('Content-Type', None)
return request, response
iopipe = IOpipe(plugins=[TracePlugin(auto_http=True, http_filter=http_filter)])
To add additional HTTP headers to your ttrace data use ``http_headers``:
.. code:: python
http_headers = ['Cache-Control', 'Etag']
iopipe = IOpipe(plugins=[TracePlugin(auto_http=True, http_headers=http_headers)
.. _plugins-1:
Plugins
-------
Creating Plugins
To create an IOpipe plugin you must implement the
iopipe.plugins.Plugin interface.
Here is a minimal example:
.. code:: python
from iopipe.plugins import Plugin
class MyPlugin(Plugin): name = 'my-plugin' version = '0.1.0' homepage = 'https://github.com/iopipe/my-plugin/' enabled = True
def pre_setup(self, iopipe):
pass
def post_setup(self, iopipe):
pass
def pre_invoke(self, event, context):
pass
def post_invoke(self, event, context):
pass
def post_response(self, response):
pass
def pre_report(self, report):
pass
def post_report(self):
pass
As you can see, this plugin doesn’t do much. If you want to see a
functioning example of a plugin check out the trace plugin at
iopipe.contrib.trace.plugin.TracePlugin.
Plugin Properties ^^^^^^^^^^^^^^^^^
A plugin has the following properties defined:
name: The name of the plugin, must be a string.version: The version of the plugin, must be a string.homepage: The URL of the plugin’s homepage, must be a string.enabled: Whether or not the plugin is enabled, must be a boolean.Plugin Methods ^^^^^^^^^^^^^^
A plugin has the following methods defined:
pre_setup: Is called once prior to the agent initialization; is
passed the iopipe instance.post_setup: Is called once after the agent is initialized; is
passed the iopipe instance.pre_invoke: Is called prior to each invocation; is passed the
event and context of the invocation.post_invoke: Is called after each invocation; is passed the
event and context of the invocation.post_response: Is called after the invocation response; is passed
the response\ value.pre_report: Is called prior to each report being sent; is passed
the report instance.post_report: Is called after each report is sent; is passed the
report instance.This package supports Python 2.7, 3.6 and 3.7, the runtimes supported by AWS Lambda.
IOpipe publishes AWS Lambda Layers <https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html>__
which are publicly available on AWS. Using a framework that supports
lambda layers (such as SAM or Serverless), you can use the following
ARNs for your runtime:
arn:aws:lambda:$REGION:146318645305:layer:IOpipePython:$VERSION_NUMBERarn:aws:lambda:$REGION:146318645305:layer:IOpipePython27:$VERSION_NUMBERWhere $REGION is your AWS region and $VERSION_NUMBER is an
integer representing the IOpipe release. You can get the version number
via the Releases <https://github.com/iopipe/iopipe-python/releases>__
page.
Then in your SAM template (for example), you can add:
.. code:: yaml
Globals: Function: Layers: - arn:aws:lambda:us-east-1:146318645305:layer:IOpipePython:1
And the IOpipe library will be included in your function automatically.
You can also wrap your IOpipe functions without a code change using layers. For example, in your SAM template you can do the following:
.. code:: yaml
Resources: YourFunctionere: Type: 'AWS::Serverless::Function' Properties: CodeUri: path/to/your/code # Automatically wraps the handler with IOpipe Handler: iopipe.handler.wrapper Runtime: python3.6 Environment: Variables: # Specifies which handler IOpipe should run IOPIPE_HANDLER: path.to.your.handler IOPIPE_TOKEN: 'your token here'
We also have an example app <https://github.com/iopipe/iopipe-python/tree/master/acceptance/serverless-layers>__
using layers with Serverless. It also demonstrates how to use layers
without a code change.
NEW: We have also released a Serverless Plugin <https://www.github.com/iopipe/serverless-iopipe-layers>__ to do
all this for you automatically.
IOpipe integrates with popular serverless frameworks. See below for examples. If you don’t see a framework you’d like to see supported, please create an issue.
Chalice
Using IOpipe with the `Chalice <https://github.com/aws/chalice>`__
framework is easy. Wrap your ``app`` like so:
.. code:: python
from chalice import Chalice
from iopipe import IOpipe
iopipe = IOpipe()
app = Chalice(app_name='helloworld')
@app.route('/')
def index():
return {'hello': 'world'}
# Do this after defining your routes
app = iopipe(app)
Serverless
Using IOpipe with
Serverless <https://github.com/serverless/serverless>__ is easy.
First, we highly recommend the
serverless-python-requirements <https://github.com/UnitedIncome/serverless-python-requirements>__
plugin:
.. code:: bash
$ npm install --save-dev serverless-python-requirements
This plugin will add requirements.txt support to Serverless. Once
installed, add the following to your serverless.yml:
.. code:: yaml
plugins: - serverless-python-requirements
Then add iopipe\ to your requirements.txt:
.. code:: bash
$ echo "iopipe" >> requirements.txt
Now Serverless will pip install -r requirements.txt when packaging
your functions.
Keep in mind you still need to add the @iopipe decorator to your
functions. See Usage <https://github.com/iopipe/iopipe-python#usage>__
for details.
Be sure to check out the
serverless-python-requirements <https://github.com/UnitedIncome/serverless-python-requirements>__
README as the plugin has a number of useful features for compiling
AWS Lambda compatible Python packages.
If you’re using the
serverless-wsgi <https://github.com/logandk/serverless-wsgi>__ plugin,
you will need to wrap the wsgi handler it bundles with your function.
The easiest way to do this is to create a wsgi_wrapper.py module in
your project’s root with the following:
.. code:: python
import imp
from iopipe import IOpipe
wsgi = imp.load_source('wsgi', 'wsgi.py')
iopipe = IOpipe() handler = iopipe(wsgi.handler)
Then in your serverless.yml, instead of this:
.. code:: yaml
functions: api: handler: wsgi.handler ...
Use this:
.. code:: yaml
functions: api: handler: wsgi_wrapper.handler ...
Zappa
Using IOpipe with `Zappa <https://github.com/Miserlou/Zappa>`__ is easy.
In your project add the following:
.. code:: python
from iopipe import IOpipe
from zappa.handler import lambda_handler
iopipe = IOpipe()
lambda_handler = iopipe(lambda_handler)
Then in your ``zappa_settings.json`` file include the following:
.. code:: js
{
"lambda_handler": "module.path.to.lambda_handler"
}
Where ``module-path.to.lambda_handler`` is the Python module path to the
``lambda_handler`` you created above. For example, if you put it in
``myproject/__init__.py`` the path would be
``myproject.lambda_handler``.
Accessing Context
If the framework you’re using makes it non-trivial to access the Lamba
context, you can use iopipe.context. The iopipe.context is
None if outside of an invocation.
.. code:: python
from iopipe import IOpipe
iopipe = IOpipe()
if iopipe.context: # do something with context
Contributions are welcome. We use the
black <https://github.com/ambv/black>__ code formatter.
.. code:: bash
pip install black
We recommend using it with
pre-commit <https://pre-commit.com/#install>__:
.. code:: bash
pip install pre-commit pre-commit install
Using these together will auto format your git commits.
If you have tox installed, you can run the Python 2.7 and 3.6 tests
with:
.. code:: bash
tox
If you don’t have tox installed you can also run:
.. code:: bash
python setup.py test
We also have make tasks to run tests in lambci/lambda:build-python
Docker containers:
.. code:: bash
make test
Apache 2.0
FAQs
IOpipe agent for serverless Application Performance Monitoring
We found that iopipe demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 4 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Fluent Assertions is facing backlash after dropping the Apache license for a commercial model, leaving users blindsided and questioning contributor rights.
Research
Security News
Socket researchers uncover the risks of a malicious Python package targeting Discord developers.
Security News
The UK is proposing a bold ban on ransomware payments by public entities to disrupt cybercrime, protect critical services, and lead global cybersecurity efforts.