.. ===============LICENSE_START======================================================= .. Acumos CC-BY-4.0 .. =================================================================================== .. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved. .. =================================================================================== .. This Acumos documentation file is distributed by AT&T and Tech Mahindra .. under the Creative Commons Attribution 4.0 International License (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://creativecommons.org/licenses/by/4.0 .. .. This file is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. .. ===============LICENSE_END=========================================================
|Build Status|
acumos is a client library that allows modelers to push their Python models
to the Acumos platform <https://www.acumos.org/>__.
You will need a Python 3.6 or 3.7 environment in order to install acumos.
Python 3.8 and later can also be used starting with version 0.9.5, some AI
framework like Tensor Flow was not supported in Python 3.8 and later.
You can use Anaconda <https://www.anaconda.com/download/>__
(preferred) or pyenv <https://github.com/pyenv/pyenv>__ to install and
manage Python environments.
If you’re new to Python and need an IDE to start developing, we
recommend using Spyder <https://github.com/spyder-ide/spyder>__ which
can easily be installed with Anaconda.
The acumos package can be installed with pip:
.. code:: bash
pip install acumos
The acumos package uses protocol buffers and assumes you have
the protobuf compiler protoc installed. Please visit the protobuf repository <https://github.com/google/protobuf/releases/tag/v3.4.0>__
and install the appropriate protoc for your operating system.
Installation is as easy as downloading a binary release and adding it to
your system $PATH. This is a temporary requirement that will be
removed in a future version of acumos.
Anaconda Users: You can easily install protoc from an Anaconda package <https://anaconda.org/anaconda/libprotobuf>__ via:
.. code:: bash
conda install -c anaconda libprotobuf
.. |Build Status| image:: https://jenkins.acumos.org/buildStatus/icon?job=acumos-python-client-tox-verify-master :target: https://jenkins.acumos.org/job/acumos-python-client-tox-verify-master/
.. ===============LICENSE_START======================================================= .. Acumos CC-BY-4.0 .. =================================================================================== .. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved. .. =================================================================================== .. This Acumos documentation file is distributed by AT&T and Tech Mahindra .. under the Creative Commons Attribution 4.0 International License (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://creativecommons.org/licenses/by/4.0 .. .. This file is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. .. ===============LICENSE_END=========================================================
This tutorial provides a brief overview of acumos for creating
Acumos models. The tutorial is meant to be followed linearly, and some
code snippets depend on earlier imports and objects. Full examples are
available in the examples/ directory of the Acumos Python client repository <https://gerrit.acumos.org/r/gitweb?p=acumos-python-client.git;a=summary>__.
#. Importing Acumos_
#. Creating A Session_
#. A Simple Model_
#. Exporting Models_
#. Defining Types_
#. Using DataFrames with scikit-learn_
#. Declaring Requirements_
#. Declaring Options_
#. Keras and TensorFlow_
#. Testing Models_
#. More Examples_
First import the modeling and session packages:
.. code:: python
from acumos.modeling import Model, List, Dict, create_namedtuple, create_dataframe
from acumos.session import AcumosSession
An AcumosSession allows you to export your models to Acumos. You can
either dump a model to disk locally, so that you can upload it via the
Acumos website, or push the model to Acumos directly.
If you’d like to push directly to Acumos, create a session with the push_api argument:
.. code:: python
session = AcumosSession(push_api="https://my.acumos.instance.com/push")
See the onboarding page of your Acumos instance website to find the correct
push_api URL to use.
If you’re only interested in dumping a model to disk, arguments aren’t needed:
.. code:: python
session = AcumosSession()
Any Python function can be used to define an Acumos model using Python type hints <https://docs.python.org/3/library/typing.html>__.
Let’s first create a simple model that adds two integers together. Acumos needs to know what the inputs and outputs of your functions are. We can use the Python type annotation syntax to specify the function signature.
Below we define a function add_numbers with int type parameters
x and y, and an int return type. We then build an Acumos
model with an add method.
Note: Function
docstrings <https://www.python.org/dev/peps/pep-0257/>__ are included
with your model and used for documentation, so be sure to include one!
.. code:: python
def add_numbers(x: int, y: int) -> int:
'''Returns the sum of x and y'''
return x + y
model = Model(add=add_numbers)
We can now export our model using the AcumosSession object created
earlier. The push and dump_zip APIs are shown below. The dump_zip method will
save the model to disk so that it can be onboarded via the Acumos website. The
push method pushes the model directly to Acumos.
.. code:: python
session.push(model, 'my-model')
session.dump_zip(model, 'my-model', '~/my-model.zip') # creates ~/my-model.zip
For more information on how to onboard a dumped model via the Acumos website,
see the web onboarding guide <https://docs.acumos.org/en/latest/submodules/portal-marketplace/docs/user-guides/portal-user/portal/portal-onboarding-intro.html#on-boarding-by-web>__.
Note: Pushing a model to Acumos will prompt you for an onboarding token if
you have not previously provided one. The interactive prompt can be avoided by
exporting the ACUMOS_TOKEN environment variable, which corresponds to an
authentication token that can be found in your account settings on the Acumos
website.
In this example, we make a model that can read binary images and output
some metadata about them. This model makes use of a custom type
ImageShape.
We first create a NamedTuple type called ImageShape, which is
like an ordinary tuple but with field accessors. We can then use
ImageShape as the return type of get_shape. Note how
ImageShape can be instantiated as a new object.
.. code:: python
import io
import PIL
ImageShape = create_namedtuple('ImageShape', [('width', int), ('height', int)])
def get_format(data: bytes) -> str:
'''Returns the format of an image'''
buffer = io.BytesIO(data)
img = PIL.Image.open(buffer)
return img.format
def get_shape(data: bytes) -> ImageShape:
'''Returns the width and height of an image'''
buffer = io.BytesIO(data)
img = PIL.Image.open(buffer)
shape = ImageShape(width=img.width, height=img.height)
return shape
model = Model(get_format=get_format, get_shape=get_shape)
Note: Starting in Python 3.6, you can alternatively use this simpler syntax:
.. code:: python
from acumos.modeling import NamedTuple
class ImageShape(NamedTuple):
'''Type representing the shape of an image'''
width: int
height: int
The create_namedtuple function allows us to create types with structure,
however sometimes it's useful to work with unstructured data, such as plain
text, dictionaries or byte strings. The new_type function allows for just
that.
For example, here's a model that takes in unstructured text, and returns the number of words in the text:
.. code:: python
from acumos.modeling import new_type
Text = new_type(str, 'Text')
def count(text: Text) -> int:
'''Counts the number of words in the text'''
return len(text.split(' '))
def create_text(x: int, y: int) -> Text:
'''Returns a string containing ints from x to y'''
return " ".join(map(str, range(x, y+1)))
def reverse_text(text: Text) -> Text:
'''Returns an empty image buffer from dimensions'''
return text[::-1]
By using the new_type function, you inform acumos that Text is
unstructured, and therefore acumos will not create any structured types or
messages for the count function.
You can use the new_type function to create dictionaries or byte string
type unstructured data as shown below.
.. code:: python
from acumos.modeling import new_type
Dict = new_type(dict, 'Dict')
Image = new_type(byte, 'Image')
In this example, we train a RandomForestClassifier using
scikit-learn and use it to create an Acumos model.
When making machine learning models, it’s common to use a dataframe data
structure to represent data. To make things easier, acumos can
create NamedTuple types directly from pandas.DataFrame objects.
NamedTuple types created from pandas.DataFrame objects store
columns as named attributes and preserve column order. Because
NamedTuple types are like ordinary tuple types, the resulting
object can be iterated over. Thus, iterating over a NamedTuple
dataframe object is the same as iterating over the columns of a
pandas.DataFrame. As a consequence, note how np.column_stack can
be used to create a numpy.ndarray from the input df.
Finally, the model returns a numpy.ndarray of int corresponding
to predicted iris classes. The classify_iris function represents
this as List[int] in the signature return.
.. code:: python
import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
iris = load_iris()
X = iris.data
y = iris.target
clf = RandomForestClassifier(random_state=0)
clf.fit(X, y)
# here, an appropriate NamedTuple type is inferred from a pandas DataFrame
X_df = pd.DataFrame(X, columns=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'])
IrisDataFrame = create_dataframe('IrisDataFrame', X_df)
# ==================================================================================
# # or equivalently:
#
# IrisDataFrame = create_namedtuple('IrisDataFrame', [('sepal_length', List[float]),
# ('sepal_width', List[float]),
# ('petal_length', List[float]),
# ('petal_width', List[float])])
# ==================================================================================
def classify_iris(df: IrisDataFrame) -> List[int]:
'''Returns an array of iris classifications'''
X = np.column_stack(df)
return clf.predict(X)
model = Model(classify=classify_iris)
Check out the sklearn examples in the examples directory for full
runnable scripts.
If your model depends on another Python script or package that you wrote, you can
declare the dependency via the acumos.metadata.Requirements class:
.. code:: python
from acumos.metadata import Requirements
Note that only pure Python is supported at this time.
Custom scripts can be included by giving Requirements a sequence of paths
to Python scripts, or directories containing Python scripts. For example, if the
model defined in model.py depended on helper1.py:
::
model_workspace/
├── model.py
├── helper1.py
└── helper2.py
this dependency could be declared like so:
.. code:: python
from helper1 import do_thing
def transform(x: int) -> int:
'''Does the thing'''
return do_thing(x)
model = Model(transform=transform)
reqs = Requirements(scripts=['./helper1.py'])
# using the AcumosSession created earlier:
session.push(model, 'my-model', reqs)
session.dump(model, 'my-model', '~/', reqs) # creates ~/my-model
Alternatively, all Python scripts within model_workspace/ could be included
using:
.. code:: python
reqs = Requirements(scripts=['.'])
Custom packages can be included by giving Requirements a sequence of paths to
Python packages, i.e. directories with an __init__.py file. Assuming that the
package ~/repos/my_pkg contains:
::
my_pkg/
├── __init__.py
├── bar.py
└── foo.py
then you can bundle my_pkg with your model like so:
.. code:: python
from my_pkg.bar import do_thing
def transform(x: int) -> int:
'''Does the thing'''
return do_thing(x)
model = Model(transform=transform)
reqs = Requirements(packages=['~/repos/my_pkg'])
# using the AcumosSession created earlier:
session.push(model, 'my-model', reqs)
session.dump(model, 'my-model', '~/', reqs) # creates ~/my-model
Python packaging and PyPI <https://pypi.org/>__ aren’t
perfect, and sometimes the name of the Python package you import in your
code is different than the package name used to install it. One example
of this is the PIL package, which is commonly installed using a fork called pillow <https://pillow.readthedocs.io>_ (i.e.
pip install pillow will provide the PIL package).
To address this inconsistency, the Requirements
class allows you to map Python package names to PyPI package names. When
your model is analyzed for dependencies by acumos, this mapping is
used to ensure the correct PyPI packages will be used.
In the example below, the req_map parameter is used to declare a
requirements mapping from the PIL Python package to the pillow
PyPI package:
.. code:: python
reqs = Requirements(req_map={'PIL': 'pillow'})
The acumos.metadata.Options class is a collection of options that users may
wish to specify along with their Acumos model. If an Options instance is not
provided to AcumosSession.push, then default options are applied. See the
class docstring for more details.
Below, we demonstrate how options can be used to include additional model metadata
and influence the behavior of the Acumos platform. For example, a license can be
included with a model via the license parameter, either by providing a license
string or a path to a license file. Likewise, we can specify whether or not the Acumos
platform should eagerly build the model microservice via the create_microservice
parameter. Then thanks to the deploy parameter you can specifiy if you want to deploy
this microservice automatically. (Please refer to the appropriate documentation on Acumos
wiki to use this functionality based on an external jenkins server). if create_microservice=True,
deploy can be True or False. But if create_microservice=False, deploy must be set to False
if not, create_microservice will be force to True to create the micro-service and deploy it.
.. code:: python
from acumos.metadata import Options
opts = Options(license="Apache 2.0", # "./path/to/license_file" also works
create_microservice=True, # Build the microservice just after the on-boarding
deploy=True) # Deploy the microservice based on an external Jenkins server
session.push(model, 'my-model', options=opts)
Check out the Keras and TensorFlow examples in the examples/ directory of
the Acumos Python client repository <https://gerrit.acumos.org/r/gitweb?p=acumos-python-client.git;a=summary>__.
The acumos.modeling.Model class wraps your custom functions and
produces corresponding input and output types. This section shows how to
access those types for the purpose of testing. For simplicity, we’ll
create a model using the add_numbers function again:
.. code:: python
def add_numbers(x: int, y: int) -> int:
'''Returns the sum of x and y'''
return x + y
model = Model(add=add_numbers)
The model object now has an add attribute, which acts as a
wrapper around add_numbers. The add_numbers function can be
invoked like so:
.. code:: python
result = model.add.inner(1, 2)
print(result) # 3
The model.add object also has a corresponding wrapped function
that is generated by acumos.modeling.Model. The wrapped function is
the primary way your model will be used within Acumos.
We can access the input_type and output_type attributes to test
that the function works as expected:
.. code:: python
AddIn = model.add.input_type
AddOut = model.add.output_type
add_in = AddIn(1, 2)
print(add_in) # AddIn(x=1, y=2)
add_out = AddOut(3)
print(add_out) # AddOut(value=3)
model.add.wrapped(add_in) == add_out # True
Below are some additional function examples. Note how numpy types
can even be used in type hints, as shown in the numpy_sum function.
.. code:: python
from collections import Counter
import numpy as np
def list_sum(x: List[int]) -> int:
'''Computes the sum of a sequence of integers'''
return sum(x)
def numpy_sum(x: List[np.int32]) -> np.int32:
'''Uses numpy to compute a vectorized sum over x'''
return np.sum(x)
def count_strings(x: List[str]) -> Dict[str, int]:
'''Returns a count mapping from a sequence of strings'''
return Counter(x)
.. ===============LICENSE_START======================================================= .. Acumos CC-BY-4.0 .. =================================================================================== .. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved. .. =================================================================================== .. This Acumos documentation file is distributed by AT&T and Tech Mahindra .. under the Creative Commons Attribution 4.0 International License (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://creativecommons.org/licenses/by/4.0 .. .. This file is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. .. ===============LICENSE_END=========================================================
ACUMOS-4330 <https://jira.acumos.org/browse/ACUMOS-4330>_ACUMOS-4323 <https://jira.acumos.org/browse/ACUMOS-4323>_ACUMOS-4303 <https://jira.acumos.org/browse/ACUMOS-4303>_ACUMOS-4298 <https://jira.acumos.org/browse/ACUMOS-4298>_ACUMOS-4273 <https://jira.acumos.org/browse/ACUMOS-4273>_ACUMOS-4123 <https://jira.acumos.org/browse/ACUMOS-4123>_ACUMOS-4123 <https://jira.acumos.org/browse/ACUMOS-4123>_ACUMOS-4094 <https://jira.acumos.org/browse/ACUMOS-4094>_ACUMOS-3961 <https://jira.acumos.org/browse/ACUMOS-3961>_ACUMOS-3956 <https://jira.acumos.org/browse/ACUMOS-3956>_ACUMOS-3956 <https://jira.acumos.org/browse/ACUMOS-3956>_Gerrit-6275 <https://gerrit.acumos.org/r/c/acumos-python-client/+/6275>_ACUMOS-2712 <https://jira.acumos.org/browse/ACUMOS-2712>_Gerrit-5504 <https://gerrit.acumos.org/r/c/acumos-python-client/+/5504>_(This is the recommended version for the Clio release)
Enhancements
acumos now supports Keras models built with tensorflow.kerasSupport changes
acumos no longer supports Python 3.4Bug fixes
Authentication
Requirements
Bug fixes
Bug fixes
Bug fixes
TensorFlow
Model upload
Authentication token
ACUMOS_TOKEN can be used to short-circuit
the authentication processExtra headers
AcumosSession.push now accepts an optional extra_headers argument,
which will allow users and systems to include additional information when
pushing models to the onboarding serverModeling
Model wrapper
Protobuf and protoc
Keras
Replaced library-specific onboarding functions with “new-style” models
.. ===============LICENSE_START======================================================= .. Acumos CC-BY-4.0 .. =================================================================================== .. Copyright (C) 2017-2018 AT&T Intellectual Property & Tech Mahindra. All rights reserved. .. =================================================================================== .. This Acumos documentation file is distributed by AT&T and Tech Mahindra .. under the Creative Commons Attribution 4.0 International License (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://creativecommons.org/licenses/by/4.0 .. .. This file is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. .. ===============LICENSE_END=========================================================
We use a combination of tox, pytest, and flake8 to test
acumos. Code which is not PEP8 compliant (aside from E501) will be
considered a failing test. You can use tools like autopep8 to
“clean” your code as follows:
.. code:: bash
$ pip install autopep8
$ cd acumos-python-client
$ autopep8 -r --in-place --ignore E501 acumos/ testing/ examples/
Run tox directly:
.. code:: bash
$ cd acumos-python-client
$ export WORKSPACE=$(pwd) # env var normally provided by Jenkins
$ tox
You can also specify certain tox environments to test:
.. code:: bash
$ tox -e py36 # only test against Python 3.6
$ tox -e flake8 # only lint code
A set of integration test is also available in acumos-package/testing/integration_tests.
To run those, use acumos-package/testing/tox-integration.ini as tox config (-c flag),
onboarding tests will be ran with python 3.6 to 3.9.
You will need to set your user credentials and platform configuration in tox-integration.ini.
.. code:: bash
$ tox -c acumos-package/testing/integration_tests
The RST files in the docs/ directory are used to publish HTML pages to ReadTheDocs.io and to build the package long description in setup.py. The symlink from the subdirectory acumos-package to the docs/ directory is required for the Python packaging tools. Those tools build a source distribution from files in the package root, the directory acumos-package. The MANIFEST.in file directs the tools to pull files from directory docs/, and the symlink makes it possible because the tools only look within the package root.
FAQs
Acumos client library for building and pushing Python models
We found that acumos demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.