pyodps

ODPS Python SDK and data analysis framework

|PyPI version| |Docs| |License| |Implementation|

Elegent way to access ODPS API. Documentation <http://pyodps.readthedocs.org/>__

Installation

The quick way:

::

pip install pyodps[full]

If you don’t need to use Jupyter, just type

::

pip install pyodps

The dependencies will be installed automatically.

Or from source code (not recommended for production use):

.. code:: shell

$ virtualenv pyodps_env $ source pyodps_env/bin/activate $ pip install git+https://github.com/aliyun/aliyun-odps-python-sdk.git

Dependencies

• Python (>=2.7), including Python 3+, pypy, Python 3.7 recommended
• setuptools (>=3.0)

Run Tests

• install pytest
• copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
• run pytest odps

Usage

.. code:: python

import os from odps import ODPS

Make sure environment variable ALIBABA_CLOUD_ACCESS_KEY_ID already set to Access Key ID of user

while environment variable ALIBABA_CLOUD_ACCESS_KEY_SECRET set to Access Key Secret of user.

Not recommended to hardcode Access Key ID or Access Key Secret in your code.

o = ODPS( os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'), os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'), project='your-project', endpoint='your-endpoint', ) dual = o.get_table('dual') dual.name 'dual' dual.table_schema odps.Schema { c_int_a bigint c_int_b bigint c_double_a double c_double_b double c_string_a string c_string_b string c_bool_a boolean c_bool_b boolean c_datetime_a datetime c_datetime_b datetime } dual.creation_time datetime.datetime(2014, 6, 6, 13, 28, 24) dual.is_virtual_view False dual.size 448 dual.table_schema.columns [<column c_int_a, type bigint>, <column c_int_b, type bigint>, <column c_double_a, type double>, <column c_double_b, type double>, <column c_string_a, type string>, <column c_string_b, type string>, <column c_bool_a, type boolean>, <column c_bool_b, type boolean>, <column c_datetime_a, type datetime>, <column c_datetime_b, type datetime>]

DataFrame API

.. code:: python

from odps.df import DataFrame df = DataFrame(o.get_table('pyodps_iris')) df.dtypes odps.Schema { sepallength float64 sepalwidth float64 petallength float64 petalwidth float64 name string } df.head(5) |==========================================| 1 / 1 (100.00%) 0s sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5) |==========================================| 1 / 1 (100.00%) 12s name sepalwidth 0 Iris-setosa 3.5 1 Iris-setosa 3.2 2 Iris-setosa 3.1 3 Iris-setosa 3.6 4 Iris-setosa 3.9

Command-line and IPython enhancement

::

In [1]: %load_ext odps

In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450>

In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa

Python UDF Debugging Tool

.. code:: python

#file: plus.py from odps.udf import annotate

@annotate('bigint,bigint->bigint') class Plus(object): def evaluate(self, a, b): return a + b

::

$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5

Contributing

For a development install, clone the repository and then install from source:

::

git clone https://github.com/aliyun/aliyun-odps-python-sdk.git cd pyodps pip install -r requirements.txt -e .

If you need to modify the frontend code, you need to install nodejs/npm <https://www.npmjs.com/>__. To build and install your frontend code, use

::

python setup.py build_js python setup.py install_js

License

Licensed under the Apache License 2.0 <https://www.apache.org/licenses/LICENSE-2.0.html>__

.. |PyPI version| image:: https://img.shields.io/pypi/v/pyodps.svg?style=flat-square :target: https://pypi.python.org/pypi/pyodps .. |Docs| image:: https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat-square :target: http://pyodps.readthedocs.org/ .. |License| image:: https://img.shields.io/pypi/l/pyodps.svg?style=flat-square :target: https://github.com/aliyun/aliyun-odps-python-sdk/blob/master/License .. |Implementation| image:: https://img.shields.io/pypi/implementation/pyodps.svg?style=flat-square