Security News
The Push to Ban Ransom Payments Is Gaining Momentum
Ransomware costs victims an estimated $30 billion per year and has gotten so out of control that global support for banning payments is gaining momentum.
Readme
Python library for opening URIs as streaming file-like objects
This library provides the omnio.open()
function with an interface very
similar to that of the built in Python open
function. The difference is
that while Python's open
operates only on local filesystem paths, omnio
accepts URIs as well.
It currently supports file
, http
, and s3
URIs, though it may be
expanded to support additional schemes in the future.
In addition, it supports compression and decompression of streams with gzip or bz2.
io
modulerequests
boto3
Read a local binary file:
>>> with omnio.open('example.bin', 'r') as f:
>>> data = f.read()
Read a local text file:
>>> with omnio.open('example.txt', 'rt') as f:
>>> text = f.read()
Read a text file from the web:
>>> with omnio.open('https://example.com/example.txt', 'rt') as f:
>>> text = f.read()
Read a gzipped text file from the web, uncompressing on the fly:
>>> with omnio.open('https://example.com/example.txt.gz', 'rtz') as f:
>>> text = f.read()
Read a text file from s3:
>>> with omnio.open('s3://my-bucket/my-key', 'rt') as f:
>>> text = f.read()
Write a megabyte of random data to s3, compressing with bzip2:
>>> import os
>>> data = os.urandom(1024**2)
>>> with omnio.open('s3://my-bucket/my-key', 'wbj') as f:
>>> f.write(data)
Read a bzip2 compressed csv file into a list of data rows:
>>> import csv
>>> with omnio.open('data/example_data.csv.bz2', 'rtj') as f:
>>> reader = csv.reader(f)
>>> data = list(reader)
The top-level API provides of a single function intended to be
referenced as omnio.open()
. This function API is designed to mimic
Python 3's built-in open()
as much as possible, and should normally
be able to be used as a drop-in replacement.
Signature:
omnio.open(uri, mode='rb', encoding=None, errors=None, newline=None)
Returns:
A file-like object whose type depends on the scheme and the mode.
Parameters:
uri -- URI or local path. Supported URI schemes are file
,
http
, and s3
. Local paths may be specified by as ordinary path
strings.
mode -- Optional string that specifies the mode in which the file is opened. It defaults to 'rb' which means open for reading in binary mode. Supported modes are documented below.
Modes:
Character | Meaning |
---|---|
'r' | open for reading (default) |
'w' | open for writing, truncating the file first |
'b' | binary mode (default) |
't' | text mode |
'z' | use gzip compression |
'j' | use bzip2 compression |
These characters can be composed into valid modes. Binary mode is always the default, so some mode specifications are equivalent. The complete list of supported modes are as follows:
Mode | Meaning |
---|---|
'r', 'rb' | read bytes |
'rt' | read and decode to unicode |
'rz', 'rbz' | read, decompressing gzip to bytes |
'rj', 'rbj' | read, decompressing bzip2 to bytes |
'rtz' | read, decompress gzip to bytes, decode to unicode |
'rtj' | read, decompress bzip2 to bytes, decode to unicode |
'w', 'wb' | write bytes |
'wt' | write unicode, encoding to bytes |
'wz', 'wbz' | write bytes, compress with gzip |
'wj', 'wbj' | write bytes, compress with bzip2 |
'wtz' | write unicode, encode to bytes, compress with gzip |
'wtj' | write unicode, encode to bytes, compress with bzip2 |
Some keyword arguments may be applicable to only certain modes. For
example, encoding
only applies to 't' (text) modes.
Some schemes may not support some modes. For example, the http scheme currently does not support any 'w' (write) modes.
The glob
submodule is intended to be a drop-in replacement for the
standard library glob
module. It provides the functions escape()
,
glob()
, and iglob()
with the signature of each matching that of
the original.
The difference is that glob()
and iglob()
optionally accept URIs.
Currently, local paths, file://
, and s3://
URIs are supported.
The omnio.open
function accepts an optional config
parameter. This
allows for specifying scheme-specific configuration.
The default_config()
method returns a config dictionary with all supported
keys defined along with their default values.
>>> import omnio
>>> omnio.default_config()
{'file': {}, 'http': {'content_iter_chunk_size': 512}, 's3': {'upload_part_size': 5242880, 'boto_client_config_args': [], 'boto_client_config_kwargs': {}}}
To specify alternate values for these parameters, instantiate a default
config, update the dict with the desired values and pass it as a keyword arg
to the omnio.open()
function.
>>> config = omnio.default_config()
>>> config["s3"]["boto_client_config_kwargs"] = {"read_timeout": 600}
>>> with omnio.open("s3://my-bucket/my-key", "rt", config=config) as fd:
fd.read()
FAQs
Python library for opening URIs as streaming file-like objects
We found that omnio demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Ransomware costs victims an estimated $30 billion per year and has gotten so out of control that global support for banning payments is gaining momentum.
Application Security
New SEC disclosure rules aim to enforce timely cyber incident reporting, but fear of job loss and inadequate resources lead to significant underreporting.
Security News
The Python Software Foundation has secured a 5-year sponsorship from Fastly that supports PSF's activities and events, most notably the security and reliability of the Python Package Index (PyPI).