.. image:: https://readthedocs.org/projects/bloop/badge?style=flat-square :target: http://bloop.readthedocs.org/ .. image:: https://img.shields.io/travis/numberoverzero/bloop/master.svg?style=flat-square :target: https://travis-ci.org/numberoverzero/bloop .. image:: https://img.shields.io/gitter/room/numberoverzero/bloop.svg?style=flat-square :target: https://gitter.im/numberoverzero/bloop .. image:: https://img.shields.io/pypi/v/bloop.svg?style=flat-square :target: https://pypi.python.org/pypi/bloop
Bloop is an object mapper for DynamoDB and DynamoDBStreams.
Requires Python 3.6+
::
pip install bloop
First, we need to import all the things:
.. code-block:: python
>>> from bloop import (
... BaseModel, Boolean, Column, String, UUID,
... GlobalSecondaryIndex, Engine
... )
Next we'll define the account model (with streaming enabled), and create the backing table:
.. code-block:: python
>>> class Account(BaseModel):
... class Meta:
... stream = {
... "include": {"old", "new"}
... }
... id = Column(UUID, hash_key=True)
... name = Column(String)
... email = Column(String)
... by_email = GlobalSecondaryIndex(projection='keys', hash_key='email')
... verified = Column(Boolean, default=False)
...
>>> engine = Engine()
>>> engine.bind(Account)
Let's make a few users and persist them:
.. code-block:: python
>>> import uuid
>>> admin = Account(id=uuid.uuid4(), email="admin@domain.com")
>>> admin.name = "Admin McAdminFace"
>>> support = Account(name="this-is-fine.jpg", email="help@domain.com")
>>> support.id = uuid.uuid4()
>>> engine.save(admin, support)
Or do the same in a transaction:
.. code-block:: python
>>> with engine.transaction() as tx:
... tx.save(admin)
... tx.save(support)
...
>>>
And find them again:
.. code-block:: python
>>> q = engine.query(
... Account.by_email,
... key=Account.email=="help@domain.com"
... )
>>> q.first()
Account(email='help@domain.com',
id=UUID('d30e343f-f067-4fe5-bc5e-0b00cdeaf2ba'),
verified=False)
.. code-block:: python
>>> s = engine.scan(
... Account,
... filter=Account.name.begins_with("Admin")
... )
>>> s.one()
Account(email='admin@domain.com',
id=UUID('08da44ac-5ff6-4f70-8a3f-b75cadb4dd79'),
name='Admin McAdminFace',
verified=False)
Let's find them in the stream:
.. code-block:: python
>>> stream = engine.stream(Account, "trim_horizon")
>>> next(stream)
{'key': None,
'meta': {'created_at': datetime.datetime(...),
'event': {'id': 'cbb9a9b45eb0a98889b7da85913a5c65',
'type': 'insert',
'version': '1.1'},
'sequence_number': '100000000000588052489'},
'new': Account(
email='help@domain.com',
id=UUID('d30e343f-...-0b00cdeaf2ba'),
name='this-is-fine.jpg',
verified=False),
'old': None}
>>> next(stream)
{'key': None,
'meta': {'created_at': datetime.datetime(...),
'event': {'id': 'cbdfac5671ea38b99017c4b43a8808ce',
'type': 'insert',
'version': '1.1'},
'sequence_number': '200000000000588052506'},
'new': Account(
email='admin@domain.com',
id=UUID('08da44ac-...-b75cadb4dd79'),
name='Admin McAdminFace',
verified=False),
'old': None}
>>> next(stream)
>>> next(stream)
>>>
Check out the User Guide_ or Public API Reference_ to create your own nested types, overlapping models,
set up cross-region replication in less than 20 lines, and more!
.. _User Guide: https://bloop.readthedocs.io/en/latest/user/quickstart.html .. _Public API Reference: https://bloop.readthedocs.io/en/latest/api/public.html
This changelog structure is based on Keep a Changelog v0.3.0.
Bloop follows Semantic Versioning 2.0.0 and a draft appendix__ for its Public API.
__ http://keepachangelog.com/en/0.3.0/ __ http://semver.org/spec/v2.0.0.html __ https://gist.github.com/numberoverzero/c5d0fc6dea624533d004239a27e545ad
IMeta.columns_by_dynamo_nameFixed an issue where copying an Index would lose projection information when the projection mode was
"include". This fix should have no effect for most users. You would only run into this issue if you
were manually calling bind_index with copy=True on a projection mode "include" or you subclass
a model that has an index with that projection mode. This does not require a major version change since
there is no reasonable workaround that would be broken by making this fix. For example, a user might
decide to monkeypatch Index.__copy__, bind_index or refresh_index to preserve the projection
information. Those workarounds will not be broken by this change. For an example of the issue, see
Issue #147_.
Index.projection is now a set instead of a list`. Since Columnimplementshashthis won't affect any existing calls that pass in lists. To remain consistent, this change is reflected inEngine.search, Search.init, Index.init``, and any docs or examples that refer to passing
lists/sets of Columns.Index.__copy__ preserves Index.projection["included"] when projection mode is "include"... _Issue #147: https://github.com/numberoverzero/bloop/issues/147
Remove deprecated keyword atomic= from Engine.save and Engine.delete, and Type._dump must return
a bloop.actions.Action instance. See the Migration Guide for context on these changes, and sample code to
easily migrate your existing custom Types.
util.default_context can be used to create a new load/dump context and respects existing dict
objects and keys (even if empty).Type._dump must return a bloop.actions.Action now. Most users won't need to change any code since custom
types usually override dynamo_dump. If you have implemented your own _dump function, you can probably
just use actions.wrap and actions.unwrap to migrate:
.. code-block:: python
def _dump(self, value, *, context, **kwargs): value = actions.unwrap(value) # the rest of your function here return actions.wrap(value)
atomic= keyword has been removed from Engine.save and Engine.delete.bloop.exceptions.UnknownType is no longer raised and has been removed.BaseModel._load and BaseModel._dump have been removed. These were not documented or used
anywhere in the code base, and unpack_from_dynamodb should be used where _load was anyway.Engine._load and Engine._dump have been removed. These were not documented and are trivially
replaced with calls to typedef._load and typedef._dump instead.dumped attr for Conditions is no longer needed since there's no need to dump objects except
at render time.Bug fix. Thanks to @wilfre in PR #141_!
.. _PR #141: https://github.com/numberoverzero/bloop/pull/141
bloop.stream.shard.py::unpack_shards no longer raises when a Shard in the DescribeStream has a ParentId
that is not also available in the DescribeStream response (the parent shard has been deleted). Previously the
code would raise while trying to link the two shard objects in memory. Now, the shard will have a ParentId of
None.The atomic= keyword for Engine.save and Engine.delete is deprecated and will be removed in 3.0.
In 2.4 your code will continue to work but will raise DeprecationWarning when you specify a value for atomic=.
The Type._dump function return value is changing to Union[Any, bloop.Action] in 2.4 to prepare for the
change in 3.0 to exclusively returning a bloop.Action. For built-in types and most custom types that only
override dynamo_dump this is a no-op, but if you call Type._dump you can use bloop.actions.unwrap() on
the result to get the inner value. If you have a custom Type._dump method it must return an action in 3.0. For
ease of use you can use bloop.actions.wrap() which will specify either the SET or REMOVE action to match
existing behavior. Here's an example of how you can quickly modify your code:
.. code-block:: python
# current pre-2.4 method, continues to work until 3.0
def _dump(self, value, **kwargs):
value = self.dynamo_dump(value, **kwargs)
if value is None:
return None
return {self.backing_type: value}
# works in 2.4 and 3.0
from bloop import actions
def _dump(self, value, **kwargs):
value = actions.unwrap(value)
value = self.dynamo_dump(value, **kwargs)
return actions.wrap(value)
Note that this is backwards compatible in 2.4: Type._dump will not change unless you opt to pass the new
Action object to it.
SearchIterator.token provides a way to start a new Query or Scan from a previous query/scan's state.
See Issue #132_.SearchIterator.move_to takes a token to update the search state. Count/ScannedCount state are lost when
moving to a token.Engine.delete and Engine.save take an optional argument sync= which can be used to update objects with
the old or new values from DynamoDB after saving or deleting. See the Return Values_ section of the User Guide
and Issue #137_.bloop.actions expose a way to manipulate atomic counters and sets. See the Atomic Counters_ section of the
User Guide and Issue #136_... _Issue #132: https://github.com/numberoverzero/bloop/issues/132 .. _Return Values: https://bloop.readthedocs.io/en/latest/user/engine.html#return-values .. _Issue #137: https://github.com/numberoverzero/bloop/issues/137 .. _Atomic Counters: https://bloop.readthedocs.io/en/latest/user/engine.html#actions .. _Issue #136: https://github.com/numberoverzero/bloop/issues/136
atomic= keyword for Engine.save and Engine.delete emits DeprecationWarning and will be
removed in 3.0.Type._dump will return a bloop.action.Action object if one is passed in, in preparation for the
change in 3.0.Engine.bind is much faster for multi-model tables. See Issue #130_.
.. _Issue #130: https://github.com/numberoverzero/bloop/issues/130
SessionWrapper caches DescribeTable responses. You can clear these with
SessionWrapper.clear_cache; mutating calls such as .enable_ttl will invalidate the cached description.Engine.bind will call CreateTable at most once per table. Subsequent calls to bind
will call CreateTable again.Minor bug fix.
bloop.conditions.iter_columns no longer yields None on Condition() (or
any other condition whose .column attribute is None).This release adds support for Transactions_ and On-Demand Billing_. Transactions can include changes across
tables, and provide ACID guarantees at a 2x throughput cost and a limit of 10 items per transaction.
See the User Guide__ for details.
.. code-block:: python
with engine.transaction() as tx:
tx.save(user, tweet)
tx.delete(event, task)
tx.check(meta, condition=Metadata.worker_id == current_worker)
__ https://bloop.readthedocs.io/en/latest/user/transactions.html
Engine.transaction(mode="w") returns a transaction object which can be used directly or as a context manager.
By default this creates a WriteTransaction, but you can pass mode="r" to create a read transaction.
WriteTransaction and ReadTransaction can be prepared for committing with .prepare() which returns a
PreparedTransaction which can be committed with .commit() some number of times. These calls are usually
handled automatically when using the read/write transaction as a context manager::
tx = engine.transaction() tx.save(user) p = tx.prepare() p.commit()
with engine.transaction() as tx: tx.save(user)
Meta supports On-Demand Billing_::
class MyModel(BaseModel): id = Column(String, hash_key=True) class Meta: billing = {"mode": "on_demand"}
(internal) bloop.session.SessionWrapper.transaction_read and
bloop.session.SessionWrapper.transaction_write can be used to call TransactGetItems and TransactWriteItems
with fully serialized request objects. The write api requires a client request token to provide idempotency guards,
but does not provide temporal bounds checks for those tokens.
Engine.load now logs at INFO instead of WARNING when failing to load some objects.Meta.ttl["enabled"] will now be a literal True or False after binding the model, rather than the string
"enabled" or "disabled".Meta.encryption or Meta.backups is None or missing, it will now be set after binding the model.Meta and GSI read/write units are not validated if billing mode is "on_demand" since they will be 0 and the
provided setting is ignored... _Transactions: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transactions.html .. _On-Demand Billing: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.OnDemand
DynamicList and DynamicMap types can store arbitrary values, although they will only be loaded as their
primitive, direct mapping to DynamoDB backing types. For example::
class MyModel(BaseModel): id = Column(String, hash_key=True) blob = Column(DynamicMap) i = MyModel(id="i") i.blob = {"foo": "bar", "inner": [True, {1, 2, 3}, b""]}
Meta supports Continuous Backups_ for Point-In-Time Recovery::
class MyModel(BaseModel): id = Column(String, hash_key=True) class Meta: backups = {"enabled": True}
SearchIterator exposes an all() method which eagerly loads all results and returns a single list.
Note that the query or scan is reset each time the method is called, discarding any previously buffered state.
String and Binary types load None as "" and b"" respectively."" or b"") will no longer throw a botocore exception, and will instead
be treated as None. This brings behavior in line with the Set, List, and Map types... _Continuous Backups: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BackupRestore.html
Added support for Server-Side Encryption. This uses an AWS-managed Customer Master Key (CMK) stored in KMS
which is managed for free_: "You are not charged for the following: AWS-managed CMKs, which are automatically
created on your behalf when you first attempt to encrypt a resource in a supported AWS service."
Meta supports Server Side Encryption::
class MyModel(BaseModel): id = Column(String, hash_key=True) class Meta: encryption = {"enabled": True}
.. _Server-Side Encryption: https://aws.amazon.com/blogs/aws/new-encryption-at-rest-for-dynamodb/ .. _KMS: https://console.aws.amazon.com/iam/#/encryptionKeys .. _managed for free: https://aws.amazon.com/kms/pricing/
Fix a bug where the last records in a closed shard in a Stream were dropped. See Issue #87_ and
PR #112_.
.. _Issue #111: https://github.com/numberoverzero/bloop/issues/111 .. _PR #112: https://github.com/numberoverzero/bloop/pull/112
Stream no longer drops the last records from a closed Shard when moving to the child shard.2.0.0 introduces 4 significant new features:
Model inheritance and mixins
Table name templates: table_name_template="prod-{table_name}"
TTL_ support: ttl = {"column": "not_after"}
Column defaults::
verified=Column(Boolean, default=False) not_after = Column( Timestamp, default=lambda: ( datetime.datetime.now() + datetime.timedelta(days=30) ) )
Python 3.6.0 is now the minimum required version, as Bloop takes advantage of __set_name__ and
__init_subclass__ to avoid the need for a Metaclass.
A number of internal-only and rarely-used external methods have been removed, as the processes which required them have been simplified:
Column.get, Column.set, Column.delete in favor of their descriptor protocol counterpartsbloop.Type._register is no longer necessary before using a custom TypeIndex._bind is replaced by helpers bind_index and refresh_index. You should not need to call these.Engine takes an optional keyword-only arg "table_name_template" which takes either a string used to format
each name, or a function which will be called with the model to get the table name of. This removes the need to
connect to the before_create_table signal, which also could not handle multiple table names for the same model.
With this change BaseModel.Meta.table_name will no longer be authoritative, and the engine must be consulted to
find a given model's table name. An internal function Engine._compute_table_name is available, and the
per-engine table names may be added to the model.Meta in the future. (see Issue #96_)
A new exception InvalidTemplate is raised when an Engine's table_name_template is a string but does
not contain the required "{table_name}" formatting key.
You can now specify a TTL_ (see Issue #87_) on a model much like a Stream::
class MyModel(BaseModel): class Meta: ttl = { "column": "expire_after" }
id = Column(UUID, hash_key=True)
expire_after = Column(Timestamp)
A new type, Timestamp was added. This stores a datetime.datetime as a unix timestamp in whole seconds.
Corresponding Timestamp types were added to the following extensions, mirroring the DateTime extension:
bloop.ext.arrow.Timestamp, bloop.ext.delorean.Timestamp, and bloop.ext.pendulum.Timestamp.
Column takes an optional kwarg default, either a single value or a no-arg function that returns a value.
Defaults are applied only during BaseModel.__init__ and not when loading objects from a Query, Scan, or Stream.
If your function returns bloop.util.missing, no default will be applied. (see PR #90, PR #105
for extensive discussion)
(internal) A new abstract interface, bloop.models.IMeta was added to assist with code completion. This
fully describes the contents of a BaseModel.Meta instance, and can safely be subclassed to provide hints to your
editor::
class MyModel(BaseModel): class Meta(bloop.models.IMeta): table_name = "my-table" ...
(internal) bloop.session.SessionWrapper.enable_ttl can be used to enable a TTL on a table. This SHOULD NOT
be called unless the table was just created by bloop.
(internal) helpers for dynamic model inheritance have been added to the bloop.models package:
bloop.models.bind_columnbloop.models.bind_indexbloop.models.refresh_indexbloop.models.unbindDirect use is discouraged without a strong understanding of how binding and inheritance work within bloop.
.. _TTL: https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-dynamodb-now-supports-automatic-item-expiration-with-time-to-live-ttl/ .. _Issue #96: https://github.com/numberoverzero/bloop/issues/96 .. _Issue #87: https://github.com/numberoverzero/bloop/issues/87 .. _PR #90: https://github.com/numberoverzero/bloop/pull/90 .. _PR #105: https://github.com/numberoverzero/bloop/pull/105
Python 3.6 is the minimum supported version.
BaseModel no longer requires a Metaclass, which allows it to be used as a mixin to an existing class which
may have a Metaclass.
BaseModel.Meta.init no longer defaults to the model's __init__ method, and will instead use
cls.__new__(cls) to obtain an instance of the model. You can still specify a custom initialization function::
class MyModel(BaseModel): class Meta: @classmethod def init(_): instance = MyModel.new(MyModel) instance.created_from_init = True id = Column(...)
Column and Index support the shallow copy method __copy__ to simplify inheritance with custom subclasses.
You may override this to change how your subclasses are inherited.
DateTime explicitly guards against tzinfo is None, since datetime.astimezone started silently allowing
this in Python 3.6 -- you should not use a naive datetime for any reason.
Column.model_name is now Column.name, and Index.model_name is now Index.name.
Column(name=) is now Column(dynamo_name=) and Index(name=) is now Index(dynamo_name=)
The exception InvalidModel is raised instead of InvalidIndex.
The exception InvalidSearch is raised instead of the following: InvalidSearchMode, InvalidKeyCondition,
InvalidFilterCondition, and InvalidProjection.
(internal) bloop.session.SessionWrapper methods now require an explicit table name, which is not read from the
model name. This exists to support different computed table names per engine. The following methods now require
a table name: create_table, describe_table (new), validate_table, and enable_ttl (new).
Column.get, Column.set, and Column.delete helpers have been removed in favor of using the Descriptor
protocol methods directly: Column.__get__, Column.__set__, and Column.__delete__.bloop.Type no longer exposes a _register method; there is no need to register types before using them,
and you can remove the call entirely.Column.model_name, Index.model_name, and the kwargs Column(name=), Index(name=) (see above)InvalidIndex has been removed.InvalidComparisonOperator was unused and has been removed.UnboundModel is no longer raised during Engine.bind and has been removed.InvalidSearchMode, InvalidKeyCondition, InvalidFilterCondition, and InvalidProjection
have been removed.Index._bind has been replaced with the more complete solutions in bloop.models.bind_column and
bloop.models.bind_index.__ https://pypi.python.org/pypi/declare
This release is exclusively to prepare users for the name/model_name/dynamo_name changes coming in 2.0;
your 1.2.0 code will continue to work as usual but will raise DeprecationWarning when accessing model_name on
a Column or Index, or when specifying the name= kwarg in the __init__ method of Column,
GlobalSecondaryIndex, or LocalSecondaryIndex.
Previously it was unclear if Column.model_name was the name of this column in its model, or the name of the model
it is attached to (eg. a shortcut for Column.model.__name__). Additionally the name= kwarg actually mapped to
the object's .dynamo_name value, which was not obvious.
Now the Column.name attribute will hold the name of the column in its model, while Column.dynamo_name will
hold the name used in DynamoDB, and is passed during initialization as dynamo_name=. Accessing model_name or
passing name= during __init__ will raise deprecation warnings, and bloop 2.0.0 will remove the deprecated
properties and ignore the deprecated kwargs.
Column.name is the new home of the Column.model_name attribute. The same is true for
Index, GlobalSecondaryIndex, and LocalSecondaryIndex.__init__ method of Column, Index, GlobalSecondaryIndex, and LocalSecondaryIndex now takes
dynamo_name= in place of name=.Column.model_name raises DeprecationWarning, and the same for Index/GSI/LSI.Column(name=) raises DeprecationWarning, and the same for Index/GSI/LSI.When a Model's Meta does not explicitly set read_units and write_units, it will only default to 1/1 if the
table does not exist and needs to be created. If the table already exists, any throughput will be considered
valid. This will still ensure new tables have 1/1 iops as a default, but won't fail if an existing table has more
than one of either.
There is no behavior change for explicit integer values of read_units and write_units: if the table does
not exist it will be created with those values, and if it does exist then validation will fail if the actual values
differ from the modeled values.
An explicit None for either read_units or write_units is equivalent to omitting the value, but allows
for a more explicit declaration in the model.
Because this is a relaxing of a default only within the context of validation (creation has the same semantics) the
only users that should be impacted are those that do not declare read_units and write_units and rely on the
built-in validation failing to match on values != 1. Users that rely on the validation to succeed on tables with
values of 1 will see no change in behavior. This fits within the extended criteria of a minor release since there
is a viable and obvious workaround for the current behavior (declare 1/1 and ensure failure on other values).
When a Query or Scan has projection type "count", accessing the count or scanned properties will
immediately execute and exhaust the iterator to provide the count or scanned count. This simplifies the previous
workaround of calling next(query, None) before using query.count.
Issue #95_)__next__
to raise botocore.exceptions.ParamValidationError (see Issue #95_).. _Issue #95: https://github.com/numberoverzero/bloop/issues/95
Engine.bind takes optional kwarg skip_table_setup
to skip CreateTable and DescribeTable calls (see Issue #83_)Issue #71_).. _Issue #83: https://github.com/numberoverzero/bloop/issues/83 .. _Issue #71: https://github.com/numberoverzero/bloop/issues/71
Bug fix.
as documented__ we should be using lexicographical sorting
on the opaque string. However, without leading 0s that sort fails, and we must assume the string represents an
integer to sort on. Particularly annoying, tomorrow the SequenceNumber could start with non-numeric characters
and still conform to the spec, but the sorting-as-int assumption breaks. However, we can't properly sort without
making that assumption.Minor bug fix.
ext.arrow, ext.delorean, and ext.pendulum now load and dump None correctly.Bug fixes.
arrow, delorean, and pendulum extensions now have a default timezone of "utc" instead of
datetime.timezone.utc. There are open issues for both projects to verify if that is the expected behavior.1.0.0 is the culmination of just under a year of redesigns, bug fixes, and new features. Over 550 commits, more than 60 issues closed, over 1200 new unit tests. At an extremely high level:
EngineView and its associated temporary config changes.See the Migration Guide above for specific examples of breaking changes and how to fix them, or the
User Guide__ for a tour of the new Bloop. Lastly, the Public and Internal API References are
finally available and should cover everything you need to extend or replace whole subsystems in Bloop
(if not, please open an issue).
__ https://bloop.readthedocs.io/en/latest/user/quickstart.html#user-quickstart
bloop.signals exposes Blinker signals which can be used to monitor object changes, when
instances are loaded from a query, before models are bound, etc.
before_create_tableobject_loadedobject_savedobject_deletedobject_modifiedmodel_boundmodel_createdmodel_validatedEngine.stream can be used to iterate over all records in a stream, with a total ordering over approximate
record creation time. Use engine.stream(model, "trim_horizon") to get started. See the
User Guide__ for details.
New exceptions RecordsExpired and ShardIteratorExpired for errors in stream state
New exceptions Invalid* for bad input subclass BloopException and ValueError
DateTime types for the three most common date time libraries:
bloop.ext.arrow.DateTimebloop.ext.delorean.DateTimebloop.ext.pendulum.DateTimemodel.Meta has a new optional attribute stream which can be used to enable a stream on the model's table.
model.Meta exposes the same projection attribute as Index so that (index or model.Meta).projection
can be used interchangeably
New Stream class exposes DynamoDBStreams API as a single iterable with powerful seek/jump options, and simple
json-friendly tokens for pausing and resuming iteration.
Over 1200 unit tests added
Initial integration tests added
(internal) bloop.conditions.ReferenceTracker handles building #n0, :v1, and associated values.
Use any_ref to build a reference to a name/path/value, and pop_refs when backtracking (eg. when a value is
actually another column, or when correcting a partially valid condition)
(internal) bloop.conditions.render is the preferred entry point for rendering, and handles all permutations
of conditions, filters, projections. Use over ConditionRenderer unless you need very specific control over
rendering sequencing.
(internal) bloop.session.SessionWrapper exposes DynamoDBStreams operations in addition to previous
bloop.Client wrappers around DynamoDB client
(internal) New supporting classes streams.buffer.RecordBuffer, streams.shard.Shard, and
streams.coordinator.Coordinator to encapsulate the hell^Wjoy that is working with DynamoDBStreams
(internal) New class util.Sentinel for placeholder values like missing and last_token
that provide clearer docstrings, instead of showing func(..., default=object<0x...>) these will show
func(..., default=Sentinel<[Missing]>)
__ https://bloop.readthedocs.io/en/latest/user/streams.html#user-streams
bloop.Column emits object_modified on __set__ and __del__
Conditions now check if they can be used with a column's typedef and raise InvalidCondition when they can't.
For example, contains can't be used on Number, nor > on Set(String)
bloop.Engine no longer takes an optional bloop.Client but instead optional dynamodb and
dynamodbstreams clients (usually created from boto3.client("dynamodb") etc.)
Engine no longer takes **config -- its settings have been dispersed to their local touch points
atomic is a parameter of save and delete and defaults to Falseconsistent is a parameter of load, query, scan and defaults to Falseprefetch has no equivalent, and is baked into the new Query/Scan iterator logicstrict is a parameter of a LocalSecondaryIndex, defaults to TrueEngine no longer has a context to create temporary views with different configuration
Engine.bind is no longer by keyword arg only: engine.bind(MyBase) is acceptable in addition to
engine.bind(base=MyBase)
Engine.bind emits new signals before_create_table, model_validated, and model_bound
Engine.delete and Engine.save take *objs instead of objs to easily save/delete small multiples of
objects (engine.save(user, tweet) instead of engine.save([user, tweet]))
Engine guards against loading, saving, querying, etc against abstract models
Engine.load raises MissingObjects instead of NotModified (exception rename)
Engine.scan and Engine.query take all query and scan arguments immediately, instead of using the builder
pattern. For example, engine.scan(model).filter(Model.x==3) has become
engine.scan(model, filter=Model.x==3).
bloop.exceptions.NotModified renamed to bloop.exceptions.MissingObjects
Any code that raised AbstractModelException now raises UnboundModel
bloop.types.DateTime is now backed by datetime.datetime instead of arrow. Only supports UTC now, no
local timezone. Use the bloop.ext.arrow.DateTime class to continue using arrow.
The query and scan interfaces have been entirely refactored: count, consistent, ascending and other
properties are part of the Engine.query(...) parameters. all() is no longer needed, as Engine.scan and
.query immediately return an iterable object. There is no prefetch setting, or limit.
The complete property for Query and Scan have been replaced with exhausted, to be consistent with the Stream
module
The query and scan iterator no longer cache results
The projection parameter is now required for GlobalSecondaryIndex and LocalSecondaryIndex
Calling Index.__set__ or Index.__del__ will raise AttributeError. For example,
some_user.by_email = 3 raises if User.by_email is a GSI
bloop.Number replaces bloop.Float and takes an optional decimal.Context for converting numbers.
For a less strict, lossy Float type see the Patterns__ section of the User Guide
bloop.String.dynamo_dump no longer calls str() on the value, which was hiding bugs where a non-string
object was passed (eg. some_user.name = object() would save with a name of <object <0x...>)
bloop.DateTime is now backed by datetime.datetime and only knows UTC in a fixed format. Adapters for
arrow, delorean, and pendulum are available in bloop.ext
bloop.DateTime does not support naive datetimes; they must always have a tzinfo
docs:
(internal) Path lookups on Column (eg. User.profile["name"]["last"]) use simpler proxies
(internal) Proxy behavior split out from Column's base class bloop.conditions.ComparisonMixin
for a cleaner namespace
(internal) bloop.conditions.ConditionRenderer rewritten, uses a new bloop.conditions.ReferenceTracker
with a much clearer api
(internal) ConditionRenderer can backtrack references and handles columns as values (eg.
User.name.in_([User.email, "literal"]))
(internal) _MultiCondition logic rolled into bloop.conditions.BaseCondition, AndCondition and
OrCondition no longer have intermediate base class
(internal) AttributeExists logic rolled into bloop.conditions.ComparisonCondition
(internal) bloop.tracking rolled into bloop.conditions and is hooked into the object_* signals.
Methods are no longer called directly (eg. no need for tracking.sync(some_obj, engine))
(internal) update condition is built from a set of columns, not a dict of updates to apply
(internal) bloop.conditions.BaseCondition is a more comprehensive base class, and handles all manner of
out-of-order merges (and(x, y) vs and(y, x) where x is an and condition and y is not)
(internal) almost all *Condition classes simply implement __repr__ and render; BaseCondition
takes care of everything else
(internal) bloop.Client became bloop.session.SessionWrapper
(internal) Engine._dump takes an optional context, **kwargs, matching the
signature of Engine._load
(internal) BaseModel no longer implements __hash__, __eq__, or __ne__ but ModelMetaclass will
always ensure a __hash__ function when the subclass is created
(internal) Filter and FilterIterator rewritten entirely in the bloop.search module across multiple
classes
__ https://bloop.readthedocs.io/en/latest/user/patterns.html#patterns-float
AbstractModelException has been rolled into UnboundModelall() method has been removed from the query and scan iterator interface. Simply iterate with
next(query) or for result in query:Query.results and Scan.results have been removed and results are no longer cached. You can begin the
search again with query.reset()new_base() function has been removed in favor of subclassing BaseModel directlybloop.Float has been replaced by bloop.Numberbloop.engine.LoadManager logic was rolled into bloop.engine.load(...)EngineView has been removed since engines no longer have a baseline config and don't need a
context to temporarily modify itEngine._update has been removed in favor of util.unpack_from_dynamodbEngine._instance has been removed in favor of directly creating instances from
model.Meta.init() in unpack_from_dynamodbColumn.contains(value) now renders value with the column typedef's inner type. Previously, the container
type was used, so Data.some_list.contains("foo")) would render as (contains(some_list, ["f", "o", "o"]))
instead of (contains(some_list, "foo"))Set renders correct wire format -- previously, it incorrectly sent {"SS": [{"S": "h"}, {"S": "i"}]} instead
of the correct {"SS": ["h", "i"]}Set and List expose an inner_typedef for conditions to force rendering of inner values
(currently only used by ContainsCondition)Set was rendering an invalid wire format, and now renders the correct "SS", "NS", or "BS" values.Set and List were rendering contains conditions incorrectly, by trying to dump each value in the
value passed to contains. For example, MyModel.strings.contains("foo") would render contains(#n0, :v1)
where :v1 was {"SS": [{"S": "f"}, {"S": "o"}, {"S": "o"}]}. Now, non-iterable values are rendered
singularly, so :v1 would be {"S": "foo"}. This is a temporary fix, and only works for simple cases.
For example, List(List(String)) will still break when performing a contains check.
This is fixed correctly in 1.0.0 and you should migrate as soon as possible.model.Meta now exposes gsis and lsis, in addition to the existing indexes. This simplifies code that
needs to iterate over each type of index and not all indexes.engine_for_profile was no longer necessary, since the client instances could simply be created with a given
profile.bloop.Client now takes boto_client, which should be an instance of boto3.client("dynamodb") instead of
a boto3.session.Session. This lets you specify endpoints and other configuration only exposed during the
client creation process.Engine no longer uses "session" from the config, and instead takes a client param which should be an
instance of bloop.Client. bloop.Client will be going away in 1.0.0 and Engine will simply take the boto3
clients directly.AbstractModelException is raised when attempting to perform an operation which requires a
table, on an abstract model. Raised by all Engine functions as well as bloop.Client operations.Engine operations raise AbstractModelException when attempting to perform operations on abstract models.model.Meta.abstract was False, or there was no value.
Now, ModelMetaclass will explicitly set abstract to False so that model.Meta.abstract can be used
everywhere, instead of getattr(model.Meta, "abstract", False).Column has a new attribute model, the model it is bound to. This is set during the model's creation by
the ModelMetaclass.Engine.bind will now skip intermediate models that are abstract. This makes it easier to pass abstract models,
or models whose subclasses may be abstract (and have non-abstract grandchildren).(no public changes)
Conditions implement __eq__ for checking if two conditions will evaluate the same. For example::
large = Blob.size > 10242 small = Blob.size < 10242 large == small False also_large = Blob.size > 1024**2 large == also_large True large is also_large False
.. _changelog-v0.9.6:
0.9.6 is the first significant change to how Bloop binds models, engines, and tables. There are a few breaking changes, although they should be easy to update.
Where you previously created a model from the Engine's model:
.. code-block:: python
from bloop import Engine
engine = Engine()
class MyModel(engine.model):
...
You'll now create a base without any relation to an engine, and then bind it to any engines you want:
.. code-block:: python
from bloop import Engine, new_base
BaseModel = new_base()
class MyModel(BaseModel):
...
engine = Engine()
engine.bind(base=MyModel) # or base=BaseModel
engine_for_profile takes a profile name for the config file and creates an appropriate session.
This is a temporary utility, since Engine will eventually take instances of dynamodb and dynamodbstreams
clients. This will be going away in 1.0.0.BloopException which can be used to catch anything thrown by Bloop.new_base() creates an abstract base for models. This replaces Engine.model now that multiple
engines can bind the same model. This will be going away in 1.0.0 which will provide a BaseModel class.session parameter to Engine is now part of the config kwargs. The underlying bloop.Client is
no longer created in Engine.__init__, which provides an opportunity to swap out the client entirely before
the first Engine.bind call. The semantics of session and client are unchanged.Engine._load, Engine._dump, and all Type signatures now pass an engine explicitly through the context
parameter. This was mentioned in 0.9.2 and context is now required.Engine.bind now binds the given class and all subclasses. This simplifies most workflows, since you can
now create a base with MyBase = new_base() and then bind every model you create with
engine.bind(base=MyBase).BloopException instead of Exception.Set, List, Map, and TypedMap accept a typedef of None so they can raise a more
helpful error message. This will be reverted in 1.0.0 and will once again be a required parameter.model, unbound_models, or models attributes. Engine.model has been replaced
by the new_base() function, and models are bound directly to the underlying type engine without tracking
on the Engine instance itself.EngineView attributes are now properties, and point to the underlying engine's attributes; this includes
client, model, type_engine, and unbound_models. This fixed an issue when using
with engine.context(...) as view: to perform operations on models bound to the engine but not the engine view.
EngineView will be going away in 1.0.0.Engine functions now take optional config parameters to override the engine's config. You should update your code to
use these values instead of engine.config, since engine.config is going away in 1.0.0. Engine.delete
and Engine.save expose the atomic parameter, while Engine.load exposes consistent.
Added the TypedMap class, which provides dict mapping for a single typedef over any number of keys.
This differs from Map, which must know all keys ahead of time and can use different types. TypedMap only
supports a single type, but can have arbitrary keys. This will be going away in 1.0.0.
.. _changelog-v0.9.2:
_load, _dump, dynamo_load, dynamo_dump now take an optional keyword-only arg
context. This dict will become required in 0.9.6, and contains the engine
instance that should be used for recursive types. If your type currently uses cls.Meta.bloop_engine,
you should start using context["engine"] in the next release. The bloop_engine attribute is being removed,
since models will be able to bind to multiple engines.(no public changes)
.. _changelog-v0.9.0:
FAQs
ORM for DynamoDB
We found that bloop demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.
Security News
Research
Socket researchers found a malicious Maven package impersonating the legitimate ‘XZ for Java’ library, introducing a backdoor for remote code execution.