zodbpickle
README
.. image:: https://github.com/zopefoundation/zodbpickle/actions/workflows/tests.yml/badge.svg
:target: https://github.com/zopefoundation/zodbpickle/actions/workflows/tests.yml
.. image:: https://coveralls.io/repos/github/zopefoundation/zodbpickle/badge.svg
:target: https://coveralls.io/github/zopefoundation/zodbpickle
:alt: Coverage status
.. image:: https://img.shields.io/pypi/v/zodbpickle.svg
:target: https://pypi.python.org/pypi/zodbpickle
:alt: PyPI
.. image:: https://img.shields.io/pypi/pyversions/zodbpickle.svg
:target: https://pypi.python.org/pypi/zodbpickle
:alt: Python versions
This package presents a uniform pickling interface for ZODB:
-
Under Python2, this package forks both Python 2.7's pickle
and
cPickle
modules, adding support for the protocol 3
opcodes.
It also provides a new subclass of bytes
, zodbpickle.binary
,
which Python2 applications can use to pickle binary values such that
they will be unpickled as bytes
under Py3k.
-
Under Py3k, this package forks the pickle
module (and the supporting
C extension) from both Python 3.2 and Python 3.3. The fork add support
for the noload
operations used by ZODB.
Caution
zodbpickle
relies on Python's pickle
module.
The pickle
module is not intended to be secure against erroneous or
maliciously constructed data. Never unpickle data received from an
untrusted or unauthenticated source as arbitrary code might be executed.
Also see https://docs.python.org/3.6/library/pickle.html
General Usage
To get compatibility between Python 2 and 3 pickling, replace::
import pickle
by::
from zodbpickle import pickle
This provides compatibility, but has the effect that you get the fast implementation
in Python 3, while Python 2 uses the slow version.
To get a more deterministic choice of the implementation, use one of::
from zodbpickle import fastpickle # always C
from zodbpickle import slowpickle # always Python
Both modules can co-exist which is helpful for comparison.
But there is a bit more to consider, so please read on!
Loading/Storing Python 2 Strings
In all their wisdom, the Python developers have decided that Python 2 str
instances should be loaded as Python 3 str
objects (i.e. unicode
strings). Patches were proposed in Python issue 6784
__ but were never
applied. This code base contains those patches.
.. __: http://bugs.python.org/issue6784
Example 1: Loading Python 2 pickles on Python 3 ::
$ python2
>>> import pickle
>>> pickle.dumps('\xff', protocol=0)
"S'\\xff'\np0\n."
>>> pickle.dumps('\xff', protocol=1)
'U\x01\xffq\x00.'
>>> pickle.dumps('\xff', protocol=2)
'\x80\x02U\x01\xffq\x00.'
$ python3
>>> from zodbpickle import pickle
>>> pickle.loads(b"S'\\xff'\np0\n.", encoding='bytes')
b'\xff'
>>> pickle.loads(b'U\x01\xffq\x00.', encoding='bytes')
b'\xff'
>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', encoding='bytes')
b'\xff'
Example 2: Loading Python 3 pickles on Python 2 ::
$ python3
>>> from zodbpickle import pickle
>>> pickle.dumps(b"\xff", protocol=0)
b'c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.'
>>> pickle.dumps(b"\xff", protocol=1)
b'c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.'
>>> pickle.dumps(b"\xff", protocol=2)
b'\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.'
$ python2
>>> import pickle
>>> pickle.loads('c_codecs\nencode\np0\n(V\xff\np1\nVlatin1\np2\ntp3\nRp4\n.')
'\xff'
>>> pickle.loads('c_codecs\nencode\nq\x00(X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02tq\x03Rq\x04.')
'\xff'
>>> pickle.loads('\x80\x02c_codecs\nencode\nq\x00X\x02\x00\x00\x00\xc3\xbfq\x01X\x06\x00\x00\x00latin1q\x02\x86q\x03Rq\x04.')
'\xff'
Example 3: everything breaks down ::
$ python2
>>> class Foo(object):
... def __init__(self):
... self.x = 'hello'
...
>>> import pickle
>>> pickle.dumps(Foo(), protocol=0)
"ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb."
>>> pickle.dumps(Foo(), protocol=1)
'ccopy_reg\n_reconstructor\nq\x00(c__main__\nFoo\nq\x01c__builtin__\nobject\nq\x02Ntq\x03Rq\x04}q\x05U\x01xq\x06U\x05helloq\x07sb.'
>>> pickle.dumps(Foo(), protocol=2)
'\x80\x02c__main__\nFoo\nq\x00)\x81q\x01}q\x02U\x01xq\x03U\x05helloq\x04sb.'
$ python3
>>> from zodbpickle import pickle
>>> class Foo(object): pass
...
>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", encoding='bytes')
>>> foo.x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Foo' object has no attribute 'x'
wait what? ::
>>> foo.__dict__
{b'x': b'hello'}
oooh. So we use encoding='ASCII'
(the default) and errors='bytes'
and
hope it works::
>>> foo = pickle.loads("ccopy_reg\n_reconstructor\np0\n(c__main__\nFoo\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nS'hello'\np7\nsb.", errors='bytes')
>>> foo.x
'hello'
falling back to bytes if necessary ::
>>> pickle.loads(b'\x80\x02U\x01\xffq\x00.', errors='bytes')
b'\xff'
Support for noload()
The ZODB uses cPickle
's noload()
method to retrieve all persistent
references from a pickle without loading any objects. This feature was removed
from Python 3's pickle. Unfortuantely, this unnecessarily fills the pickle
cache.
This module provides a noload()
method again.
===========
Changelog
4.0 (2024-05-30)
- Drop support for Python 3.7.
3.3 (2024-04-16)
3.2 (2024-02-16)
- Add preliminary support for Python 3.13 as of 3.13a3.
3.1 (2023-10-05)
- Add support for Python 3.12.
3.0.1 (2023-03-28)
- Fix
NameError
in .fastpickle
and .slowpickle
.
3.0 (2023-03-24)
-
Build Linux binary wheels for Python 3.11.
-
Add preliminary support for Python 3.12a5.
-
Drop support for Python 2.7, 3.5, 3.6.
-
Drop support for deprecated python setup.py test
.
2.6 (2022-11-17)
- Add support for building arm64 wheels on macOS.
2.5 (2022-11-03)
- Add support for the final Python 3.11 release.
2.4 (2022-09-15)
2.3 (2022-04-22)
- Add support for Python 3.11 (as of 3.11.0a7).
2.2.0 (2021-09-29)
- Add support for Python 3.10.
2.1.0 (2021-09-24)
- Add support for Python 3.9.
2.0.0 (2019-11-13)
- CPython 2: Make
zodbpickle.binary
objects smaller and untracked
by the garbage collector. Now they behave more like the native bytes
object. Just like it, and just like on Python 3, they cannot have
arbitrary attributes or be weakly referenced. See issue 53 <https://github.com/zopefoundation/zodbpickle/issues/53>
_.
1.1 (2019-11-09)
1.0.4 (2019-06-12)
- Fix pickle corruption under certain conditions. See
pull request 47 <https://github.com/zopefoundation/zodbpickle/pull/47>
_.
1.0.3 (2018-12-18)
- Fix a bug: zodbpickle.slowpickle assigned
_Pickler
to Unpickler
.
1.0.2 (2018-08-10)
- Add support for Python 3.7.
1.0.1 (2018-05-16)
- Fix a memory leak in pickle protocol 3 under Python 2. See
issue 36 <https://github.com/zopefoundation/zodbpickle/issues/36>
_.
1.0 (2018-02-09)
0.7.0 (2017-09-22)
-
Drop support for Python 2.6 and 3.2.
-
Add support for Jython 2.7.
-
Add support for Python 3.5 and 3.6.
0.6.0 (2015-04-02)
-
Restore the noload
behaviour from Python 2.6 and provide the
noload
method on the non-C-accelerated unpicklers under PyPy and
Python 2.
-
Add support for PyPy, PyPy3, and Python 3.4.
0.5.2 (2013-08-17)
0.5.1 (2013-07-06)
-
Update all code and tests to Python 2.6.8, 2.7.5, 3.2.5, 3.3.2 .
-
Add the modules zodbpickle.fastpickle
and zodbpickle.slowpickle
.
This provides a version-independent choice of the C or Python
implementation.
-
Fix a minor bug on OS X
0.5.0 (2013-06-14)
- Removed support for the
bytes_as_strings
arguments to pickling APIs:
the pickles created when that argument was true might not be unpickled
without passing encoding='bytes'
, which ZODB couldn't reliably enforce.
On Py3k, ZODB will be using protocol=3
pickles anyway.
0.4.4 (2013-06-07)
- Add protocol 3 opcodes to the C version of the
noload()
dispatcher.
0.4.3 (2013-06-07)
- Packaging error: remove spurious
-ASIDE
file from sdist.
0.4.2 (2013-06-07)
0.4.1 (2013-04-29)
- Fix typo in Python2 version of
zodbpickle.pickle
module.
0.4 (2013-04-28)
-
Support the common pickle module interface for Python 2.6, 2.7, 3.2, and 3.3.
-
Split the Python implementations / tests into Python2- and Py3k-specific
variants.
-
Added a fork of the Python 2.7 _pickle.c
, for use under Python2.
The fork adds support for the Py3k protocol 3
opcodes.
-
Added a custom binary
type for use in Python2 apps.
Derived from bytes
, the binary
type allows Python2 apps to pickle
binary data using opcodes which will cause it to be unpickled as bytes
on Py3k. Under Py3k, the binary
type is just an alias for bytes
.
0.3 (2013-03-18)
-
Added noload
code to Python 3.2 version of Unpickler
. As with
the Python 3.3 version, this code remains untested.
-
Added bytes_as_strings
option to the Python 3.2 version of
Pickler
, dump
, and dumps
.
0.2 (2013-03-05)
-
Added bytes_as_strings
option to Pickler
, dump
, and dumps
.
-
Incomplete support for Python 3.2:
-
Move _pickle.c
-> _pickle_33.c
.
-
Clone Python 3.2.3's _pickle.c
-> _pickle_32.c
and apply the
same patch.
-
Choose between them at build time based on sys.version_info
.
-
Disable some tests of 3.3-only features.
-
Missing: implementation of noload()
in _pickle_32.c
.
-
Missing: implementation of bytes_as_strings=True
in _pickle_32.c
.
0.1.0 (2013-02-27)
- Initial release of Python 3.3's pickle with the patches of Python
issue 6784
__ applied.
.. __: http://bugs.python.org/issue6784#msg156166
- Added support for
errors="bytes"
.