townforge/python
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

merged valid HTML from trunk

Browse Source 
[SVN r46845]
This commit is contained in:
Dave Abrahams 2008-06-29 12:55:50 +00:00
parent 55e9ff14a1
commit 8d86dc199c
1 changed files with 175 additions and 224 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

399
doc/v2/pickle.html
View File

@ -1,112 +1,93 @@
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0//EN"
"http://www.w3.org/TR/REC-html40/strict.dtd">
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<title>Boost.Python Pickle Support</title>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Boost.Python Pickle Support</title>
</head>
<div>
<body>
<div>
<img src="../../../../boost.png" alt="boost.png (6897 bytes)" align=
"center" width="277" height="86" />
<hr />
<img src="../../../../boost.png"
alt="boost.png (6897 bytes)"
align="center"
width="277" height="86">
<h1>Boost.Python Pickle Support</h1>Pickle is a Python module for object
serialization, also known as persistence, marshalling, or flattening.
<hr>
<h1>Boost.Python Pickle Support</h1>
<p>It is often necessary to save and restore the contents of an object to
a file. One approach to this problem is to write a pair of functions that
read and write data from a file in a special format. A powerful
alternative approach is to use Python's pickle module. Exploiting
Python's ability for introspection, the pickle module recursively
converts nearly arbitrary Python objects into a stream of bytes that can
be written to a file.</p>
Pickle is a Python module for object serialization, also known
as persistence, marshalling, or flattening.
<p>The Boost Python Library supports the pickle module through the
interface as described in detail in the <a href=
"http://www.python.org/doc/current/lib/module-pickle.html">Python Library
Reference for pickle.</a> This interface involves the special methods
<tt>__getinitargs__</tt>, <tt>__getstate__</tt> and <tt>__setstate__</tt>
as described in the following. Note that Boost.Python is also fully
compatible with Python's cPickle module.</p>
<hr />
<p>
It is often necessary to save and restore the contents of an object to
a file. One approach to this problem is to write a pair of functions
that read and write data from a file in a special format. A powerful
alternative approach is to use Python's pickle module. Exploiting
Python's ability for introspection, the pickle module recursively
converts nearly arbitrary Python objects into a stream of bytes that
can be written to a file.
<h2>The Boost.Python Pickle Interface</h2>At the user level, the
Boost.Python pickle interface involves three special methods:
<p>
The Boost Python Library supports the pickle module
through the interface as described in detail in the
<a href="http://www.python.org/doc/current/lib/module-pickle.html"
>Python Library Reference for pickle.</a> This interface
involves the special methods <tt>__getinitargs__</tt>,
<tt>__getstate__</tt> and <tt>__setstate__</tt> as described
in the following. Note that Boost.Python is also fully compatible
with Python's cPickle module.
<dl>
<dt><strong><tt>__getinitargs__</tt></strong></dt>
<hr>
<h2>The Boost.Python Pickle Interface</h2>
<dd>
When an instance of a Boost.Python extension class is pickled, the
pickler tests if the instance has a <tt>__getinitargs__</tt> method.
This method must return a Python tuple (it is most convenient to use
a boost::python::tuple). When the instance is restored by the
unpickler, the contents of this tuple are used as the arguments for
the class constructor.
At the user level, the Boost.Python pickle interface involves three special
methods:
<p>If <tt>__getinitargs__</tt> is not defined, <tt>pickle.load</tt>
will call the constructor (<tt>__init__</tt>) without arguments;
i.e., the object must be default-constructible.</p>
</dd>
<dl>
<dt>
<strong><tt>__getinitargs__</tt></strong>
<dd>
When an instance of a Boost.Python extension class is pickled, the
pickler tests if the instance has a <tt>__getinitargs__</tt> method.
This method must return a Python tuple (it is most convenient to use
a boost::python::tuple). When the instance is restored by the
unpickler, the contents of this tuple are used as the arguments for
the class constructor.
<dt><strong><tt>__getstate__</tt></strong></dt>
<p>
If <tt>__getinitargs__</tt> is not defined, <tt>pickle.load</tt>
will call the constructor (<tt>__init__</tt>) without arguments;
i.e., the object must be default-constructible.
<dd>When an instance of a Boost.Python extension class is pickled, the
pickler tests if the instance has a <tt>__getstate__</tt> method. This
method should return a Python object representing the state of the
instance.</dd>
<p>
<dt>
<strong><tt>__getstate__</tt></strong>
<dt><strong><tt>__setstate__</tt></strong></dt>
<dd>
When an instance of a Boost.Python extension class is pickled, the
pickler tests if the instance has a <tt>__getstate__</tt> method.
This method should return a Python object representing the state of
the instance.
<dd>When an instance of a Boost.Python extension class is restored by
the unpickler (<tt>pickle.load</tt>), it is first constructed using the
result of <tt>__getinitargs__</tt> as arguments (see above).
Subsequently the unpickler tests if the new instance has a
<tt>__setstate__</tt> method. If so, this method is called with the
result of <tt>__getstate__</tt> (a Python object) as the argument.</dd>
</dl>The three special methods described above may be <tt>.def()</tt>'ed
individually by the user. However, Boost.Python provides an easy to use
high-level interface via the
<strong><tt>boost::python::pickle_suite</tt></strong> class that also
enforces consistency: <tt>__getstate__</tt> and <tt>__setstate__</tt>
must be defined as pairs. Use of this interface is demonstrated by the
following examples.
<hr />
<p>
<dt>
<strong><tt>__setstate__</tt></strong>
<h2>Examples</h2>There are three files in <tt>boost/libs/python/test</tt>
that show how to provide pickle support.
<hr />
<dd>
When an instance of a Boost.Python extension class is restored by the
unpickler (<tt>pickle.load</tt>), it is first constructed using the
result of <tt>__getinitargs__</tt> as arguments (see above). Subsequently
the unpickler tests if the new instance has a <tt>__setstate__</tt>
method. If so, this method is called with the result of
<tt>__getstate__</tt> (a Python object) as the argument.
<h3><a href="../../test/pickle1.cpp"><tt>pickle1.cpp</tt></a></h3>The C++
class in this example can be fully restored by passing the appropriate
argument to the constructor. Therefore it is sufficient to define the
pickle interface method <tt>__getinitargs__</tt>. This is done in the
following way:
</dl>
The three special methods described above may be <tt>.def()</tt>'ed
individually by the user. However, Boost.Python provides an easy to use
high-level interface via the
<strong><tt>boost::python::pickle_suite</tt></strong> class that also
enforces consistency: <tt>__getstate__</tt> and <tt>__setstate__</tt>
must be defined as pairs. Use of this interface is demonstrated by the
following examples.
<hr>
<h2>Examples</h2>
There are three files in
<tt>boost/libs/python/test</tt> that show how to
provide pickle support.
<hr>
<h3><a href="../../test/pickle1.cpp"><tt>pickle1.cpp</tt></a></h3>
The C++ class in this example can be fully restored by passing the
appropriate argument to the constructor. Therefore it is sufficient
to define the pickle interface method <tt>__getinitargs__</tt>.
This is done in the following way:
<ul>
<li>1. Definition of the C++ pickle function:
<pre>
<ul>
<li>1. Definition of the C++ pickle function:
<pre>
struct world_pickle_suite : boost::python::pickle_suite
{
static
@ -117,26 +98,28 @@ provide pickle support.
}
};
</pre>
<li>2. Establishing the Python binding:
<pre>
</li>
<li>2. Establishing the Python binding:
<pre>
class_&lt;world&gt;("world", args&lt;const std::string&amp;&gt;())
// ...
.def_pickle(world_pickle_suite())
// ...
</pre>
</ul>
</li>
</ul>
<hr />
<hr>
<h3><a href="../../test/pickle2.cpp"><tt>pickle2.cpp</tt></a></h3>
<h3><a href="../../test/pickle2.cpp"><tt>pickle2.cpp</tt></a></h3>The C++
class in this example contains member data that cannot be restored by any
of the constructors. Therefore it is necessary to provide the
<tt>__getstate__</tt>/<tt>__setstate__</tt> pair of pickle interface
methods:
The C++ class in this example contains member data that cannot be
restored by any of the constructors. Therefore it is necessary to
provide the <tt>__getstate__</tt>/<tt>__setstate__</tt> pair of
pickle interface methods:
<ul>
<li>1. Definition of the C++ pickle functions:
<pre>
<ul>
<li>1. Definition of the C++ pickle functions:
<pre>
struct world_pickle_suite : boost::python::pickle_suite
{
static
@ -161,92 +144,76 @@ provide pickle support.
}
};
</pre>
<li>2. Establishing the Python bindings for the entire suite:
<pre>
</li>
<li>2. Establishing the Python bindings for the entire suite:
<pre>
class_&lt;world&gt;("world", args&lt;const std::string&amp;&gt;())
// ...
.def_pickle(world_pickle_suite())
// ...
</pre>
</ul>
</li>
</ul>
<p>
For simplicity, the <tt>__dict__</tt> is not included in the result
of <tt>__getstate__</tt>. This is not generally recommended, but a
valid approach if it is anticipated that the object's
<tt>__dict__</tt> will always be empty. Note that the safety guard
described below will catch the cases where this assumption is violated.
<p>For simplicity, the <tt>__dict__</tt> is not included in the result of
<tt>__getstate__</tt>. This is not generally recommended, but a valid
approach if it is anticipated that the object's <tt>__dict__</tt> will
always be empty. Note that the safety guard described below will catch
the cases where this assumption is violated.</p>
<hr />
<hr>
<h3><a href="../../test/pickle3.cpp"><tt>pickle3.cpp</tt></a></h3>
<h3><a href="../../test/pickle3.cpp"><tt>pickle3.cpp</tt></a></h3>This
example is similar to <a href=
"../../test/pickle2.cpp"><tt>pickle2.cpp</tt></a>. However, the object's
<tt>__dict__</tt> is included in the result of <tt>__getstate__</tt>.
This requires a little more code but is unavoidable if the object's
<tt>__dict__</tt> is not always empty.
<hr />
This example is similar to <a
href="../../test/pickle2.cpp"><tt>pickle2.cpp</tt></a>. However, the
object's <tt>__dict__</tt> is included in the result of
<tt>__getstate__</tt>. This requires a little more code but is
unavoidable if the object's <tt>__dict__</tt> is not always empty.
<h2>Pitfall and Safety Guard</h2>The pickle protocol described above has
an important pitfall that the end user of a Boost.Python extension module
might not be aware of:
<hr>
<h2>Pitfall and Safety Guard</h2>
<p><strong><tt>__getstate__</tt> is defined and the instance's
<tt>__dict__</tt> is not empty.</strong></p>
The pickle protocol described above has an important pitfall that the
end user of a Boost.Python extension module might not be aware of:
<p>
<strong>
<tt>__getstate__</tt> is defined and the instance's <tt>__dict__</tt>
is not empty.
</strong>
<p>
<p>The author of a Boost.Python extension class might provide a
<tt>__getstate__</tt> method without considering the possibilities
that:</p>
The author of a Boost.Python extension class might provide a
<tt>__getstate__</tt> method without considering the possibilities
that:
<ul>
<li>his class is used in Python as a base class. Most likely the
<tt>__dict__</tt> of instances of the derived class needs to be pickled
in order to restore the instances correctly.</li>
<p>
<ul>
<li>
his class is used in Python as a base class. Most likely the
<tt>__dict__</tt> of instances of the derived class needs to be
pickled in order to restore the instances correctly.
<li>the user adds items to the instance's <tt>__dict__</tt> directly.
Again, the <tt>__dict__</tt> of the instance then needs to be
pickled.</li>
</ul>
<p>
<li>
the user adds items to the instance's <tt>__dict__</tt> directly.
Again, the <tt>__dict__</tt> of the instance then needs to be
pickled.
</ul>
<p>
To alert the user to this highly unobvious problem, a safety guard is
provided. If <tt>__getstate__</tt> is defined and the instance's
<tt>__dict__</tt> is not empty, Boost.Python tests if the class has
an attribute <tt>__getstate_manages_dict__</tt>. An exception is
raised if this attribute is not defined:
<pre>
<p>To alert the user to this highly unobvious problem, a safety guard is
provided. If <tt>__getstate__</tt> is defined and the instance's
<tt>__dict__</tt> is not empty, Boost.Python tests if the class has an
attribute <tt>__getstate_manages_dict__</tt>. An exception is raised if
this attribute is not defined:</p>
<pre>
RuntimeError: Incomplete pickle support (__getstate_manages_dict__ not set)
</pre>
To resolve this problem, it should first be established that the
<tt>__getstate__</tt> and <tt>__setstate__</tt> methods manage the
instances's <tt>__dict__</tt> correctly. Note that this can be done
either at the C++ or the Python level. Finally, the safety guard
should intentionally be overridden. E.g. in C++ (from
<a href="../../test/pickle3.cpp"><tt>pickle3.cpp</tt></a>):
<pre>
</pre>To resolve this problem, it should first be established that the <tt>
__getstate__</tt> and <tt>__setstate__</tt> methods manage the
instances's <tt>__dict__</tt> correctly. Note that this can be done
either at the C++ or the Python level. Finally, the safety guard should
intentionally be overridden. E.g. in C++ (from <a href=
"../../test/pickle3.cpp"><tt>pickle3.cpp</tt></a>):
<pre>
struct world_pickle_suite : boost::python::pickle_suite
{
// ...
static bool getstate_manages_dict() { return true; }
};
</pre>
Alternatively in Python:
<pre>
</pre>Alternatively in Python:
<pre>
import your_bpl_module
class your_class(your_bpl_module.your_class):
__getstate_manages_dict__ = 1
@ -255,54 +222,41 @@ is not empty.
def __setstate__(self, state):
# your code here
</pre>
<hr />
<hr>
<h2>Practical Advice</h2>
<h2>Practical Advice</h2>
<ul>
<li>
In Boost.Python extension modules with many extension classes,
providing complete pickle support for all classes would be a
significant overhead. In general complete pickle support should
only be implemented for extension classes that will eventually
be pickled.
<ul>
<li>In Boost.Python extension modules with many extension classes,
providing complete pickle support for all classes would be a
significant overhead. In general complete pickle support should only be
implemented for extension classes that will eventually be pickled.</li>
<p>
<li>
Avoid using <tt>__getstate__</tt> if the instance can also be
reconstructed by way of <tt>__getinitargs__</tt>. This automatically
avoids the pitfall described above.
<li>Avoid using <tt>__getstate__</tt> if the instance can also be
reconstructed by way of <tt>__getinitargs__</tt>. This automatically
avoids the pitfall described above.</li>
<p>
<li>
If <tt>__getstate__</tt> is required, include the instance's
<tt>__dict__</tt> in the Python object that is returned.
<li>If <tt>__getstate__</tt> is required, include the instance's
<tt>__dict__</tt> in the Python object that is returned.</li>
</ul>
<hr />
</ul>
<h2>Light-weight alternative: pickle support implemented in Python</h2>
<hr>
<h2>Light-weight alternative: pickle support implemented in Python</h2>
<h3><a href="../../test/pickle4.cpp"><tt>pickle4.cpp</tt></a></h3>
The <tt>pickle4.cpp</tt> example demonstrates an alternative technique
for implementing pickle support. First we direct Boost.Python via
the <tt>class_::enable_pickling()</tt> member function to define only
the basic attributes required for pickling:
<pre>
<h3><a href="../../test/pickle4.cpp"><tt>pickle4.cpp</tt></a></h3>The
<tt>pickle4.cpp</tt> example demonstrates an alternative technique for
implementing pickle support. First we direct Boost.Python via the
<tt>class_::enable_pickling()</tt> member function to define only the
basic attributes required for pickling:
<pre>
class_&lt;world&gt;("world", args&lt;const std::string&amp;&gt;())
// ...
.enable_pickling()
// ...
</pre>
This enables the standard Python pickle interface as described
in the Python documentation. By &quot;injecting&quot; a
<tt>__getinitargs__</tt> method into the definition of the wrapped
class we make all instances pickleable:
<pre>
</pre>This enables the standard Python pickle interface as described in the
Python documentation. By "injecting" a <tt>__getinitargs__</tt> method into
the definition of the wrapped class we make all instances pickleable:
<pre>
# import the wrapped world class
from pickle4_ext import world
@ -312,18 +266,15 @@ class we make all instances pickleable:
# now inject __getinitargs__ (Python is a dynamic language!)
world.__getinitargs__ = world_getinitargs
</pre>
</pre>See also the <a href=
"../tutorial/doc/html/python/techniques.html#python.extending_wrapped_objects_in_python">
tutorial section</a> on injecting additional methods from Python.
<hr />
© Copyright Ralf W. Grosse-Kunstleve 2001-2004. Distributed under the
Boost Software License, Version 1.0. (See accompanying file
LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
See also the
<a href="../tutorial/doc/html/python/techniques.html#python.extending_wrapped_objects_in_python"
>tutorial section</a> on injecting additional methods from Python.
<hr>
&copy; Copyright Ralf W. Grosse-Kunstleve 2001-2004. Distributed under
the Boost Software License, Version 1.0. (See accompanying file
LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
<p>
Updated: Feb 2004.
</div>
<p>Updated: Feb 2004.</p>
</div>
</body>
</html>