223 lines
9.2 KiB
Plaintext
223 lines
9.2 KiB
Plaintext
[section:python Python Bindings]
|
|
[python]
|
|
|
|
Boost.MPI provides an alternative MPI interface from the _Python_
|
|
programming language via the `boost.mpi` module. The
|
|
Boost.MPI Python bindings, built on top of the C++ Boost.MPI using the
|
|
_BoostPython_ library, provide nearly all of the functionality of
|
|
Boost.MPI within a dynamic, object-oriented language.
|
|
|
|
The Boost.MPI Python module can be built and installed from the
|
|
`libs/mpi/build` directory. Just follow the [link
|
|
mpi.getting_started.config configuration] and [link mpi.getting_started.config.installation
|
|
installation] instructions for the C++ Boost.MPI. Once you have
|
|
installed the Python module, be sure that the installation location is
|
|
in your `PYTHONPATH`.
|
|
|
|
[section:quickstart Quickstart]
|
|
|
|
[python]
|
|
|
|
Getting started with the Boost.MPI Python module is as easy as
|
|
importing `boost.mpi`. Our first "Hello, World!" program is
|
|
just two lines long:
|
|
|
|
import boost.mpi as mpi
|
|
print "I am process %d of %d." % (mpi.rank, mpi.size)
|
|
|
|
Go ahead and run this program with several processes. Be sure to
|
|
invoke the `python` interpreter from `mpirun`, e.g.,
|
|
|
|
[pre
|
|
mpirun -np 5 python hello_world.py
|
|
]
|
|
|
|
This will return output such as:
|
|
|
|
[pre
|
|
I am process 1 of 5.
|
|
I am process 3 of 5.
|
|
I am process 2 of 5.
|
|
I am process 4 of 5.
|
|
I am process 0 of 5.
|
|
]
|
|
|
|
Point-to-point operations in Boost.MPI have nearly the same syntax in
|
|
Python as in C++. We can write a simple two-process Python program
|
|
that prints "Hello, world!" by transmitting Python strings:
|
|
|
|
import boost.mpi as mpi
|
|
|
|
if mpi.world.rank == 0:
|
|
mpi.world.send(1, 0, 'Hello')
|
|
msg = mpi.world.recv(1, 1)
|
|
print msg,'!'
|
|
else:
|
|
msg = mpi.world.recv(0, 0)
|
|
print (msg + ', '),
|
|
mpi.world.send(0, 1, 'world')
|
|
|
|
There are only a few notable differences between this Python code and
|
|
the example [link mpi.tutorial.point_to_point in the C++
|
|
tutorial]. First of all, we don't need to write any initialization
|
|
code in Python: just loading the `boost.mpi` module makes the
|
|
appropriate `MPI_Init` and `MPI_Finalize` calls. Second, we're passing
|
|
Python objects from one process to another through MPI. Any Python
|
|
object that can be pickled can be transmitted; the next section will
|
|
describe in more detail how the Boost.MPI Python layer transmits
|
|
objects. Finally, when we receive objects with `recv`, we don't need
|
|
to specify the type because transmission of Python objects is
|
|
polymorphic.
|
|
|
|
When experimenting with Boost.MPI in Python, don't forget that help is
|
|
always available via `pydoc`: just pass the name of the module or
|
|
module entity on the command line (e.g., `pydoc
|
|
boost.mpi.communicator`) to receive complete reference
|
|
documentation. When in doubt, try it!
|
|
[endsect:quickstart]
|
|
|
|
[section:user_data Transmitting User-Defined Data]
|
|
Boost.MPI can transmit user-defined data in several different ways.
|
|
Most importantly, it can transmit arbitrary _Python_ objects by pickling
|
|
them at the sender and unpickling them at the receiver, allowing
|
|
arbitrarily complex Python data structures to interoperate with MPI.
|
|
|
|
Boost.MPI also supports efficient serialization and transmission of
|
|
C++ objects (that have been exposed to Python) through its C++
|
|
interface. Any C++ type that provides (de-)serialization routines that
|
|
meet the requirements of the Boost.Serialization library is eligible
|
|
for this optimization, but the type must be registered in advance. To
|
|
register a C++ type, invoke the C++ function [funcref
|
|
boost::mpi::python::register_serialized
|
|
register_serialized]. If your C++ types come from other Python modules
|
|
(they probably will!), those modules will need to link against the
|
|
`boost_mpi` and `boost_mpi_python` libraries as described in the [link
|
|
mpi.getting_started.config.installation installation section]. Note that you do
|
|
*not* need to link against the Boost.MPI Python extension module.
|
|
|
|
Finally, Boost.MPI supports separation of the structure of an object
|
|
from the data it stores, allowing the two pieces to be transmitted
|
|
separately. This "skeleton/content" mechanism, described in more
|
|
detail in a later section, is a communication optimization suitable
|
|
for problems with fixed data structures whose internal data changes
|
|
frequently.
|
|
[endsect:user_data]
|
|
|
|
[section:collectives Collectives]
|
|
|
|
Boost.MPI supports all of the MPI collectives (`scatter`, `reduce`,
|
|
`scan`, `broadcast`, etc.) for any type of data that can be
|
|
transmitted with the point-to-point communication operations. For the
|
|
MPI collectives that require a user-specified operation (e.g., `reduce`
|
|
and `scan`), the operation can be an arbitrary Python function. For
|
|
instance, one could concatenate strings with `all_reduce`:
|
|
|
|
mpi.all_reduce(my_string, lambda x,y: x + y)
|
|
|
|
The following module-level functions implement MPI collectives:
|
|
all_gather Gather the values from all processes.
|
|
all_reduce Combine the results from all processes.
|
|
all_to_all Every process sends data to every other process.
|
|
broadcast Broadcast data from one process to all other processes.
|
|
gather Gather the values from all processes to the root.
|
|
reduce Combine the results from all processes to the root.
|
|
scan Prefix reduction of the values from all processes.
|
|
scatter Scatter the values stored at the root to all processes.
|
|
[endsect:collectives]
|
|
|
|
[section:skeleton_content Skeleton/Content Mechanism]
|
|
Boost.MPI provides a skeleton/content mechanism that allows the
|
|
transfer of large data structures to be split into two separate stages,
|
|
with the skeleton (or, "shape") of the data structure sent first and
|
|
the content (or, "data") of the data structure sent later, potentially
|
|
several times, so long as the structure has not changed since the
|
|
skeleton was transferred. The skeleton/content mechanism can improve
|
|
performance when the data structure is large and its shape is fixed,
|
|
because while the skeleton requires serialization (it has an unknown
|
|
size), the content transfer is fixed-size and can be done without
|
|
extra copies.
|
|
|
|
To use the skeleton/content mechanism from Python, you must first
|
|
register the type of your data structure with the skeleton/content
|
|
mechanism *from C++*. The registration function is [funcref
|
|
boost::mpi::python::register_skeleton_and_content
|
|
register_skeleton_and_content] and resides in the [headerref
|
|
boost/mpi/python.hpp <boost/mpi/python.hpp>] header.
|
|
|
|
Once you have registered your C++ data structures, you can extract
|
|
the skeleton for an instance of that data structure with `skeleton()`.
|
|
The resulting `skeleton_proxy` can be transmitted via the normal send
|
|
routine, e.g.,
|
|
|
|
mpi.world.send(1, 0, skeleton(my_data_structure))
|
|
|
|
`skeleton_proxy` objects can be received on the other end via `recv()`,
|
|
which stores a newly-created instance of your data structure with the
|
|
same "shape" as the sender in its `"object"` attribute:
|
|
|
|
shape = mpi.world.recv(0, 0)
|
|
my_data_structure = shape.object
|
|
|
|
Once the skeleton has been transmitted, the content (accessed via
|
|
`get_content`) can be transmitted in much the same way. Note, however,
|
|
that the receiver also specifies `get_content(my_data_structure)` in its
|
|
call to receive:
|
|
|
|
if mpi.rank == 0:
|
|
mpi.world.send(1, 0, get_content(my_data_structure))
|
|
else:
|
|
mpi.world.recv(0, 0, get_content(my_data_structure))
|
|
|
|
Of course, this transmission of content can occur repeatedly, if the
|
|
values in the data structure--but not its shape--changes.
|
|
|
|
The skeleton/content mechanism is a structured way to exploit the
|
|
interaction between custom-built MPI datatypes and `MPI_BOTTOM`, to
|
|
eliminate extra buffer copies.
|
|
[endsect:skeleton_content]
|
|
|
|
[section:compatibility C++/Python MPI Compatibility]
|
|
Boost.MPI is a C++ library whose facilities have been exposed to Python
|
|
via the Boost.Python library. Since the Boost.MPI Python bindings are
|
|
build directly on top of the C++ library, and nearly every feature of
|
|
C++ library is available in Python, hybrid C++/Python programs using
|
|
Boost.MPI can interact, e.g., sending a value from Python but receiving
|
|
that value in C++ (or vice versa). However, doing so requires some
|
|
care. Because Python objects are dynamically typed, Boost.MPI transfers
|
|
type information along with the serialized form of the object, so that
|
|
the object can be received even when its type is not known. This
|
|
mechanism differs from its C++ counterpart, where the static types of
|
|
transmitted values are always known.
|
|
|
|
The only way to communicate between the C++ and Python views on
|
|
Boost.MPI is to traffic entirely in Python objects. For Python, this
|
|
is the normal state of affairs, so nothing will change. For C++, this
|
|
means sending and receiving values of type `boost::python::object`,
|
|
from the _BoostPython_ library. For instance, say we want to transmit
|
|
an integer value from Python:
|
|
|
|
comm.send(1, 0, 17)
|
|
|
|
In C++, we would receive that value into a Python object and then
|
|
`extract` an integer value:
|
|
|
|
[c++]
|
|
|
|
boost::python::object value;
|
|
comm.recv(0, 0, value);
|
|
int int_value = boost::python::extract<int>(value);
|
|
|
|
In the future, Boost.MPI will be extended to allow improved
|
|
interoperability with the C++ Boost.MPI and the C MPI bindings.
|
|
[endsect:compatibility]
|
|
|
|
[section:reference Reference]
|
|
The Boost.MPI Python module, `boost.mpi`, has its own
|
|
[@boost.mpi.html reference documentation], which is also
|
|
available using `pydoc` (from the command line) or
|
|
`help(boost.mpi)` (from the Python interpreter).
|
|
|
|
[endsect:reference]
|
|
|
|
[endsect:python]
|