PyMw

PyMW: Python master/worker system

The PyMW framework is a master-worker computing framework for Python. Using PyMW in conjunction with a special interpreter (PyBOINC) allows you to send Python scripts as work units and does not require the client to have Python installed. PyMW can also be used without the portable Python interpreter, however, the client is responsible for installing the correct version of Python (2.4 or greater) and adding the interpreter to their system PATH environment variable.

In addition to BOINC, PyMW also provides non-BOINC interfaces for Condor, MPI, and simple multi-core processing which can be selected using a command line switch. The multi-core interface is particularly useful for debugging BOINC applications before submitting them to your project.

The following is a brief explanation of how to setup PyMW using the BOINC virtual server.

Installing and Testing PyMW

Before starting this tutorial you should setup the BOINC virtual server following the instructions in the Quick Start guide. It's also highly recommended that you ssh into your VM instead of using a VM viewer so that you can copy and paste commands. You can start the ssh daemon with the following command:

 $ su -c "/usr/sbin/sshd"

Once your server is setup, login to your VM as "boincadm" (with password "boincadmpw" if you haven't changed it). To install PyMW, first download the PyMW package and unzip it into your home folder on the BOINC VM and run the setup script to install it into your local Python "site-packages" directory. This can be done with the following commmands:

$ cd ~
$ wget http://downloads.sourceforge.net/project/pymw/pymw/PyMW%200.2/pymw-0.2.tar.gz
$ tar -xzf pymw-0.2.tar.gz
$ rm pymw-0.2.tar.gz
$ cd pymw-0.2
$ su -c "python setup.py install"

(the password for root is rootpw, unless you changed it)

Note that the URL for PyMW was current as of the time of this writing (v0.2), but you should check for and use the newest available release.

At this point, PyMW is installed and so it's a good idea to test the installation. While still in the PyMW directory that you extracted above, try the following example distributed with PyMW:

$ cd examples
$ python monte_pi.py

(some output should appear)

$ cd ../../

After a moment, you should see some vague estimate of PI printed out to the screen. This means that PyMW is installed and working properly. By default, the example that was run above uses the "generic" interface and does not interact with the BOINC server.

Next, you should install PyBOINC to allow your PyMW apps to run without Python being installed on the BOINC client's machines. The following downloads the zip file, extracts it and moves it into a convenient location:

$ wget http://bitbucket.org/jeremycowles/pyboinc/downloads/pyboinc-0.01.zip
$ unzip pyboinc-0.01.zip
$ mv pyboinc-0.01/bin-python26 pyboinc
$ rm -r pyboinc-0.01
$ rm pyboinc-0.01.zip

With PyMW and PyBOINC setup, it's time to run a real BOINC application with PyMW.

Running PyMW Applications

Note: the following assumes that you have a folder in your home directory called "pyboinc" containing the Python 2.6 binaries and that you have installed PyMW as described in the previous section.

First, you must setup a project for testing, I will call mine "sandbox" and it will be run on the local address I have given my VM, 192.168.1.190, but you should use whatever address or domain name that is setup on your VM:

$ cd ~/boinc/tools
$ ./make_project --url_base http://192.168.1.190 sandbox

(answer yes to all questions/warnings)

This may take a while to generate the server keys. After it's completed, you should go into your project directory and add it to your cron jobs and to the Apache server:

$ cd ~/projects/sandbox
$ su -c 'cat sandbox.httpd.conf >> /etc/apache2/httpd.conf'
  (you'll be prompted for the root password, which is "rootpw")

$ su -c 'apache2ctl -k restart'
$ crontab sandbox.cronjob

The project should not be created, but there is no application installed (notice, we never called xadd or update_versions). Luckily, PyMW will do this for us. Now all we have to do is run our application with the boinc interface and point it to our home directory:

$ cd ~/pymw-0.2/examples
$ python monte_pi.py -n 1 -i boinc -p /home/boincadm/projects/sandbox -c /home/boincadm/pyboinc -a 1.00_python26.zip

(lots of information will scroll out and then it will eventually wait)

The Monte Carlo PI estimation application is waiting for BOINC clients to process it's work units. So go back to your main PC (not the VM) and open up the BOINC client (install it if needed).

Once you have the BOINC manager open, click on "advanced view", when debugging it's best if you stay in this view. Next, click "Tools->Attach to project" from the main application menu. Click "Next" and then enter your project URL ("http://192.168.1.190/sandbox" if you are following this exactly) and then click next again.

Now that you are attached to the project, it should be listed in the project view. Click on the project and click the "update" button to force it to request work from the server. The PyBOINC interpreter will download and then a handful of tasks should appear. Once these tasks complete you can hit "update" on the project again to force the client to send them back to the server.

Back at the BOINC VM, you should now see the results of the computation.

Talking to BOINC from Python

The PyBOINC interpreter also includes a custom module for interfacing with the BOINC client API under the namespace "boinc". Most common BOINC API functions have been implemented, to see a full list, run the following on the BOINC VM:

$ cd ~/pyboinc/linux

(you may or may not need the next line)

$ chmod +x pymw_1.03_i686-pc-linux-gnu
$ echo -e "import boinc\n""print dir(boinc)" > boinc_test.py
$ ./pymw_1.03_i686-pc-linux-gnu boinc_test.py 1.00_python26.zip

Accessing the boinc API in your code is as easy as importing the boinc namespace and then using the function you need, for example:

import boinc
print boinc.time_to_checkpoint()

There are two things worth noting here, first, all boinc functions have the "boinc_" part of the C name removed, which is redundant when using the "boinc" namespace as a prefix. Secondly, the function boinc.get_app_name() is not actually part of the BOINC API, but is provided by PyBOINC to let you discover the true application name that was launched and not just your script name.

See the PyMW /examples folder for more examples on using PyMW or go to the PyMW web site for official documentation.