Shed Skin and Parallel Python

Shed Skin and Parallel Python

Shed Skin is an experimental Python to C++ compiler. Parallel Python allows for clean and simple parallelization of Python processes over multiple cores. Wouldnt it be cool if we could combine the two? Of course I wouldnt be writing this if I hadnt tried. Heres how to do it.

Create an extension module with Shed Skin, containing function(s) youd like to use in parallel. For example, we might use the partial sum function from the Parallel Python website:

def part_sum(start, end):
..
return sum

For Shed Skins type inference to work, part_sum must be called from somewhere:

if __name__ == __main__:
part_sum(1, 1)

Creating an extension module is simple (suppose the module is named meuk.py):

ss -e meuk
make

Because Parallel Python expects pure-Python code, we must call our compiled function via a pure-Python wrapper:

def part_sum(start, end):
import meuk
return meuk.part_sum(start, end)

In order for Parallel Python to find our extension module (at least on my Ubuntu system), we must issue this in advance:

export PYTHONPATH=$pwd

And there you have it. Here are some timings:

no extension module, 1 worker: 11.3 seconds
no extension module, 2 workers: 6.2 seconds
extension module, 1 worker: 0.6 seconds
extension module, 2 workers: 0.3 seconds


Shed Skin and Parallel Python