Stack Overflow
Stack Overflow

Questions tagged [parallel-python]

Parallel Python provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network)

Parallel Python (PP) is a Python module which provides mechanism for parallel execution of python code on SMP (systems with multiple processors or cores) and clusters (computers connected via network).

PP module overcomes the limitations imposed by the GIL (Global Interpreter Lock) on multithreading and provides a simple way to write parallelPpython applications. Internally ppsmp uses processes and IPC (Inter Process Communications) to organize parallel computations.

All the details and complexity of the latter are completely taken care of, and your application just submits jobs and retrieves their results (the easiest way to write parallel applications).

The software written with PP works in parallel even on many computers connected via local network or Internet (clusters). Cross-platform portability and dynamic load-balancing allows PP to parallelize computations efficiently even on heterogeneous and multi-platform clusters.

65 questions
9
votes
1 answer

Python sharing class instance among threads

I have a class, that loads all resources into memory that are needed for my application (mostly images). Then several threads need to access these resources through this class. I don't want every instance to reload all resources, so I thought I use…
user2078645
  • 189
  • 1
  • 1
  • 9
8
votes
1 answer

logging while using parallel python

I'm using parallel python to execute a big function (executePipeline) multiple times. This function is also using multiprocessing (with the multiprocessing module). I'm having some trouble to display correctly the logging messages on my console…
Romanzo Criminale
  • 1,289
  • 2
  • 14
  • 21
6
votes
2 answers

Parallel Python: How do I supply arguments to 'submit'?

This is only the second question with the parallel-python tag. After looking through the documentation and googling for the subject, I've come here as it's where I've had the best luck with answers and suggestions. The following is the API (I think…
Peter Stewart
  • 2,857
  • 6
  • 28
  • 30
5
votes
0 answers

Parallel Python - RuntimeError: Communication pipe read error

I'm using parallel python to run multiple dynamic simulations using a module called OrcFxAPI. The program works perfectly if it is run as a python program on my machine however if I convert it to an exe file using py2exe and then run I get the…
Gareth Fuller
  • 328
  • 2
  • 12
4
votes
2 answers

Moving Parallel Python code to the cloud

Upon hearing that the scientific computing project (happens to be the stochastic tractography method described here) I'm currently running for an investigator would take 4 months on our 50 node cluster, the investigator has asked me to examine other…
Thomas
  • 6,515
  • 1
  • 31
  • 47
3
votes
1 answer

using ppserver with parallel python

i'm using parallel python for stearing Monte Carlo simulations in Modelica. Everything works perfectly if i'm working on a single computer with 8 ncpus, however, i fail in adding ppservers to the required job_server. I start ppserver.py on the…
ruben baetens
  • 2,806
  • 6
  • 25
  • 31
3
votes
1 answer

Why does Parallel Python work the way it does?

In Parallel Python, why is it necessary to wrap any modules the function passed will need along with variables and namespaces in that job submission call - how necessary is it to preserve module level "global" variables? (if that's all that's going…
Thomas
  • 6,515
  • 1
  • 31
  • 47
3
votes
2 answers

Logging worker processes with Parallel Python

I've inherited the maintenance of some scientific computing using Parallel Python on a cluster. With Parallel Python, jobs are submitted to a ppserver, which (in this case) talks to already-running ppserver processes on other computers, dishing…
Thomas
  • 6,515
  • 1
  • 31
  • 47
3
votes
1 answer

Using parallel python to parallelize loop over nodes in networkx

I am doing some complex calculation in networkx. The calculation involves calculating some quantity again and again for each node in the network. Just as an example of such a calculation, suppose we want to calculate the average degree of neighbors…
Peaceful
  • 4,920
  • 15
  • 54
  • 79
3
votes
0 answers

parallel-python error: RuntimeError("Socket connection is broken")

I am using a simple program to send a function: import pp nodes=('mosura02','mosura03','mosura04','mosura05','mosura06', 'mosura09','mosura10','mosura11','mosura12') nodes=('miner:60001',) def pptester(): js=pp.Server(ppservers=nodes) …
Wolfgang Kerzendorf
  • 724
  • 1
  • 9
  • 24
3
votes
1 answer

Parallel Python: Passing a function written in another module to 'submit'

I am using the Parallel Python module (pp), and want to submit a job to a worker. However, the function that I want to execute is in another module (written with Cython), and I don't know how to import the function name to the new worker. The method…
Abhinav
  • 268
  • 1
  • 7
  • 19
3
votes
1 answer

When using Parallel Python, is there any way to tell on which machine the job has run?

I have written a simple program using parallel python, and all works well. However, mainly for curiosities sake, I would like to know on which machine each task ran, and how long it took. Is there any way to programmatically get this information for…
Paul Wagland
  • 27,756
  • 10
  • 52
  • 74
3
votes
0 answers

Parallel Python (pp) error SystemError: error return without exception set

I am using Parallel Python (pp library) to run a rather large (and embarassingly parallel) job. My code: # Parallel fit with pp def fit_label(label, trainX, trainY, label_index): # print 'Fitting', i, 'label out of', len(label_index) from…
mchangun
  • 9,814
  • 18
  • 71
  • 101
3
votes
0 answers

Parallel Python (pp) and subprocess : task never ends on the client

I'm trying to use the module "Parallel Python" (pp) to distribute the computation of a "big" tool (gdal2tiles, if you've heard of it). I was running a few simple tests to get familiar with pp, but I ran into a problem that I can't explain. Here is…
IzAndCuddles
  • 31
  • 2
3
votes
3 answers

Structuring a program. Classes and functions in Python

I'm writing a program that uses genetic techniques to evolve equations. I want to be able to submit the function 'mainfunc' to the Parallel Python 'submit' function. The function 'mainfunc' calls two or three methods defined in the Utility…
Peter Stewart
  • 2,857
  • 6
  • 28
  • 30
1
2 3 4 5