Stack Overflow
Stack Overflow

Questions tagged [ipython-parallel]

Use this tag for questions related to IPython's architecture for parallel and distributed computing.

Quoting the IPython parallel overview:

IPython has a sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism . [...]. Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython.

is used for all sort of questions that are engaged to using IPython's parallel capabilities.

191 questions
25
votes
4 answers

How do I add a kernel on a remote machine in IPython (Jupyter) Notebook?

Dropdown menu in the top-right of the UI on a local machine (PC): Kernel-> Change kernel-> Python 2 (on a local PC) Python 3 (on a local PC) My new kernel (on a remote PC)
korniichuk
  • 550
  • 1
  • 8
  • 13
24
votes
4 answers

Ipython Notebook: where is jupyter_notebook_config.py in Mac?

I just started using a Mac, so please forgive me if this sounds too naive. I'm trying to install Interactive Parallel. From https://github.com/ipython/ipyparallel, it says I need to find jupyter_notebook_config.py. I've already installed python and…
ZK Zhao
  • 19,885
  • 47
  • 132
  • 206
19
votes
2 answers

How to best share static data between ipyparallel client and remote engines?

I am running the same simulation in a loop with different parameters. Each simulation makes use a pandas DataFrame (data) which is only read, never modified. Using ipyparallel (IPython parallel), I can put this DataFrames into the global variable…
hamx0r
  • 4,081
  • 1
  • 33
  • 46
14
votes
2 answers

How to pin threads to cores with predetermined memory pool objects? (80 core Nehalem architecture 2Tb RAM)

I've run into a minor HPC problem after running some tests on a 80core (160HT) nehalem architecture with 2Tb DRAM: A server with more than 2 sockets starts to stall a lot (delay) as each thread starts to request information about objects on the…
root-11
  • 1,727
  • 1
  • 19
  • 33
12
votes
1 answer

How to enable ipyparallel for Jupyter?

There're docs scattered under Jupyter or ipyparallel, but there's no a single piece document illustrating the entire process from beginning to end. I am really confused. Can anyone has experience to share? Here are checks about my…
etlolap
  • 531
  • 1
  • 6
  • 14
11
votes
1 answer

IPython.parallel not using multicore?

I am experimenting with IPython.parallel and just want to launch several shell command on different engines. I have the following Notebook: Cell 0: from IPython.parallel import Client client = Client() print len(client) 5 And launch the…
zermelozf
  • 545
  • 1
  • 8
  • 16
10
votes
2 answers

Large memory consumption by iPython Parallel module

I am using the ipyparallel module to speed up an all by all list comparison but I am having issues with huge memory consumption. Here is a simplified version of the script that I am running: From a SLURM script start the cluster and run the python…
Pierre Joubert
  • 101
  • 2
10
votes
2 answers

Can't create ipyparallel clusters on Jupyter Notebook

I have the following: ipyparallel (5.0.0) ipython (4.0.3) I have enabled ipcluster by typing in the command line: ipcluster nbextension enable I'm trying to create a new cluster on the IPython Clusters tab on the Jupyter notebook, but this is…
JPN
  • 632
  • 12
  • 24
9
votes
1 answer

Import custom modules on IPython.parallel engines with sync_imports()

I've been playing around with IPython.parallel and I wanted to use some custom modules of my own, but haven't been able to do it as explained on the cookbook using dview.sync_imports(). The only thing that has worked for me was something like def…
Alex S
  • 1,027
  • 1
  • 10
  • 17
8
votes
1 answer

Printing to stdout in IPython parallel processes

I'm new to IPython and would like to print intermediate results to stdout while running IPython parallel cluster functions. (I'm aware that with multiple processes, this might mangle the output, but that's fine--it's just for testing/debugging, and…
user1470788
  • 206
  • 2
  • 7
7
votes
2 answers

IPython-IPyparallel import error

I have copied all the codes to the work directory on all my engine machines. And my code are: my_test.py my_startegy.py main.py So, main.py will be ran on Client Machine, the codes in main.py are: from ipyparallel import Client import…
GoingMyWay
  • 16,802
  • 32
  • 96
  • 149
7
votes
1 answer

Restarting Remote Kernels when using IPython.parallel

How do I force the remote kernels to restart when I am using IPython.parallel? For example in the normal IPython notebook, I can restart a kernel right from the tool bar. My question is when I have remote kernels, how do I force the same operation?
Alex Rothberg
  • 10,243
  • 13
  • 60
  • 120
6
votes
0 answers

launching ipyparallel cluster across multiple nodes using MPI

I am trying to start a ipyparallel cluster using MPI. The ipcluster_config has following lines modified as such: c.MPILauncher.mpi_cmd = ['mpiexec'] c.MPIControllerLauncher.controller_args = ['--ip=*'] c.MPILauncher.mpi_args = ["-machinefile",…
Kabira K
  • 1,916
  • 2
  • 22
  • 38
6
votes
1 answer

Loading lightgbm model and using predict with parallel for loop freezes (Python)

I have the need to use my model to do predictions in batches and in parallel in python. If I load the model and create the data frames in a regular for loop and use the predict function it works with no issues. If I create disjoint data frames in…
RDizzl3
  • 846
  • 8
  • 18
6
votes
0 answers

ipyparallel error when using decorator

I am attempting to run a job using the ipyparallel system with lru_cache and am running into issues. From the terminal: ipcluster start -n 2 In an ipython notebook: from ipyparallel import Client clients = Client() def world(): return hello()…
johnchase
  • 13,155
  • 6
  • 38
  • 64
1
2 3
…
12 13