Remote parallel kernels and LibraryLink -- how to make them work together?

Question

Does anyone have experience using C extensions to Mathematica (LibraryLink or MathLink -- currently I'm using LibraryLink) with remote parallel kernels?

In short: How can I transparently use a LibraryLink-defined function (see CreateLibrary and LibraryFunctionLoad) in both parallel and non-parallel evaluations when the subkernels run on a remote machine?

I am looking for some setup steps that will allow me to have a libraryFun function (written in C) that can be called either normally as libraryFun[args], or in parallel (both of Parallelize@Table[libraryFun[arg], {arg, 0, 100}] and the same with ParallelTable[]) when the subkernels run on a remote machine.

^{Running the main kernel remotely too might be better if I weren't having trouble with that as well.}

Update

In the meantime I made some progress. I'll describe it here.

First, ParallelEvaluate will evaluate an expression in all parallel kernels. If the source files for the C extension are copied to the remote machine, we can compile them there like this:

ParallelNeeds["CCompilerDriver`"]
k1 = First@Kernels[]
ParallelEvaluate[SetDirectory["/path/to/source/files"]]
ParallelEvaluate[CreateLibrary["sourefile", "myLibrary"]]

This needs to be done one time only. I assume that the library has been already compiled on the main kernel machine.

After this, in all subsequent sessions we can use FindLibrary on both the main and the remote machines to load the library.

LibraryFunctionLoad[myFun = FindLibrary["myLibrary"], "fun", ...]
ParallelEvaluate[myFun = LibraryFunctionLoad[FindLibrary["myLibrary"], "fun", ...]]

And here comes the trouble. Because of different paths, myFun will have different values in the main and in the parallel kernels.

So the question is: How can ensure that the value of myFun will not accidentally get synchronized between the main and the parallel kernels?

I'll show in an isolated examples how this might accidentally happen:

In[1]:= LaunchKernels[2]
Out[1]= {KernelObject[1, "local"], KernelObject[2, "local"]}

Set value of x in main kernel:

In[2]:= x = 1
Out[2]= 1

Note that it gets the same value in remote kernels too:

In[3]:= ParallelEvaluate[x]
Out[3]= {1, 1}

Set a different value for x in the parallel kernels and verify that they keep it:

In[4]:= ParallelEvaluate[x = 2]
Out[4]= {2, 2}

In[5]:= {x, ParallelEvaluate[x]}
Out[5]= {1, {2, 2}}

Now "innocently" use Parallelize on something containing x:

In[6]:= Parallelize[Table[x, {10}]]
Out[6]= {1, 1, 1, 1, 1, 1, 1, 1, 1, 1}

And see how the value of x got re-synced between the main and subkernels.

In[7]:= {x, ParallelEvaluate[x]}
Out[7]= {1, {1, 1}}

The new question is: How can I prevent a certain symbol from ever auto-syncing between the main and the subkernels?

Can we just compile and load the library in the main kernel, and then distribute the library function to remote subkernels? — xslittlegrass, Mar 04 '14 at 07:29
@xslittlegrass Not necessarily if the subkernels run on a different computer. The `LibraryFunction` has the path to a shared library hard-coded, and the path might not be the same on a remote machine. — Szabolcs, Mar 04 '14 at 13:43
I see, I was asking because I use subkernels on different nodes on a HPC and it works just distribute the library function. Now I know the reason because all the nodes on HPC share the same file system. — xslittlegrass, Mar 04 '14 at 17:32
@xslittlegrass Exactly! For this reason it's easier to do this on clusters and I do take advantage of this too. — Szabolcs, Mar 04 '14 at 17:54

score 5 · Answer 1 · answered Jan 05 '12 at 23:18

I hope this answers your question:

For the moment, I will assume the main kernel and the parallel kernels are on the same architecture, which for me is Windows 7. First you compile a function; you can do this outside of Mathematica using a C compiler, or directly in Mathematica:

f = Compile[{x}, x^2, CompilationTarget -> "C"]

You can examine by looking at the InputForm of f where the generated dll is located:

f // InputForm

Gives something like:

CompiledFunction[{8, 8., 5468}, {_Real}, {{3, 0, 0}, {3, 0, 1}}, {}, {0, 0, 2, 0, 0}, {{40, 
56, 3, 0, 0, 3, 0, 1}, {1}}, Function[{x}, x^2], Evaluate,
LibraryFunction["C:\\Users\\arnoudb\\AppData\\Roaming\\Mathematica\\ApplicationData\\CCompilerDriver\\BuildFolder\\arnoudb2win-5184\\compiledFunction0.dll", 
"compiledFunction0", {{Real, 0, "Constant"}}, Real]]

You can copy this file to a location where the parallel (possibly remote) kernel can find it, for example:

CopyFile["C:\\Users\\arnoudb\\AppData\\Roaming\\Mathematica\\ApplicationData\\CCompilerDriver\\BuildFolder\\arnoudb2win-3316\\compiledFunction1.dll",
"c:\\users\\arnoudb\\compiledFunction1.dll"]

Then you can load the library in all the parallel kernels like so:

ParallelEvaluate[
 ff = LibraryFunctionLoad["C:\\users\\arnoudb\\compiledFunction1.dll",
 "compiledFunction1", {Real}, Real]
]

And check if this works:

ParallelEvaluate[ff[3.4]]

Which returns {11.56,11.56} for me.

If the parallel kernel is on a different architecture, you will need to compile the C code for that architecture (or evaluate the Compile[..., CompilationTarget->"C"] on the parallel kernel).

Thanks Arnoud! In the meantime I also made some progress on this. I would like to be able to use `ff` from both the main and the parallel kernels. Please see my update to the question on how this might cause problems. — Szabolcs, Jan 06 '12 at 09:34

Szabolcs · Accepted Answer · 2012-01-09T16:28:40.490

I seem to have found a solution to my question in Update above. It seems to work but I cannot confirm yet that it is not fragile.

The solution is to put symbols that we don't want synchronized into a separate context. Using c`x in my example in place of x prevents synchronizing the value of x when it is used inside Parallelize. Then we can add this context to the $ContextPath to make the symbol easily accessible.

The most convenient way to do this is probably putting all definitions in a package that loads the library functions using LibraryFunctionLoad[FindLibrary[...], ...]. For this to work, the library must have been compiled manually first on both the local and remote machine, however, the package code can be the exactly same both for the main and the subkernels.

I am still interested if someone can confirm that variables not in $Context are guaranteed not to be auto-synchronized.

Update It has been confirmed here.

Remote parallel kernels and LibraryLink -- how to make them work together?

Update

2 Answers2