2

I've been using the sendto() method to send variables between workers

I was under the impression that in Julia, functions were 'first class citizens', but I cannot transfer functions between workers using the sendto() method

What is the recommended way of transferring predefined methods between workers?

Update

I seem to be able to use the sendto() method to send anonymous functions

a = function(x)
x*2
end

sendto(2,a=a)

remotecall_fetch(2,a,5)
4

as opposed to

function g(x)
x*2
end

sendto(2,g=g)

remotecall_fetch(2,g,10)
ERROR: On worker 2:
function g not defined on process 2

This suffices in some cases, but I need to be able to send defined functions over, as most of them are complex and defined in scripts that get included when starting julia

Community
  • 1
  • 1
isebarn
  • 3,812
  • 5
  • 22
  • 38
  • I suggest making this a self answered question. (I.e. remove your answer from the question and put it in an answer) – Frames Catherine White Aug 08 '16 at 11:49
  • I dont want to send anonymous methods over, I want to send functions over – isebarn Aug 08 '16 at 12:20
  • Can you give more redetails on your use case (also your julia version would be good). `remote_call` is the more usual way to send a function to a worker, then call it. And as of 0.5 it only sends it once. https://github.com/JuliaLang/julia/pull/16808 – Frames Catherine White Aug 08 '16 at 12:29
  • 0.4 and 0.6. I've got a toolbox that consists of ~40 and growing different scripts. most scripts contain a collection of multiple dispach methods. most methods are used inside other methods. The most 'complex' of methods sometimes get so complex that I allow the user to choose using multiple workers. When a user chooses to uses many workers, I need to move a few methods over to those workers. Sometimes that takes up alot of memory and time. So, I'd like to transfer exactly the method necessary instead of including the entire script that contains it. I cannot send a function using remotecall – isebarn Aug 08 '16 at 13:28
  • Did you mean to say that you can send named functions but not anonymous ones, as opposed to the other way around? What would be the reason for not making your functions named objects? – Michael Ohlrogge Aug 08 '16 at 14:31
  • I can send anonymous functions, not named ones – isebarn Aug 08 '16 at 14:56
  • 1
    In v0.4, the best way would be to `@everywhere` at the function definition, or use `@spawnat`. – Chris Rackauckas Aug 08 '16 at 15:24
  • 1
    For your package, you may want to have a `setup_parallel()` function which calls `include` on a file which simply `@everywhere` defines the appropriate methods (you may need to `@eval @everywhere` to get it in the global scope) – Chris Rackauckas Aug 08 '16 at 15:26

1 Answers1

1

There is no way to send a subset of the methods in a package to another machine. Very often methods refer to other types and functions in the same module, so the system would have to at least send all dependencies as well. That could work, but the bigger problem is deciding whose responsibility it is to distribute code, and when. For example, initially your library might decide to send itself (or parts of itself) to other nodes, but then the user might later want to do a parallel map of your library functions, such that the whole library is needed on every node. This gets very complex, so it is far simpler for everybody just to load all needed code on all nodes as early as possible.

It will help to factor your code into smaller modules. Code that defines reusable functions should be wrapped in a module separate from anything that does "real work", so that loading it does not take significant time.

If the code loaded on the client node really needs to be very different from what's loaded on the workers, you can put only the definitions that need to be sent in their own module, say Helpers. Your main code would contain using Helpers. Then after your code is loaded on the client node, you can start workers and run @everywhere include("Helpers.jl") as suggested by Chris. Then references to functions inside Helpers should work across machines.

Jeff Bezanson
  • 3,147
  • 23
  • 26