Julia: Variable names inside of the function need to match the names outside of the function when distributing an array among workers?

Question

I have the following Julia function which takes an input array and distributes it among available workers.

function DistributeArray(IN::Array,IN_symb::Symbol;mod=Main)    # Distributes an array among workers
    dim = length(size(IN))
    size_per_worker = floor(Int,size(IN,1) / nworkers())
    StartIdx = 1
    EndIdx = size_per_worker
    for (idx, pid) in enumerate(workers())
        if idx == nworkers()
            EndIdx = size(IN,1)
        end
        if dim == 3
            @spawnat(pid, eval(mod, Expr(:(=), IN_symb, IN[StartIdx:EndIdx,:,:])))
        elseif dim == 2
            @spawnat(pid, eval(mod, Expr(:(=), IN_symb, IN[StartIdx:EndIdx,:])))
        elseif dim == 1
            @spawnat(pid, eval(mod, Expr(:(=), IN_symb, IN[StartIdx:EndIdx])))
        else 
            error("Invalid dimensions for input array.")
        end
        StartIdx = EndIdx + 1
        EndIdx = EndIdx + size_per_worker - 1
    end
end

I call this function inside some of my other functions to distribute an array. As an example, here is a test function:

function test(IN::Array,IN_symb::Symbol)   
    DistributeArray(IN,IN_symb)
    @everywhere begin
        if myid() != 1
            println(size(IN))
        end
    end
end

I expect this function to take the 'IN' array and distribute it among all available workers, then print the size allocated to each worker. The following set of commands (where the names of the inputs match the names used inside the functions) works correctly:

addprocs(3)
IN = rand(27,33)
IN_symb = :IN
test(IN,IN_symb)

# From worker 2:    (9,33)
# From worker 3:    (8,33)
# From worker 4:    (10,33)

However, when I change the names of the inputs so that they are different from the names used in the functions, I get an error (start a new julia session before running the follow commands):

addprocs(3)
a = rand(27,33)
a_symb = :a
test(a,a_symb)

ERROR: On worker 2:
UndefVarError: IN not defined
 in eval at ./sysimg.jl:14
 in anonymous at multi.jl:1378
 in anonymous at multi.jl:907
 in run_work_thunk at multi.jl:645
 [inlined code] from multi.jl:907
 in anonymous at task.jl:63
 in remotecall_fetch at multi.jl:731
 in remotecall_fetch at multi.jl:734
 in anonymous at multi.jl:1380

...and 3 other exceptions.

 in sync_end at ./task.jl:413
 [inlined code] from multi.jl:1389
 in test at none:2

I don't understand what is causing this error. It appears to me that the functions are not using the inputs that I give them?

score 1 · Accepted Answer · edited Sep 25 '17 at 17:11

1

In your function test() you are running println(size(IN)). Thus, you are looking on each of the processes for a specific object named IN. In the second example, however, you are naming your objects a rather than IN (since the symbol you supply is :a). The symbol that you supply to the DistributeArray() function is what defines the name that the objects will have on the workers, so that is the name you use to refer to those objects in the future.

You could achieve the results that I think you're looking for, though, with a slight modification to your test() function:

function test(IN::Array,IN_symb::Symbol)   
    DistributeArray(IN,IN_symb)
    for (idx, pid) in enumerate(workers())
        @spawnat pid println(size(eval(IN_symb)))
    end
end

In my opinion, @spawnat can be a bit more flexible at times in letting you better specify the expressions you want it to evaluate.

edited Sep 25 '17 at 17:11

Graham

7,431
18
59
84

answered Aug 25 '16 at 20:59

Michael Ohlrogge

10,559
5
48
76

@user6443519 I'll confess, the whole `DistributeArray()` function is rather hacky. It's possible that there is some better way out there that I just don't know of. And if there isn't, I wish there were. But, it has been enough to let me scrape by on for what I've needed it for. I'm fixing on writing up some stuff on this to query people on the GitHub page for Julia, since they tend to be quite knowledgeable. I haven't gotten around to it yet though. You could also try doing so, if you are inclined. – Michael Ohlrogge Aug 25 '16 at 22:47
@user6443519 Of course, the whole infrastructure for moving data around between workers, as articulated [here](http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-another-processor-in-julia) is quite hacky - the `DistributeArray()` function isn't really much more so than that, since it is heavily based off those principles. So, in a tiny bit of my defense, it may be hacky, but it seems that a lot of the parallel stuff with Julia is hacky right now. – Michael Ohlrogge Aug 25 '16 at 22:51

Julia: Variable names inside of the function need to match the names outside of the function when distributing an array among workers?

1 Answers1