Confused on how Java shares variables during multiprocessing

Question

I just started using java so sorry if this question's answer is obvious. I can't really figure out how to share variables in java. I have been playing around with python and wanted to try to port some code over to Java to learn the langauge a bit better. Alot of my code is ported but I'm unsure how exactly multiprocessing and sharing of variables works in Java(my process is not disk bound, and uses alot of cpu and searching of a list).

In Python, I can do this:

from multiprocessing import Pool, Manager
manager = Manager()
shared_list = manager.list()
pool = Pool(process=4) 
for variables_to_send in list_of_data_to_process:
       pool.apply_async(function_or_class, (variables_to_send, shared_list))
pool.close()
pool.join()

I've been having a bit of trouble figuring out how to do multiprocessing and sharing like this in Java. This question helped me understand a bit(via the code) how implementing runnable can help and I'm starting to think java might automatically multiprocess threads(correct me if I'm wrong on this I read that once threads exceed capacity of a cpu they are moved to another cpu? The oracle docs seem to be more focused on threads than multiprocessing). But it doesn't explain how to share lists or other variables between proceses(and keep them in close enough sync).

Any suggestions or resources? I am hoping I'm searching for the wrong thing(multiprocessing java) and that this is hopefully as easy(or similarly straightforward) as it is in my above code.

Thanks!

Answer 1

There is an important difference between a thread and a process, and you are running into it now: with some exceptions, threads share memory, but processes do not.

Note that real operating systems have ways around just about everything I'm about to say, but these features aren't used in the typical case. So, to fire up a new process, you must clone the current process in some way with a system call (on *nix, this is fork()), and then replace the code, stack, command-line arguments, etc. of the child process with another system call (on *nix, this is the exec() family of system calls). Windows has rough equivalents of both these system calls, so everything I'm saying is cross-platform. Also, the Java Runtime Environment takes care of all these system calls under the covers, and without JNI or some other interop technology you can't really execute them yourself.

There are two important things to note about this model: the child process doesn't share the address space of the parent process, and the entire address space of the child process gets replaced on the exec() call. So, variables in the parent process are unavailable to the child process, and vice versa.

The thread model is quite different. Threads are kind of like lite processes, in that each thread has its own instruction pointer, and (on most systems) threads are scheduled by the operating system scheduler. However, a thread is a part of a process. Each process has at least one thread, and all the threads in the process share memory.

Now to your problem:

The Python multiprocessing module spawns processes with very little effort, as your code example shows. In Java, spawning a new process takes a little more work. It involves creating a new Process object using ProcessBuilder.start() or Runtime.exec(). Then, you can pipe strings to the child process, get back its output, wait for it to exit, and a few other communication primitives. I would recommend writing one program to act as the coordinator and fire up each of the child processes, and writing a worker program that roughly corresponds to function_or_class in your example. The coordinator can open multiple copies of the worker program, give each a task, and wait for all the workers to finish.

Answer 2

You can use Java Thread for this purpose. You need to create one user defined class. That class should have setter method through which you can set shared_list object. Implement Runnable interface and perform processing task in run() method. You can find good example on internet. If you are sharing the same instance of shared_list then you need to make sure that access to this variable is synchronized.

Answer 3

This is not the easiest way to work with threads in java but its the closed to the python code you posted. The task class is an instance of the callable interface and it has a call method. When we create each of the 10000 Task instances we pass them a reference to the same list. So when the call method of all those objects is called they will use the same list.

We are using a fixed size thread pool of 4 threads here so all the tasks we are submitting get queued and wait for a thread to be available.

public class SharedListRunner {
    public void RunList() {
        ExecutorService executerService = Executors.newFixedThreadPool(4);
        List<String> sharedList = new List<String>();
        sharedList.add("Hello");
        for(int i=0; i < 10000; i++)
            executerService.submit(new Task(list));
    }
}

public class Task implements Callable<String> {

    List<String> sharedList;    

    public Task(List<String> sharedList) {
            this.sharedList = sharedList;
    }

    @Override
    public String call() throws Exception {
            //Do something to shared list
            sharedList.size();  
            return "World";
    }
}

At any one time 4 threads are accessing the list. If you want to dig further 4 Java threads are accessing the list, There are probably fewer OS threads servicing those 4 java threads and there are even fewer processor threads normally 2 or 4 per core of your cpu.