Erlang takes ages to count?

Question

Hi guys I'm trying to make a load generator and my goal is to compare how much of my system's resources are consumed when spawning Erlang processes as compared to spawning threads (Java). I am doing this by having the program count to 1000000000 10 times. Java takes roughly 35 seconds to finish the whole process with 10 threads created, Erlang takes ages with 10 processes, I grew impatient with it because it spent over 4 minutes counting. If I just make Erlang and Java count to 1000000000 without spawning threads/processes, Erlang takes 1 minute and 32 seconds and Java takes a good 3 or so seconds. I know Erlang is not made for crunching numbers but that much of a difference is alarming, why is there such a big difference ? Both use my CPU to 100% but no spike in RAM. I am not sure what other methods can be used to make this comparison, I am open to any suggestions as well.

here is the code for both versions

-module(loop).

-compile(export_all).


start(NumberOfProcesses) ->
loop(0, NumberOfProcesses).

%%Processes to spawn
loop(A, NumberOfProcesses) -> 
if A < NumberOfProcesses -> 
    spawn(loop, outerCount, [0]),
    loop(A+1, NumberOfProcesses);   

    true -> ok
end.

%%outer loop 
outerCount(A) ->
if A < 10 ->

    innerCount(0),
    outerCount(A + 1);

    true -> ok
end.

%%inner loop 
innerCount(A) ->
if A < 1000000000 ->
    innerCount(A+1);

    true -> ok
end.

and java

import java.util.Scanner;

class Loop implements Runnable 
{

public static void main(String[] args) 
{

    System.out.println("Input number of processes");
    Scanner scan = new Scanner(System.in);        
    String theNumber = scan.nextLine();   


    for (int t = 0; t < Integer.parseInt(theNumber); t++)
    {
        new Thread(new Loop()).start();
    }
}

public void run() 
{
    int i;
    for (i = 0; i < 10; i++)
    {
        for (int j = 0; j < 1000000000; j++);
    }
}
}

Answer 1

Are you running a 32- or 64-bit version of Erlang? If it's 32 bit, then the inner loop limit 1000000000 won't fit in a single-word fixnum (max 28 bits incl. sign), and the loop will start to do bignum arithmetic on the heap which is way way more expensive than just incrementing a word and looping (it will also cause garbage collection to happen now and then, to get rid of old unused numbers from the heap). Changing the outer loop from 10 to 1000 and removing 2 zeros correspondingly from the inner loop should make it use fixnum arithmetic only even on a 32-bit BEAM.

Then, it's also a question of whether the Java version is actually doing any work at all, or if the loop gets optimized away to a no-op at some point. (The Erlang compiler doesn't do that sort of trick - at least not yet.)

Answer 2

RichardC answer gives some clue to understand the difference of execution time. I will add also that if your java code is compiled, it may benefits a lot from the predictive branching of the microprocessor, and thus make a better use of the cache memories.

But the more important in my opinion is that you are not choosing the right ratio of Process/processing to evaluate the cost of process spawning.

The test use 10 processes that does some significant work. I would have chosen a test where many processes are spawned (some thousands? I don't know how much threads the JVM can manage) each process doing very few things, for example this code which spawn at each step twice the number of process and wait for the deepest processes to send back the message done. With a depth of 17, which means 262143 processes in total and 131072 returned messages, it takes less than 0.5 s on my very slow PC, that is less than 2µs per process (of course the dual core dual thread should be used)

-module (cascade).

-compile([export_all]).


test() ->
    timer:tc(?MODULE,start,[]).

start() ->
    spawn(?MODULE,child,[self(),17]),
    loop(1024*128).

loop(0) -> done;
loop(N) ->
    receive
        done -> loop(N-1)
    end.

child(P,0) -> P ! done;
child(P,N) ->
    spawn(?MODULE,child,[P,N-1]),
    spawn(?MODULE,child,[P,N-1]).

Answer 3

There are a few problems here.

I don't know how you can evaluate what the Java compiler is doing, but I'd wager it's optimizing the loop out of existence. I think you'd have to have the loop do something meaningful to make any sort of comparison.

More importantly, the Erlang code is not doing what you think it's doing, as best as I can tell. It appears that each process is counting up to 1000000000, and then doing it again for a total of 10 times.

Perhaps worse, your functions are not tail recursive, so your functions keep accumulating in memory waiting for the last one to execute. (Edit: I may be wrong about that. Unaccustomed to the if statement.)

Here's Erlang that does what you want it to do. It's still very slow.

-module(realloop).
-compile(export_all).

start(N) ->
    loop(0, N).

loop(N, N) ->
    io:format("Spawned ~B processes~n", [N]);
loop(A, N) ->
    spawn(realloop, count, [0, 1000000000]),
    loop(A+1, N).

count(Upper, Upper) ->
    io:format("Reached ~B~n", [Upper]);
count(Lower, Upper) ->
    count(Lower+1, Upper).