Erlang VM killed when creating millions of processes

Question

So after Joe Armstrongs' claims that erlang processes are cheap and vm can handle millions of them. I decided to test it on my machine:

process_galore(N)->
    io:format("process limit: ~p~n", [erlang:system_info(process_limit)]),
    statistics(runtime),
    statistics(wall_clock),
    L = for(0, N, fun()-> spawn(fun() -> wait() end) end),
    {_, Rt} = statistics(runtime),
    {_, Wt} = statistics(wall_clock),
    lists:foreach(fun(Pid)-> Pid ! die end, L),
    io:format("Processes created: ~p~n
          Run time ms: ~p~n
          Wall time ms: ~p~n
          Average run time: ~p microseconds!~n", [N, Rt, Wt, (Rt/N)*1000]).

wait()->
    receive die ->
         done
    end.

for(N, N, _)->
    [];
for(I, N, Fun) when I < N ->
    [Fun()|for(I+1, N, Fun)].

Results are impressive for million processes - I get aprox 6.6 micro! seconds average spawn time. But when starting 3m processes, OS shell prints "Killed" with erlang runtime gone. I run erl with +P 5000000 flag, system is: arch linux with quadcore i7 and 8GB ram.

Were you running out of memory when it printed "Killed" and killed the process? See http://stackoverflow.com/questions/726690/who-killed-my-process-and-why. — Dogbert, Aug 06 '16 at 17:42
that was it, thanks for the clue. dmesg shows out of memory fault — Sharas, Aug 06 '16 at 19:55

score 9 · Accepted Answer · edited May 23 '17 at 12:09

9

Erlang processes are cheap, but they're not free. Erlang processes spawned by spawn use 338 words of memory, which is 2704 bytes on a 64 bit system. Spawning 3 million processes will use at least 8112 MB of RAM, not counting the overhead of creating the linked list of pids and the anonymous function created for each process (I'm not sure if they're shared if they're created like you're creating.) You'll probably need 10-12GB of free RAM to spawn and keep alive 3 million (almost) empty processes.

As I pointed out in the comments (and you later verified), the "Killed" message was printed by the Linux Kernel when it killed the Erlang VM, most likely for using up too much RAM. More information here.

edited May 23 '17 at 12:09

Community

1
1

answered Aug 06 '16 at 20:37

Dogbert

212,659
41
396
397

1

Memory was the bottleneck here alright. I am running the example code Joe Armstrong gave in his "Programming Erlang" book. It says he was running tests on i7 with 8GB, just like me. And he gives example times of creating 3m processes without dramatic increase in spawn time as would happen if swap area were used. My machine doesn't have swap - hence the kill. – Sharas Aug 07 '16 at 15:29

Erlang VM killed when creating millions of processes

1 Answers1