3

Erlang Interoperability guide discusses different interoperability mechanisms. Here are my conclusions:

  • Ports and Erl_Interface programs: OS scheduled, limit scalability.

  • Port Drivers: dangerous because a crash in the port driver brings the emulator down too.

  • C Nodes: Node server needs to scale as well as Erlang app to avoid scalability sacrifices.

  • NIFs: Loic sums them up well.

Some advocate the use of OpenCL basically delegating resource hungry computations to GPU while letting the Erlang emulator to own the CPU. This sounds fantastic but then you have a requirement on your servers having a suitable GPU.

Using JInterface and communicating with a Java process that spawns a thread for every request might be an option.

So has anyone come across a solution that has been tested in practise and turned out to work well?

spc16670
  • 504
  • 6
  • 15
  • 2
    The description of NIFs you link to looks out of date and inaccurate. For example, a NIF running on a scheduler "for more than a few microseconds" will not cause problems; this happens only if a NIF runs for more than a few milliseconds. Also, the [`enif_schedule_nif` function](http://www.erlang.org/doc/man/erl_nif.html#enif_schedule_nif) provides a nice way to break up long NIFs to avoid scheduler issues, plus with the experimental [dirty scheduler functionality](http://www.erlang.org/doc/man/erl_nif.html#dirty_nifs) NIFs can run for as long as they want without causing scheduler hiccups. – Steve Vinoski Sep 29 '15 at 15:00
  • To use the dirty scheduler functionality you need to built the emulator with experimental dirty scheduler support enabled. I do not think the idea of running production code with experimental functionalities will have many enthusiasts. In general NIFs seem a bit dangerous. As Loic pointed, they may crash your VM. And compiling libraries that use them under Windows is a nightmare - and I am not a fan of running Erlang on WIndows. – spc16670 Sep 29 '15 at 15:25
  • 2
    The experimental dirty scheduler functionality will someday become a regular feature, hopefully in Erlang 19. I implemented the dirty scheduler functionality for Erlang/OTP and I run all my Erlang 17 and 18 VMs with it enabled, and have not had any problems with it. Also, see [these measurements](https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7), which show the tiny amount of overhead involved in switching a process over to a dirty scheduler. – Steve Vinoski Sep 29 '15 at 15:52

1 Answers1

3

Actually all solutions take place. As I've been working tightly with some of them I could say the following:

  • Ports are safe but port communication is slow. If port crashes, VM continues working. If you do not communicate with your port extensively or you do not trust the port - this is your choice

  • NIFs are extremely fast. If your data flow is great you should use them. Of course they are unsafe so you have to program NIF library carefully and you'd better learn some C (the point that most of NIF creators skip). Actually scheduling problems are easily overcome with the specific pattern. You should start the new C thread that does actual job just after receiving data from Erlang and detach processing from Erlang thread. So you quit NIF function very quickly returning back in Erlang and waiting for a message from C code.

  • Java Nodes or C nodes are for tasks that can be moved to the node completely. That are some long and heavy jobs.

Bearing in mind above considerations you decide the way that fits your task best.

spc16670
  • 504
  • 6
  • 15
Lol4t0
  • 12,444
  • 4
  • 29
  • 65
  • Would you agree there is no silver bullet then? Ports are a trade off between scalability v reliability and NIFs are a risk to Erlang resiliency unless written very well. A scalable C or Java node might be a good fit from integration perspective. – spc16670 Sep 30 '15 at 08:40
  • 1
    @coolfeature, Standalone nodes still slower to communicate to comparing to NIFs. Erlang only have limited sockets pool (if I'm not mistaken) and you'll have to serialize/deserialize data. If time to process data is signifactly greater than time to serialize/transfer/deserialize, standalone nodes will fit you. – Lol4t0 Sep 30 '15 at 08:49