4

I don't want to have a long running process. But at the same time, want to use Supervisor, and worker processes. How should I go about it?

Scenario: Every midnight I fetch data from an external API using supervised processes.

But I don't want the process to run until next day if the workers complete their job.

In my case, 1 Supervisor, 5 poolboy processes always exist.

Since the erlang processes are cheap, and these are idle, should I have the supervisor part of my applications and start every time, and leave it idle?

Or should I try to kill the Supervisor when the worker processes are done?

When I do :observer.start I could see few of my dependency packages have processes started. Is this okay? When should I worry about the number of processes?

Vysakh Sreenivasan
  • 1,719
  • 1
  • 16
  • 27

1 Answers1

4

A few processes really isn't a big deal. For one application I worked on, we tested starting it up with a few million poolboy processes, and it only took a little over 1GB of RAM to do so. According to the docs:

A newly spawned Erlang process uses 309 words of memory in the non-SMP emulator without HiPE support. (SMP support and HiPE support both add to this size.)

So really, don't worry about dependencies spawning too many processes. It's not a big deal.

As for your structural question, one possibility is to use a simple_one_for_one supervisor. You then just have a cron job that sends a message (perhaps a cURL POST request if you have a web server going) to the running VM to execute whatever your batch job is, or use the timer moduleto run jobs at specified intervals.

EDIT: Elixir has similar functionality in the Process module. See Jose Valim's answer to a similar question.

However you choose to start your job, you can them simply call supervisor:start_child or Supervisor.start_child, depending on which language you're working in, to start a worker for your job.

Community
  • 1
  • 1
Cody Poll
  • 7,690
  • 3
  • 27
  • 22