3

I've been struggling to run multiple instances of Puppeteer on DigitalOcean for quite some time with little luck. I'm able to run ~5 concurrently using tools like puppeteer-cluster, but for some reason the whole thing just chokes with little helpful messaging. So, I switched to spawning ~5 child processes without any additional library -- just Puppeteer itself. Same issue. Chokes with no helpful errors.

I'm able to run all of these jobs just fine locally, but after I deploy, I hit these walls. So, my hunch is that it's a resource/performance issue, but I can't say for sure.

I'm running a droplet with 1GB and 3CPUs on Digital Ocean.

Basically, I'm just looking for ways to start troubleshooting something like this. is there a way I can know for sure that I'm hitting resource walls? I've tried pm2 and the DO dashboard graphs, but I feel like those are all leaving a lot of information out, or else I'm missing something else altogether.

Thomas Dondorf
  • 23,416
  • 6
  • 84
  • 105
Alex MacArthur
  • 2,220
  • 1
  • 18
  • 22

2 Answers2

4

Author of puppeteer-cluster here. You are right, 1 GB of memory is likely not enough for running 5 browser windows (or tabs) in addition to your operating system and maybe even other background tasks.

Here is a list of resources you should check:

  • Memory: Use a tool like htop to check your memory usage while your application is running.
  • CPU: Again, you can use htop for that, 3 vCPUs should be more than enough for 5 windows.
  • Disk space: Use a tool like df to check if there is enough space on the disk. I know of multiple cases in which there was not enough space on the disk (like some old kernels filling the disk), and Chrome needs at least some space to run.
  • Network throughput: Rarely the problem, but sometimes the network just does not have the bandwidth to support many open browser. Use a tool like nload to check the network throughput.

To use htop or nload, you start your script in the background (node script.js &) or use a terminal multiplexer (like tmux). Resource problems should then be easy to spot.

Thomas Dondorf
  • 23,416
  • 6
  • 84
  • 105
  • Thanks, I'll try those out! – Alex MacArthur Jul 31 '19 at 18:58
  • 1
    Your `htop` recommendation helped resolve the issue! Amazing, helpful tool. Found out that CPU usage was the main thing I was underestimating. I resized my droplet accordingly and now things are running very reliably. Thanks again for the insight! – Alex MacArthur Aug 13 '19 at 17:05
1

Most probably you're running out of memory, 5 puppeteer processes are a lot for a 1GB VM.

You can run 

grep -i 'killed process' /var/log/messages

to confirm that the OOM killer terminated your processes.

Oliver
  • 11,857
  • 2
  • 36
  • 42