14

I am having a problem when running multiple instances of PhantomJS on Ubuntu 14. After a few minutes, the processes become unresponsive.

Brief background: Using PhantomJS 2.0 to render a web page that ultimately gets saved as a PDF using Wkhtmtopdf. PhantomJS is only responsible for loading the page, making ajax requests, and waiting for a response after the PDF is saved on the server. It does not generate the PDF itself. There are hundreds of web pages that need to be generated into PDF, so I want to run as many PhantomJS instances in parallel as the system allows.

Each PhantomJS process is started by a shell script like so: {path to phantomjs} {path to js file} --data {some argument} >> {path to log file} 2>&1 &

The problem occurs after a couple of minutes where I stop getting any output from the PhantomJS processes, and looking at top I can see they are just laying there not doing anything. The JS script has timers that retry to load a page if it takes longer than a minute, and eventually call phantom.exit() if the page can't load / PDF generation fails. So even if something goes wrong, the process should still exit - but it doesn't.

I tried changing the number of PhantomJS instances running in parallel. Tried 20 -> 10 -> 5 -> 3, but it doesn't seem to matter. I can actually get many more jobs execute successfully when maintaining 20 instances at a time.

When running with --debug=true I can see that at some point it gets stuck at [DEBUG] WebPage - updateLoadingProgress: Also going through the output I see several of these warnings: [WARNING] QIODevice::write: device not open which makes me believe that is the source of the problem.

I thought there might be some contention for file resources so I tried without redirecting output to a log file, and not using --local-storage-path, but that didn't help.

As a side note, I have been using PhantomJS for several years now doing the same procedure, only sequentially (run a single PhantomJS process at a time). And although there were a few snags to overcome, it worked great.

Any idea what's causing this? Anyone faced with a similar problem? Any advice on running multiple PhantomJS instances in parallel?

Thanks!

user2300061
  • 171
  • 7
  • I'm having the exact same issues! Did you manage to find a solution? Thanks! – YuviDroid Jan 20 '16 at 22:39
  • @YuviDroid, I haven't found a solid solution. I was able to avoid the problem by limiting the number of instances running at a any given time, and closing / creating a new page object when retrying a job instead of using the same page. It was a while ago, so I don't remember exactly what made it "stable" but, it took a lot of trial and error. And I would like to be able to run many more instances than I currently am running, so a proper solution would be appreciated. – user2300061 Jan 22 '16 at 11:05
  • What version of PhantomJS are you using? It may be related to this image rendering bug which is supposed to have been fixed as of 2.1.1 -> http://code.google.com/p/phantomjs/issues/detail?id=54 – Cahit Jan 25 '16 at 23:16
  • 1
    Have you considered the possibility it's actually the web server causing the issue? Many servers limit concurrent connections (or even do ip-based throttling) and if phantomjs was exceeding the server's own "requests per whatever" limit then it would have to wait (ie, "hang") until the server released more data. – SpliFF Jan 27 '16 at 09:46
  • abandon phantomJS and use electron or nightmare https://github.com/segmentio/nightmare http://electron.atom.io/ – antpaw Jan 27 '16 at 22:19

2 Answers2

0

I faced the exact same issue both locally and on my CI server (which was also Ubuntu). Uninstalling 2.0.0 and upgrading to to 2.1.1 resolved the problem for me.

Anthony E
  • 11,072
  • 2
  • 24
  • 44
  • It didn't resolve my issue. On the contrary, it seems there is some nasty bug in version 2.1.1 that keeps single instances hanging forever. I noticed that I wasn't getting the PDFs on time, and when I ran `top` I saw there were dozens of phantomjs processes just hanging. I had to downgrade back to version 2.0 – user2300061 May 29 '16 at 10:24
0

I was facing the same issue. Use driver.quit() instead of driver.close(). That solved the issue for me.

Sunil L
  • 56
  • 6