3

Im using PhantomJS to collect data about a Html page. My code it`s something like this:

from selenium import webdriver

class PageElements():

    def __init__(self, url):
        self.driver = webdriver.PhantomJS()
        self.driver.get(url)
        self.elements, self.attribute_types = self._load_elements(self.driver)

    def _load_elements(self, self.driver)
        """"This is not relevant"""

So, after I execute the code on IPython Notebook sometimes, to test things out. After a while, i get this on my Activity Monitor:

enter image description here

And this:

enter image description here

The processes still run even after i add a destroyer like:

def __del__(self):
    self.driver.close()

What is happening? I would really appreciate a "why this is happening" answer, instead a "do this" one. Why my destroyer isn't working?

I opened @forivall links, and saw the Selenium code. The PhantomJS webdriver has it`s own destructor (thus making mine redundant). Why aren't they working in this case?

Lucas Ribeiro
  • 6,132
  • 2
  • 25
  • 28
  • Is the code exiting before the end, maybe due to uncaught exceptions? – Richard Sep 25 '13 at 18:51
  • Nope. Added a print "exit" in the destructor, and it has been executed. – Lucas Ribeiro Sep 25 '13 at 19:03
  • 1
    Check out this page. http://stackoverflow.com/questions/13287490/is-there-a-way-to-use-phantomjs-in-python Seems phantomJS support was possibly dropped (as suggested on that page.) Also, I noticed that code on that page was using .quit(), not .close() - Are multiple browsers being opened during the code? If that is the case, .close() will close the current window, but not all of the opened windows. .quit() will exit the current driver. – Richard Sep 25 '13 at 19:09

5 Answers5

3

__del__() tends to be unreliable in python. Not only do you not know when it will be called, you don't even have any guarantees that it will ever be called. try/finally constructs, or (even better) with-blocks (a.k.a. context managers), are much more reliable.

That said, I had a similar issue even using context managers. phantomjs processes were left running all over the place. I was invoking phantomjs through selenium as follows:

from selenium import webdriver
from contextlib import closing
with closing(webdriver.PhantomJS()) as driver:
    do_stuff(driver)

contextlib's closing() function ensures that the close() method of its argument gets called whatever happens, but as it turns out, driver.close(), while available, is the wrong method for cleaning up a webdriver session. driver.quit() is the proper way to clean up. So instead of the above, do one of the following:

from selenium import webdriver
from contextlib import contextmanager

@contextmanager
def quitting(quitter):
    try:
        yield quitter
    finally:
        quitter.quit()

with quitting(webdriver.PhantomJS()) as driver:
    do_stuff(driver)

or

from selenium import webdriver
driver = webdriver.PhantomJS()
try:
    do_stuff(driver)
finally:
    driver.quit()

(The above two snippets are equivalent)

Credit goes to @Richard's comment on the original question for pointing me toward .quit().

jcdyer
  • 18,616
  • 5
  • 42
  • 49
  • 1
    tried to edit the second code example to remove the extra ")" for `driver = webdriver.PhantomJS())` but edits need to be 6 chars or more. – Eitan Nov 16 '15 at 16:11
1

As of July 2016, following the discussion on this GitHub issue, the best solution is to run:

import signal

driver.service.process.send_signal(signal.SIGTERM)
driver.quit()

Instead of driver.close(). Just running driver.quit() will kill the node process but not the phantomjs child process that it spawned.

leekaiinthesky
  • 5,413
  • 4
  • 28
  • 39
0
    self.driver = webdriver.PhantomJS()

This creates a web browser that is then used by Selenium to run the tests. Each time Selenium runs, it opens a new instance of the web browser, rather than looking to see if there is a previous one it could re-use. If you do not use .close at the end of the test, then the browser will continue to run in the background.

As you have seen, running the test multiple times leaves multiple browsers orphaned.

Richard
  • 8,961
  • 3
  • 38
  • 47
0

What the difference between this case, and objects that Python usually destroy automatically with it`s garbage collector?

The difference is that it's creating something outside of Python's domain: it's creating a new OS-level process. Perhaps webdriver.PhantomJS should have its own __del__ that will shut itself down Perhaps the behaviour should be more robust, but that's not the design decision that the selenium developers went with, probably because most of the other drivers are not headless (so it's obvious that the windows are open).

Unfortunately, neither the selenium (or unofficial) documentation has much clarification/best practices on this. (see comments below on __del__ behaviour).

links to source: phantomjs/webdriver.py remote/webdriver.py (superclass)

forivall
  • 9,504
  • 2
  • 33
  • 58
  • Thats's odd. I edited my question. Selenium Webdriver for PhantomJS has its destructor in the implementation. But it`s not working either. – Lucas Ribeiro Sep 25 '13 at 18:48
  • It's been a while since I've played with `__del__`, and it turns out it has some edge cases: http://stackoverflow.com/questions/3554952/del-at-program-end/3555013#3555013 . Perhaps the selenium python api would be nicer if it supported context manager usage (`__enter__`, `__exit__`, `with ... as :`. Using a context manager would require you to re-organize your code though, but for the better. – forivall Sep 25 '13 at 20:19
  • Huh, I got a drive-by downvote last week. Please add a comment on how my answer can be improved, or how it may be incorrect. – forivall Jan 31 '14 at 23:28
0

I was also struggling for the same problem and I solved it from this source link.

By replacing self.process.kill() in selenium/webdriver/phantomjs/service.py with self.process.send_signal(signal.SIGTERM).

By using driver.quit() will kill all process of phantomjs on completing program or cancel the program using CTR+C

Mukesh
  • 421
  • 6
  • 20