I am attempting to you PhantomJS in order to run my webscrapes with Selenium and Python without needing to open up a new window when the scrape loops into the next page.
I consulted this post initially: Is there a way to use PhantomJS in Python?
However I had to modify my path to get to node.exe in my code below:
for link in soup1.findAll('a', {'property_title'}):
#print 'https://www.tripadvisor.com/Restaurant_Review-g294217-' + link.get('href')
restaurant_url = 'https://www.tripadvisor.com/Restaurant_Review-g188590-' + link.get('href')
driver = webdriver.PhantomJS(r"C:\Program Files (x86)\nodejs\node.exe")
driver.get(restaurant_url)
neighborhood = driver.find_element_by_xpath(r'//*[@id="BODYCON"]/div[2]/div/div[2]/div[2]/div[1]/div[1]/div[2]/div[2]/div[2]/ul/li[3]')
restneighborhood = neighborhood.text
print restneighborhood
I get this error:
Traceback (most recent call last):
File "C:/Users/dtrinh/PycharmProjects/TripAdvisorData/LinkPull-HK.py", line 23, in <module>
driver = webdriver.PhantomJS(r"C:\Program Files (x86)\nodejs\node.exe")
File "C:\Python27\lib\site-packages\selenium-3.0.1-py2.7.egg\selenium\webdriver\phantomjs\webdriver.py", line 52, in __init__
self.service.start()
File "C:\Python27\lib\site-packages\selenium-3.0.1-py2.7.egg\selenium\webdriver\common\service.py", line 86, in start
self.assert_process_still_running()
File "C:\Python27\lib\site-packages\selenium-3.0.1-py2.7.egg\selenium\webdriver\common\service.py", line 99, in assert_process_still_running
% (self.path, return_code)
selenium.common.exceptions.WebDriverException: Message: Service C:\Program Files (x86)\nodejs\node.exe unexpectedly exited. Status code was: 9
I have no idea what I am doing because I usually use the ChromeDriver, but if anyone could help me that would be awesome.
