I want to deploy my Python script on Ubuntu server and call it via cron. On my local Windows machine, I tried headless and it is working perfectly fine, I can even take screenshots in it. But running the script on the server results in errors like the element cannot be found. Can someone tell me what is happening with this?
Error Reproduced:
File "DomainScraper.py", line 30, in <module>
login = driver.find_element_by_xpath('//a[@href="'+login_url+'"]').click()
File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
'value': value})['value']
File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[@href="https://account.domaintools.com/log-in/?r=https%3A%2F%2Freversewhois.domaintools.com%2F%3Frefine"]"}
(Session info: headless chrome=81.0.4044.17)
My code, which is I deployed in the server:
#imports...
options = Options()
options.add_argument('--incognito')
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-extensions')
options.add_argument('--disable-infobars')
options.add_argument('--allow-running-insecure-content')
driver = webdriver.Chrome(options=options)
driver.delete_all_cookies()
driver.implicitly_wait(3)
url = "https://reversewhois.domaintools.com/?refine#q=%5B%5B%5B%22whois%22%2C%222%22%2C%22VerifiedID%40SG-Mandatory%22%5D%5D%5D"
driver.get(url)
driver.save_screenshot("sample.png")
login_url = 'https://account.domaintools.com/log-in/?r=https%3A%2F%2Freversewhois.domaintools.com%2F%3Frefine'
login = driver.find_element_by_xpath('//a[@href="'+login_url+'"]').click()
username = driver.find_element_by_id("username")
password = driver.find_element_by_id("password")
username.send_keys("**********************")
password.send_keys("***************")
# time.sleep(5)
driver.find_element_by_id("password").send_keys(Keys.ENTER)
pageNumber = 0
while True:
driver.implicitly_wait(3)
driver.get('https://reversewhois.domaintools.com/?ajax=mReverseWhois&call=ajaxGetPreviewPage&q=%5B%5B%5B%22whois%22%2C%222%22%2C%22VerifiedID%40SG-Mandatory%22%5D%5D%5D&o='+str(pageNumber))
time.sleep(3)
pre = driver.find_element_by_tag_name("pre").text
data = json.loads(pre)
if data['body']:
table = data['body']
tables = pd.read_html(table,skiprows=1)
df = tables[-1]
df.to_csv('Domains.csv', mode='a', sep=',',index=False)
print(df.to_string(index=False))
pageNumber += 1
# print(pageNumber)
continue
else:
break
Update:
Tried using and installing both libraries
sudo apt install -y xvfb
pip install pyvirtualdisplay
and added this before launching Chrome
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
seems not to be working after all. I took a screenshot and got this output:
When I don't use the xvfb library, I only get white screen.
I think Selenium cant open the url. What should I do about this?
