Both the server and my computer have geckodriver 0.26.0, Firefox 71, and Selenium 3.141.0.
My computer has MacOS Mojave with python 3.8 and the server is CentOS 7 with python 3.7. The code runs perfectly on my computer, but it returns errors on the server.
I don't remember how, but I have been getting different errors depending on if I add breakpoints to it, or if I run it in terminal or submit the job in SLURM.
On terminal:
File "Main.py", line 230, in <module>
main()
File "Main.py", line 179, in main
dfs=get_data(stations, inidate, findate)
File "Main.py", line 113, in get_data
list_files=return_list_day(date)
File "Main.py", line 66, in return_list_day
driver.get(webdir)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
or
Submitted on SLURM (and most often error):
Traceback (most recent call last):
File "Main.py", line 230, in <module>
main()
File "Main.py", line 179, in main
dfs=get_data(stations, inidate, findate)
File "Main.py", line 113, in get_data
list_files=return_list_day(date)
File "Main.py", line 66, in return_list_day
driver.get(webdir)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Timeout loading page after 300000ms
I can use Selenium without issues in another unrelated code that finishes perfectly fine. So I don't understand why Selenium is having a hard time here or where it comes from.
The code fails here:
## Commented parts are cuz I tried running with the MOZ_HEADLESS tag on terminal
## Didnt make a difference as far as I could tell.
#options = Options()
#options.headless = True
driver = webdriver.Firefox()#options=options)
driver.get(webdir) ##<-- HERE
## Added these because of other replies to this issue I found
driver.implicitly_wait(7)
time.sleep(3)
.
.
.
driver.close()
That bit of codes retrieves a list from the webpage and is ran inside a for-loop. The error is not in the retrieving part, is in driver.get(webdir). I can't share the website since it's literally like looking into the server of a partner institution. webdir is a directory and I am basically waiting till its contents are loaded so I can retrieve its file names.
I know you can't help much without a website, but do the errors I've shown give any indication as to what the problem might be?. Can the retrieving of a specific website behave differently based on OS? I have googled the errors, found questions here, read through them and applied them to see how it changed, and nothing did or I got a different error (either one of the two above).
I found this that states incompatibility between gecko and Mozilla, but since I can successfully run another (unrelated) code with the exact calling and usage of Selenium (only different URL given) then I don't think that's my issue.
Thanks for any help! Let me know what other information I could give that might help.
Edit:
It is not the same as the question that was linked, since I have given it sleep time and didn't change anything. It has 40 GB of ram allocated so its not dying out of too little memory. Which are the solutions shown in this question.
