"get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp using Selenium GeckoDriver on remote Ubuntu ligthsail

Question

I'm struggling with the Firefox webdriver implementation on a Lightsail instance since few days.

I'm trying to make run this simple code:

opts = FirefoxOptions()
opts.add_argument('-headless')

driver = webdriver.Firefox(options=opts)

print(f"Driver creation: {time.perf_counter() - ts}")
ts = time.perf_counter()


driver.install_addon(os.path.abspath('metamask-10.2.2-an+fx.xpi'), temporary=True)

print(f"Add on installation: {time.perf_counter() - ts}")
ts = time.perf_counter()

driver.get('https://www.youtube.com')

print(f"Go to Youtube: {time.perf_counter() - ts}")

driver.close()

On my local machine, I got for result:

Driver creation: 1.2246361820007223
Add on installation: 0.9261693749995175
Go to Youtube: 1.8188569550002285

And the geckodriver.log:

1644664439885   geckodriver INFO    Listening on 127.0.0.1:48035
1644664439888   mozrunner::runner   INFO    Running command: "/usr/bin/firefox" "--marionette" "-headless" "--remote-debugging-port" "55337" "-no-remote" "-profile" "/tmp/rust_mozprofileyCDIKp"
*** You are running in headless mode.
1644664440148   Marionette  INFO    Marionette enabled
[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp/rust_mozprofileyCDIKp/search.json.mozlz4", (void 0)))
DevTools listening on ws://localhost:55337/devtools/browser/1817941e-57a7-42f0-92dc-cc6c9aabaf0b
1644664441030   Marionette  INFO    Listening on port 43901
1644664441101   RemoteAgent WARN    TLS certificate errors will be ignored for this session
JavaScript warning: moz-extension://4ba69c0a-421f-4439-bc86-32f0a1c5a733/background-6.js, line 46: asm.js type error: expecting argument type declaration for 'e' of the form 'arg = arg|0' or 'arg = +arg' or 'arg = fround(arg)'
JavaScript warning: moz-extension://4ba69c0a-421f-4439-bc86-32f0a1c5a733/ui-3.js, line 7: unreachable code after return statement
JavaScript warning: moz-extension://4ba69c0a-421f-4439-bc86-32f0a1c5a733/ui-3.js, line 7: unreachable code after return statement
JavaScript warning: moz-extension://4ba69c0a-421f-4439-bc86-32f0a1c5a733/ui-3.js, line 7: unreachable code after return statement
JavaScript warning: moz-extension://4ba69c0a-421f-4439-bc86-32f0a1c5a733/ui-5.js, line 1: unreachable code after return statement
JavaScript warning: moz-extension://4ba69c0a-421f-4439-bc86-32f0a1c5a733/ui-5.js, line 1: unreachable code after return statement
JavaScript warning: moz-extension://4ba69c0a-421f-4439-bc86-32f0a1c5a733/ui-5.js, line 1: unreachable code after return statement
1644664442980   Marionette  WARN    Ignoring event 'pageshow' because document has an invalid readyState of 'uninitialized'.

I would like to have approximatly the same result on my remote Lightsail instance. I tried a lot of solution I found on google, but nothing worked, so this is what I did:

Delete instance -> Create a new one (Ubuntu 20.04LTS)
Make server routine:
- sudo apt-get update
- sudo apt-get upgrade
Install GeckoDriver:
- wget https://github.com/mozilla/geckodriver/releases/download/v0.30.0/geckodriver-v0.30.0-linux64.tar.gz
- tar -xvzf geckodriver*
- chmod +x geckodriver
- sudo mv geckodriver /usr/local/bin/
Install Firefox:
- sudo apt install firefox
Install python3-pip and the requirements for selenium.

The result after launching a test:

Driver creation: 6.041947366000045
Add on installation: 11.09006381100005
==> Server block after few minutes

and the geckodriver.log:

1644660948039   geckodriver     INFO    Listening on 127.0.0.1:41557
1644660948538   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "--marionette" "-headless" 
"--disable-dev-shm-usage" "--no-sandbox" "--window-size=1920,1080" "--ignore-certificate-errors" "--allow-runni
ng-insecure-content" "--proxy-server=\'direct://\'" "--proxy-bypass-list=*" "--start-maximized" "--disable-gpu"
 "--disable-setuid-sandbox" "--remote-debugging-port" "58861" "-no-remote" "-profile" "/tmp/rust_mozprofilejTQJ
s8"
*** You are running in headless mode.
[GFX1-]: glxtest: libGL.so.1 missing
[GFX1-]: glxtest: libEGL missing
1644660949544   Marionette      INFO    Marionette enabled
[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open t
he file at /tmp/rust_mozprofilejTQJs8/search.json.mozlz4", (void 0)))
DevTools listening on ws://localhost:58861/devtools/browser/73d9a07f-26e3-45aa-b118-79158a25d2dd
1644660953547   Marionette      INFO    Listening on port 46187
1644660953617   RemoteAgent     WARN    TLS certificate errors will be ignored for this session

The difference I found from the local geckodriver.log, is about libGL and libEgl, so I install:

apt install libgl1-mesa-glx
apt install libegl-dev

After testing:

Driver creation: 5.604287772000134
Add on installation: 52.37673208100023
==> Block after few minutes

The geckodriver.log:

1644671693878   geckodriver     INFO    Listening on 127.0.0.1:38405
1644671694374   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "--marionette" "-headless" 
"--remote-debugging-port" "35887" "-no-remote" "-profile" "/tmp/rust_mozprofileNhzPGd"
*** You are running in headless mode.
[GFX1-]: glxtest: Unable to open a connection to the X server
[GFX1-]: glxtest: libEGL missing
1644671695500   Marionette      INFO    Marionette enabled
[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open t
he file at /tmp/rust_mozprofileNhzPGd/search.json.mozlz4", (void 0)))
DevTools listening on ws://localhost:35887/devtools/browser/b99cd6dd-92a1-4c98-aec0-8c2dfb6c29b1
1644671699397   Marionette      INFO    Listening on port 39313
1644671699537   RemoteAgent     WARN    TLS certificate errors will be ignored for this session
JavaScript warning: moz-extension://2c7aa371-d63f-4896-8c5f-dab9902639ff/background-6.js, line 46: asm.js type 
error: expecting argument type declaration for 'e' of the form 'arg = arg|0' or 'arg = +arg' or 'arg = fround(a
rg)'

I really don't know how to make it work on my Lightsail instance. I don't know what I can add to help someone to find a solution, maybe the versionning on my remote:

firefox -v: Mozilla Firefox 96.0

I tried a lot of other solutions found on the web, but as I don't understand what's the problem, everything else does nothing more. Thanks advance.

undetected Selenium · Answer 1 · 2022-02-12T20:13:49.207

This error message...

1644671694374   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "--marionette" "-headless" "--remote-debugging-port" "35887" "-no-remote" "-profile" "/tmp/rust_mozprofileNhzPGd"
*** You are running in headless mode.
[GFX1-]: glxtest: Unable to open a connection to the X server
[GFX1-]: glxtest: libEGL missing
1644671695500   Marionette      INFO    Marionette enabled
[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp/rust_mozprofileNhzPGd/search.json.mozlz4", (void 0)))

...implies that the GeckoDriver instance was unable to initiate/spawn a new Browsing Context.

The most probable reason is there is/are dangling/zombie rust_mozprofile instances clinging within the /tmp directory. As a result, though /tmp/rust_mozprofileNhzPGd was created by GeckoDriver but /tmp/rust_mozprofileNhzPGd/search.json.mozlz4 can't be accessed.

Solution

You need to invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully at the end of each execution. Incase the dangling/zombie GeckoDriver process still continues to occupy your system memory you need to forcefully kill them.

References

You can find a relevant detailed discussions in:

Additionally, you may need to update the firefox version to the latest release of v97.0.

Thx for answer. I don't understand many things: - I code in Python, I did not find a tearDown methods in Selenium. - I don't have any folder that starts with "rust" in my /tmp/ folder. I created a new Lightsail Instance only to try this simple code, and it still doesn't work. I don't know what to do. Also; I have the same error in my local machine but it works well .. I'm not able to update firefox too with the command "sudo apt install --only-upgrade firefox" it tell me 96.0 is the newest version. — LeNomade, Feb 12 '22 at 20:08
Check the links mentioned in the _References_ section you will find discussions about `tearDown()` method. — undetected Selenium, Feb 12 '22 at 20:15
I used your solution with psutils you gave to another question. It changes nothing, it still doesn't works. I still have the "glxtest: libEGL" error and the "Javascript warning: moz-extnesion" error too. — LeNomade, Feb 12 '22 at 20:36
[Selenium](https://stackoverflow.com/a/54482491/7429447) version? GeckoDriver version? — undetected Selenium, Feb 12 '22 at 20:37
selenium==4.1.0 /// geckodriver = 0.30.0 : https://github.com/mozilla/geckodriver/releases/download/v0.30.0/geckodriver-v0.30.0-linux64.tar.gz — LeNomade, Feb 12 '22 at 20:43

"get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /tmp using Selenium GeckoDriver on remote Ubuntu ligthsail

1 Answers1

Solution

References