1

I'm using selenium to access a page that requires logging in. Currently I'm doing something like this:

import json
from selenium.webdriver import Chrome

cookies_path = 'tmp/cookies.json'
cookies = json.load(open(cookies_path))
driver = Chrome()
driver.get('https://www.example.com/')
for cookie in cookies:
    driver.add_cookie(cookie)
driver.get('https://www.example.com/user-data')
# do stuff
json.dump(driver.get_cookies(), open(cookies_path, 'w'))
driver.quit()

The script is scheduled to run every hour. However, it can only run successfully 23 times. At the 24th time, when driver.get('https://www.example.com/user-data') is executed, the driver will be redirected to the login page and my code breaks. Then I have to manually log in again, manually save the cookies, but the new cookies will still be invalid after 24 hours. But on my own laptop, I can stay logged in on this website indefinitely and have never been redirected to log in. I tried skipping expired ones when adding cookies to the driver, but it did not help.

FYI the website is using OAuth2 and I guess my cookies expire when the session token expires on the server. Is there a way to keep the cookies valid indefinitely with selenium like with my browser?

Edit:

Just saw this person had the same problem as mine, but no answer so far.

2 Answers2

1

Its because cookies are expiring after 24 hours in case of selenium whereas on your laptop the browser is likely storing sessions token in secure session storage which is separate from cookies and which persists. You can achieve the same effect with selenium by storing the session token in a secure storage on your local machine(serialise in a file) and then retrieving(deserialise) it each time you run the script.

Saheed Hussain
  • 1,096
  • 11
  • 12
0

Thanks to @Saheed Hussain's answer and this answer, I solved it by doing

from selenium.webdriver import Chrome, ChromeOptions

options = ChromeOptions()
options.add_argument('--user-data-dir=tmp/user_data')
options.add_argument('--profile-directory=Profile 1')
driver = Chrome(options=options)
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Apr 21 '23 at 04:41