Keep sessions and cookies saved (profile) on Chrome in headless mode via chromedriver and selenium to keep logged in

Question

I have written myself a little bot which determines if the bot is logged in for the application or not. If not, it will log in the user and do some stuff. However, I would like to run this bot in background now without having the bot to need to log in every time. The bot should only be logged in once and then kept logged in via session cookies.

This actually does even work when running my bot not in the headless mode. However, as soon as I add the argument options.add_argument("--headless") the bot has to log in every time the script and browser starts. This means, my session cookies are not saved in the designated user profil but are saved in the designated user profile when running the bot without the headless argument.

Those are my settings:

# Browser
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")
options.add_argument("user-data-dir=/Users/helloworld/Library/Application Support/Google/Chrome/User Data/Profile 1")

s = Service("/Users/helloworld/Desktop/Programmieren/chromedriver")
browser = webdriver.Chrome(service=s, options=options)

I already came across this stack post which suggest to append the user to the end of the user-data-dir argument which I also did. But still, it has no effect for me. The user doesn't kept logged in!

I am running on the following versions:

• Python 3.9.10
• Chrome 98.0.4758.102
• ChromeDriver 98.0.4758.80
• Selenium 4.1.0
• MacOS 11.6

Does anyone know how to run selenium Chrome in headless mode and the given user profile?

Kind regards and thank you!