1

I'm trying to make a cookie scanner that gets all cookies from a website. My approach was to open a website with Selenium ChromeDriver and read the sqlite3 database which the Chrome browser creates.

My problem is that the database is apparently empty, but in the Chrome browser window, I can see the cookies.

Here is my code and a screenshot of the browser:

from selenium import webdriver
import os, shutil, sqlite3


browser_list_place = 0
browser_list = []
profiles_folder = "profiles"


def getcookies(url):
    if os.path.isdir(profiles_folder):
        shutil.rmtree(profiles_folder)

    co = webdriver.ChromeOptions()
    co.add_argument("--no-sandbox")
    co.add_argument("--user-data-dir=" + profiles_folder + "/" + str(browser_list_place))

    browser_list.append(webdriver.Chrome('D:\crawler\chromedriver.exe', options=co))
    browser_list[browser_list_place].set_page_load_timeout(30)

    browser_list[browser_list_place].get(url)

    #browser_list[browser_list_place].quit()

    for folder in range(0, browser_list_place + 1):
        con = sqlite3.connect(profiles_folder + "/" + str(folder) + "/Default/Cookies")
        cur = con.cursor()
        cur.execute("SELECT * FROM cookies")
        rows = cur.fetchall()
        for row in rows:
            print(row)


getcookies('https://developer.mozilla.org/de/')

enter image description here

Why is the database empty for my program?

aschultz
  • 1,658
  • 3
  • 20
  • 30
Basti G.
  • 411
  • 1
  • 5
  • 26

1 Answers1

1

Use pickle to save and load cookies

import pickle as pk
pk.dump(driver.get_cookies(), open("cookies.pkl", "wb"))  # save cookies

cookies = pk.load(open("cookies.pkl", "rb"))  # load cookies
for cookie in cookies:
    driver.add_cookie(cookie)
Kostas Charitidis
  • 2,991
  • 1
  • 12
  • 23
  • But gives me `driver.get_cookies()` all cookies also includes the third party cookies? Because Google Chrome shows 7 Cookies and `driver.get_cookies()` only 5 for the url above. And if i open the sqlite3 database with a sqlite browser it shows 6 cookies. Why? – Basti G. Aug 14 '19 at 12:31
  • Nope. I think selenium allows you to interact with cookies from current domain only. – Kostas Charitidis Aug 14 '19 at 12:35
  • Yes and this was the reason why i must read the sqlite3 database from Google Chrome because i think these includes all cookies... – Basti G. Aug 14 '19 at 12:38
  • Not sure if this could help: https://stackoverflow.com/questions/22532870/encrypted-cookies-in-chrome – Kostas Charitidis Aug 14 '19 at 12:42
  • Maybe later. My current problem is that the database is sometimes empty and sometimes not also i open it with a sqlite browser and i don't know why. Deletes Google Chrome the cookies after a specific time or do i something wrong? – Basti G. Aug 14 '19 at 12:48
  • Or is there a more efficient way to get all cookies (first and third party) without selenium? The values is not so important for me but all other information like name, host, expires, .... – Basti G. Aug 14 '19 at 12:53
  • I cannot give you a very specific answer. Another package is `browsercookie` https://bitbucket.org/richardpenman/browsercookie/ but when i tried it it didnt have permissions to the sqlite file so I quit trying. – Kostas Charitidis Aug 14 '19 at 13:08
  • Okay.. but have you an idea why the cookie database from chrome browser sometimes is empty? – Basti G. Aug 14 '19 at 14:44