2

I am currently building a reporting tool with Selenium on Adwords. (And no, I cannot use Google's API because it doesn't provide the functionality I want).

I am trying to avoid logging in and logging out, because I understand frequent logins and logouts are not as convenient as cookie based authentication.

I have the following code: save.py

try:
  driver = webdriver.Chrome()
  driver.get('https://adwords.google.com')
  time.sleep(90)
  # Manually login to adwords page and wait
  pickle.dump( driver.get_cookies() , open("cookies.pkl","wb"))

finally:
  driver.close()

And this: load.py

try:
  driver = webdriver.Chrome()
  cookies = pickle.load(open("cookies.pkl", "rb"))
  for cookie in cookies:
    driver.add_cookie(cookie)

  driver.get('https://adwords.google.com/')
  time.sleep(60)

finally:
  driver.close()

When I first run load.py, I am actually able to see the spinner that shows up when one logs into Adwords. Shortly after however, I get logged out!

I don't know what is causing Google's authentication system to log me out. What do you think is the cause of this?

Code Repo: https://gist.github.com/anonymous/63d32e49f59a09ab82fac1f9f972d555

Tinker
  • 4,165
  • 6
  • 33
  • 72

1 Answers1

3

The method driver.add_cookie works for the current domain only. So you'll first have to set the domain with driver.get(...) before calling driver.add_cookie:

import json, re

def save_cookies(driver, file_path):
  with open(file_path, 'w') as file:
    cfg = {
      'url': driver.current_url,
      'cookies': driver.get_cookies()
    }
    json.dump(cfg, file, indent=2)

def load_cookies(driver, file_path):
  with open(file_path, 'r') as file :
    cfg = json.load(file)
    driver.get(re.match(".+?//[^/]+", cfg['url'])[0] + '/favicon.ico')

    for cookie in cfg['cookies']:
      cookie['expiry'] = cookie.get('expiry', -1)
      driver.add_cookie(cookie);

# save the cookies
save_cookies(driver, r"cookies.json");

# restore cookies
load_cookies(driver, r"cookies.json")

Or by calling the devtool API to save/restore the cookies for all the domains (Chrome only):

from selenium import webdriver
import json, base64

def send_devtools(driver, cmd, params={}):
  resource = "/session/%s/chromium/send_command_and_get_result" % driver.session_id
  url = driver.command_executor._url + resource
  body = json.dumps({'cmd': cmd, 'params': params})
  response = driver.command_executor._request('POST', url, body)
  if response['status']:
    raise Exception(response.get('value'))
  return response.get('value')

def save_cookies(driver, file_path):
  cookies = send_devtools(driver, "Network.getAllCookies", {})  
  with open(file_path, 'w') as file:
    json.dump(cookies, file, indent=2)

def load_cookies(driver, file_path):
  with open(file_path, 'r') as file :
    cookies = json.load(file)
    send_devtools(driver, "Network.setCookies", cookies)

# save cookies
save_cookies(driver, r"c:\temp\cookies.json");

# restore cookies
load_cookies(driver, r"c:\temp\cookies.json")
Florent B.
  • 41,537
  • 7
  • 86
  • 101
  • Can you explain why you chose to get the favicon? Is it because you don't actually want to load the site? – Tinker Oct 30 '17 at 19:56
  • /favicon.ico is the cheapest way to set the domain. loading the site directly will set the default cookies and you'll still have to reload the site to get the page as logged in. – Florent B. Oct 30 '17 at 20:00
  • I tried using your code but it seems like adwords is still not letting me: https://gist.github.com/anonymous/e9250e49c847b6a2e1eaad45143044d8 – Tinker Oct 31 '17 at 05:09
  • I had a look at the cookies created from adword and it seems that the page is somehow using cross domain cookies (.doubleclick.net). Another way would be to use the devtool api to save and restore all the cookie. You could also use a static profile by launching the driver with the argument user-data-dir. – Florent B. Nov 01 '17 at 16:56
  • I opened a incognito browser and logged in via adwords. Somehow, all the domains for all the cookies I see (via cookies.txt plugin) are just google.com and adwords.google.com. How did you manage to see .doubleclick.net? – Tinker Nov 01 '17 at 18:08
  • I just has a look in `Application / Cookies / https://adwords.google.com` from a simple instance with Chrome. I only managed to recover a Session from Google via the devtool api by saving/restoring all the cookies of the browser. – Florent B. Nov 01 '17 at 18:17