Load session cookies with scrapy

Question

I'm using scrapy to scrape sites that require login, but I'm no sure exactly which are the fields requires to save and load in order to keep the session.

With selenium I'm doing the following to save the cookies:

import pickle
import selenium.webdriver 

driver = selenium.webdriver.Firefox()
driver.get("http://www.google.com")
pickle.dump( driver.get_cookies() , open("cookies.pkl","wb"))

And this to load them:

import pickle import selenium.webdriver

driver = selenium.webdriver.Firefox()
driver.get("http://www.google.com")
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
    driver.add_cookie(cookie)

And it works just fine, is it possible to do exactly this using scrapy

Possible duplicate of [Scrapy - how to manage cookies/sessions](https://stackoverflow.com/questions/4981440/scrapy-how-to-manage-cookies-sessions) — Gallaecio, Aug 01 '19 at 10:42

score 1 · Answer 1 · answered Jul 12 '19 at 17:42

1

Sent a request using cookies:

request_with_cookies = Request(url="http://www.example.com", cookies={'currency': 'USD', 'country': 'UY'})

Get cookies from response:

cookies_from_response = response.headers[b'Cookies'].decode()

answered Jul 12 '19 at 17:42

0x01h

843
7
13

Load session cookies with scrapy

1 Answers1