How to get cookie from scrapy response and set the cookie to the next request?

Question

I have disabled the Default Scrapy cookie option, so that i have to set it manually.

COOKIES_ENABLED = False
COOKIES_DEBUG = True

Now, i need to set cookie with the value which is received as the response of the same site. I can able to get the cookie as below,

cookie = response.headers.getlist('Set-Cookie')[0].split(";")[0].split("=")

now i am trying to set it to the form request by

FormRequest.from_response(response,
                formdata={"username": "asldkfs", "pass": "slskd"},
                cookies={cookie[0]:cookie[1]},
                meta = {'dont_redirect': True,'handle_httpstatus_list': [302]},
                callback=self.redirection)

def redirection(self,response): 
    self.log("redirection")
    self.log(response.headers)               
    self.log("Cookie2")
    cook1 = response.headers.getlist('Set-Cookie')[0].split(";")[0].split("=")
    self.log(cook1)        
    self.log("end cookie2")
    return Request("http://something.net/some/sa/"+response.headers.getlist('Location')[0],cookies={cook1[0]:cook1[1]},
        callback=self.check_login_response)

.
.
.

So I could not set the cookie.Do i need to set any other value also or what could be the problem?

Paulo Romeira · Answer 1 · 2017-07-29T08:41:48.123

15

The cookies argument only works if you have COOKIES_ENABLED set to True, since CookiesMiddleware handles it.

Hence you have to set it manually on the headers:

cookie = response.headers.getlist('Set-Cookie')[0].split(';')[0]

FormRequest.from_response(response,
            formdata={"username": "asldkfs", "pass": "slskd"},
            headers={'Cookie': cookie}, # <---
            meta = {'dont_redirect': True,'handle_httpstatus_list': [302]},
            callback=self.redirection)

edited Jul 29 '17 at 08:41

answered Jul 28 '17 at 07:32

Paulo Romeira

751
7
11

2

hey I got error `cookie = response.headers.getlist('Set-Cookie')[0].split(';')[0] TypeError: a bytes-like object is required, not 'str'`\ – Efrat Levitan Dec 23 '20 at 11:49

score 7 · Answer 2 · edited Jan 01 '21 at 08:54

7

The answer from Paulo Romeira is correct, only the parsing from byte to string is missing:

cookie = response.headers.getlist('Set-Cookie')[0].decode("utf-8").split(";")[0].split("=")

edited Jan 01 '21 at 08:54

Tomer Shetah

8,413
7
27
35

answered Jan 01 '21 at 08:28

Mirko

113
3
7

`AttributeError: 'str' object has no attribute 'decode'` – osacognitive Aug 25 '21 at 15:57

score 1 · Answer 3 · edited Jun 15 '17 at 15:47

1

I think you could not work with cookies if you disabled it.

edited Jun 15 '17 at 15:47

Community

1
1

answered Jun 06 '17 at 17:34

Verz1Lka

406
4
15

score 1 · Answer 4 · answered Jan 19 '22 at 07:56

You can also use the concepts of creating cookiejars and handling the cookies there (for maintaining sessions in scrapy) using the following strategies documented here in the official scrapy docs: https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#std:reqmeta-cookiejar

Also refer this stackoverflow issue: Scrapy - how to manage cookies/sessions

score 0 · Answer 5 · answered Jul 07 '23 at 18:06

If you want a different way other than response.headers.getlist('Set-Cookie'), you can do something like:

from scrapy.http.cookies import CookieJar


def response_cookies(response):
    """
    Get cookies from response
    
    @param response scrapy response object
    @return: dict
    """

    obj = CookieJar(policy=None)
    jar = obj.make_cookies(response, response.request)
    cookies = {}
    for cookie in jar:
        cookies[cookie.name] = cookie.value
    return cookies

How to get cookie from scrapy response and set the cookie to the next request?

5 Answers5