Scrapy cookie accept form

Question

I've considered and tried various options, namely: - FormRequests - Passing cookies

Sadly i keep getting stuck on: https://www.marktplaats.nl/cookiewall/?target=https%3A%2F%2Fwww.marktplaats.nl%2F

class MarktplaatsSpider(CrawlSpider):
    name = 'MarktplaatsSpidertest'
    source = 'Markplaats.nl'
    allowed_domains = ['marktplaats.nl']
    start_urls = ['https://www.marktplaats.nl/']

    rules = [Rule(LinkExtractor(allow=()), callback='parse_item',follow=True)]

    def start_request(self):
        form_data = {'CookieOptIn':'true'}
        request_body = json.dumps(form_data)
        yield scrapy.Request('https://www.marktplaats.nl',
                            method="POST",
                            body=request_body,
                            headers={'Content-Type': 'application/json; charset=UTF-8'}, )

   def parse_item(self, response):
       print(response.url)
       item['URL'] = response.url

       yield item(source=self.source, URL=item['URL'], hash = get_hash(response.url))

There are several other websites where i come accross the same problem. I simply do not know how my spider can get to the page.

Can anyone help me/point me in the right direction?

Regards,

Durk

score 1 · Accepted Answer · answered Apr 09 '17 at 13:12

Try this.

cookies = {
    'CookieOptIn': 'true',
    'luckynumber': '1896761001',
    'MpSession': '9ff31f05-36fd-4570-9cdc-e1800bf682fe',
}

headers = {
    'Pragma': 'no-cache',
    'DNT': '1',
    'Accept-Encoding': 'gzip, deflate, sdch, br',
    'Accept-Language': 'en-US,en;q=0.8',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Cache-Control': 'no-cache',
    'Referer': 'https://www.marktplaats.nl/cookiewall/?target=https%3A%2F%2Fwww.marktplaats.nl%2F',
    'Connection': 'keep-alive',
}

yield Request('https://www.marktplaats.nl/', headers=headers, cookies=cookies)

Yes, get cookie manually is the solution!He can also try [selenium](http://stackoverflow.com/questions/13287490/is-there-a-way-to-use-phantomjs-in-python)! — nick, Apr 10 '17 at 03:40

Scrapy cookie accept form

1 Answers1