Post request/Form submission with scrapy leads to Error 404

Question

I am learning how to build a spider using scrapy to scrape this webpage: https://www.beesmart.city. To have access, it is necessary to do a form submission here: https://www.beesmart.city/login

Inspecting the page, I didn't find a CSRF Token so I guess it is not needed in this case.

I used the following code:

import scrapy

class BeesmartLogin(scrapy.Spider):
    name = 'beesmart_login'
    login_url = 'https://www.beesmart.city/login'
    start_urls = [login_url]

    def parse(self, response):
        # extract the csrf token value
        # create a python dictionary with the form values
        data = {
            'email': '<...@...>',
            'password': '********',
        }
        # submit a POST request to it
        yield scrapy.FormRequest(url=self.login_url, formdata=data, callback=self.parse_quotes)

    def parse_quotes(self, response):
        print("Parsing is done")

When I run it I get a 404 however. Does anyone have an idea why? I am very thankful for any help. Tobi :)

The full response is:

(base) C:\Users\tobia\OneDrive\Documents\Python Scripts\Webscraping>scrapy runspider beesmart_login.py
2019-04-05 15:21:10 [scrapy.utils.log] INFO: Scrapy 1.5.2 started (bot: scrapybot)
2019-04-05 15:21:10 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.1b  26 Feb 2019), cryptography 2.6.1, Platform Windows-10-10.0.17134-SP0
2019-04-05 15:21:10 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2019-04-05 15:21:10 [scrapy.extensions.telnet] INFO: Telnet Password: ef43803864a4422d
2019-04-05 15:21:10 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2019-04-05 15:21:11 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-04-05 15:21:11 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-04-05 15:21:11 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-04-05 15:21:11 [scrapy.core.engine] INFO: Spider opened
2019-04-05 15:21:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-04-05 15:21:11 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2019-04-05 15:21:11 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.beesmart.city/login> (referer: None)
2019-04-05 15:21:11 [scrapy.core.engine] DEBUG: Crawled (404) <POST https://www.beesmart.city/login> (referer: https://www.beesmart.city/login)
2019-04-05 15:21:11 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.beesmart.city/login>: HTTP status code is not handled or not allowed
2019-04-05 15:21:11 [scrapy.core.engine] INFO: Closing spider (finished)
2019-04-05 15:21:11 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 657,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 1,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 10500,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/404': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2019, 4, 5, 13, 21, 11, 770670),
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/404': 1,
 'log_count/DEBUG': 3,
 'log_count/INFO': 9,
 'request_depth_max': 1,
 'response_received_count': 2,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2019, 4, 5, 13, 21, 11, 50573)}
2019-04-05 15:21:11 [scrapy.core.engine] INFO: Spider closed (finished)

Scrape doesn't use JavaScript but this page uses JavaScript for login. It can send login data to different url then you expect. And it may not work without JavaScript. Open web browser, turn off JavaScript and load your page and you will see if this page works without JavaScript. You can also use DevTools in Chrome/Firefox to see all requests when you login. Maybe you see dirfferent url used when you login. — furas, Apr 05 '19 at 14:08

score 1 · Accepted Answer · answered Apr 05 '19 at 21:11

1

If you are looking to understand CSFR Tokens - they have been explained on the following SO question

As furas explained it's not possible to use pure scrapy to login over javascript.

I recommend that you read the following article about Web scraping javascript using python

answered Apr 05 '19 at 21:11

Thierrydev

141
6

Great thank you! I will post the full solution once I'm done. – Tobi Apr 08 '19 at 11:11

Post request/Form submission with scrapy leads to Error 404

1 Answers1