0

I have error scrapy.exceptions.IgnoreRequest when I want to get data from this link link. I 'm a newbie in Scrapy, I can get data from the other links but with this link, I cannot get data. I don't know why, I search and find out the error in Exceptions but how to fix thisexception:

2017-02-16 23:05:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://edirectory.matrade.gov.my/robots.txt> (referer: None)
2017-02-16 23:05:39 [scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET http://edirectory.matrade.gov.my/application/edirectory.nsf/category?

    OpenForm&query=product&code=PT&sid=BED1E22D5BE3F9B5394D6AF0E742828F>
    2017-02-16 23:05:39 [scrapy.core.engine] INFO: Closing spider (finished)
    2017-02-16 23:05:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {'downloader/exception_count': 1,
     'downloader/exception_type_count/scrapy.exceptions.IgnoreRequest': 1,
     'downloader/request_bytes': 233,
     'downloader/request_count': 1,
     'downloader/request_method_count/GET': 1,
     'downloader/response_bytes': 215,
     'downloader/response_count': 1,
     'downloader/response_status_count/200': 1,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2017, 2, 16, 16, 5, 39, 373000),
     'log_count/DEBUG': 3,
     'log_count/INFO': 7,
     'response_received_count': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2017, 2, 16, 16, 5, 33, 557000)}
    2017-02-16 23:05:39 [scrapy.core.engine] INFO: Spider closed (finished)

Thanks in advance! (My girl friend needs my help, she thinks I can do everything with all links :()

RoShan Shan
  • 2,924
  • 1
  • 18
  • 38
  • 2
    Possible duplicate of [getting Forbidden by robots.txt: scrapy](http://stackoverflow.com/questions/37274835/getting-forbidden-by-robots-txt-scrapy) – Cargo23 Feb 16 '17 at 16:36
  • @e Thanks a ton. I don't know why but I fixed by change to `ROBOTSTXT_OBEY=False`. Can you explain more. Thanks again. – RoShan Shan Feb 16 '17 at 16:44
  • http://www.robotstxt.org/robotstxt.html – Cargo23 Feb 16 '17 at 20:41

0 Answers0