I have error scrapy.exceptions.IgnoreRequest
when I want to get data from this link link. I 'm a newbie in Scrapy, I can get data from the other links but with this link, I cannot get data. I don't know why, I search and find out the error in Exceptions but how to fix thisexception:
2017-02-16 23:05:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://edirectory.matrade.gov.my/robots.txt> (referer: None)
2017-02-16 23:05:39 [scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET http://edirectory.matrade.gov.my/application/edirectory.nsf/category?
OpenForm&query=product&code=PT&sid=BED1E22D5BE3F9B5394D6AF0E742828F>
2017-02-16 23:05:39 [scrapy.core.engine] INFO: Closing spider (finished)
2017-02-16 23:05:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 1,
'downloader/exception_type_count/scrapy.exceptions.IgnoreRequest': 1,
'downloader/request_bytes': 233,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 215,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 2, 16, 16, 5, 39, 373000),
'log_count/DEBUG': 3,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2017, 2, 16, 16, 5, 33, 557000)}
2017-02-16 23:05:39 [scrapy.core.engine] INFO: Spider closed (finished)
Thanks in advance! (My girl friend needs my help, she thinks I can do everything with all links :()