Scrapy not yielding requests

Question

So my spider seems to not crawl any links beyond the start_page, i have this code

 for link in response.css('#ContentPlaceHolder1_divNewsContent a::attr(href)').extract():
        yield scrapy.Request(response.urljoin(link), callback=self.parse)

EDIT : I deleted the bit about the unicode, turns out it has nothing to do with unicode, the links are definitely getting fetched (i checked through adding a print statement before yielding a request) and they can be requested just fine, but within the spider it seems like scrapy just isn't requesting the links at all.

Here's an example output, as you can see lots of urls in the log but nothing is being crawled beyond the start links i provided.

2016-11-17 21:39:18 [scrapy] INFO: Scrapy 1.2.1 started (bot: newspapersScraping)
2016-11-17 21:39:18 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'newspapersScraping.spiders', 'SPIDER_MODULES': ['newspapersScraping.spiders'], 'DEPTH_LIMIT': 1, 'BOT_NAME': 'newspapersScraping'}
2016-11-17 21:39:18 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2016-11-17 21:39:18 [py.warnings] WARNING: /usr/local/lib/python2.7/dist-packages/scrapy/utils/deprecate.py:156: ScrapyDeprecationWarning: `scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware` class is deprecated, use `scrapy.downloadermiddlewares.useragent.UserAgentMiddleware` instead
  ScrapyDeprecationWarning)

2016-11-17 21:39:18 [py.warnings] WARNING: /home/starstuff/Development/newspapersScraping/newspapersScraping/comm/rotate_useragent.py:5: ScrapyDeprecationWarning: Module `scrapy.contrib.downloadermiddleware.useragent` is deprecated, use `scrapy.downloadermiddlewares.useragent` instead
  from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware

    2016-11-17 21:39:18 [scrapy] INFO: Enabled downloader middlewares:
    ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
     'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
     'newspapersScraping.comm.rotate_useragent.RotateUserAgentMiddleware',
     'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
     'scrapy.downloadermiddlewares.retry.RetryMiddleware',
     'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
     'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
     'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
     'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
     'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
     'scrapy.downloadermiddlewares.stats.DownloaderStats']
    2016-11-17 21:39:18 [scrapy] INFO: Enabled spider middlewares:
    ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
     'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
     'scrapy.spidermiddlewares.referer.RefererMiddleware',
     'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
     'scrapy.spidermiddlewares.depth.DepthMiddleware']
    2016-11-17 21:39:18 [py.warnings] WARNING: /home/starstuff/Development/newspapersScraping/newspapersScraping/pipelines.py:3: ScrapyDeprecationWarning: Module `scrapy.log` has been deprecated, Scrapy now relies on the builtin Python library for logging. Read the updated logging entry in the documentation to learn more.
      from scrapy import log

    2016-11-17 21:39:18 [scrapy] INFO: Enabled item pipelines:
    ['newspapersScraping.pipelines.NewsAPIpipeline']
    2016-11-17 21:39:18 [scrapy] INFO: Spider opened
    2016-11-17 21:39:18 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2016-11-17 21:39:18 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
    2016-11-17 21:39:20 [scrapy] DEBUG: Crawled (200) <GET http://www.ahram.org.eg/Category/202085/145/%D9%85%D8%AC%D9%84%D8%B3-%D8%A7%D9%84%D9%86%D9%88%D8%A7%D8%A8.aspx> (referer: None)
    2016-11-17 21:39:20 [scrapy] DEBUG: Crawled (200) <GET http://www.ahram.org.eg/Category/202085/3/%D8%AA%D8%AD%D9%82%D9%8A%D9%82%D8%A7%D8%AA.aspx> (referer: None)
    [My log] http://www.ahram.org.eg/News/202085/145/561506/مجلس-النواب/النواب-يقر-مواد-«الجمعيات-الأهلية»-ويرسله-إلى-مجلس.aspx
    2016-11-17 21:39:20 [scrapy] DEBUG: Filtered offsite request to 'www.ahram.org.eg': <GET http://www.ahram.org.eg/News/202085/145/561506/%D9%85%D8%AC%D9%84%D8%B3-%D8%A7%D9%84%D9%86%D9%88%D8%A7%D8%A8/%D8%A7%D9%84%D9%86%D9%88%D8%A7%D8%A8-%D9%8A%D9%82%D8%B1-%D9%85%D9%88%D8%A7%D8%AF-%C2%AB%D8%A7%D9%84%D8%AC%D9%85%D8%B9%D9%8A%D8%A7%D8%AA-%D8%A7%D9%84%D8%A3%D9%87%D9%84%D9%8A%D8%A9%C2%BB-%D9%88%D9%8A%D8%B1%D8%B3%D9%84%D9%87-%D8%A5%D9%84%D9%89-%D9%85%D8%AC%D9%84%D8%B3.aspx>
    [My log] http://www.ahram.org.eg/News/202085/145/561508/مجلس-النواب/العجاتى-قوانين-الصحافة-والإعلام-أمام-البرلمان-خلال.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561510/مجلس-النواب/يحيى-راشد-السياحة-العربية-مستقبل-مصر-وخطة-لتنمية-ا.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561517/مجلس-النواب/سحر-نصر--مليار-دولار-للمشروعات-الصغيرة-ونستهدف-الص.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561515/مجلس-النواب/غياب-الوزير-أو-من-ينوب-عنه-يثير-الغضب-فى-لجنة-الصح.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561513/مجلس-النواب/جلسة-استماع-لممثلى-المجتمع-المدنى-حول-قانون-الجمعي.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561512/مجلس-النواب/قرطام--نص-الاستقالة-المتداول-حاليا-قديم.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561520/مجلس-النواب/قبل-تقديمها-للرئاسةشباب-الأحزاب-يضعون-اللمسات-الأخ.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561522/مجلس-النواب/سؤال-لوزير-الثقافة-حول-صندوق-تمويل-مشروعات-دار-الأ.aspx
    [My log] http://www.ahram.org.eg/News/202085/145/561507/مجلس-النواب/بعد-الاعتذار-لعلاء-عبدالمنعم-على-اتهامه-بتزوير-الت.aspx
    [My log] http://www.ahram.org.eg/News/202085/3/561573/تحقيقات/الاختنـاقـات-المـــرورية--حـــالة-مـزمنـــة.aspx
    [My log] http://www.ahram.org.eg/News/202085/3/561571/تحقيقات/البحث-عــن-كيس-دم--رحلــة-عــذاب.aspx
    2016-11-17 21:39:20 [scrapy] DEBUG: Crawled (200) <GET http://www.ahram.org.eg/Category/202085/16/%D8%A7%D9%84%D8%A7%D8%AE%D9%8A%D8%B1%D8%A9.aspx> (referer: None)
    2016-11-17 21:39:20 [scrapy] DEBUG: Crawled (200) <GET http://www.ahram.org.eg/Category/202085/29/%D8%A7%D9%84%D9%85%D8%AD%D8%A7%D9%81%D8%B8%D8%A7%D8%AA.aspx> (referer: None)
    [My log] http://www.ahram.org.eg/News/202085/16/561559/الاخيرة/روح-«الساحر»-تضىء-«القاهرة-السينمائى»جائزة-فاتن-حم.aspx
    [My log] http://www.ahram.org.eg/News/202085/16/561561/الاخيرة/بدأ-اعدادها-عام-اليونسكو-يصدر-موسوعة-عن-الثقافة-ال.aspx
    [My log] http://www.ahram.org.eg/News/202085/16/561564/الاخيرة/«القمر-العملاق»يصفى-الذهن-ويجدد-طاقة-الجسم.aspx
    [My log] http://www.ahram.org.eg/News/202085/16/561566/الاخيرة/«كن-نفسك»-عرض-أزياء-لمصابى-متلازمة-داون.aspx
    [My log] http://www.ahram.org.eg/News/202085/16/561556/الاخيرة/أخبار-الصباح.aspx
    2016-11-17 21:39:20 [scrapy] DEBUG: Crawled (200) <GET http://www.ahram.org.eg/Category/202085/136/%D9%85%D8%AA%D8%A7%D8%A8%D8%B9%D8%A7%D8%AA.aspx> (referer: None)
    [My log] http://www.ahram.org.eg/News/202085/29/561539/المحافظات/خلافات-بين-وزير-التنمية-المحلية-ومحافظ-البحيرة-بسب.aspx
    [My log] http://www.ahram.org.eg/News/202085/29/561543/المحافظات/رغم-استقرار-حاله-الجو--«طوارئ-السيول»-مازالت-مستمر.aspx
    [My log] http://www.ahram.org.eg/News/202085/29/561545/المحافظات/السبت-«إجازة-»-بمدارس-الفترة-الواحدة-بقنا.aspx
    [My log] http://www.ahram.org.eg/News/202085/29/561537/المحافظات/احتفالية-باليوم-العالمى-للسكر-بالمنصورة.aspx
    [My log] http://www.ahram.org.eg/News/202085/136/561525/متابعات/السيسى-يبحث-سبل-النهوض-بأوضاع-مدينة-الإسكندرية--مش.aspx
    [My log] http://www.ahram.org.eg/News/202085/136/561527/متابعات/سرعة-تنمية-اكتشافات-الغاز-بشرق-المتوسطالرئيس-يوجه-.aspx
    [My log] http://www.ahram.org.eg/News/202085/136/561529/متابعات/الإمام-الأكبر-خلال-استقباله-وفدا-نيجيرياتعاليم-الإ.aspx
    [My log] http://www.ahram.org.eg/News/202085/136/561533/متابعات/القوات-المسلحة-تقوم-بعلاج-شباب-التجنيد-المصابين-بف.aspx
    [My log] http://www.ahram.org.eg/News/202085/136/561535/متابعات/بروتوكول-للتدريب-على-تشغيل-البيوت-والصوب-الزراعية-.aspx
    2016-11-17 21:39:21 [scrapy] DEBUG: Crawled (200) <GET http://www.ahram.org.eg/Category/202085/27/%D9%85%D8%B5%D8%B1.aspx> (referer: None)
    [My log] http://www.ahram.org.eg/News/202085/27/561514/مصر/وزير-الإسكان-الانتهاء-من--آلاف-وحدة-من-المشروع-الا.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561516/مصر/سعفان-لعمال-«الشوربجى»--تطوير-صناعة-الغزل-والنسيج-.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561518/مصر/وزير-الرى-تطوير-منظومة-الرصد-والمراقبة-للسد-العالى.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561519/مصر/خلال-جولة-تفقدية-للسيدة-والخليفةمحافظ-القاهرة-يشدد.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561521/مصر/-زيادة-فى-صادرات-الخضر-والفاكهة-من-ميناء-الإسكندري.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561523/مصر/بمناسبة-مرور--عاما-على-افتتاحهالمتحف-المصرى-يفتح-أ.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561524/مصر/انطلاق-الحوار-المجتمعى-لتجديد-الخطاب-الدينى.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561526/مصر/الهلالى--ضرورة-وضع-علم-مصر-فى-مكان-بارز-بالمدارس.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561528/مصر/«مايا»-تتفقد-تجارب-ناجحة-لمشروعات-المرأة-فى-المنيا.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561530/مصر/مناقشة-قوانين-الإعلام--فى-آداب-عين-شمس-غدا.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561531/مصر/القضاء-الإدارى-تأجيل-دعوى-إلزام-وزير-الآثار-بفسخ-ع.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561532/مصر/الرئيس-السيسى-يهنئ-فلسطين-بالعيد-الوطني.aspx
    [My log] http://www.ahram.org.eg/News/202085/27/561534/مصر/مندوب-الرئيس-للتعزية.aspx
    2016-11-17 21:39:21 [scrapy] INFO: Closing spider (finished)
    2016-11-17 21:39:21 [scrapy] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 2219,
     'downloader/request_count': 6,
     'downloader/request_method_count/GET': 6,
     'downloader/response_bytes': 495932,
     'downloader/response_count': 6,
     'downloader/response_status_count/200': 6,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2016, 11, 18, 5, 39, 21, 136828),
     'log_count/DEBUG': 8,
     'log_count/INFO': 7,
     'log_count/WARNING': 3,
     'offsite/domains': 1,
     'offsite/filtered': 39,
     'request_depth_max': 1,
     'response_received_count': 6,
     'scheduler/dequeued': 6,
     'scheduler/dequeued/memory': 6,
     'scheduler/enqueued': 6,
     'scheduler/enqueued/memory': 6,
     'start_time': datetime.datetime(2016, 11, 18, 5, 39, 18, 923369)}
    2016-11-17 21:39:21 [scrapy] INFO: Spider closed (finished)

Maybe http://stackoverflow.com/questions/4389572/how-to-fetch-a-non-ascii-url-with-python-urlopen will help? — Mark Ransom, Nov 16 '16 at 04:07
What version of Scrapy are you using? There have been a few major improvements on how scrapy handles URLs with non-ASCII characters. The sample URL you shared is fetched ok with scrapy 1.2 for me: http://pastebin.com/fmJzAqcC — paul trmbrth, Nov 16 '16 at 10:22

Eugene Lisitsky · Accepted Answer · 2016-11-21T11:47:32.247

2

Scrapy ignores urls because they are not in allowed_domain list. Try to write, for ex:

class ArticleSpider(scrapy.Spider):
    name = "ahram"
    allowed_domains = ['ahram.org', 'www.ahram.org.eg']
    ...

In this case scrapy crawl will follow required links.

edited Nov 21 '16 at 11:47

answered Nov 18 '16 at 16:00

Eugene Lisitsky

12,113
5
38
59

Had to change the imports to work with python 2 to `from urlparse import urlparse, urlunparse` and `from urllib import quote`, then i did as you said and i'm getting this error http://pastebin.com/DwhFXg7G – user2968505 Nov 19 '16 at 02:32
i also tried on another spider that has the same problem, as it links that are clearly extracted are not being crawled and it didn't fix the problem for it : ( – user2968505 Nov 19 '16 at 03:16
Could you put somewhere whole code? For ex on github? – Eugene Lisitsky Nov 19 '16 at 09:47
here's a link from my onedrive for a zip of the project note that the only spiders i'm interested in for now are (ahram) and (alanba) ignore the other ones, https://1drv.ms/u/s!AiVC5X3jJPL1a3SSaGo7YJJNUuo – user2968505 Nov 19 '16 at 14:28
I've looked. Scrapy ignores urls because they are not in allowed_domain list. Try to write: `allowed_domains = ['ahram.org', 'www.ahram.org.eg']`. In this case scrapy crawl all found links. – Eugene Lisitsky Nov 19 '16 at 19:07

Scrapy not yielding requests

1 Answers1