1

In my scrapy I just want the html response inside a variable from custom url.

Suppose I have the url

url = "http://www.example.com"

Now I want to get the html of that page for parsing

pageHtml = scrapy.get(url)

I want something like this

page = urllib2.urlopen('http://yahoo.com').read()

The only problem that I can't use above line in my crawler is because my session is already authenticated by scrapy so I can't use any other function for getting the html of that function

I don't want response in any callback but simply straight inside the variable

satoru
  • 31,822
  • 31
  • 91
  • 141
user19140477031
  • 363
  • 1
  • 4
  • 13
  • Hi! Please show us your spider code so far and the section where you'd like to access the HTML of the page and I'm sure someone will be happy to help. – Talvalin Dec 19 '12 at 01:04
  • I don't have the full crawler code yet but i want what this guy says but i didn't understood his answer http://stackoverflow.com/questions/12879216/use-scrapy-parse-function-to-parse-a-specific-url?rq=1 – user19140477031 Dec 19 '12 at 01:21

1 Answers1

1

Basically, you just need to add the relevant imports for the code in that question to work. You'll also need to add a link variable which is used but not defined in that example code.

import httplib
from scrapy.spider import BaseSpider
from scrapy.http import TextResponse

bs = BaseSpider('some')
# etc
Talvalin
  • 7,789
  • 2
  • 30
  • 40
  • If this doesn't help answer the question posted, then please let me know how I can improve my answer to help better. :) – Talvalin Dec 19 '12 at 21:28
  • sorry for delayed response. i want to know that if i use that TextResponse will by session will still use same authtication as used with request and will send cookies. or this url will be separate and if that page is behind the login system then i will be redirected to the login page – user19140477031 Dec 20 '12 at 00:58
  • @Talvalin: re: the code from nik-v (in the question that the asker refers to) - would be much obliged if you could edit your example above with a working edit of nik-v's. I've been trying, but so far no luck. – scharfmn May 31 '13 at 16:59