How to make a scrapy request on a url that have more then one parameter?

Question

I'm trying to make a request using scrap shell first, just to test the Xpaths to this kind of url:

site.com/search/?action=search&description=MY SEARCH &e_author=

but the requests only goes to 'action=search' ... the other parameters is been requested ..

how do I build this requests?

Have you tried using quotes while opening it from scrapy shell ? `scrapy shell "site.com/search/?action=search&description=MY SEARCH &e_author="` — Rahul, Nov 22 '15 at 04:36

score 1 · Answer 1 · edited May 23 '17 at 10:27

1

You have to normalize the URL before making a request, see:

I would go with urllib.quote():

> import urllib

> quoted_url = urllib.quote(url)
> fetch(quoted_url)

edited May 23 '17 at 10:27

Community

answered Nov 22 '15 at 01:35

alecxe

1 Answers1