6

I would like to pass multiple user-defined arguments to my scrapy spyder, so I tried to follow this post: How to pass a user defined argument in scrapy spider

However, when I follow the advice there I get an error:

root@ scrapy crawl dmoz -a address= 40-18 48th st -a borough=4
Usage
=====
  scrapy crawl [options] <spider>

crawl: error: running 'scrapy crawl' with more than one spider is no longer supported

I also tried with various permutations of quotation marks:

root@ scrapy crawl dmoz -a address= "40-18 48th st" -a borough="4"
Usage
=====
  scrapy crawl [options] <spider>
crawl: error: running 'scrapy crawl' with more than one spider is no longer supported

What is the correct way to pass parameters to the Scrapy spider? I would like to pass a username and password for the spider's login/scraping process. Thanks for any suggestions.

Community
  • 1
  • 1
sunny
  • 3,853
  • 5
  • 32
  • 62

1 Answers1

13

No scrapy problem, I guess. It's how your shell interprets input, spliting tokens in spaces. So, you must not have any of them between the key and its value. Try with:

scrapy crawl dmoz -a address="40-18 48th st" -a borough="4"
Birei
  • 35,723
  • 2
  • 77
  • 82
  • I did try that option as well. I am using Ubuntu in case that matters, but I have tried every permutation of quotation marks on the input and names of the inputs. Unless I am missing something, what you suggested is exactly what I have at the end of my question as an example of what I have already tried? – sunny Jun 23 '15 at 13:08
  • @sunny: Sure you tried my solution and didn't work either? It's different. The `-a` switch accepts a `key=value` argument, but the shell process the input before `scrapy`, and the space after the `=` sign makes it to generate two different tokens, that `scrapy` interprets as another spider. – Birei Jun 23 '15 at 13:13
  • I just copied and pasted your line, and it gave me the exact same error as what I've pasted above. – sunny Jun 23 '15 at 13:22
  • @sunny: Then, sorry. I had tested yours with same error, and mine with success, so I cannot help much more because cannot reproduce your issue. – Birei Jun 23 '15 at 13:39
  • thanks a lot. I really appreciate your help. I'll try rebooting, but that doesn't generally matter on Linux. I think you're right it must be a shell issue, so I'll point my inquiries that way. – sunny Jun 23 '15 at 13:40
  • I'm sorry I was totally wrong. Once I restarted everything was fine. Maybe it was something to do with my putty connection or I don't know? Your solution is working now. – sunny Jun 23 '15 at 18:34