8

Creating a scraper for fantasy team. Looking for a way to pass a list of the players names as arguments, and then for each player_name in player_list run the parsing code.

I currently have something like this

class statsspider(BaseSpider):
name = 'statsspider'

def __init__ (self, domain=None, player_list=""):
    self.allowed_domains = ['sports.yahoo.com']
    self.start_urls = [
        'http://sports.yahoo.com/nba/players',
    ]
    self.player_list= "%s" % player_list


def parse(self, response):
    example code
    yield request

I'm assuming entering a list of arguments is the same as just one argument through the command line so I enter something like this: 

scrapy crawl statsspider -a player_list=['xyz','abc']

Problem 2!

Solved the first issue by inputting a comma delimited list of arguments like so

scrapy crawl statsspider -a player_list="abc def,ghi jkl"

I now want to go through each "name" (i.e. 'abc def') to find the first initial of their last name (in this case 'd').

I use the code

array = []
for player_name in self.player_list:
    array.append(player_name)
print array

And I end up with the result [["'",'a','b','c',... etc]] Why does python not assign player_name to each 'name' (e.g. 'abc def' and 'ghi jkl')? can someone explain this logic to me, and I will probably understand the right way to do it afterwards!

Python Learner
  • 185
  • 1
  • 12

1 Answers1

16

Shell arguments are string-based. You need to parse arg in your code.

command line:

scrapy crawl statsspider -a player_list=xyz,abc

python code:

self.player_list = player_list.split(',')
kev
  • 155,172
  • 47
  • 273
  • 272
  • Hey @kev thanks for the reply! I now know how to pass lists into the command line. It works, but now I have a new problem regarding me not understand python's logic. If you could see the appended "Problem 2" that would be greatly appreciated! – Python Learner Dec 11 '13 at 00:55