0
url_list = ['www.scrape.com/file1', 'www.scrape.com/file2', ''www.scrape.com/file3'] 

category_id_list = ['12345','abcde','ABCDE']

zip_list = ['10075','10017','10028']

I have three variables I use to create a URL to be requested. in the order: url_list+zip+categoryid

the url is then passed into a function which has the scrape code

I was using 3 for loops to iterate over these lists but that is highly redundant

for url_ in url_list:
   for category_id in category_id_list:
       for zip_ in zip_list:

           request_url = url_+category_zip_
           func(request_url)

This does the job, but is there a more optimal way to do it? Thank you!

Carcigenicate
  • 43,494
  • 9
  • 68
  • 117

3 Answers3

3

You may use itertools.product

import itertools

for url in (str.join("",url) for url in itertools.product(url_list,category_id_list,zip_list)):
    func(url)
abc
  • 11,579
  • 2
  • 26
  • 51
0

This might be a bit late, but this is the way I did it:

cats = ["a","b","c","d"]

zips = ["25320","53902","59607","53123"]

base = "https://example.com"

for i in range(4):
   url = "{}/{}/{}".format(base, cats[i], zips[i])
   print(url)

Output:

https://example.com/a/25320
https://example.com/b/53902
https://example.com/c/59607
https://example.com/d/53123
Aero Blue
  • 518
  • 2
  • 14
0

One way to avoid writing multiple for loops is using zip.It allows you to access ith element from each list at once. So you can do something like:

url_list = ['www.scrape.com/file1', 'www.scrape.com/file2', 'www.scrape.com/file3']
category_id_list = ['12345','abcde','ABCDE']
zip_list = ['10075','10017','10028']

for url, id, zip in zip(url_list, category_id_list, zip_list):
    request_url = url + id + zip
    func(request_url)
dans
  • 23
  • 5