1

I am trying to run a scraper I found online but receive a ValueError: too many values to unpack on this line of code

 k, v = piece.split("=")

This line is part of this function 

def format_url(url):
# make sure URLs aren't relative, and strip unnecssary query args
u = urlparse(url)

scheme = u.scheme or "https"
host = u.netloc or "www.amazon.com"
path = u.path

if not u.query:
    query = ""
else:
    query = "?"
    for piece in u.query.split("&"):
        k, v = piece.split("=")
        if k in settings.allowed_params:
            query += "{k}={v}&".format(**locals())
    query = query[:-1]

return "{scheme}://{host}{path}{query}".format(**locals())

If you have any input it would be appreciated, thank you.

emmodz
  • 13
  • 4
  • 2
    The error is stating that there are more than 2 values after splitting so it cannot accommodate it in 2 variables. There are more than one = in your query – Rajan Chauhan Jul 13 '17 at 21:53
  • Just add a `print piece` before that call and it will print out the offending piece. Without knowing what url you're trying to parse, we can't offer specific help. – TemporalWolf Jul 13 '17 at 21:54
  • 1
    Include the query here in your question if you need help. – Rajan Chauhan Jul 13 '17 at 21:54

3 Answers3

2

Instead of parsing the urls yourself, you can use urlparse.parse_qs function:

>>> from urlparse import urlparse, parse_qs
>>> URL = 'https://someurl.com/with/query_string?i=main&mode=front&sid=12ab&enc=+Hello'
>>> parsed_url = urlparse(URL)
>>> parse_qs(parsed_url.query)
{'i': ['main'], 'enc': [' Hello '], 'mode': ['front'], 'sid': ['12ab']}

(source)

Dennis Golomazov
  • 16,269
  • 5
  • 73
  • 81
1

This is due to the fact that one of the pieces contains two or more '=' characters. In that case you thus return a list of three or more elements. And you cannot assign it to the two values.

You can solve that problem, by splitting at most one '=' by adding an additional parameter to the .split(..) call:

k, v = piece.split("=",1)

But now we still do not have guarantees that there is an '=' in the piece string anyway.

We can however use the urllib.parse module in (urlparse in ):

from urllib.parse import urlparse, parse_qsl

purl = urlparse(url)
quer = parse_qsl(purl.query)

for k,v in quer:
    # ...
    pass

Now we have decoded the query string as a list of key-value tuples we can process separately. I would advice to build up a URL with the urllib as well.

Willem Van Onsem
  • 443,496
  • 30
  • 428
  • 555
0

You haven't shown any basic debugging: what is piece at the problem point? If it has more than a single = in the string, the split operation will return more than 2 values -- hence your error message.

If you want to split on only the first =, then use index to get the location, and grab the slices you need:

pos = piece.index('=')
k = piece[:pos]
v = piece[pos+1:]
Prune
  • 76,765
  • 14
  • 60
  • 81