Using Python request.get param but it gives me wrong url

Question

I'm trying to use Python request library to scrap some data from Steam. But first I need to modify my URL

for example, if I want to access games with tags

2D [id : 3871]
1980s [id : 7743]

https://store.steampowered.com/search?tags=7743%2C3871

this is the link I need. But when I do this

steam_url = "https://store.steampowered.com/search?"
search = request.get(steam_url, params = {'tags' : [7743, 3871]})

I get this URL

https://store.steampowered.com/search?tags=7743&tags=3871

which is showing me only 2D games [id : 3871]

To solve this I tried to do this

steam_url = "https://store.steampowered.com/search?"
search = get(steam_url, params = {'tags' : '%2C'.join(list(map(str,[7743, 3871])))})

Then I get this URL

https://store.steampowered.com/search?tags=7743%252C3871

I couldn't understand why there is %252C between those ids.

What should I do?

Possible duplicate of [What is %2C in a URL?](https://stackoverflow.com/questions/6182356/what-is-2c-in-a-url) — Johan, Sep 30 '18 at 14:55

score 2 · Answer 1 · answered Sep 30 '18 at 14:30

2

I couldn't understand why there is %252C between those ids.

Because you are using '%2C'.join which gets escaped as %252C with %25 being the percent sign. Use ','.join and let requests.get do the escaping.

answered Sep 30 '18 at 14:30

stackeks

66
5

Johan · Accepted Answer · 2018-09-30T14:36:24.557

1

%2C is the URL Encoded hexadecimal value of ,. When you throw in %2C manually, the request package will try to make the actual value %2C safely URL-encoded, which will result into %252C (more specifically it will try to convert % into %25).

You're trying to url encode values, but so is the request package, which means that it's double encoded.

If you just write the , manually in the function it should work.

steam_url = "https://store.steampowered.com/search?"
search = get(steam_url, params = {'tags' : ','.join(list(map(str,[7743, 3871])))})

However, just as Hassan Voyeau says, you don't need to make the function so complicated, you can just write the values normally by {'tags': '7743,3871'}, as long as you don't try to URL-encode any values manually.

edited Sep 30 '18 at 14:36

answered Sep 30 '18 at 14:31

Johan

3,577
1
14
28

1

Exactly. I wondered why my code was downvoted. I actually tested it and know it works. – Hassan Voyeau Sep 30 '18 at 14:41
Same with my answer, I tried to explain why it happens and how to avoid confusion in the future. Yours solves the problem more elegantly because it's not as much unnecessary code. – Johan Sep 30 '18 at 14:46
1

If I'm gonna get downvoted without any explanation for correctly answering a question and how to solve it I'll rather just delete my answer alltogether instead of helping. – Johan Sep 30 '18 at 15:12
1

thanks to you both, those comments really enlightened me. this problem was a huge setback for me, so thanks a lot! – gokdumano Sep 30 '18 at 15:39
Thank you @ShayzeOn1. Please consider marking the answer that helped you the most, as solution for your question (assuming your problem got resolved). – Johan Sep 30 '18 at 16:01
1

If there is no explanatory comment, and you're confident that your answer is good then don't mind the negative votes. Some people may downvote just because they can, with no particular reason. +1 form me to cheer you up :) – t.m.adam Sep 30 '18 at 17:25
1

Thanks @t.m.adam, it was childish of me, was in a bad mood already. Thanks for the positivity and the +1 :) – Johan Sep 30 '18 at 20:23

score 1 · Answer 3 · answered Sep 30 '18 at 14:36

Try the below code. My testing of my code gives https://store.steampowered.com/search?tags=7743%2C3871

import requests

steam_url = "https://store.steampowered.com/search?"
search = requests.get(steam_url, params = {'tags' : '7743,3871'})

print(search.url)

Using Python request.get param but it gives me wrong url

3 Answers3