3

I have a simple script that scrapes Google for a link and then scrapes that link. However, some links contain dashes in them, and for some reason it comes out like this %25E2%2580%2593 in my script (in the url). So it would now look like this: http://myaddress.com/search?q=The_%25E2%2580%2593_World when I want it to look like this http://myaddress.com/search?q=The_–_World. How can I go about doing this? Should I be using UTF-8 encoding/decoding?

Edit:
I tried double unquoting (with reference to this link) but to no avail. Instead I get a result that looks like this: http://myaddress.com/search?q=The_–_World.

Lachlan Mather
  • 273
  • 1
  • 5
  • 16

1 Answers1

3

The URL appears to be double URL encoded.

To decode to its raw form use the urllib libraries' parse.unquote function to perform double URL decoding:

import urllib.parse

url = 'http://myaddress.com/search?q=The_%25E2%2580%2593_World'
urllib.parse.unquote(urllib.parse.unquote(url))

which decodes to the desired 'http://myaddress.com/search?q=The_–_World' URL.

EDIT:

As you have explained that you are using Python 2.7, the equivalent decode function would be unquote(url) (refer to the documentation here).

import urllib

url = 'http://myaddress.com/search?q=The_%25E2%2580%2593_World'
print(urllib.unquote(urllib.unquote(url))).decode('utf-8')

Output:

http://myaddress.com/search?q=The_–_World

smoggers
  • 3,154
  • 6
  • 26
  • 38
  • I forgot to mention that I'm using Python 2.7; and I know that `urllib.parse` is a python 3 library. I tried using `urllib.url2pathname` instead, and this did show promising signs at first but instead gave me this `P:\\myaddress.com\search?q=The_–_World` (when I double decode it). I assume this has something to do with not decoding in UTF-8? – Lachlan Mather Jun 18 '18 at 03:48