Expand short urls in python using requests library

Question

I have a large number of short URLs and I want to expand them. I found somewhere online (I missed the source) the following code:

short_url = "t.co/NHBbLlfCaa"
r = requests.get(short_url)
if r.status_code == 200:
    print("Actual url:%s" % r.url)

It works perfectly. But I get this error when I ping the same server for many times:

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='www.fatlossadvice.pw', port=80): Max retries exceeded with url: /TIPS/KILLED-THAT-TREADMILL-WORKOUT-WORD-TO-TIMMY-GACQUIN.ASP (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 11004] getaddrinfo failed',))

I tried many solutions like the set here: Max retries exceeded with URL in requests, but nothing worked.

I was thinking about another solution, which is to pass an useragent in the request, and each time I change it randomly (using a large number of useragents):

user_agent_list = [
        'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0',
        'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0',
        'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36',
    ]

r = requests.get(short_url, headers={'User-Agent': user_agent_list[np.random.randint(0, len(user_agent_list))]})
if r.status_code == 200:
    print("Actual url:%s" % r.url)

My problem is that r.url always return the short url instead of the long one (the expanded one).

What am I missing?

What do you mean "short url"? Are you using url shortener service, like e.g. tinyurl? — Andrej Kesely, Jun 28 '19 at 15:04
@AndrejKesely, I added an example. I wasn't able to add it with "https://" , stackoverflow terms of use prevents me. — Minions, Jun 28 '19 at 15:09

Andrej Kesely · Accepted Answer · 2019-06-28T15:58:07.643

1

You can prevent the error by adding allow_redirects=False to requests.get() method to prevent redirecting to page that doesn't exist (and thus raising the error). You have to examine the header sent by server yourself (replace XXXX by https, remove spaces):

import requests

short_url = ["XXXX t.co /namDL4YHYu",
 'XXXX t.co /MjvmV',
 'XXXX t.co /JSjtxfaxRJ',
 'XXXX t.co /xxGSANSE8K',
 'XXXX t.co /ZRhf5gWNQg']

for url in short_url:
    r = requests.get(url, allow_redirects=False)
    try:
        print(url, r.headers['location'])
    except KeyError:
        print(url, "Page doesn't exist!")

Prints:

XXXX t.co/namDL4YHYu http://gottimechillinaround.tumblr.com/post/133931725110/tip-672
XXXX t.co/MjvmV Page doesn't exist!
XXXX t.co/JSjtxfaxRJ http://www.youtube.com/watch?v=rE693eNyyss
XXXX t.co/xxGSANSE8K http://www.losefattips.pw/Tips/My-stretch-before-and-after-my-workout-is-just-as-important-to-me-as-my-workout.asp
XXXX .co/ZRhf5gWNQg http://www.youtube.com/watch?v=3OK1P9GzDPM

edited Jun 28 '19 at 15:58

answered Jun 28 '19 at 15:15

Andrej Kesely

168,389
15
48
91

It doesn't work for many URL's, like: XXXX:// t.co /namDL4YHYu ! – Minions Jun 28 '19 at 15:23
@Ghanem It works for me, it prints `http://gottimechillinaround.tumblr.com/post/133931725110/tip-672` – Andrej Kesely Jun 28 '19 at 15:27
Sorry, maybe I copied incorrect one. Try these: XXXXs:// t.co/MjvmV, XXXX:// t.co/JSjtxfaxRJ, XXXXs:// t.co/xxGSANSE8K, XXXXs:// t.co/ZRhf5gWNQg – Minions Jun 28 '19 at 15:48
@Ghanem The `t.co/MjvmV` dosn't contain key `location` because the page doesn't exist. When I put it into the firefox, it's error page. – Andrej Kesely Jun 28 '19 at 15:58

Expand short urls in python using requests library

1 Answers1