Python Requests library redirect new url

Question

I've been looking through the Python Requests documentation but I cannot see any functionality for what I am trying to achieve.

In my script I am setting allow_redirects=True.

I would like to know if the page has been redirected to something else, what is the new URL.

For example, if the start URL was: www.google.com/redirect

And the final URL is www.google.co.uk/redirected

How do I get that URL?

Check out [this answer](https://stackoverflow.com/a/4902578/4607733) for dealing with `urllib2 ` — logi-kal, Nov 23 '19 at 14:12
check my solution using webbrowser here (https://stackoverflow.com/questions/62503861/to-get-redirected-url-with-requests/70869177#70869177) — Shahin Shirazi, Jan 26 '22 at 19:41

score 210 · Answer 1 · edited Jun 11 '20 at 19:34

You are looking for the request history.

The response.history attribute is a list of responses that led to the final URL, which can be found in response.url.

response = requests.get(someurl)
if response.history:
    print("Request was redirected")
    for resp in response.history:
        print(resp.status_code, resp.url)
    print("Final destination:")
    print(response.status_code, response.url)
else:
    print("Request was not redirected")

Demo:

>>> import requests
>>> response = requests.get('http://httpbin.org/redirect/3')
>>> response.history
(<Response [302]>, <Response [302]>, <Response [302]>)
>>> for resp in response.history:
...     print(resp.status_code, resp.url)
... 
302 http://httpbin.org/redirect/3
302 http://httpbin.org/redirect/2
302 http://httpbin.org/redirect/1
>>> print(response.status_code, response.url)
200 http://httpbin.org/get

httpbin.org is giving 404s for some reason, but httpbingo.org (same URL scheme) worked just fine for me. — Preston Badeer, Oct 26 '20 at 19:33
@PrestonBadeer: This is a known issue: https://github.com/postmanlabs/httpbin/issues/617. It's not crucial that the demo works for the answer, luckily. — Martijn Pieters, Oct 26 '20 at 21:29

score 96 · Answer 2 · edited Dec 20 '16 at 16:06

96

This is answering a slightly different question, but since I got stuck on this myself, I hope it might be useful for someone else.

If you want to use allow_redirects=False and get directly to the first redirect object, rather than following a chain of them, and you just want to get the redirect location directly out of the 302 response object, then r.url won't work. Instead, it's the "Location" header:

r = requests.get('http://github.com/', allow_redirects=False)
r.status_code  # 302
r.url  # http://github.com, not https.
r.headers['Location']  # https://github.com/ -- the redirect destination

edited Dec 20 '16 at 16:06

unkulunkulu

11,576
2
31
49

answered Sep 11 '15 at 17:05

hwjp

15,359
7
71
70

Thank you - this boosted my URL referral script (which had thousands of urls) by several seconds. – ahinkle Jan 12 '17 at 17:33
Do you know what is up with `r.next`? I thought that would contain a `PreparedRequest` pointing to the redirect URL, but that does not seem to be the case... – Elias Strehle Aug 17 '18 at 13:55
Worth adding that this answer will only give you the first redirect URL. if this url, when visited, would have normally redirect again to a new url, you will miss it. – Nioooooo Jun 14 '21 at 11:23

score 51 · Answer 3 · edited Jul 18 '21 at 22:03

51

I think requests.head instead of requests.get will be more safe to call when handling url redirect. Check a GitHub issue here:

r = requests.head(url, allow_redirects=True)
print(r.url)

edited Jul 18 '21 at 22:03

Asclepius

57,944
17
167
143

answered Jun 01 '15 at 02:32

Geng Jiawen

8,904
3
48
37

4

This should be the accepted answer. Short and sweet. – Volatil3 Apr 09 '17 at 00:27
8

@Volatil3: Not all servers respond to a HEAD request the same way the would with a GET. – Blender Apr 26 '17 at 05:47
For me this method worked in extracting the final redirect URL, saved considerable amount of manual effort for 30K URL – Ashish Tripathi Jan 27 '23 at 14:36

score 47 · Answer 4 · edited Mar 04 '20 at 12:38

47

the documentation has this blurb https://requests.readthedocs.io/en/master/user/quickstart/#redirection-and-history

import requests

r = requests.get('http://www.github.com')
r.url
#returns https://www.github.com instead of the http page you asked for

edited Mar 04 '20 at 12:38

FelixEnescu

4,664
2
33
34

answered Dec 09 '13 at 16:31

Back2Basics

7,406
2
32
45

score 13 · Answer 5 · answered Jan 16 '17 at 04:20

13

For python3.5, you can use the following code:

import urllib.request
res = urllib.request.urlopen(starturl)
finalurl = res.geturl()
print(finalurl)

answered Jan 16 '17 at 04:20

Shuai.Z

376
3
5

this is the correct answer for Python 3.5, it took me a while to find, thanks – jjj Oct 28 '19 at 01:25
Can you finish your answer please, If you found out how to do redirect with python3? Thanks. – Vladimir Despotovic Jun 06 '21 at 20:36

score 2 · Answer 6 · edited Jul 07 '23 at 14:31

I wrote the following function to get the full URL from a short URL (bit.ly, t.co, ...)

import requests

def expand_short_url(url):
    r = requests.head(url, allow_redirects=False)
    r.raise_for_status()
    if 300 < r.status_code < 400:
        url = r.headers.get('Location', url)

    return url

Usage (short URL is this question's url):

short_url = 'https://tinyurl.com/' + '4d4ytpbx'
full_url = expand_short_url(short_url)
print(full_url)

Output:

https://stackoverflow.com/questions/20475552/python-requests-library-redirect-new-url

score 0 · Answer 7 · answered Nov 05 '22 at 12:23

All the answers are applicable where the final url exists/working fine. In case, final URL doesn't seems to work then below is way to capture all redirects. There was scenario where final URL isn't working anymore and other ways like url history give error.
Code Snippet

long_url = ''
url = 'http://example.com/bla-bla'
try:
    while True:
        long_url = requests.head(url).headers['location']
        print(long_url)
        url = long_url
except:
    print(long_url)

score -1 · Answer 8 · answered Jan 26 '22 at 20:13

I wasn't able to use requests library and had to go different way. Here is the code that I post as solution to this post. (To get redirected URL with requests)

This way you actually open the browser, wait for your browser to log the url in the history log and then read last url in your history. I wrote this code for google chrom, but you should be able to follow along if you are using different browser.

import webbrowser
import sqlite3
import pandas as pd
import shutil

webbrowser.open("https://twitter.com/i/user/2274951674")
#source file is where the history of your webbroser is saved, I was using chrome, but it should be the same process if you are using different browser
source_file = 'C:\\Users\\{your_user_id}\\AppData\\Local\\Google\\Chrome\\User Data\\Default\\History'
# could not directly connect to history file as it was locked and had to make a copy of it in different location
destination_file = 'C:\\Users\\{user}\\Downloads\\History'
time.sleep(30) # there is some delay to update the history file, so 30 sec wait give it enough time to make sure your last url get logged
shutil.copy(source_file,destination_file) # copying the file.
con = sqlite3.connect('C:\\Users\\{user}\\Downloads\\History')#connecting to browser history
cursor = con.execute("SELECT * FROM urls")
names = [description[0] for description in cursor.description]
urls = cursor.fetchall()
con.close()
df_history = pd.DataFrame(urls,columns=names)
last_url = df_history.loc[len(df_history)-1,'url']
print(last_url)

>>https://twitter.com/ozanbayram01

Python Requests library redirect new url

8 Answers8

Linked

Related