How to Python requests to follow URL like my browser

Question

I've noticed that the requests library for Python does not follow certain URL redirects like my browser.

For example, when I do:

response = requests.get('http://www.bbmt.org/', verify=False, allow_redirects=True)

The final URL is https://secure.jbs.elsevierhealth.com/action/consumeSsoCookie?redirectUri=http%3A%2F%2Fwww.bbmt.org%2Faction%2FconsumeSharedSessionAction%3FMAID%3DJ3%252BqsjOKzWZhWAeF2bXl%252FA%253D%253D%26JSESSIONID%3DaaaorUqRLHgAe4WCenKv%26SERVER%3DWZ6myaEXBLEt1UgI9cIkvA%253D%253D%26ORIGIN%3D470200154%26RD%3DRD&acw=&utt=

However, in my browser, I am eventually redirected back to http://www.bbmt.org/.

Is there a way to have requests behave like my browser in these scenarios?

Answer 1

The redirect inside the https://secure.jbs.elsevierhealth.com is a javascript redirect.
You can see it inside the source-code:

window.location.href = "http://www.bbmt.org/action/consumeSharedSessionAction?SERVER=WZ6myaEXBLHj3ZzqSv9HPw%3D%3D&MAID=IBS8Eq6B1iRWhf2ywTW5pg%3D%3D&JSESSIONID=aaa8eY-zM394XcPptT_Kv&ORIGIN=670572791&RD=RD";

You will need to run javascript in order to do this redirect (and this is something requests do not do).

If you need a solution for this specific redirect, you can parse the content of the response (in python) and take that specific URL and use it to create a new request.

If you need a general solution - you will need to use a headless browser to do that. You can find more information in this question.