How to follow a redirect with urllib?

Question

I'm creating a script in Python 3 which access a page like:

example.com/daora/zz.asp?x=qqrzzt

using the urllib.request.urlopen("example.com/daora/zz.asp?x=qqrzzt"), but this code just give me the same page(example.com/daora/zz.asp?x=qqrzzt) and on the browser i get a redirect to a page like:

example.com/egg.aspx

What could i do to retrieve the

example.com/egg.aspx

and not the

example.com/daora/zz.asp?x=qqrzzt

I think this is relevant code, this is the code from "example.com/daora/zz.asp?x=qqrzzt":

<head>

<script language="JavaScript">

<!--
    function Submit()

    {
        document.formzz.submit();
    }
-->
</script>

</head>

<body bgcolor="#FFFFFF" leftmargin="0" topmargin="0" marginwidth="0" marginheight="0" onLoad="javascript:Submit();">

<form name="formZZ" method="post" action="http://example.com/egg.aspx">

<input type="hidden" name="token" value="UFASGFJKASGDJFGAJS">

</form>

First, why is this tagged both `urllib2` and `python-3.x`? There is no `urllib2` in 3.x, so… which one of those two is wrong? — abarnert, Apr 23 '13 at 00:28
I can't help but notice the `method="post"` in the form - if urllib is to be used the standard compliant method involving a HTTP 307 redirect is to NOT automatically redirect, and thus will involve an exception - see [thread](https://stackoverflow.com/questions/62384020/python-3-7-urllib-request-doesnt-follow-redirect-url) for details. — metatoaster, Jan 15 '23 at 03:36

score 4 · Accepted Answer · answered Apr 23 '13 at 00:49

urllib.request follows redirects automatically; you don't need to do anything.

The problem here is that there is no redirect to follow. The web page uses Javascript to fake a form submission as soon as it's loaded. urllib just fetches the page; it doesn't implement a browser DOM and run Javascript code.

Depending on how general you need your script to be, the simplest solution may be something hacky. For example, if you're just trying to spider 500 pages that all have a similar structure but different details, just find the action of the first form and navigate to that.

Also, if fetching the pages and processing them are two distinct steps, you may want to write a fetcher with super-simple Javascript/Greasemonkey (running in the browser, so it's already got a working DOM implementation, etc.) and a separate fancy processing script in Python (which just operates on the finally-fetched/generated HTML pages).

If you need to be fully general, the simplest solution is probably to use the selenium browser automation framework. (Or, maybe, PyWin32 or PyObjC to automate IE or Webkit directly.)

If you want the best possible solution, and have infinite resources… write your own implementation of the DOM and hook up your favorite Javascript interpreter (probably spidermonkey or v8). That's only about 2/3rds as much work as writing a new browser. (And you may be able to find pieces that get you 80% of the way there. For example, if you're willing to use Jython instead of CPython as your Python interpreter, HtmlUnit is pretty slick.)

How to follow a redirect with urllib?

1 Answers1

Linked