Downloading an older version of Java using Python requests

Question

I am trying to download an older version of java using python requests. Code below:

import json

import requests
from requests.auth import HTTPBasicAuth


def main():
    with open('auth.json', 'r') as jf:
        auth_data = json.load(jf)['jre']
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    r = requests.get('http://download.oracle.com/otn/java/jdk/10.0.1+10/fb4372174a714e6b8c52526dc134031e/jre-10.0.1_windows-x64_bin.exe',
                     auth=HTTPBasicAuth(**auth_data),
                     cookies={'oraclelicense': 'accept-securebackup-cookie'},
                     headers=headers,
                     allow_redirects=True)
    print(r.content)
    if r.status_code == 200:
        print('Saving file')
        with open('tmp.exe', 'wb') as f:
            f.write(r.content)
    else:
        raise Exception('{}: {}'.format(r.status_code, r.reason))


if __name__ == '__main__':
    main()

The username and password is correct since I have tested the same via browser. However when running above code, I get the below HTML in "r.content" object:

<html>\n<script language="javascript" type="text/javascript">\nfunction submitForm()\n{\nvar hash = location.hash;\nif (hash) {\nif(hash.indexOf("#") == -1){\nhash="#"+hash\n}\ndocument.myForm.action = document.myForm.action+hash;\n}\ndocument.myForm.submit();\n}\n</script><head><base target="_self"></head><body onLoad="submitForm()"><noscript><p>JavaScript is required. Enable JavaScript to use OAM Server.</p></noscript><form action="https://login.oracle.com/mysso/signon.jsp" method="post" name="myForm"><!------------ DO NOT REMOVE -------------><!----- loginform renderBrowserView -----><!-- Required for SmartView Integration --><input type="hidden" name="bmctx" value="9E1E7DBD1B88FA0EDC1F7E37E421616D1FDB9B73CA87F72A1DAF42D4AD693562"><input type="hidden" name="contextType" value="external"><input type="hidden" name="username" value="string"><input type="hidden" name="contextValue" value="%2Foam"><input type="hidden" name="challenge_url" value="https%3A%2F%2Flogin.oracle.com%2Fmysso%2Fsignon.jsp"><input type="hidden" name="password" value="sercure_string"><input type="hidden" name="request_id" value="5622122954907487022"><input type="hidden" name="authn_try_count" value="0"><input type="hidden" name="site2pstoretoken" value="v1.2~EC062A1473FF0629A4~C21B44F7E520492680446D4092E64F6B8844655283266266F6C39D875F74A979AF022A3ED4F814701BB31E0022AB1EDD77BA6A3BEEFC60BB487D9848741B16030781192396B846794AF3BFCE084EAB4A698F202537B8F222C8C9D3BFB3F9F11E37D96E520257E1244C551BFA0E82CE05423D6DEADFD3ECD401587B711580B438E4C4715DB6890E8CBEC5978F7054A158D33857A2F8350D909BE4D4F77FFF58EA9E8D7083E375E4BDC232BE9344BEFD7966FA06AED7D1659AD7E0923F91D9D83EACD67CE2E308B79366DB7704B7F9BDE4199A2C90381705622B08ADC0AEB11517EFC56092F7B432AD745DE4D95657D216CD6A1FA2C4504DA2BEC0212EA407F6ED3E572839540F5BF5C70CA53D6F8C6A6C3E930B9F233983FFF70830E9CFF6EBBA163D5328C5212D4C"><input type="hidden" name="v" value="v1.4"><input type="hidden" name="p_submit_url" value="https://login.oracle.com:443/sso/auth"><input type="hidden" name="p_cancel_url" value="https://login.oracle.com:443"><input type="hidden" name="p_error_code" value=""><input type="hidden" name="ssousername" value=""><input type="hidden" name="subscribername" value=""><input type="hidden" name="OAM_REQ" value="VERSION_4~TqtsVldhnamztgMQ8zN2zaIvP1KeT%2f6jpCTJRmnKMeAFWPmfDdceQk1TNAiFDmwPum0RX3z96hhtaoV9gq4Emv141fuyx%2bZZrPR%2bYo%2bTYwPfgCsgQYis06yKSGIDkX%2faUmEaZWk1A4jZrnc1NLooTDegrDH7IvReX1DtmQtMiVs%2fSJ%2b34HvMGTBd2mVs%2f8TSpvTU%2bph%2fiHuYD2mtThqvla7iMprkcZXDU2G3XkBGAh5k2pn0w39beaH0Lp6854GYm8xLIyvKTA63cethU9xHOlNw7kMpArbYFichq9xwUTgmZmDcHOHN%2fR%2fR9yHJAQ2%2b255%2bTiv3wOjqhK8HZoessj0KMhDr1It9%2fMxn2Bf%2b7bIf6lp8Gcj5%2ffjCfV3wjK8kpYV39atr6OxD9x%2f7tgEgZILoVGRXObSVX06ga2XR%2bj9JxUZiggEf3II94K1EMZ9W8l3IOhZACzn0n40dpY2tyZ6B3nOICjxVpCch%2fYReA95Fhgg00qGaJSy4fRuChPS%2fbc6OCJUGPKN8smYMYH8C9nzCr4K6OuXSK762x2KLMReFxejTf74mC0XxRnhW7K2AznxEXzCmpTYatlrUVSJILiAybCu953QrL19i7vHDb7AkFPbvXZUyjEw0xGSrbZTNs6rKCI9NOQfynu6IG8Ksot886DAIRYHNjj7oeibCORqJNCbaL546kN%2bZHK%2fP%2bD8pRSdTK6zNTOo%2b8CMYPhGKAnJBji4gyXBGCt5DxJA1JTac2%2fqA0XjFF0DS9kdJy0TMxasMKfQOKZGxwakQ%2btUYUPWUp%2bJ%2f%2f3xE03PZVSs%2b59cBlp8jYsxUq8nyzppIHhOqwMKO%2bfZqsjnGgX9VEWYsQryc5kqpD3EjMNsUSL4lGZFFsQlRmr2QXW7ov1lNSHmVGx4dkZ6jKq%2fa5TW4fq0bFbwMeneqyrPwtntJQcMbdF77YZjLbA%2fjMkVjONy%2fu9vBfD6sgl%2bZpTvSFL%2f5a3XoToe%2fAdcEgPPubpTmZWDyUMceIFc4mApF1qs9j20uMFbVGe60D89Vis%2f6SmzJeA4e4Cesovy0w9QXg5S1droQVBde3%2bfixYHnqSdOpgO6s5RqQm2dqmN%2fHl6nm74odL73zHnZD1CCHtdXIgbW0glOyMdi3%2bWxLU0Qp5XybFLfAvkVVm0%2b7WEE%2biXs1UewHHHkDg9QzcsoKNwIytQuWTxspUJxag8fv35kCqrXx0etEpBechvT1PDd32APQVmchHqsZBWe3uJoDnY3M0IJoA7wuoGI6pTkJRtbDNLrkMS%2fxZqH18qE2G4DAyuJ%2fFpx5ntV1Y3KJW812nGTia2ZsGG2k%2bmpcXsVP4GShauoTyAP2NMt%2fuNSJ5L%2bCHfuve4MtRBekNSjQ1NymGE8AewWjouCTT1qhwfWqsBUiIyahmHgIrLEi%2bJ5VP941qmWviUN6Iv7vMpTQrfRvhxRqZQRMa5gcjKNH6ShO8U9eLIoTrN6aIkjime%2fcWeZQKPG3hqKWwIq3jrZ78wAJB42CGlfm7GHtTGm2trevtHA1sGWu5s%2fYsPBjLWS3CoKWwa7Pb3Q%2bCSsLjE4yREB7hwiOrylh2aXajRg3cG8ujcpEj3T00rWM1q40fPTHel5kl1X9CQgDqqbkt3mbFHnUK%2bdVkXACINi7aHIpjWuphF%2bNn7gNSP9yRsJUBUijcpB%2bLi8x0wAFpOHO8ySao00S9xHyzC0Mi1Kb2Ux5m5ssCBStkMskcXNQpo1zb%2fKvxNHd%2f1dPxGVdw5eBfogRM6JP8B0DzzskDJv7MDj5JovAop1FN9d%2ftNg4de%2fmrCdTfUphJ1aNm2mB2mF1ep24inT0fY6OGpPxeRmWnMreT0JM7eWTqmdsF8zKbP0tx9C7EusQqwdgM1Z1%2fODAdBWsR3QJ06ineEI4z39lzj4C3SuxAroS6aDjlu98BWb1u8jFLqbih0PD37YGrGP7QsZahWvNCWS8AONjV8p%2f2G1vOlYR8Nb12ojRNrFFINelwp9tYpn8VX1BOMX57H6ONM1NBDuWk59PJLeqn7PreBOOy2gYZEU75IqJyoilOQ8slGD4rJaJI%2ftXINly3Tc0nVr0dMV1cgNI8XviaeKQh9RYn%2finqmgKH%2fewhb1kokq2D%2bmt%2biO%2baBlpJVU45qq099iVc9KItsDEm%2fyO7c0clIqMRCNJlmyW7aib8uiH8MvcNzPTxqHQr69Q55IIbR3HMYIfi1TEwCz%2fwfiHe2vJQNMkLvtfnspKcEUwi4x3k9J4qC1bOexsosllS8e2kxXEw%3d%3d"><input type="hidden" name="locale" value="en"><input type="hidden" name="resource_url" value="https%253A%252F%252Fedelivery.oracle.com%252Fakam%252Fotn%252Fjava%252Fjdk%252F10.0.1%252B10%252Ffb4372174a714e6b8c52526dc134031e%252Fjre-10.0.1_windows-x64_bin.exe"></form></body></html>

This question is actually not about Python and requests, but about web pages parsing. Try to save r.content in to .html file and open it in your webbrowser, maybe you'll find download link inside of it, and could parse html file for it. — MihanEntalpo, Oct 26 '18 at 06:44
I need a way to download the exe file and not get any HTML content like above. I'm ok with a solution using wget or cURL as well. — Prashanth S Sriram, Oct 26 '18 at 06:51
Did you google for the solution, that someone did already make? I've found this one https://gist.github.com/hgomez/4697585 — MihanEntalpo, Oct 26 '18 at 06:53
There are a conversation in comments of that gist, and there are people complaining about problems, and fixind it. Did you try something they recommend? — MihanEntalpo, Oct 26 '18 at 07:07
Also, 5th link in google is leads to https://stackoverflow.com/questions/10268583/downloading-java-jdk-on-linux-via-wget-is-shown-license-page-instead - where the answer with 1500+ likes located. — MihanEntalpo, Oct 26 '18 at 07:09
I have checked all these solutions. But ever since Oracle started adding authentication for such requests, none of these are working anymore. Note, I even tried changing "otn" to "otn-pub" in the URL, but that doesn't work for all old versions. I get a 404 in a lot of cases. — Prashanth S Sriram, Oct 26 '18 at 07:18
Ok, if oracle make the things so hard for automatic downloads, but not so hard for users, the decision may be: Use PhantomJS headless browser, and command it to act like a user on a oracle's page, to get the actual download link. — MihanEntalpo, Oct 26 '18 at 07:32

Downloading an older version of Java using Python requests

0 Answers0