1

I'm trying to dowload a file using Python 3's urllib, but I get some html garbage instead of the actual file. However, if I use the browser, I can download the file just fine. A minimum non-working example:

import urllib.request

url = 'https://contrataciondelestado.es/wps/wcm/connect/PLACE_es/Site/area/docAccCmpnt?srv=cmpnt&cmpntname=GetDocumentsById&source=library&DocumentIdParam=ecd194a4-82e1-4fd2-8135-616622234f9b'

urllib.request.urlretrieve(url,'blah.pdf')

I also tried the two answers in this thread (creating an user agent and using the requestsmodule)...but the same nothing.

Using requests

import requests

url = 'https://contrataciondelestado.es/wps/wcm/connect/PLACE_es/Site/area/docAccCmpnt?srv=cmpnt&cmpntname=GetDocumentsById&source=library&DocumentIdParam=ecd194a4-82e1-4fd2-8135-616622234f9b'
r = requests.get(url, allow_redirects=True)
with open('test.pdf', 'wb') as f:
    f.write(r.content)
print(r.is_redirect)

Same gibberish, and the requests module says the passed URL is not a redirect.

I also tried more "sophisticated" stuff like the download_file function suggested here......same old.

Any clue?

Cheers.

Community
  • 1
  • 1
manu
  • 1,333
  • 2
  • 11
  • 24

2 Answers2

0

The URL you used to perform download is not the final one, there is a redirection (from HTML source).

$ curl -I "https://contrataciondelestado.es/wps/wcm/connect/PLACE_es/Site/area/docAccCmpnt?srv=cmpnt&cmpntname=GetDocumentsById&source=library&DocumentIdParam=ecd194a4-82e1-4fd2-8135-616622234f9b"
HTTP/1.1 200 OK
Date: Tue, 11 Sep 2018 09:44:41 GMT
ETag: "-1462112711"
Content-Length: 435
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Cache-Control: no-cache="set-cookie, set-cookie2"
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Content-Language: en-US
Location: https://contrataciondelestado.es
Set-Cookie: JSESSIONID=0000t4dCqcAT246C2R0a6jwMpsq:prodnod5; Path=/; Domain=.contrataciondelestado.es; Secure
Set-Cookie:  JSESSIONID=0000t4dCqcAT246C2R0a6jwMpsq:prodnod5; Path=/; Domain=.contrataciondelestado.es; Secure
Connection: close


<link
    href='/wps/CacheProxyServlet/colorPalette/default/browserVendor/unknown/browserName/Default+HTML+Client/browserVersion/unknown/locale/es/forwardurl/TemaPlace/themes/html/TemaPlace/./styles.jsp'
    rel="styleSheet" type="text/css">

<meta http-equiv="refresh" content="0;url='/wps/wcm/connect/bb876769-1b16-4f8b-84fc-b85d5f864e07/DOC20120619092407Pliego+campamentos+verano.pdf?MOD=AJPERES'">

You have to extract the right URL from the meta http-equiv="refresh" tag:

https://contrataciondelestado.es/wps/wcm/connect/bb876769-1b16-4f8b-84fc-b85d5f864e07/DOC20120619092407Pliego+campamentos+verano.pdf?MOD=AJPERES

Antwane
  • 20,760
  • 7
  • 51
  • 84
  • That could work, but I was aiming at something more portable. That means, if possible, using only Python's standard (or some well known) library – manu Sep 11 '18 at 14:50
0

Finally, I ended up using Antwane's solution. Just for future reference:

I downloaded the file using Python's standard library

urllib.request.urlretrieve(url, 'aux')

which gave me a text (html, actually) file (named 'aux') containing the redirection. I read that

with open('aux') as f:
    html = f.read()

and built a regular expression

regex_url_from_http_equiv = re.compile('<meta http-equiv="refresh" content="0;url=\'/(.*)\'')

to extract the working URL

redirection_match = self.regex_url_from_http_equiv.search(html)

# here you would write the usual checks and whatelse...
if redirection_match:
    redirection = redirection_match.group(1)

Not a cool solution, but good enough for now.

Thank you all for your input!!

manu
  • 1,333
  • 2
  • 11
  • 24