urllib2 does not read entire page

Question

A portion of code that I have that will parse a web site does not work.

I can trace the problem to the .read function of my urllib2.urlopen object.

page = urllib2.urlopen('http://magiccards.info/us/en.html')
data = page.read()

Until yesterday, this worked fine; but now the length of the data is always 69496 instead of 122989, however when I open smaller pages my code works fine.

I have tested this on Ubuntu, Linux Mint and windows 7. All have the same behaviour.

I'm assuming that something has changed on the web server; but the page is complete when I use a web browser. I have tried to diagnose the issue with wireshark but the page is received as complete.

Does anybody know why this may be happening or what I could try to determine the issue?

If you think it has to do with the web server, how could you do anything? — Florin Stingaciu, Oct 24 '12 at 15:20
This is the error I get with urllib in py3k: `http.client.IncompleteRead: IncompleteRead(69496 bytes read, 53493 more expected)` — SilentGhost, Oct 24 '12 at 15:25
since a web browser was still able to show the complete page, my code also needed to be able to process en entire page. — kannerke, Oct 25 '12 at 15:15

score 4 · Answer 1 · answered Oct 24 '12 at 15:51

4

The page seems to be misbehaving unless you request the content encoded as gzip. Give this a shot:

import urllib2
import zlib

request = urllib2.Request('http://magiccards.info/us/en.html')
request.add_header('Accept-Encoding', 'gzip')
response = urllib2.urlopen(request)
data = zlib.decompress(response.read(), 16 + zlib.MAX_WBITS)

As Nathan suggested, you could also use the great Requests library, which accepts gzip by default.

import requests

data = requests.get('http://magiccards.info/us/en.html').text

answered Oct 24 '12 at 15:51

Solon

41
2

It may be that `keep-alive` had nothing to do with this at all. I need to investigate with Wireshark. – Nathan Villaescusa Oct 24 '12 at 16:16
Yeah, could be. curl supports keep-alive and seems to have the same issue. – Solon Oct 24 '12 at 16:25
curl has the same issue, but it works if you pass the `--compressed` option. It's definitely a server issue. – Adam Rosenfield Oct 24 '12 at 17:13
I added the http header to the request and my code works again. Many thanks for your suggestion. – kannerke Oct 25 '12 at 15:12

score 0 · Answer 2 · edited May 23 '17 at 11:48

Yes, the server is closing connection and you need keep-alive to be sent. urllib2 does not have that facility ( :-( ). There used be urlgrabber which you could use have a HTTPHandler that works alongside with urllib2 opener. But unfortunately, I dont find that working too. At the moment, you could be other libraries, like requests as demonstrated in the other answer or httplib2.

import httplib2
h = httplib2.Http(".cache")
resp, content = h.request("http://magiccards.info/us/en.html", "GET")
print len(content)

urllib2 does not read entire page

2 Answers2