2

When I use the python requests module, calling requests.get(url), I have found that the response from the url is being truncated.

import requests

url = 'https://gtfsrt.api.translink.com.au/Feed/SEQ'
response = requests.get(url)
print response.text

The response I get from the URL is being truncated. Is there a way to get requests to retrieve the full set of data and not truncate it?

Note: The given URL is a public transport feed which puts out a huge quantity data during the peak of day.

gustafbstrom
  • 1,622
  • 4
  • 25
  • 44
Tom
  • 925
  • 3
  • 10
  • 24
  • 1
    `requests` itself does not truncate data, no. Have you verified that the server is sending correct headers? – Martijn Pieters Jul 07 '15 at 09:40
  • When I load that URL, `len(response.content)` matches `response.headers['content-length']`. – Martijn Pieters Jul 07 '15 at 09:42
  • So I did a check where I did a request via the above script, then also did a manual download from the url myself. The manual feed was considerably longer then the one return via the script. – Tom Jul 07 '15 at 09:42
  • It might not be truncating it anymore as the vehicle numbers have dropped considerably, from its peak number. – Tom Jul 07 '15 at 09:44
  • Any HTTP server is free to *completely alter* its response from second to second. It is a black box, we don't know what criteria it uses to serve content. Most likely `requests` request differs from your browser request in some way that makes the server respond differently. That could be any number of factors; check for differences in query parameters, headers, and the time of day. – Martijn Pieters Jul 07 '15 at 10:01
  • But there is nothing *inherent in `requests` here*. This is a server behaviour issue. You may well get a different response again if you used `curl`. Or `urllib2`, or a library in a different programming language. Or even if you requested the data from a different IP address. – Martijn Pieters Jul 07 '15 at 10:02
  • Ok thanks. Then the bug must be somewhere else in the broader script this belongs to. – Tom Jul 07 '15 at 10:52
  • 1
    I'm tempted to close this as "can't reproduce". – Burhan Khalid Jul 07 '15 at 11:11
  • Fire up a packet capture utility like Wireshark and verify whether the server is actually truncating the response, or whether the data is getting lost somewhere on your machine. – bsa Jul 07 '15 at 13:47

2 Answers2

1

I ran into the same issue. The problem is not your Python code. It might be PyCharm or Utility you are using - The console has a buffer limit. You may have to increase that to see your full response.

Refer to this article for more help: Increase output buffer when running or debugging in PyCharm

SamG
  • 11
  • 1
-1

Add "gzip":true to your request options. That fixed the issue for me.

Esther Sheina Agishtein
  • 1
  • Do you mean 'Content-Encoding': 'gzip'? Still, looking at the discussion above, this does not seem the solution to the issue at hand. – M.G.Poirot Apr 12 '22 at 10:00
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Apr 12 '22 at 12:40