2

To use timeout I use requests in next mode:

import requests


r = requests.get(url, stream=True, timeout=2)
r.content()

BUT to use a size limit, I need to use stream:

import requests


r = requests.get(currentUrl,stream=True)
for chunk in r.iter_content(chunk_size=1024):
    buffer.extend(chunk)
    if len(buffer) > 1024 * 100:
        r.close()
        break

On streaming requests, the timeout only applies to the connection attempt. On regular requests, the timeout is applied to the connection process and on to all attempts to read data from the underlying socket. It does not apply to the total download time for the request.

Another option is to use Content-Length, but not all websites has this...

My question is how can I use both limitation in the same time? Or wich is suitable package for this?

Martin Gergov
  • 1,556
  • 4
  • 20
  • 29
user3817928
  • 95
  • 1
  • 4
  • Did you try setting a global timeout for blocking socket operations yourself by using [`socket.settimeout`](https://docs.python.org/2/library/socket.html#socket.socket.settimeout)? – Lukas Graf Jul 08 '14 at 20:28
  • @LukasGraf Please give me an code sample – user3817928 Jul 08 '14 at 20:34
  • Sorry, I meant [`socket.setdefaulttimeout`](https://docs.python.org/2/library/socket.html#socket.setdefaulttimeout). So something like `import socket; socket.setdefaulttimeout(0.5)` - you would need to decrease that timeout as you read chunks. – Lukas Graf Jul 08 '14 at 20:48
  • A related question is here: http://stackoverflow.com/questions/23514256/http-request-with-timeout-maximum-size-and-connection-pooling – Adrian B Jul 08 '14 at 21:08

0 Answers0