3

I am new to Python, I want to understand if the Http request is Synchronous or Async? do i need to implement callbacks ?

I am using urllib2 module and below is the syntax:

content = urllib2.urlopen(urlnew).read()

At my server there are more than 30,000 records and for each one there will be an http call and the response received will be stored.

Any help appreciated.

AlG
  • 14,697
  • 4
  • 41
  • 54
Dragon
  • 719
  • 1
  • 9
  • 18
  • Yes, it is synchronous. – BlackMamba Apr 10 '15 at 14:09
  • The `urllib2` library is synchronous. What makes you think it might be asynchronous? – Martijn Pieters Apr 10 '15 at 14:09
  • See [Asynchronous Requests with Python requests](http://stackoverflow.com/q/9110593) for an asynchronous option. – Martijn Pieters Apr 10 '15 at 14:10
  • thanks @Martijn i checked the docs and could not find anything related to its behaviour can you provide any links for the same? – Dragon Apr 10 '15 at 14:12
  • 2
    In most environments synchronous behaviour is the norm; documentation will only explicitly call out asynchronous behaviour instead. So no, I cannot link you to documentation that unequivocally states that Python's `urllib2` library is synchronous. You can look at the [source code](https://hg.python.org/cpython/file/0db36098b908/Lib/urllib2.py) instead if you really want to verify this. – Martijn Pieters Apr 10 '15 at 14:15
  • Also relevant: [Asynchronous HTTP calls in Python](http://stackoverflow.com/q/4962808) – Martijn Pieters Apr 10 '15 at 14:20

1 Answers1

7

Like most Python stuff, unless explicitely mentioned, urllib2 is synchronous. So the execution will block until the server responded.

So if you want to make 30,000 requests, you will have to do one request after the other one. An alternative would be to launch the requests in multiple processes (using multiprocessing) to parallelize it.

But the better option, especially since you seem to be in control of the server, would be to have it provide some kind of batch request that allows you to query multiple (or all) records at once.

poke
  • 369,085
  • 72
  • 557
  • 602