1

I am looking to implement the threading solution suggested here: What is the fastest way to send 100,000 HTTP requests in Python?

My question is, if my "do something" after making the URL call is going to be appending a new row to a list (using data.append) - will this cause any issues?

Also, I will also be Python 2 and using urllib2 -- would the recommended solution change at all if it was urllib2 instead of httplib?

Community
  • 1
  • 1
reese0106
  • 2,011
  • 2
  • 16
  • 46

2 Answers2

0

Since you are trying to implement IO bound operation,I would suggets you to use gevent. Here is the example.

Or use this answer. Gevent support not only Python 2 and Python 3, also PyPy which also can speed up your program.

Community
  • 1
  • 1
gorros
  • 1,411
  • 1
  • 18
  • 29
0

Asynchronous IO is the solution to this exact problem. Since your problem is completely IO bound there is no reason for using multiple threads. If you are using python 3.4 using asyncio otherwise you can use tulips. You can also use Trollius library which provides asyncio likes syntax in Python 2x.

hspandher
  • 15,934
  • 2
  • 32
  • 45
  • do you have an example using tulips that you would recommend? I am not using python 3 so I am interested in seeing how I can apply this. – reese0106 Apr 06 '16 at 18:57
  • I have added the link for documentation to trollius. As an afterthought using trollius would be easier than tulips – hspandher Apr 07 '16 at 18:48