1

I have created a load generator to test a server, written in Python. The load generator creates a socket, connects to the server socket, requests a file, then disconnects. The load generator runs as expected when I instantiate < ~300 threads, but when I instantiate any more past that, I get the following errors client-side:

  • [Errno 32] Broken pipe on sendall
  • [Errno 54] Connection reset by peer on recv

occasionally,

  • [Errno 41] Protocol wrong type for socket

Why is the server closing its read side of the socket prematurely, forcing the client-side to get a broken pipe error? I'm assuming the other errors are related to this.

I have read another thread stating that this might be due to lack of memory client-side because so many threads are instantiated: How to fix java.net.SocketException: Broken pipe?

I'm looking for an understanding of why these errors are occurring when many threads are instantiated, but do not occur when fewer threads are instantiated, and how to narrow down the problem and fix it.

Community
  • 1
  • 1
cosmosa
  • 743
  • 1
  • 10
  • 23
  • What platform are you on? Which Python version? Do your threads do any CPU-bound work at all? – abarnert May 01 '15 at 02:03
  • Platform: mac OSX, Python 2.7. No, my threads just request a file and that is it. – cosmosa May 01 '15 at 02:04
  • OK, OS X can handle 300 threads no problem—but the old (either before 3.2 or before 3.3) CPython global interpreter lock can't. You end up stalling threads all over the place if even one thread ever briefly does any non-trivial CPU-bound work. If you port your code to Python 3 and run it under the 3.4 interpreter, or just run your existing code under Jython or IronPython, the problem might just go away. – abarnert May 01 '15 at 02:06
  • However, using a thread per client may not be the best way to run a server that takes care of that many concurrent connections. Especially in Python, where accidentally doing CPU-bound work doesn't just choke things up, it serializes the whole program into running one thread at a time. – abarnert May 01 '15 at 02:07
  • @abarnert, interesting. Thanks for your input. – cosmosa May 01 '15 at 02:07
  • It may be worth, at least as an experiment, using `gevent`, which lets you write code that looks just like threaded code but actually runs it in an event loop in a single thread. That may not be the best design for your program, but it may help show whether the problem is in the GIL and context switching or somewhere else, and whether you're accidentally doing too much CPU work in your handlers. – abarnert May 01 '15 at 02:10
  • One last easy experiment: use `multiprocessing` instead of `threading`. That will definitely increase your overhead, but it will eliminate all problems with CPU-bound threads, and with other things (like unintended sharing between threads causing races), so it'll help narrow down where the problem is. – abarnert May 01 '15 at 02:11
  • @abarnert, I have never used multiprocessing before, but I am going to give it a try. – cosmosa May 01 '15 at 02:29

0 Answers0