3

I'm trying to simulate a network (only TCP) using python's multiprocessing, multi-threading and raw socket.

What I'm doing?

  1. Create a child process which sniffs the network for new connections.
  2. A child process processes 1000 connections and terminates itself after spawning another child process (via main process obviously.) which will do the same job.
  3. All the connection information is stored in a dictionary which is process specific.
  4. Per connection, I'm creating a Timer thread which will delete the connection from the dictionary if the connection is idle for 5 seconds. (And I'm returning from the Thread also.)
  5. While deleting the connection, I'm de-referencing the timer thread and all other dictionary parameters for that connection.
  6. After deleting the last element from the dictionary, I'm calling a gc.collect() to collect the garbage and calling os._exit(0) so the child process will terminate. (Keep in mind that another sibling process has already taken over.)

Why I'm making it so complicated?

  1. Each connection needs to have its own Timer as it has to die after 5 secs of inactivity.
  2. I've tried with a single process processing all the connections which kept on eating up the memory. (Though I followed the above method, memory was not being released.) And at the end, the machine with 4GB memory used to hang making me unable to use it. (Keyboard and mouse interrupts were very very slow.)
  3. So I made a child process for only 1000 connections, and then terminate it. Which releases the memory (I've seen it as the memory used by Python mostly remains a constant).
  4. Since I'm terminating the child process, all threads associated to it should be deleted. (I've read that a thread is not removed in Python unless the parent process dies, which is not the case here.)

What is my problem?

  1. Many times I'm seeing an error (Multiple times per child process), Can't start new thread. Though I'm explicitly terminating the child process using os._exit(). I know that there might be a limit in creating new threads. But I don't think it'd be too less like 2000 or 3000. Since I'm terminating the child processes, I hope the threads are also getting deleted. (Correct me if I'm wrong.)
  2. Occasionally, This Can't start new thread. error comes while spawning a new child process. I couldn't understand why it's showing a thread error while creating a child process?
  3. Rarely I've seen, at the beginning of the first child process itself the above error comes. (No doubt the previous instances of Python is being killed just before few seconds.) Since no instance of Python is there in the process list (Of Linux), It's obvious that all threads from the previous instance have been cleared which is not reflected in this rare scenario.

No doubt I'm ignoring the errors with try except, It'd be good to know:

  1. Why that error comes with so many less threads?
  2. On which parameters (of OS and python), creation of a new thread depends?
  3. How to avoid the above error? (I've one, which will create just one background thread as daemon instead of a timer thread for each connection. So other than that any better solution?)
RatDon
  • 3,403
  • 8
  • 43
  • 85
  • not sure but: don't you have to explicitly exit/return from the threads? Besides, `os._exit` exits the process without calling cleanup handlers, flushing stdio buffers, etc. I wouldn't rely on that call to stop all my threads... – Jean-François Fabre Aug 22 '16 at 06:29
  • I'm returning from the threads. And before calling `os._exit` I'm de-referencing all the varaibles I'm using. Not sure about the stdio buffers and calling `gc.collect` which I think should do the cleanups as there were no references to any of the memory. – RatDon Aug 22 '16 at 06:37
  • There are limits to threads. See for example http://stackoverflow.com/questions/18647230/cant-create-new-threads-in-python. 4 gig of ram isn't much, especially considering that a significant amount of that is tied up with system resources. You may be using tools that also create threads .... For instance Popen.communicate .... Which would grow your thread count considerably. – tdelaney Aug 22 '16 at 06:48
  • http://www.blog.pythonlibrary.org/2016/08/02/python-201-a-multiprocessing-tutorial/ – buhtz Aug 22 '16 at 06:49
  • @tdelaney I switched to 32GB of RAM, still the problem persists. And I don't think it's related to memory, as the memory used by Python is at max 2%. (Combining all the processes.) It's not like I'm creating all the threads at once. By the time it reaches to 500, some of them might already be removed. – RatDon Aug 22 '16 at 07:39

0 Answers0