I'm trying to simulate a network (only TCP) using python's multiprocessing, multi-threading and raw socket.
What I'm doing?
- Create a child process which sniffs the network for new connections.
- A child process processes 1000 connections and terminates itself after spawning another child process (via main process obviously.) which will do the same job.
- All the connection information is stored in a dictionary which is process specific.
- Per connection, I'm creating a Timer thread which will delete the connection from the dictionary if the connection is idle for 5 seconds. (And I'm returning from the Thread also.)
- While deleting the connection, I'm de-referencing the timer thread and all other dictionary parameters for that connection.
- After deleting the last element from the dictionary, I'm calling a
gc.collect()
to collect the garbage and callingos._exit(0)
so the child process will terminate. (Keep in mind that another sibling process has already taken over.)
Why I'm making it so complicated?
- Each connection needs to have its own Timer as it has to die after 5 secs of inactivity.
- I've tried with a single process processing all the connections which kept on eating up the memory. (Though I followed the above method, memory was not being released.) And at the end, the machine with 4GB memory used to hang making me unable to use it. (Keyboard and mouse interrupts were very very slow.)
- So I made a child process for only 1000 connections, and then terminate it. Which releases the memory (I've seen it as the memory used by Python mostly remains a constant).
- Since I'm terminating the child process, all threads associated to it should be deleted. (I've read that a thread is not removed in Python unless the parent process dies, which is not the case here.)
What is my problem?
- Many times I'm seeing an error (Multiple times per child process),
Can't start new thread.
Though I'm explicitly terminating the child process usingos._exit()
. I know that there might be a limit in creating new threads. But I don't think it'd be too less like 2000 or 3000. Since I'm terminating the child processes, I hope the threads are also getting deleted. (Correct me if I'm wrong.) - Occasionally, This
Can't start new thread.
error comes while spawning a new child process. I couldn't understand why it's showing a thread error while creating a child process? - Rarely I've seen, at the beginning of the first child process itself the above error comes. (No doubt the previous instances of Python is being killed just before few seconds.) Since no instance of Python is there in the process list (Of Linux), It's obvious that all threads from the previous instance have been cleared which is not reflected in this rare scenario.
No doubt I'm ignoring the errors with try except
, It'd be good to know:
- Why that error comes with so many less threads?
- On which parameters (of OS and python), creation of a new thread depends?
- How to avoid the above error? (I've one, which will create just one background thread as daemon instead of a timer thread for each connection. So other than that any better solution?)