Multi threaded application is not outputting properly

Question

I have recently been introduced to the threading module in python so I decided to play around with it I opened a python socket server on port 7000:

import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('127.0.0.1',7000))
s.listen(1)
c, a = s.accept()

and made my client server try connecting to every port from 1 to 65535 until it establishes connection on port 7000. Obviously this would take very long so I multi-threaded it:

import threading
import socket
import sys
host = None
def conn(port):
    try:
        s.connect((host,port))
        print 'Connected'
        sys.exit(1)
    except:
        pass
    global host
host = '127.0.0.1'
for i in range(65535):
    t = threading.Thread(target=conn, args=(i,))
    t.start()

When the client connects its suppose to return the message 'connected' however when debugging I noticed some very strange behavior with the program. Sometimes the program would return that it connected, other times the program would fail to output that it was connected to the server instead it would just terminate without printing anything.

Its obviously a problem with the threads. As when i make the client connect to port 7000 only it works 100% of the time. However threading it through all 65535 ports causes the client to sometimes not print anything. What is the reason for this and how can I prevent or circumvent it.

Edit: I realized making it try to connect to a smaller number of ports, ports 1-10 and port 7000, gives it a higher chance of printing out connected.

Starting 65535 threads probably isn't a good way of tackling this. A better approach might be to spawn fewer threads that each try to connect to a subset of ports. In any case, you should probably add a check to your loop so that you stop trying to connect/spawn new threads once you've made a connection. Having a high number of threads is liable to make your program a bit unstable/unpredictable. — hnefatl, Aug 15 '17 at 10:50
Also, do you ever `join` on the threads you've created (see [here](https://stackoverflow.com/questions/11968689/python-multithreading-wait-till-all-threads-finished))? If you're just spawning them then exiting, it's possible that you're exiting before they're able to connect. This would explain the variance in behaviour, as it comes down to processor scheduling and the like. — hnefatl, Aug 15 '17 at 10:55
In the current edit it looks like `def conn(port):` calls itself recursively. — u354356007, Aug 15 '17 at 10:59
doing a thread.join() would nullify the purpose of threading i want to make this process ad fast as possible — , Aug 15 '17 at 11:09
It won't. Create and run threads, store references, and then join them in a separate loop. — u354356007, Aug 15 '17 at 11:16
When you failed connect to a port (e.g: 6999), it raise CONNECT_REFUSED. This exception will change the socket itself. When It try to connect 7000 (which is open), it will raise ```socket.error: [Errno 22] Invalid argument```. You should create a new socket when failed. — gushitong, Aug 15 '17 at 12:16

u354356007 · Answer 1 · 2017-08-15T12:04:08.233

0

65535 is a huge number.

Any performance gain you might get will be dwarfed by this amount of threads. An OS should plan processor time for each of this thread and then it takes time to switch between threads. In the worst case (and 7k is pretty much bad) all OS does is thread switching with little real work in between. 2-8 threads (or just a thread per physical core) would be much more performant.

Also, make sure you wait until your threads exit, and don't silence out errors with except: pass. I bet, there are a lot of interesting things happening there. At least [selectively] log these exceptions somewhere.

Edit. Use join in order to make sure that all spawned threads exit before the main thread.

threads = [threading.Thread(target=conn, args=(i,)) for i in range(8)]
for thread in threads:
    thread.start()

# do whatever

for thread in threads:
    thread.join()

edited Aug 15 '17 at 12:04

answered Aug 15 '17 at 10:59

u354356007

3,205
15
25

I've tried this but the behavior I get is still the same, the thread just cant seem to be able to output properly so its not stemming from the threads exiting too early, lowering the amount of ports allows the thread to output properly so is there another reason? – Aug 16 '17 at 12:21
And do you still silence all the exceptions with `except: pass`? – u354356007 Aug 16 '17 at 12:37
Yes still 65000 threads however the exceptions arent silenced – Aug 29 '17 at 11:59
so should I only run a few threads like 3 to 8 at a time until i finish all 65k in order to maximize thread efficiency? – Aug 30 '17 at 12:56

gushitong · Answer 2 · 2017-08-15T11:06:14.660

If connect() fails, consider the state of the socket as unspecified. Portable applications should close the socket and create a new one for reconnecting.

>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('127.0.0.1', 6999))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 61] Connection refused
>>>
>>> s.connect(('127.0.0.1', 7000))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 22] Invalid argument
>>> 
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('127.0.0.1', 7000))
# Connect success.

Multi threaded application is not outputting properly

2 Answers2