How "generate " multiple TCP clients using Threads instead of opening multiple instances of the terminal and run the script several times?

Question

I wrote the code for a simple TCP client:

from socket import *

# Configurações de conexão do servidor
# O nome do servidor pode ser o endereço de
# IP ou o domínio (ola.python.net)
serverHost = 'localhost'#ip do servidor
serverPort = 50008

# Mensagem a ser mandada codificada em bytes
menssagem = [b'Ola mundo da internet!']

# Criamos o socket e o conectamos ao servidor
sockobj = socket(AF_INET, SOCK_STREAM)
sockobj.connect((serverHost, serverPort))

# Mandamos a menssagem linha por linha
for linha in menssagem:
    sockobj.send(linha)

    # Depois de mandar uma linha esperamos uma resposta
    # do servidor
    data = sockobj.recv(1024)
    print('Cliente recebeu:', data)

# Fechamos a conexão
sockobj.close()

I would like to know, how " generate " multiple clients TCP using Threads instead of opening multiple instances of the terminal and run the script several times.

`threading` Python module might help you. Or try `multiprocessing` if you wanna break out of the GIL (Global Interpreter Lock - better Google that term). If you're using Python 3.5, you should almost definitely try `asyncio` and that fancy `async def` syntax for asynchronous functions definition. — ForceBru, Aug 17 '16 at 20:42
@ForceBru, thanks. Could you give me a example? I am using Python 3.4. — Paul Sigonoso, Aug 17 '16 at 20:52
simply wrap your current code into a function and spawn several threads running it (something like `threading.Thread(target=func).start()`) — ForceBru, Aug 17 '16 at 20:54
here's the documentation on `threading`: https://docs.python.org/3.4/library/threading.html — ForceBru, Aug 17 '16 at 20:58
consider that python isn't the best tool for this job. You might want to consider a threadpool approach and asynchronous io (boost::asio is a good choice). — stack user, Sep 02 '16 at 06:23

score 2 · Accepted Answer · answered Aug 29 '16 at 13:21

2

Try this: The worker method will be set as target for the thread. So every thread will use the code of the method. After starting all thread, the for loop at the bottom will wait for all threads to finish.

In the worker method you can use arrays or lists of data from outside the method. So you can iterate for example over a list of Urls or append the fetched data to a new output array.

import threading

threads = []
maxNrOfThreads = 5

def worker():
    do_stuff()

for _ in range(maxNrOfThreads):
    thr = threading.Thread(target=worker)
    threads.append(thr)
    thr.setDaemon(True)
    thr.start()

for thread in threads:
    thread.join()

answered Aug 29 '16 at 13:21

Alu

727
5
16

why for _ in? I dont understand _ – Paul Sigonoso Aug 29 '16 at 14:16
It means you don't need the index of the iteration. Underscore is a convention to mark the variable as unused. You can also use `i` instead of `_` and ignore it. – Alu Aug 29 '16 at 14:17
def worker(): do_stuff() -> it will be my TCP client? – Paul Sigonoso Aug 29 '16 at 15:45
Yes, there you can open the socket and do your stuff ;) – Alu Aug 29 '16 at 16:08
@Alu Could you explain why you set the threads as Daemons when you are going to immediately follow by calling join on every thread. Doesn't that negate the point of setting them as Daemons? – bravosierra99 Sep 01 '16 at 17:11
Good point. In that example you are right. For a good explanation look at http://stackoverflow.com/questions/190010/daemon-threads-explanation – Alu Sep 01 '16 at 17:15

Nabeel Ahmed · Answer 2 · 2016-09-05T14:18:20.550

I have just wrapped your code/what you want to do, into a function i.e. worker(). Next, I have added extra code for thread spawning and set the worker() function as the target (the function/work/code each spawned thread will execute - this is why it's a convention to name it worker).

th = threading.Thread(target=worker)

The multithreaded version of you above example can be as follows:

import threading
from socket import *

serverHost = 'localhost'#ip do servidor
serverPort = 50008

threads = []
input = input("Enter the number of threads:")
num_of_threads = int(input)

def worker():
    # Criamos o socket e o conectamos ao servidor
    sockobj = socket(AF_INET, SOCK_STREAM)
    sockobj.connect((serverHost, serverPort))

    # Mandamos a menssagem linha por linha
    for linha in menssagem:
        sockobj.send(linha)
        # Depois de mandar uma linha esperamos uma resposta
        # do servidor
        data = sockobj.recv(1024)
        print('Cliente recebeu:', data)

    sockobj.close()

# thread generation block
for t in range(num_of_threads):
    th = threading.Thread(target=worker)
    threads.append(th)
    th.setDaemon(True)
    th.start()

for thread in threads:
    thread.join()

There's no `raw_input` in Python 3. Notice the tag. – saaj Sep 05 '16 at 11:38 — saaj, Sep 05 '16 at 11:38

Raul R. · Answer 3 · 2016-09-06T03:30:16.057

Here is one solution with queues to send distinct messages.

#!/usr/bin/python

import Queue
import threading
import time
from socket import *


DEF_HOST = 'localhost'
DEF_PORT = 50008


queueLock = threading.Lock()


class myThreadTCP (threading.Thread):

    def __init__(self, host=DEF_HOST, port=DEF_PORT, q=None):
        threading.Thread.__init__(self)
        self.host = host
        self.port = port
        self.q = q

    def run(self):
        global queueLock
        print("Starting Thread")
        # Criamos o socket e o conectamos ao servidor
        sockobj = socket(AF_INET, SOCK_STREAM)
        sockobj.connect((self.host, self.port))
        while not workQueue.empty():
            with queueLock:
                data = q.get()
            if data:
                print("sending %s" % data)
                sockobj.send(data)
                # Depois de mandar uma linha esperamos uma resposta
                # do servidor
                data = sockobj.recv(1024)
                print('Cliente recebeu:', data)
        # Fechamos a conexão
        sockobj.close()
        print("Exiting Thread")


workQueue = Queue.Queue()

# Mensagem a ser mandada codificada em bytes
menssagem = [b'Ola mundo da internet!', b'Ola mundo da internet #2!']
for msg in menssagem:
    workQueue.put(msg)

threads = []

# Create 10 new threads
for i in range(0, 10):
    thread = myThreadTCP(host=DEF_HOST, port=DEF_PORT, q=workQueue)
    thread.daemon = True
    thread.start()
    threads.append(thread)

# Wait for all threads to complete
for t in threads:
    t.join()
print("Exiting Main Thread")

In Python 3 `print` is no longer an operator. Notice the tag. — saaj, Sep 05 '16 at 11:40
thanks, the code is more a pseudo-code, not tested, but have the ideas that should work. — Raul R., Sep 06 '16 at 03:50

score 0 · Answer 4 · answered Sep 05 '16 at 12:29

The simplest and most pythonic way is to use multiprocessing thread pool implementation, and then call pool.map. The former will allow you effortlessly swap from threads to processes when needed. The latter will provide you a clean interface hiding synchronisation chores behind the scenes.

#!/usr/bin/env python3 


import socket
from pprint import pprint
from contextlib import closing
from multiprocessing.dummy import Pool as ThreadPool


serverHost = 'localhost'
serverPort = 80


messageGroups = [
    [b'GET / HTTP/1.0\n\n'],
    [b'GET /some-path HTTP/1.0\n\n'],
]

def send(messages):
    result = []
    options = socket.AF_INET, socket.SOCK_STREAM
    with closing(socket.socket(*options)) as sockobj:
        sockobj.connect((serverHost, serverPort))
        for message in messages:
            sockobj.send(message)
            response = sockobj.recv(1014)
            result.append((message, response))

    return result


if __name__ == '__main__':
    size = 10
    pool = ThreadPool(size)
    result = pool.map(send, messageGroups)
    pool.close()
    pool.join()

    pprint(result)

How "generate " multiple TCP clients using Threads instead of opening multiple instances of the terminal and run the script several times?

4 Answers4