multiple threads and multiple tcp connections

Question

I have an application with multiple threads, one thread is creating 4 tcp connections, then this thread creates another thread that will handle the receive function for these 4 connections using poll or anything else, and the original thread (the first one) start sending messages to these 4 connections(round-robin). it's like below pseudo code,

main()
{

    pthread_create( &thread1, NULL, sending_thread, (void*) args);

}

void *sending_thread( void *ptr )
{
    int i=0;
    int *connections;
    connections = create_connections();

    pthread_create( &thread1, NULL, receiving_thread, (void*)&connections);
        while(1) {

        send(connections[i], "test"); 
        i++;
        if (i > 3)
        i=0;             

        }
}

void *receiving_thread( void *ptr )
{
    char *msg; 
        msg = receive(connections[i]);
        save_received_msg_to_disk(msg);

}

My question is How can I check my connections and bring up the disconnected one? for example, let's say connection1 went down, do I need to create another connection with the same fd, which is connection[1] in this case? or is there other ways to handle this case?

Environment is C/pthread in Linux

Do you *have to* do it like this with separate sending and receiving threads? — Nikolai Fetissov, Jan 23 '13 at 21:17
Not necessarily. Try `epoll(7)` over non-blocking sockets from a single thread. Measure, compare. — Nikolai Fetissov, Jan 23 '13 at 22:07
I'm using that because receive is asynchronous, not sure how will use epoll to send and receive, do you have an example or a tutorial to share? — user1832809, Jan 23 '13 at 22:09
Start here http://stackoverflow.com/questions/27247 , but of course it all depends on what you are trying to do. — Nikolai Fetissov, Jan 23 '13 at 22:15
Correct,what I'm trying to do is this, I need to send all the time and handle the received packets with multiple tcp connections, max number of connections is 4. — user1832809, Jan 23 '13 at 22:19

Nikolai Fetissov · Accepted Answer · 2013-01-25T04:27:58.533

Here are some points based on your code and comments.

While threads do allow some things happen in parallel, they usually have a non-zero cost in terms of more involved design that always boils down to synchronization. Like in your case there's no easy way of re-establishing one of the connections and sharing it between two threads again.
I haven't ever seen an app where input and output streams are completely independent. I can imagine something like a tunneling proxy or a TCP-based VPN where that might make sense, but in general case some higher level protocol still imposes some request-response semantics that again would impose/require arbitration among sending and receiving threads.
Things are different when you need throughput and when you try to minimize latency. Sleep-waiting on a mutex is usually OK for the former, but is rarely a good idea for the latter. Round-robin writing to several blocking sockets from a single thread hurts both.
If you have heavy output stream(s) and sparse input, it only makes sense to use something like epoll(7) to detect saturated connections and be notified when they are available again then starve others.

I know this doesn't answer your question directly, but my ~~rant~~ list did not fit into the comments. Hope this helps a little.

Edit 0:

Here's the usual setup with epoll(7):

Make your sockets non-blocking (fcntl(2) with O_NONBLOCK).
Set epoll_data.fd to your socket descriptor for each potential channel (four sockets in your example). Other options are possible with union epoll_data if you want to keep more complex structures then just socket descriptors.
Use EPOLLIN and EPOLLET to get edge-triggered behavior, i.e. be woken up when input buffer becomes not empty.
Only set EPOLLOUT if you get EWOULDBLOCK from a write(2), otherwise do output as usual. Same logic here with EPOLLET to detect output buffer space becoming available.
Use EPOLLRDHUP to detect other side disconnecting cleanly (for abrupt disconnects you need to handle EPIPE error form write(2)).
epoll_wait(2) gives you back number of events to iterate through. Do separate checks for input (events & EPOLLIN) and output (events & EPOLLOUT).
On input read from data.fd (or otherwise associated socket) until you get EWOULDBLOCK.
On output write until you get EWOULDBLOCK or you don't have more output data pending (remove EPOLLOUT in that case).

It looks like a lot, but is pretty simple once you get a hang of it.

You can also do non-blocking connect(2), which is probably a good idea if you ever want to re-establish broken streams without hurting others that are still chugging along (connect(2) returns -1 with errno(3) set to EINPROGRESS and you wait for the socket to become writable as above).

thanks a lot Nikolai, based on you comment which makes sense to me I'm gonna use epoll, my question here is I'm not sure how to use, never used it before actually, I've been reading about it since yesterday and the only thing I can think about is using EPOLLIN/EPOLLOUT, but not really sure how to monitor both events, is like this "if array[i].events & (EPOLLIN | EPOLLOUT)) {do_somthing;}, how can I tell this is for input and the other is for ouput — user1832809, Jan 24 '13 at 22:18
Thanks a lot, I almost did that yesterday with the help of this http://epollwait.blogspot.com/2010/11/, I only commented this part if (epoll_ctl(efd, EPOLL_CTL_MOD, fd, &ev) < 0) { printf("Couldn't modify client socket %d in epoll set: %m\n", fd); goto cleanup; and it worked perfectly. — user1832809, Jan 25 '13 at 21:16

multiple threads and multiple tcp connections

1 Answers1

Edit 0: