5

I am coding sockets server for 1000 clients maxmimum, the server is about my game, i'm using non-blocking sockets and about 10 threads that receive data simultaneously from different sockets (first thread receives from 0-100,second from 101-200 and so on..)

but if thread 1 wants to send data to all 1000 clients and thread 2 also wants to send data to all 1000 clients at the same time, is that safe? are there any chances of the data being messed in the other (client) side?

if yes, i guess the only problem that can happen is that sometimes client would receive 2 or 10 packets as 1 packet, is that correct? if yes, is there any solution to that :(

Tenev
  • 251
  • 6
  • 17
  • Agree with the answer offered by ckv, but I would suggest to reconsider the approach. Which is the benefit of having 10 threads, that read from different sockets at different time intervals? It may be easier to have threads reading from sockets and an asynchronous processing component in your architecture. – Daniel H. Jul 13 '10 at 08:59
  • @Daniel I guess u mean one thread for every socket. Right? – ckv Jul 13 '10 at 09:21
  • 1
    Most of the threads will be waiting when one of them is reading. Using something like `select()` is probably a better choice and decreases complexity. In fact, it might even be faster. – ereOn Jul 13 '10 at 09:26
  • yeah, well im using select() and fd_set for the 10 threads, but i have no idea how to make the new 10 threads to use select() for sending.. how would i trigger events in fd_set..? – Tenev Jul 13 '10 at 09:55

4 Answers4

2

The usual pattern of dealing with many sockets is to have a dedicated thread polling for I/O events with select(2), poll(2), or better kqueue(2) or epoll(4) (depending on the platform) acting as socket event dispatcher. The sockets are usually handled in non-blocking mode. Then one might have pool of threads reacting to the events and either do reads and writes directly or via lower level buffers/queues.

All sorts of techniques are applicable here - from queues to event subscription whiteboards. It gets tricky with multiplexing accepts/reads/writes/EOFs on the I/O level and with event arbitration on the application level. Several libraries like libevent and boost::asio help structure the lower level (the ACE library is also in this space, but I'd hate recommending it to anybody). You would have to come up with application-level protocols and state machines yourself (again boost::statechart might be of help).

Some good links to get better understanding of what you are up against (this is probably the millionth time they are mentioned here on SO):

Apologies for not offering a concrete solution, but this is a very wide design question and most decisions depend heavily on the context (lots of fun though). Hope this helps a bit.

Nikolai Fetissov
  • 82,306
  • 11
  • 110
  • 171
1

Since you are sending data using different sockets, there must not be any problem. Rather when these different threads access same data you have to ensure data integrity.

ckv
  • 10,539
  • 20
  • 100
  • 144
  • not really, each thread receives data on different sockets, but it can send data to all the sockets.. – Tenev Jul 13 '10 at 09:06
  • 1
    Sorry but i didnt understand your comment. Can you be more clear or can you edit your question so as to make it more clear. – ckv Jul 13 '10 at 09:12
  • threads can send to all sockets, but can receive from certain sockets – Tenev Jul 13 '10 at 09:43
0

send() is not atomic in most implementations, so sending to 1000 different sockets from multiple threads could lead to mixed-up messages arriving on the client side, and all kinds of weirdness. (I know nothing, see Nicolai's and Robert's comments below the rest of my comment still stands though (in terms of being a solution to your problem))

What I would do is use threads for sending like you use them for receiving. One thread to manage sending to one (or more) sockets that ensures that you don't write to one socket from multiple threads at the same time.

Also look here for some additional discussion and more interesting links. If you're on windows, the winsock programmers faq is an invaluable resource, for your issue see here.

Community
  • 1
  • 1
jilles de wit
  • 7,060
  • 3
  • 26
  • 50
  • hmm i was thinking about that, i would do that if i knew how, im using fd_set's for receiving with select(), but i dont know how to trigger events in fd_set's if i use select() for write/sending – Tenev Jul 13 '10 at 09:53
  • 1
    Are you sure about this? `send(2)` is a syscall - it is has to be **atomic**. Thread-safety is sort of a different question though in regard to how many bytes each `send(2)` actually consumes. – Nikolai Fetissov Jul 14 '10 at 04:24
  • @tenev, take a look at **libevent** : http://monkey.org/~provos/libevent/ - makes coding protocol state machines so much easier. – Nikolai Fetissov Jul 14 '10 at 04:26
  • @Nikolai, I'm not sure what you mean by send(2). send(...) includes a buffer of arbitrary length as one of it's parameters and I think the thread calling it could easily be paused half-way through dealing with that buffer. – jilles de wit Jul 14 '10 at 09:52
  • @tenev, if you have only one thread calling send() for any one socket (regardless of how many other sockets that thread handles) you should be fine. – jilles de wit Jul 14 '10 at 09:54
  • 1
    @jilles de wit: -1 Sorry, you're wrong. Send is 100% an atomic operation. If the data is too large and won't fit in the output buffer, then the call will block. If the call to send is interrupted, then no data is sent and an error is returned. Calls to send always completely succeed, or completely fail. – Robert S. Barnes Jul 14 '10 at 12:36
  • @jilles, you are confusing *system call reentrancy* and *application race conditions*. Btw `(2)` means section two of the manual, i.e. system calls. – Nikolai Fetissov Jul 14 '10 at 14:04
  • @Robert, Nikolai, thank you for setting me straight. I was not aware of this. What would be your answer to the original question? – jilles de wit Jul 15 '10 at 14:09
  • 1
    Take a look at this excellent paper: http://pl.atyp.us/content/tech/servers.html "High-Performance Server Architecture". – Nikolai Fetissov Jul 16 '10 at 02:25
0

Are you using UDP or TCP sockets?

If UDP, each write should be encapsulated in a separate packet and should be carried to the other side intact. The order may be swapped (as it may for any UDP packet) but they should be whole.

If TCP, there's no concept of packets on the transport layer and any 10 writes on one side may be bundled up on the other side in one read. TCP writes may also only accept part of your buffer so even if the send() function is atomic, your write isn't necessarily. In this case you'd need to synchronize it.

dascandy
  • 7,184
  • 1
  • 29
  • 50
  • TCP, synchronize... must i set non-blocking or blocking the sockets? if they are non-blocking i think i wont be able to synchronize them, since im using Select() before SEND(), i am using pthread_rwlock_wrlock(&players[socket].islocked); after select SEND(players[socket].socket) finally unlocking(&players[socket].islocked) is that a good solution ^^? looks like the fastest one to me – Tenev Jul 14 '10 at 17:53
  • With synchronize I meant that you have to separate the bytestream into packets manually. Your client will receive a byte stream no matter how you send it (asynchronous / nonblocking doesn't matter). Try writing 5 bytes 10 times in short succession - the other side receives a 50 byte block most likely. TCP is a byte stream. If you want packets you're going to have to make them. – dascandy Jul 19 '10 at 22:08