4

I'm having problems with getting winsock RIO working. It seems that every time I post a RIOReceive it returns immediately with 0 bytes transferred, and my peer can't get a message through.

After posting a RIOReceive, I wait on the RIODequeCompletion, which deques immediately with numResults = 1, but when I inspect the bytesTransferred of the RIORESULT struct, it's 0. This tells me that I'm not setting this thing up properly, but I can't find docs or examples that tell me what else I should be doing.

The internet seems to have very little on RIO. I've looked through MSDN, Len Holgate with TheServerFramework, this site, and two GitHub RIO servers.

RIOEchoServer and RIOServer_sm9 are on GitHub, but I can't post more than two links (this is my first question on this site).

This code is just to get things proven. It's currently not set to use the sendCQ, doesn't handle errors well, etc...

Here's the prep work:

void serverRequestThread() {

//init buffers 

//register big buffers
recvBufferMain = rio.RIORegisterBuffer(pRecvBufferMain, bufferSize);
sendBufferMain = rio.RIORegisterBuffer(pSendBufferMain, bufferSize);

if (recvBufferMain == RIO_INVALID_BUFFERID) {
    cout << "RIO_INVALID_BUFFERID" << endl;
}

if (sendBufferMain == RIO_INVALID_BUFFERID) {
    cout << "RIO_INVALID_BUFFERID" << endl;
}

//create recv buffer slice
recvBuffer1.BufferId = recvBufferMain;
recvBuffer1.Offset = 0;
recvBuffer1.Length = 10000;

//create send buffer slice
sendBuffer1.BufferId = sendBufferMain;
sendBuffer1.Offset = 0;
sendBuffer1.Length = 10000;

//completion queue
recvCQ = rio.RIOCreateCompletionQueue(CQsize, NULL);
sendCQ = rio.RIOCreateCompletionQueue(CQsize, NULL);

if (recvCQ == RIO_INVALID_CQ) {
    cout << "RIO_INVALID_CQ" << endl;
}

if (sendCQ == RIO_INVALID_CQ) {
    cout << "RIO_INVALID_CQ" << endl;
}

//start a loop for newly accept'd socket
while (recvCQ != RIO_INVALID_CQ && sendCQ != RIO_INVALID_CQ) {

    //get accept'd socket
    struct sockaddr_in saClient;
    int iClientSize = sizeof(saClient);
    acceptSocket = accept(listenSocket, (SOCKADDR*)&saClient, &iClientSize);

    if (acceptSocket == INVALID_SOCKET) {
        cout << "Invalid socket" << endl;
        printError();
    }

    //register request queue
    requestQueue = rio.RIOCreateRequestQueue(
        acceptSocket,       //socket
        10,                 //max RECVs on queue
        1,                  //max recv buffers, set to 1
        10,                 //max outstanding sends
        1,                  //max send buffers, set to 1
        recvCQ,             //recv queue
        recvCQ,             //send queue
        pOperationContext   //socket context
        );

    if (requestQueue == RIO_INVALID_RQ) {
        cout << "RIO_INVALID_RQ" << endl;
        printError();
    }

I now post a RIOReceive:

    //start a loop to repin recv buffer for socket
    while (acceptSocket != INVALID_SOCKET) {

        //pin a recv buffer to wait on data
        recvSuccess = rio.RIOReceive(
            requestQueue,           //socketQueue
            &recvBuffer1,           //buffer slice
            1,                      //set to 1
            RIO_MSG_WAITALL,                        //flags
            0);                     //requestContext

        if (recvSuccess == false) {
            cout << "RECV ERROR!!!!!!!!\n";
            printError();
        }

        //wait for recv to post in queue

        //std::this_thread::sleep_for(std::chrono::milliseconds(3000));

As soon as I call RIODequeCompletion, it returns 1:

        numResults = 0;
        while (numResults == 0) numResults = rio.RIODequeueCompletion(recvCQ, recvArray, 10);

        if (numResults == RIO_CORRUPT_CQ) {

            cout << "RIO_CORRUPT_CQ" << endl;

        } else if (numResults == 0) {

            cout << "no messages on queue\n";

        } else if (numResults > 0) {

but when I inspect the bytesTransferred of the RIORESULT, it's always 0:

            if (recvArray[0].BytesTransferred > 0) {

                //process results
                if (pRecvBufferMain[0] == 'G') {

                    //set respnose html
                    strcpy(pSendBufferMain, responseHTTP);

                    sendSuccess = rio.RIOSend(
                        requestQueue,   //socketQueue
                        &sendBuffer1,   //buffer slice
                        1,              //set to 1
                        0,              //flags
                        0);             //requestContext

                } else if (pRecvBufferMain[0] == 'P') {

                    //process post 


                } else {
                    //recv'd a bad message

                }

            } //end bytesTransferred if statement

            //reset everything and post another recv

        }//end response if statement

        std::this_thread::sleep_for(std::chrono::milliseconds(100));

    }//end while loop for recv'ing

    std::this_thread::sleep_for(std::chrono::milliseconds(100));

}//end while loop for accept'ing

}// end function

Like I said, I'm probably not using RIOReceive correctly, and/or I'm not setting the correct socket options that I need to (none right now).

I appreciate any help with this.

Michael220
  • 193
  • 12
  • Why are you using RIO in the first place, instead of traditional socket I/O, or Overlapped I/O, or I/O Completion Ports? Why are you polling the RIO queues instead of being notified when things are ready? And you are using `recvCQ` for both send and recv queues in `RIOCreateRequestQueue()`, you are not using `sendCQ` for the send queue. – Remy Lebeau Feb 05 '15 at 20:41
  • I'm just looking into RIO to see if it's appropriate for my server. Polling isn't efficient, as you know, but under heavy load it provides the highest performance since an IOCP would eat cycles. It's also simplest, so that's why I'm using it here, just to get some numbers on the board. Ya, the sendCQ isn't hooked up. I removed it temporarily for debugging, but it'll go back in. Thanks for asking... Do you have any ideas on why I'm not getting bytes off of RIOReceive/RIODequeueCompletion/RIORESULT? – Michael220 Feb 05 '15 at 20:54
  • I have never used RIO before. IOCP has always been the preferred high-performance solution. I don't know why MS felt the need to introduce yet another socket API when they already have several. In any case, a result of 0 bytes received usually indicates the socket was closed by the peer. – Remy Lebeau Feb 05 '15 at 21:03
  • I'm not getting any WSAENETRESET or WSAENOTCONN errors though. Is it still possible to have a broken/disconnected connection without getting those errors? – Michael220 Feb 05 '15 at 21:36
  • 1
    @RemyLebeau: Because the C10K problem was yesterday we are now in a C100K or even a C1M problem area. You can get servers with 40GBit bandwidth now. Every cycle might count and yes RIO is faster. We will see a full userland Ethernet stack soon. The 1970 Berkley API and the also now 20 year old IOCP are not the future. – Lothar Nov 28 '15 at 19:40

2 Answers2

1

Try removing RIO_MSG_WAITALL. There may be a bug whereby you're only getting the close notification (bytes == 0) rather than getting a completion with the data in it. Either way it would be interesting to see if the code works without the flag.

Do my example servers and tests work on your hardware?

Len Holgate
  • 21,282
  • 4
  • 45
  • 92
  • Thanks Len. This was one of the things that was contributing. After I removed that I also discovered that I did not correctly size my CQ to be big enough for all my RQs. I did read through all your blogs, they've been very helpful, but I haven't actually run the code yet. I've been focusing on TCP, and I thought I could just learn the theory and construct a simple server. Reality has been different of course, so I may go back and run your UDP server/clients and increment from there. Thanks for sharing your work! – Michael220 Feb 10 '15 at 02:39
  • The reason I asked about my tests working on your hardware was that I've had some clients who had strange things installed on various machines and that affected RIO, apparently. So it's good to get a baseline example that's known to work working first. Glad you found the problem. There's an issue with CQ's getting corrupted if you post more requests (and they complete) than there is space in the CQ; so you need to be careful about that. – Len Holgate Feb 10 '15 at 09:16
0

I encountered a similar issue of having zero bytesReceived in my dequeued completion result while using RioReceive with RioNotify and RioDequeueCompletion. I would also see the 'Status' value of WSAEINVAL (Invalid Parameter = 10022) in my dequeued completion result, this seems to indicate the WSA error code for the Receive call.

The particular reason I had the error is because I had allocated memory for a receiveBuffer and I was trying to pass that buffer pointer as my buffer handle in the RIO_BUFFER_SEGMENT given to RioReceive instead of passing the IntPtr returned by RioRegisterBuffer.

I fully blame myself for using too many untyped IntPtrs and losing type checking. :)

Tim Lovell-Smith
  • 15,310
  • 14
  • 76
  • 93