TcpClient.GetStream().DataAvailable returns false, but stream has more data

Question

So, it would seem that a blocking Read() can return before it is done receiving all of the data being sent to it. In turn we wrap the Read() with a loop that is controlled by the DataAvailable value from the stream in question. The problem is that you can receive more data while in this while loop, but there is no behind the scenes processing going on to let the system know this. Most of the solutions I have found to this on the net have not been applicable in one way or another to me.

What I have ended up doing is as the last step in my loop, I do a simple Thread.Sleep(1) after reading each block from the stream. This appears to give the system time to update and I am not getting accurate results but this seems a bit hacky and quite a bit 'circumstantial' for a solution.

Here is a list of the circumstances I am dealing with: Single TCP Connection between an IIS Application and a standalone application, both written in C# for send/receive communication. It sends a request and then waits for a response. This request is initiated by an HTTP request, but I am not having this issue reading data from the HTTP Request, it is after the fact.

Here is the basic code for handling an incoming connection

protected void OnClientCommunication(TcpClient oClient)
{
    NetworkStream stream = oClient.GetStream();
    MemoryStream msIn = new MemoryStream();

    byte[] aMessage = new byte[4096];
    int iBytesRead = 0;

    while ( stream.DataAvailable )
    {
        int iRead = stream.Read(aMessage, 0, aMessage.Length);
        iBytesRead += iRead;
        msIn.Write(aMessage, 0, iRead);
        Thread.Sleep(1);
    }
    MemoryStream msOut = new MemoryStream();

    // .. Do some processing adding data to the msOut stream

    msOut.WriteTo(stream);
    stream.Flush();

    oClient.Close();
}

All feedback welcome for a better solution or just a thumbs up on needing to give that Sleep(1) a go to allow things to update properly before we check the DataAvailable value.

Guess I am hoping after 2 years that the answer to this question isn't how things still are :)

Answer 1

You have to know how much data you need to read; you cannot simply loop reading data until there is no more data, because you can never be sure that no more is going to come.

This is why HTTP GET results have a byte count in the HTTP headers: so the client side will know when it has received all the data.

Here are two solutions for you depending on whether you have control over what the other side is sending:

1. Use "framing" characters: (SB)data(EB), where SB and EB are start-block and end-block characters (of your choosing) but which CANNOT occur inside the data. When you "see" EB, you know you are done.

2. Implement a length field in front of each message to indicate how much data follows: (len)data. Read (len), then read (len) bytes; repeat as necessary.

This isn't like reading from a file where a zero-length read means end-of-data (that DOES mean the other side has disconnected, but that's another story).

A third (not recommended) solution is that you can implement a timer. Once you start getting data, set the timer. If the receive loop is idle for some period of time (say a few seconds, if data doesn't come often), you can probably assume no more data is coming. This last method is a last resort... it's not very reliable, hard to tune, and it's fragile.

Answer 2

I'm seeing a problem with this.
You're expecting that the communication will be faster than the while() loop, which is very unlikely.
The while() loop will finish as soon as there is no more data, which may not be the case a few milliseconds just after it exits.

Are you expecting a certain amount of bytes?
How often is OnClientCommunication() fired? Who triggers it?

What do you do with the data after the while() loop? Do you keep appending to previous data?

DataAvailable WILL return false because you're reading faster than the communication, so that's fine only if you keep coming back to this code block to process more data coming in.

Answer 3

I was trying to check DataAvailable before reading data from a network stream and it would return false, although after reading a single byte it would return true. So I checked the MSDN documentation and they also read before checking. I would re-arrange the while loop to a do while loop to follow this pattern.

http://msdn.microsoft.com/en-us/library/system.net.sockets.networkstream.dataavailable.aspx

        // Check to see if this NetworkStream is readable. 
        if(myNetworkStream.CanRead){
            byte[] myReadBuffer = new byte[1024];
            StringBuilder myCompleteMessage = new StringBuilder();
            int numberOfBytesRead = 0;

            // Incoming message may be larger than the buffer size. 
            do{
                 numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);

                 myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));

            }
            while(myNetworkStream.DataAvailable);

            // Print out the received message to the console.
            Console.WriteLine("You received the following message : " +
                                         myCompleteMessage);
        }
        else{
             Console.WriteLine("Sorry.  You cannot read from this NetworkStream.");
        }

Answer 4

When I have this code:

    var readBuffer = new byte[1024];
    using (var memoryStream = new MemoryStream())
    {
        do
        {
            int numberOfBytesRead = networkStream.Read(readBuffer, 0, readBuffer.Length);
            memoryStream.Write(readBuffer, 0, numberOfBytesRead);
        }
        while (networkStream.DataAvailable);
    }

From what I can observe:

• When sender sends 1000 bytes and reader wants to read them. Then I suspect that NetworkStream somehow "knows" that it should receive 1000 bytes.
• When I call .Read before any data arrives from NetworkStream then .Read should be blocking until it gets more than 0 bytes (or more if .NoDelay is false on networkStream)
• Then when I read first batch of data I suspect that .Read is somehow updating from its result the counter of those 1000 bytes at NetworkStream and before this happens I suspect, that in this time the .DataAvailable is set to false and after the counter is updated then the .DataAvailable is then set to correct value if the counter data is less than 1000 bytes. It makes sense when you think about it. Because otherwise it would go to the next cycle before checking that 1000 bytes arrived and the .Read method would be blocking indefinitely, because reader could have already read 1000 bytes and no more data would arrive.
• This I think is the point of failure here as already James said:

Yes, this is just the way these libraries work. They need to be given time to run to fully validate the data incoming. – James Apr 20 '16 at 5:24

• I suspect that the update of internal counter between end of .Read and before accessing .DataAvailable is not as atomic operation (transaction) so the TcpClient needs more time to properly set the DataAvailable.

When I have this code:

    var readBuffer = new byte[1024];
    using (var memoryStream = new MemoryStream())
    {
        do
        {
            int numberOfBytesRead = networkStream.Read(readBuffer, 0, readBuffer.Length);
            memoryStream.Write(readBuffer, 0, numberOfBytesRead);

            if (!networkStream.DataAvailable)
                System.Threading.Thread.Sleep(1); //Or 50 for non-believers ;)
        }
        while (networkStream.DataAvailable);
    }

Then the NetworkStream have enough time to properly set .DataAvailable and this method should function correctly.

Fun fact... This seems to be somehow OS Version dependent. Because the first function without sleep worked for me on Win XP and Win 10, but was failing to receive whole 1000 bytes on Win 7. Don't ask me why, but I tested it quite thoroughly and it was easily reproducible.

Answer 5

Using TcpClient.Available will allow this code to read exactly what is available each time. TcpClient.Available is automatically set to TcpClient.ReceiveBufferSize when the amount of data remaining to be read is greater than or equal to TcpClient.ReceiveBufferSize. Otherwise it is set to the size of the remaining data. Hence, you can indicate the maximum amount of data that is available for each read by setting TcpClient.ReceiveBufferSize (e.g., oClient.ReceiveBufferSize = 4096;).

        protected void OnClientCommunication(TcpClient oClient)
        {
            NetworkStream stream = oClient.GetStream();
            MemoryStream msIn = new MemoryStream();

            byte[] aMessage;
            oClient.ReceiveBufferSize = 4096;
            int iBytesRead = 0;

            while (stream.DataAvailable)
            {
                int myBufferSize = (oClient.Available < 1) ? 1 : oClient.Available;
                aMessage = new byte[oClient.Available];

                int iRead = stream.Read(aMessage, 0, aMessage.Length);
                iBytesRead += iRead;
                msIn.Write(aMessage, 0, iRead);
            }
            MemoryStream msOut = new MemoryStream();

            // .. Do some processing adding data to the msOut stream

            msOut.WriteTo(stream);
            stream.Flush();

            oClient.Close();
        }

Answer 6

public class NetworkStream
{
    private readonly Socket m_Socket;

    public NetworkStream(Socket socket)
    {
        m_Socket = socket ?? throw new ArgumentNullException(nameof(socket));
    }

    public void Send(string message)
    {
        if (message is null)
        {
            throw new ArgumentNullException(nameof(message));
        }

        byte[] data = Encoding.UTF8.GetBytes(message);
        SendInternal(data);
    }

    public string Receive()
    {
        byte[] buffer = ReceiveInternal();
        string message = Encoding.UTF8.GetString(buffer);
        return message;
    }

    private void SendInternal(byte[] message)
    {
        int size = message.Length;

        if (size == 0)
        {
            m_Socket.Send(BitConverter.GetBytes(size), 0, sizeof(int), SocketFlags.None);
        }
        else
        {
            m_Socket.Send(BitConverter.GetBytes(size), 0, sizeof(int), SocketFlags.None);
            m_Socket.Send(message, 0, size, SocketFlags.None);
        }
    }

    private byte[] ReceiveInternal()
    {
        byte[] sizeData = CommonReceiveMessage(sizeof(int));
        int size = BitConverter.ToInt32(sizeData);

        if (size == 0)
        {
            return Array.Empty<byte>();
        }

        return CommonReceiveMessage(size);
    }

    private byte[] CommonReceiveMessage(int messageLength)
    {
        if (messageLength < 0)
        {
            throw new ArgumentOutOfRangeException(nameof(messageLength), messageLength, "Размер сообщения не может быть меньше нуля.");
        }

        if (messageLength == 0)
        {
            return Array.Empty<byte>();
        }

        byte[] buffer = new byte[m_Socket.ReceiveBufferSize];
        int currentLength = 0;
        int receivedDataLength;

        using (MemoryStream memoryStream = new())
        {
            do
            {
                receivedDataLength = m_Socket.Receive(buffer, 0, m_Socket.ReceiveBufferSize, SocketFlags.None);
                currentLength += receivedDataLength;
                memoryStream.Write(buffer, 0, receivedDataLength);
            }
            while (currentLength < messageLength);

            return memoryStream.ToArray();
        }
    }
}

Answer 7

This example presents an algorithm for sending and receiving data, namely text messages. You can also send files.

using System;
using System.IO;
using System.Net.Sockets;
using System.Text;

namespace Network
{
    /// <summary>
    /// Represents a network stream for transferring data.
    /// </summary>
    public class NetworkStream
    {
        #region Fields
        private static readonly byte[] EmptyArray = Array.Empty<byte>();
        private readonly Socket m_Socket;
        #endregion

        #region Constructors
        /// <summary>
        /// Initializes a new instance of the class <seealso cref="NetworkStream"/>.
        /// </summary>
        /// <param name="socket">
        /// Berkeley socket interface.
        /// </param>
        public NetworkStream(Socket socket)
        {
            m_Socket = socket ?? throw new ArgumentNullException(nameof(socket));
        }
        #endregion

        #region Properties

        #endregion

        #region Methods
        /// <summary>
        /// Sends a message.
        /// </summary>
        /// <param name="message">
        /// Message text.
        /// </param>
        /// <exception cref="ArgumentNullException"/>
        public void Send(string message)
        {
            if (message is null)
            {
                throw new ArgumentNullException(nameof(message));
            }

            byte[] data = Encoding.UTF8.GetBytes(message);
            Write(data);
        }

        /// <summary>
        /// Receives the sent message.
        /// </summary>
        /// <returns>
        /// Sent message.
        /// </returns>
        public string Receive()
        {
            byte[] data = Read();
            return Encoding.UTF8.GetString(data);
        }

        /// <summary>
        /// Receives the specified number of bytes from a bound <seealso cref="Socket"/>.
        /// </summary>
        /// <param name="socket">
        /// <seealso cref="Socket"/> for receiving data.
        /// </param>
        /// <param name="size">
        /// The size of the received data.
        /// </param>
        /// <returns>
        /// Returns an array of received data.
        /// </returns>
        private byte[] Read(int size)
        {
            if (size < 0)
            {
                // You can throw an exception.
                return null;
            }

            if (size == 0)
            {
                // Don't throw an exception here, just return an empty data array.
                return EmptyArray;
            }

            // There are many examples on the Internet where the
            // Socket.Available property is used, this is WRONG!

            // Important! The Socket.Available property is not working as expected.
            // Data packages may be in transit, but the Socket.Available property may indicate otherwise.
            // Therefore, we use a counter that will allow us to receive all data packets, no more and no less.
            // The cycle will continue until we receive all the data packets or the timeout is triggered.

            // Note. This algorithm is not designed to work with big data.

            SimpleCounter counter = new(size, m_Socket.ReceiveBufferSize);
            byte[] buffer = new byte[counter.BufferSize];
            int received;

            using MemoryStream storage = new();

            // The cycle will run until we get all the data.
            while (counter.IsExpected)
            {
                received = m_Socket.Receive(buffer, 0, counter.Available, SocketFlags.None);
                // Pass the size of the received data to the counter.
                counter.Count(received);
                // Write data to memory.
                storage.Write(buffer, 0, received);
            }

            return storage.ToArray();
        }

        /// <summary>
        /// Receives the specified number of bytes from a bound <seealso cref="Socket"/>.
        /// </summary>
        /// <returns>
        /// Returns an array of received data.
        /// </returns>
        private byte[] Read()
        {
            byte[] sizeData;
            // First, we get the size of the master data.
            sizeData = Read(sizeof(int));
            // We convert the received data into a number.
            int size = BitConverter.ToInt32(sizeData);

            // If the data size is less than 0 then throws an exception.
            // We inform the recipient that an error occurred while reading the data.

            if (size < 0)
            {
                // Or return the value null.
                throw new SocketException();
            }

            // If the data size is 0, then we will return an empty array.
            // Do not allow an exception here.

            if (size == 0)
            {
                return EmptyArray;
            }

            // Here we read the master data.
            byte[] data = Read(size);
            return data;
        }

        /// <summary>
        /// Writes data to the stream.
        /// </summary>
        /// <param name="data"></param>
        private void Write(byte[] data)
        {
            if (data is null)
            {
                // Throw an exception.
                // Or send a negative number that will represent the value null.
                throw new ArgumentNullException(nameof(data));
            }

            byte[] sizeData = BitConverter.GetBytes(data.Length);

            // In any case, we inform the recipient about the size of the data.
            m_Socket.Send(sizeData, 0, sizeof(int), SocketFlags.None);

            if (data.Length != 0)
            {
                // We send data whose size is greater than zero.
                m_Socket.Send(data, 0, data.Length, SocketFlags.None);
            }
        }
        #endregion

        #region Classes
        /// <summary>
        /// Represents a simple counter of received data over the network.
        /// </summary>
        private class SimpleCounter
        {
            #region Fields
            private int m_Received;
            private int m_Available;
            private bool m_IsExpected;
            #endregion

            #region Constructors
            /// <summary>
            /// Initializes a new instance of the class <seealso cref="SimpleCounter"/>.
            /// </summary>
            /// <param name="dataSize">
            /// Data size.
            /// </param>
            /// <param name="bufferSize">
            /// Buffer size.
            /// </param>
            /// <exception cref="ArgumentOutOfRangeException"/>
            public SimpleCounter(int dataSize, int bufferSize)
            {
                if (dataSize < 0)
                {
                    throw new ArgumentOutOfRangeException(nameof(dataSize), dataSize, "Data size cannot be less than 0");
                }

                if (bufferSize < 0)
                {
                    throw new ArgumentOutOfRangeException(nameof(dataSize), bufferSize, "Buffer size cannot be less than 0");
                }

                DataSize = dataSize;
                BufferSize = bufferSize;

                // Update the counter data.
                UpdateCounter();
            }
            #endregion

            #region Properties
            /// <summary>
            /// Returns the size of the expected data.
            /// </summary>
            /// <value>
            /// Size of expected data.
            /// </value>
            public int DataSize { get; }

            /// <summary>
            /// Returns the size of the buffer.
            /// </summary>
            /// <value>
            /// Buffer size.
            /// </value>
            public int BufferSize { get; }

            /// <summary>
            /// Returns the available buffer size for receiving data.
            /// </summary>
            /// <value>
            /// Available buffer size.
            /// </value>
            public int Available
            {
                get
                {
                    return m_Available;
                }
            }

            /// <summary>
            /// Returns a value indicating whether the thread should wait for data.
            /// </summary>
            /// <value>
            /// <see langword="true"/> if the stream is waiting for data; otherwise, <see langword="false"/>.
            /// </value>
            public bool IsExpected
            {
                get
                {
                    return m_IsExpected;
                }
            }
            #endregion

            #region Methods
            // Updates the counter.
            private void UpdateCounter()
            {
                int unreadDataSize = DataSize - m_Received;
                m_Available = unreadDataSize < BufferSize ? unreadDataSize : BufferSize;
                m_IsExpected = m_Available > 0;
            }

            /// <summary>
            /// Specifies the size of the received data.
            /// </summary>
            /// <param name="bytes">
            /// The size of the received data.
            /// </param>
            public void Count(int bytes)
            {
                // NOTE: Counter cannot decrease.

                if (bytes > 0)
                {
                    int received = m_Received += bytes;
                    // NOTE: The value of the received data cannot exceed the size of the expected data.
                    m_Received = (received < DataSize) ? received : DataSize;

                    // Update the counter data.
                    UpdateCounter();
                }
            }

            /// <summary>
            /// Resets counter data.
            /// </summary>
            public void Reset()
            {
                m_Received = 0;
                UpdateCounter();
            }
            #endregion
        }
        #endregion
    }
}

Answer 8

Use a do-while loop. This will make sure the memory stream pointers have moved. The first Read or ReadAsync will cause the memorystream pointer to move and then onwards the ".DataAvailable" property will continue to return true until we hit the end of the stream.

An example from microsoft docs:

        // Check to see if this NetworkStream is readable.
        if(myNetworkStream.CanRead){
            byte[] myReadBuffer = new byte[1024];
            StringBuilder myCompleteMessage = new StringBuilder();
            int numberOfBytesRead = 0;
    
            // Incoming message may be larger than the buffer size.
            do{
               numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
    
               myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
           }
          while(myNetworkStream.DataAvailable);
    
          // Print out the received message to the console.
          Console.WriteLine("You received the following message : " +
                                     myCompleteMessage);
       }
       else{
         Console.WriteLine("Sorry.  You cannot read from this NetworkStream.");
      }

Original Micorosoft Doc