1

I have an application that receives data from GPRS clients in the field on a TCP connection. From time to time the GPRS client devices loses connection and buffers the data, when the connection is restored all the buffered data is send to the TCP connection, cauing my application to throw a System.OutOfMemoryException.

I presume this is because the data received is bigger than my buffer size (which is set to int.MaxValue).

  1. How do I prevent my application to run out of memory?
  2. How do I make sure that I do not lose any of the incomming data?

Below is the code used to listen for, and handle incomming data

public void Listen(string ip, int port)
{
    _logger.Debug("All.Tms.SensorDataServer : SensorDataListener : Listen");

    try
    {
        var listener = new TcpListener(IPAddress.Parse(ip), port);
        listener.Start();

        while (true)
        {
            var client = listener.AcceptTcpClient();
            client.SendBufferSize = int.MaxValue;

            var thread = new Thread(() => ReadAndHandleIncommingData(client));
            thread.IsBackground = true;
            thread.Start();
        }
    }
    catch (Exception ex)
    {
        _logger.Error("TMS.Sensor.SensorDataServer : SensorDataListener : Error : ", ex);
    }  
}

and

private void ReadAndHandleIncommingData(TcpClient connection)
{
     try
     {
         var stream = connection.GetStream();
         var data = new byte[connection.ReceiveBufferSize];

         var bytesRead = stream.Read(data, 0, System.Convert.ToInt32(connection.ReceiveBufferSize));

         var sensorDataMapper = new SensorDataMapperProvider().Get(data);

         if (sensorDataMapper != null)
         {
             _sensorDataHandler.Handle(sensorDataMapper.Map(data));
         }
    }
    catch (Exception ex)
    {
        _logger.Error("TMS.Sensor.SensorDataServer : SensorDataListener : ReadAndHandleIncommingData : Error : ", ex);
    }
    finally
    {
        try
        {
            connection.Close();
        }
        catch(Exception ex)
        {
            _logger.Error("All.Tms.SensorDataServer : SensorDataListener : ReadAndHandleIncommingData : Error : ", ex);
        }
    }
}
Mike Pennington
  • 41,899
  • 19
  • 136
  • 174
Andre Lombaard
  • 6,985
  • 13
  • 55
  • 96

2 Answers2

4

About buffers

OutOfMemoryExceptions are thrown when there is no sequential memory left.
In your case, that means connection.ReceiveBufferSize is too big to keep in memory in one piece. Not because the data received is bigger than your buffer size.

You can use a smaller, fixed buffer to get the received bytes, append it somewhere and use the same buffer to receive the rest of the data, until you have it all.

One thing to look out for is the collection you use to store the received data. You can't use List<byte> for example because it stores its elements in a single array under the hood which makes no difference than getting everything in one go - like you do now.

You can see MemoryTributary, a stream implementation meant to replace MemoryStream. You can copy your stream to this one and keep it as a stream. This page also contains a lot of information that can help you understand the causes of the OutOfMemoryExceptions.

In addition, I wrote a buffer manager to provide fixed sized buffers a while ago. You can find that in Code Review, here.

About threads

Creating a thread for every single connection is brutal. They cost you 1 MB each so you should either use ThreadPool or better, IOCP (via asynchronous methods of the Socket class).

You may want to look into these for common pitfalls and best practices about socket programming:

Community
  • 1
  • 1
Şafak Gür
  • 7,045
  • 5
  • 59
  • 96
2

Use a fixed receive buffer, and parse data incrementally

miniBill
  • 1,743
  • 17
  • 41