6

Recently, our entire QA environment moved from VMWare to Hyper-V virtual machines.

One of our applications sends UDP packets to a multicast cloud by a rate of 20K packets per second.

While that worked perfectly on the VMWare environment, the Hyper-V makes the application to throw the following exception after a couple minutes of work:

System.Net.Sockets.SocketException (0x80004005): An invalid argument was supplied
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)

I've also managed to mimic this issue instantly when defining the socket's sending buffer size to 1,000,000 bytes.

How can I resolve this issue?

UPDATE 1: This is a log entry from the event viewer once the exception happens:

Faulting application name: Agent.exe, version: 1.0.12.7366, time stamp: 0x51389f69
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b83c8a
Exception code: 0xe0434352
Fault offset: 0x0000c41f
Faulting process id: 0xaf0
Faulting application start time: 0x01ce1b4ce509dc7a
Faulting application path: C:\Users\DevUser\Desktop\QA\Agent.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: d2b45dce-8740-11e2-86f9-00155d022804

UPDATE 2: The size of the UDP packet is 100-200 bytes.

UPDATE 3: Here is the problematic code:

    m_socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
    m_socket.Ttl = 1;

    if (GetRawParameter("send") != null)
    {
      Log("Starting sender...");

      StartSender();
    }

    ...snip...

private static void StartSender()
{
  m_lastPacketNumber = 0;

  m_socket.Connect(new IPEndPoint(m_ipAddress, m_port));

  if (m_bufferSize > 0)
    m_socket.SetSocketOption(
      SocketOptionLevel.Socket, SocketOptionName.SendBuffer, m_bufferSize);

  byte[] dataPad = null;

  if (m_packetSize > 8)
  {
    dataPad = new byte[m_packetSize - sizeof(long)];

    for (int i = 0; i < dataPad.Length; i++)
    {
      dataPad[i] = 0xFF;
    }
  }

  while (true)
  {
    Log("Sending data...");

    for (int i = 0; i < m_packetsPerSec; i++)
    {
      var data = BitConverter.GetBytes(m_lastPacketNumber.Value);

      if (dataPad != null)
        data = data.Concat(dataPad).ToArray();

      if (m_packetDump != null)
        m_packetDump.Add(m_lastPacketNumber.Value);

      m_socket.Send(data);

      if (m_usePerformanceCounters)
        IncreaseSendCounters(1);

      m_lastPacketNumber++;
    }

    Log(m_lastPacketNumber + " packets sent.");

    Thread.Sleep(1000);
  }
}

UPDATE 4: the failed Send() seems to happen on the #14156 or #32485 or #25412 packet (not the first one!) when I try to send 100K of packets per second.

Eran Betzalel
  • 4,105
  • 3
  • 38
  • 66
  • What SocketErrorCode do you get from the exception? – HasaniH Mar 07 '13 at 16:26
  • 2
    ErrorCode = 10022, SocketErrorCode = InvalidArgument. – Eran Betzalel Mar 07 '13 at 16:34
  • 1
    Of course you'll get an error when you try to send a packet that large. The largest UDP packet you can send is about 64K bytes. See http://stackoverflow.com/q/1098897/56778, for example. – Jim Mischel Mar 07 '13 at 18:40
  • You get an "invalid argument" error, but you haven't bothered to find out what the arguments were? And this while you have an instant replication method? Help us help you: run the programmer under a debugger to find the arguments! Also, why are you using the Winsock tag on this question? You're using .NET's sockets interface, not the Winsock API proper. Yes, I know .NET is using Winsock under the hood, but if that's your argument, you might as well expect to see the Winsock tag used on a good portion of the {Perl,Python,Ruby,...} questions here on SO, too. – Warren Young Mar 08 '13 at 00:28
  • @WarrenYoung, the arguments are presented above. This is a Winsock question for the simple reason that 10022 is a Winsock error. Running in debug mode won't help as the exception occurs in a low level stage of the C# winsock framework. – Eran Betzalel Mar 08 '13 at 10:45
  • 1
    I'm referring to the arguments your code passed to `System.Net.Sockets.Socket.Send()`. Or if you aren't calling that directly, the name and args for the last .NET framework function you *do* call before the exception is thrown. – Warren Young Mar 08 '13 at 20:20
  • I'm calling it directly and the send parameter is just a byte array of 100 characters. – Eran Betzalel Mar 09 '13 at 17:49
  • It seems that this question is a bit on the high level scale, so I also posted it here: http://social.technet.microsoft.com/Forums/en-US/winserverhyperv/thread/8ead5259-fd3a-44b7-9239-f47677707654. – Eran Betzalel Mar 10 '13 at 11:07
  • @JimMischel, as stated, I'm trying to send 100 bytes of data - it's not that much :) – Eran Betzalel Mar 10 '13 at 11:24
  • So you're calling `Socket.Send(Buffer)`? Showing some code is helpful. Otherwise you're just playing "stump the chumps" with us ... and I have better things to do. – Jim Mischel Mar 10 '13 at 13:35
  • Sure @JimMischel, I've just uploaded the code. – Eran Betzalel Mar 10 '13 at 13:48
  • It would be instructive to know what `data.Length` is when you call `Send`. Also, a full stack trace when the exception is thrown. I suspect, though, that the problem is somewhere else in the stack. I'm trying to understand why this works on VMWare but fails on Hyper-V. You might consider downloading the Framework source from http://referencesource.microsoft.com/netframework.aspx and using that to step into `Socket.Send`, just so you can see where it's failing. – Jim Mischel Mar 10 '13 at 16:16
  • 1
    data.length = 100, the exception occurs on the Win32Api call and returns the above error code. – Eran Betzalel Mar 11 '13 at 08:58
  • Is it possible that your buffer is getting modified during the send? Maybe there is another thread operating that isn't shown? – Paul Farry Mar 11 '13 at 20:51
  • 1
    My test application only use one thread - no issue here. – Eran Betzalel Mar 12 '13 at 15:12
  • Does this occur when you try sending them to the local loopback, or a unicast address? – yasth Mar 26 '13 at 20:10
  • 1
    I send the packets thru the HyperV virtual network switch. – Eran Betzalel Mar 27 '13 at 23:06
  • You might be running into issues with hyper-v and VMQ which seems to happen somewhat randomly under load. I recently solved a similar issue by disabling VMQ. See http://alexappleton.net/post/77116755157/hyper-v-virtual-machines-losing-network-connectivity – AVee Sep 16 '14 at 23:11

1 Answers1

-1

Could you "roll up" the packets so that you are sending far less packets of larger size...and then un-roll them on the other side? 20k packets/second is a lot of sendto's.

raetza
  • 39
  • 8
  • I can't delay or buffer the data before sending it on. The actual requirement is 100K, which actually works using VMWare machines. – Eran Betzalel Mar 27 '13 at 23:08