0

I have a high performance server receiving raw data that needs processing. The following code, which destructs the packet, throws an AccessViolation once every few thousand times it runs. I can't find any others with the same problem. The rest of the time it works fine. But the access violation is fatal and causes this service to be unstable.

Does any one have any idea why the "Array.Copy" line would be throwing an access violation every so often? The fixed keyword should stop the GC from getting rid of the memory?

async public static Task<AsyncProcessWebFrameResult> ProcessWebFrame(SocketAsyncEventArgs SocketEvent, byte[] Packet, int BytesCnt)
    {
        AsyncProcessWebFrameResult Result = new AsyncProcessWebFrameResult() { BytesProcessed = 0, Result = ProcessResults.Failed };

        ProtocolCommands? CommandType = null;

        int WebFrameSize = Marshal.SizeOf(typeof(WebFrameStruct));

        //do we at least a enough bytes for a frame? 
        if (BytesCnt < WebFrameSize)
        {
            DebugUtils.ConsoleWriteLine(SocketEvent, "Packet: Invalid length.");
            Result.Result = ProcessResults.ProtocolError;
            Result.BytesProcessed = BytesCnt;
            return Result;
        }

        int StartIdx = 0;

        //frame start with SOD?
        int BytesToCheck = Math.Min(BytesCnt+2, Packet.Length);
        while (StartIdx < BytesToCheck && Packet[StartIdx] != 0xA5)
            StartIdx++;

        if (StartIdx > 0 && StartIdx < BytesCnt - 1)
        {
            DebugUtils.ConsoleWriteLine(SocketEvent, "Packet: Does not start with SOD. Discarding " + StartIdx +" bytes");
            Result = await ProcessWebFrame(SocketEvent, Packet.Skip(StartIdx).ToArray(), BytesCnt - StartIdx);
            Result.BytesProcessed += StartIdx;
            return Result;
        }
        else if (StartIdx == BytesCnt-1)
        { 
            DebugUtils.ConsoleWriteLine(SocketEvent, "Packet: SOD not found discarding all.");
            Result.Result = ProcessResults.ProtocolError;
            Result.BytesProcessed = BytesCnt;
            return Result;
        }
        else if (StartIdx != 0)
        {
            DebugUtils.ConsoleWriteLine(SocketEvent, "Packet: SOD not found discarding all.");
            Result.Result = ProcessResults.ProtocolError;
            Result.BytesProcessed = BytesCnt;
            return Result;
        }

        byte[] Payload = new byte[0];

        try
        {
            unsafe
            {
                fixed (byte* pFirstByte = &(Packet[0]))
                {
                    WebFrameStruct* pFrame = (WebFrameStruct*)pFirstByte;
                    //Have we received the whole packet?
                    if (BytesCnt < WebFrameSize + pFrame->Size)
                    {
                        DebugUtils.ConsoleWriteLine(SocketEvent, string.Format("Packet: Packet incomplete. Expected: {0}, Received {1}", pFrame->Size + WebFrameSize, BytesCnt));
                        Result.Result = ProcessResults.AwaitingMoreData;
                        return Result;
                    }

                    //recognised protocol version?
                    if (((pFrame->Flags >> 4) & 0xF) != PROTO_VER)
                    {
                        DebugUtils.ConsoleWriteLine(SocketEvent, "Packet: Invalid protocol version.");
                        Result.Result = ProcessResults.ProtocolError;
                        Result.BytesProcessed = 1; //false start of frame, discard SOD
                        return Result;
                    }

                    //We have a valid packet so we can mark the bytes as processed so they get discarded, regardless of the result below.
                    Result.BytesProcessed = WebFrameSize + (int)pFrame->Size;

                    //do we have a registered controller for the command type
                    if (CommandControllers.ContainsKey((ProtocolCommands)pFrame->Type))
                    {
                        CommandType = (ProtocolCommands)pFrame->Type;
                        Array.Resize(ref Payload, (int)pFrame->Size);
                        if (Payload.Length != (int)pFrame->Size)
                            DebugUtils.ConsoleWriteLine(SocketEvent, string.Format("Array size incorrect. Is: {0} Should be {1}", Payload.Length, pFrame->Size));

                        ================================================
                        Array.Copy(Packet, WebFrameSize, Payload, 0, (int)pFrame->Size);  <---- this line throws the exception
                        =================================================
                        DebugUtils.ConsoleWriteLine(SocketEvent, string.Format("Packet is {0} -> sending to controller ", (ProtocolCommands)pFrame->Type));

                    }
                    else
                    {
                        DebugUtils.ConsoleWriteLine(SocketEvent, string.Format("Packet: No registered controller for Job {0}.", (ProtocolCommands)pFrame->Type));
                        Result.Result = ProcessResults.NoController;
                        return Result;
                    }
                }
            }



        }
        catch (AccessViolationException E)
        {
            Program.HandleFatalExceptions("", E);
        }

        return Result;
    }

The above method is called as follows

await ProcessWebFrame(SocketEvent, RxBuffer.Skip(Offset).ToArray(), RXBufferUsed - Offset);
Karl
  • 1
  • 3
  • Specific line that throws the throws the error is surrounded with ========================... towards the bottom of the code. – Karl Sep 19 '17 at 11:21
  • 1
    eh, `fixed` prevents the contents at that address from being moved. The object being in scope (edit: and still referenced) will prevent the GC from removing it. – BurnsBA Sep 19 '17 at 12:07
  • @BurnsBA thanks for the comment which is noted. But I don't think that affects the above? Is that correct? – Karl Sep 19 '17 at 12:13
  • 1
    Right, don't think it matters. Also, `Array.Copy` is implemented by the [CLR](https://github.com/fixdpt/shared-source-cli-2.0/blob/master/clr/src/vm/comsystem.cpp#L822) and I don't see it throwing an `AccessViolationException` so I'm guessing there's some access to your Packet variable happening somewhere else in code. – BurnsBA Sep 19 '17 at 12:52
  • 1
    An access violation exception suggests you are accessing memory beyond the bounds of your `Packet` array. Verify that `Packet` is large enough to _contain_ `WebFrameSize + (int)pFrame->Size` bytes. Maybe you only partially read a packet? Try adding an assertion on the `Packet` size and see if it gets hit. – Mike Strobel Sep 19 '17 at 12:52
  • @MikeStrobel I will give that ago... and let you know what happens... Thanks for the comment. – Karl Sep 19 '17 at 13:21
  • @BurnsBA Agreed, I would find it strange that Array.Copy would be faulty... but I am out of ideas. The Packet array is passed in as a parameter as "RxBuffer.Skip(Offset).ToArray()". My understanding is that ToArray() creates a new array and therefore should not be manipulated elsewhere? I've added the calling code my questions above. – Karl 14 mins ago – Karl Sep 19 '17 at 13:36
  • @HansPassant Thanks for your comment. Array.Resize is resizing Payload not Packet. And since I am using Array.Copy to move data into it, it don't think that should be a problem? Do you agree? – Karl Sep 20 '17 at 07:11
  • @BurnsBA and MikeStrobel The problem is not Array.Copy. I have managed to work out that at the point of the exception being thrown the pointer pFrame and indeed pFirstByte no longer point to the first byte of Packet. Packet still contains the correct data, but in a different location. I thought this should be impossible when using the fixed keyword? Any ideas? – Karl Sep 21 '17 at 10:59
  • Two thoughts, but these are just guesses 1) I'm thinking you need another `fixed` block for pFrame and 2) I might try tweaking the variables so that Packet gets `fixed` instead of an index into Packet (e.g. Packet[0]) 3) "indexer" has higher precedence than "address of" so `&Packet[0]` is fine – BurnsBA Sep 21 '17 at 12:12
  • @BurnsBA I tried your suggestion with no luck... but have found a work around for now. See my answer below. – Karl Sep 26 '17 at 07:33

1 Answers1

0

OK, so I managed to track down the error causing the access violation. It was not in the end Array.Copy it self, but rather the attempt to access pFrame->Size.

I managed to break and debug this and for some (still unknown to me) reason pFrame and pFirstByte pointers no longer point at Packet. Accessing Packet is still possible and all its elements are still correct, but for some reason it appears Packet has been moved. Now I thought this was not possible because of the fixed keyword (I tried all sort of variations) but to no avail.

In the end I decided something was not working with the pointers and the fixed statement, so as a alternative solution I decided to pin and copy the data out of Packet in one statement without using "unsafe", "fixed" and pointers.

I now achieve this using

            WebFrameStruct Frame;
            GCHandle handle = GCHandle.Alloc(Packet, GCHandleType.Pinned);
            Frame = Marshal.PtrToStructure<WebFrameStruct>(handle.AddrOfPinnedObject());
            handle.Free(); 

I decided to do it the managed and "safe" way. No need for unsafe code. Previously it died every 10-50k connections. But I have now run it up to 4.5M connections with no errors. So I will be using this as my solution. Thanks to all for the comments.

This post made me think of using this instead... and I liked it as it avoided the use of "unsafe" code... Reading a C/C++ data structure in C# from a byte array

Karl
  • 1
  • 3