TcpClient receiving bytes that aren't being sent?

Question

I'm writing a program that uses a TcpListener as the server. Multiple clients connect to it and send/receive packets.

These Packets are just serialized custom classes containing data. I use a BinaryFormatter to serialize and deserialize packets:

[Serializable]
public class Packet
{
    public static implicit operator byte[](Packet p)
    {
        using (var memoryStream = new MemoryStream())
        {
            new BinaryFormatter().Serialize(memoryStream, p);
            return memoryStream.ToArray();
        }
    }

    protected byte[] Serialize(object o)
    {
        using (var memoryStream = new MemoryStream())
        {
            new BinaryFormatter().Serialize(memoryStream, o);
            return memoryStream.ToArray();
        }
    }

    public static object Deserialize(byte[] data)
    {
        using (var memoryStream = new MemoryStream(data))
            return new BinaryFormatter().Deserialize(memoryStream);
    }
}

Here is an example of a packet I send:

[Serializable]
    public class Message : Packet
    {
        public string Text { get; set; }
        public Message(string text)
        {
            Text = text;
        }
        public byte[] Serialize()
        {
            return Serialize(this);
        }
    }

Now the way I send this data is I first create a Header which specifies the length of the packet we're serializing, and then sends that header to the NetworkStream, and then following that, it sends the actual packet.

I wrote an extension class for such tasks:

public static class StreamExtension
{
    public static async Task WritePacket(this NetworkStream stream, byte[] packet)
    {
        var header = BitConverter.GetBytes(packet.Length); //Get packet length.
        await stream.WriteAsync(header, 0, header.Length); //Write packet length as header.
        await stream.WriteAsync(packet, 0, packet.Length); //Write actual packet.
        Console.WriteLine($"Writing packet with length {packet.Length}");
    }
}

Now when I attempt to connect, it works fine with the first client that connects, it sends and receives all the necessary packets. But when the second client joins, it ends up trying to deserialize packets that aren't even being sent.

I created a try catch to see what the problem was, and I get an exception saying:

binary stream '0' does not contain a valid binaryheader

This is the method that's receiving data in the client:

private async void ReceiveData()
        {
            var header = new byte[4]; //Indicator for how big our actual packet is.
            var stream = Server.GetStream();
            while (await stream.ReadAsync(header, 0, header.Length) != 0)
            {
                var packetLength = BitConverter.ToInt32(header, 0);
                var buffer = new byte[packetLength];
                await stream.ReadAsync(buffer, 0, buffer.Length); //Read packet.
                try {
                    var receivedPacket = Packet.Deserialize(buffer);
                    lbLog.Items.Add($"Received packet: {receivedPacket.GetType()}");
                    if (receivedPacket is SetPiece)
                    {
                        var setPiece = receivedPacket as SetPiece;
                        MyPiece = setPiece.Piece;
                        lblPiece.Text = $"You are {MyPiece}";
                    }
                    else if (receivedPacket is StartNew)
                    {
                        foreach (var box in Controls.OfType<PictureBox>())
                            box.Image = null;
                    }
                    else if (receivedPacket is SetTurn)
                    {
                        var setTurn = receivedPacket as SetTurn;
                        IsTurn = setTurn.IsMine;
                        lblTurn.Text = setTurn.IsMine ? "Your Turn" : $"{setTurn.Name}'s Turn";
                    }
                    else if (receivedPacket is Log)
                    {
                        var log = receivedPacket as Log;
                        lbLog.Items.Add(log.Message);
                    }
                }
                catch
                {
                    File.AppendAllLines("errors.txt", new[] { $"Error occurred with packet. Length of {buffer.Length}" });
                }
            }
        }

Here is the log of the server console:

And here is the error log:

Error occurred with packet. Length of 162
Error occurred with packet. Length of 256
Error occurred with packet. Length of 1952797486

How can we have an error with a packet that has a length of 1952797486, when it never even gets sent?

Any ideas on what I'm doing wrong?