Virtual USB-serial port always failing after 0x3800 bytes received

Question

I have a device sending a constant stream of data to my PC which is receiving it over a virtual USB-serial port. On the PC I use a C# program to read the virtual USB-serial port. After receiving 0x3800 bytes (14336 bytes or 14 kB) receiving stops. (See source code below on a simple version of the test program)

OS: windows 10 (also tried on a Windows 7 system, same results) Compiler: Visual Studio 2015 Com port settings: COM3, 9600/8/N/1, no handshake.

I used 'serial port monitor' which shows that data is received after 0x3800 bytes, communication over USB between the device and the PC is not breaking down.

I've searched (for several days now) with google in github and SourceForge for possible other solutions, found nothing usefull.

The simplified version of the code below was changed in several ways to check:

• different port configurations,
• different buffer sizes at device level,
• several Nuget packages which provide virtual USB-serial port objects, most read nothing at all or fail at the same point. One package could read 0x4000 bytes.
• a C++ version (which can connect but does not read one byte).
• Tried a real RS232 port and another PC with a data sourse. No problems.

How is it possible that a tool like 'serial port monitor' can read without problems and my simple C# program not? There are several assumptions I can make about it:

• There is a configuration error in my code (which I tried very hard to find).
• The two use different access paths to get to the virtual USB-serial data. (If so, what other ways are there to connect to a virtual USB-serial port?)

Added a callback OnErrorReceived. This is never trigered.

I'm stuck, I do not know what to try of how to get communication going. Any advise is welcome.

Here is some simple code to show what I use. (Eddited to follow up a suggestion from JHBonarius)

using System;
using System.Threading;

namespace TestConsole
{
    class Program
    {
        const string PORT_NAME = "COM3";
        const int BAUD_RATE = 9600;

        static System.IO.Ports.SerialPort port;

        static void Main(string[] args)
        {
            Console.WriteLine($"] port connect {PORT_NAME}");
            port = new System.IO.Ports.SerialPort(PORT_NAME, BAUD_RATE);
            port.DataReceived += OnRx;
            port.ErrorReceived += OnErrorReceived;
            port.DtrEnable = true;  // required for communiation
            port.Open();

            // and wait forever.
            while (true) Thread.Sleep(1000);
        }

        private static void OnErrorReceived(object sender, System.IO.Ports.SerialErrorReceivedEventArgs e)
        {
            Console.WriteLine();
            Console.WriteLine($"] port error = {e.EventType}");
            Console.WriteLine();
        }

        static void OnRx(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
        {
            System.IO.Ports.SerialPort sp = (System.IO.Ports.SerialPort)sender;
            while (0 < sp.BytesToRead)
            {
                var b = sp.ReadByte();
                if (-1 < b) ConsoleWriteByte((byte)b);
                else break;
            }
        }

        const int BYTES_PER_LINE = 16;

        static int count = 0;
        static int line = 0;

        static void ConsoleWriteByte(byte value)
        {
            if (count % BYTES_PER_LINE == 0)
            {
                Console.WriteLine();
                Console.Write($"{line:x6}:");
                count = 0;
                line += BYTES_PER_LINE;
            }

            Console.Write($" {value:x2}");
            count++;
        }
    }
}

Answer 1

I did some more testing on this problem and got a hold on the code used at the sender side.

It apears that the sender side is transmitting data very fast and is monitoring the connection at USB level. If there are any problems such as buffer overflow the transmission stops.

If I increase buffer size at the receiving end I can handle more data but the C++ solution runs into problems.

I do not know how to capture the data faster or in a different way. I can change code at the receiver side but not at the sender side. So, adding handshake signales is (unfortunately) not an option but it would solve the problem.

Case closed.

edit 2022-oct-7 hmm... not yet fully closed.

Today the problem came back to me again. Played around a bit with the code, not changing anyting. The C++ test program was running for more than 60 minutes uniterupted. starting some other applications made it stop.

Looked around on how to set priority and this. Added a line SetPriorityClass(GetCurrentProcess(), HIGH_PRIORITY_CLASS) and that version ran without problems for more than three hours.

Changing process priority makes a big difference.

Answer 2

To exclude possible problems from .NET SerialPort I've written a C++ version.

While doing so I noticed that many settings in the DCB structure not only do not matter but if changed are causing problems. So,... a baud rate of 0 bytes looks strange but is actually working for a virtual USB-serial port.

Monitoring COMSTAT data I can see that there is data available, exact 0x3800 bytes. And also that ReadFile(...) is not using the data.

Why does ReadFile(...) not want to read the data? I've not yet figured out why.

Here are the results that I get.

COM port = "COM3", changed
DCB DCBlength         = 0x1c = 28
    BaudRate          = 0
    fBinary           = 1
    fParity           = 0
    fOutxCtsFlow      = 0
    fOutxDsrFlow      = 0
    fDtrControl       = 1
    fDsrSensitivity   = 0
    fTXContinueOnXoff = 0
    fOutX             = 0
    fInX              = 0
    fErrorChar        = 0
    fNull             = 0
    fRtsControl       = 1
    fAbortOnError     = 0
    XonLim            = 0x0
    XoffLim           = 0x0
    ByteSize          = 0
    Parity            = 0
    StopBits          = 0
    XonChar           = 0x0
    XoffChar          = 0x0
    ErrorChar         = 0x0
    EofChar           = 0x0
    EvtChar           = 0x0

errors code       = 0x0
comstat fCtsHole  = 0
        fDsrHold  = 0
        fRlsdHold = 0
        fXoffHold = 0
        fXoffSent = 0
        fEof      = 0
        fTxim     = 0
        InQue     = 14336
        OutQue    = 0

Here is the code for my C++ version.

#include "stdafx.h"
#include <iostream>
#include <iomanip>
#include <windows.h>

#define COM_PORT "COM3" /* "\\\\.\\COM3" */

#define BUFFER_SIZE 1024
#define BYTES_PER_LINE 16

int count = 0;
int byte_count = 0;

void OnRx(byte b)
{
    if (count % BYTES_PER_LINE == 0)
    {
        std::cout << std::endl << std::setw(6) << std::hex << byte_count << ":";
        count = 0;
        byte_count += BYTES_PER_LINE;
    }
    std::cout << " " << std::setw(2) << std::hex << b;
}

void LogDCB(const DCB &serialParams,  const char * state)
{
    std::cout << std::endl
              << "COM port = \"" << COM_PORT << "\", " << state << std::endl
              << std::hex
              << "DCB DCBlength         = 0x" << std::setw(2) << serialParams.DCBlength
              << std::dec
              << " = " << serialParams.DCBlength << std::endl;
    std::cout << "    BaudRate          = " << serialParams.BaudRate << std::endl;
    std::cout << "    fBinary           = " << serialParams.fBinary << std::endl;
    std::cout << "    fParity           = " << serialParams.fParity << std::endl;
    std::cout << "    fOutxCtsFlow      = " << serialParams.fOutxCtsFlow << std::endl;
    std::cout << "    fOutxDsrFlow      = " << serialParams.fOutxDsrFlow << std::endl;
    std::cout << "    fDtrControl       = " << serialParams.fDtrControl << std::endl;
    std::cout << "    fDsrSensitivity   = " << serialParams.fDsrSensitivity << std::endl;
    std::cout << "    fTXContinueOnXoff = " << serialParams.fTXContinueOnXoff << std::endl;
    std::cout << "    fOutX             = " << serialParams.fOutX << std::endl;
    std::cout << "    fInX              = " << serialParams.fInX << std::endl;
    std::cout << "    fErrorChar        = " << serialParams.fErrorChar << std::endl;
    std::cout << "    fNull             = " << serialParams.fNull << std::endl;
    std::cout << "    fRtsControl       = " << serialParams.fRtsControl << std::endl;
    std::cout << "    fAbortOnError     = " << serialParams.fAbortOnError << std::endl;
    std::cout << std::hex
              << "    XonLim            = 0x" << serialParams.XonLim << std::endl;
    std::cout << "    XoffLim           = 0x" << serialParams.XoffLim << std::endl;
    std::cout << std::dec
              // byte values of 0 are not visible, casting to unsigned int helps.
              << "    ByteSize          = " << static_cast<unsigned int>(serialParams.ByteSize) << std::endl;
    std::cout << "    Parity            = " << static_cast<unsigned int>(serialParams.Parity) << std::endl;
    std::cout << "    StopBits          = " << static_cast<unsigned int>(serialParams.StopBits) << std::endl;
    std::cout << std::hex
              // char values are not displayed as values, double cast required to loose the sign and char type info to fix it.
              << "    XonChar           = 0x" << static_cast<unsigned int>(static_cast<unsigned char>(serialParams.XonChar)) << std::endl;
    std::cout << "    XoffChar          = 0x" << static_cast<unsigned int>(static_cast<unsigned char>(serialParams.XoffChar)) << std::endl;
    std::cout << "    ErrorChar         = 0x" << static_cast<unsigned int>(static_cast<unsigned char>(serialParams.ErrorChar)) << std::endl;
    std::cout << "    EofChar           = 0x" << static_cast<unsigned int>(static_cast<unsigned char>(serialParams.EofChar)) << std::endl;
    std::cout << "    EvtChar           = 0x" << static_cast<unsigned int>(static_cast<unsigned char>(serialParams.EvtChar)) << std::endl
              << std::dec;
}

int main()
{
    // see: https://learn.microsoft.com/nl-nl/windows/win32/devio/monitoring-communications-events
    // see: https://www.codeproject.com/Articles/2682/Serial-Communication-in-Windows
    // see: https://web.archive.org/web/20180127160838/http:/bd.eduweb.hhs.nl/micprg/pdf/serial-win.pdf

    DWORD dw;

    // Open serial port
    HANDLE serialHandle = CreateFile(
        COM_PORT,                     // FileName
        GENERIC_READ | GENERIC_WRITE, // DesiredAccess: { GENERIC_READ, GENERIC_WRITE, ... }
        0,                            // ShareMode: If this parameter is zero ..., the file or device
                                      // cannot be shared and cannot be opened again until the handle
                                      // to the file or device is closed.
                                      // { 0, FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE }
        nullptr,                      // SecurityAttributes: This parameter can be NULL
        OPEN_EXISTING,                // CreationDisposition: For devices other than files, this
                                      // parameter is usually set to OPEN_EXISTING.
                                      // { CREATE_ALWAYS, CREATE_NEW, OPEN_ALWAYS, OPEN_EXISTING, TRUNCATE_EXISTING }
        FILE_FLAG_OVERLAPPED,         // FlagsAndAttributes: The file or device attributes and flags,
                                      // FILE_ATTRIBUTE_NORMAL being the most common default value for files.
                                      //This parameter can also contain combinations of flags (FILE_FLAG_*) for
                                      // control of file or device caching behavior, access modes, and other
                                      // special-purpose flags. 
                                      // Some of the following file attributes and flags may only apply to
                                      // files and not necessarily all other types of devices
                                      // { FILE_ATTRIBUTE_ARCHIVE, FILE_ATTRIBUTE_ENCRYPTED, FILE_ATTRIBUTE_HIDDEN,
                                      //   FILE_ATTRIBUTE_NORMAL, FILE_ATTRIBUTE_OFFLINE, FILE_ATTRIBUTE_READONLY,
                                      //   FILE_ATTRIBUTE_SYSTEM, FILE_ATTRIBUTE_TEMPORARY } | {
                                      //   FILE_FLAG_BACKUP_SEMANTICS, FILE_FLAG_DELETE_ON_CLOSE, FILE_FLAG_NO_BUFFERING,
                                      //   FILE_FLAG_OPEN_NO_RECALL, FILE_FLAG_OPEN_REPARSE_POINT, FILE_FLAG_OVERLAPPED,
                                      //   FILE_FLAG_POSIX_SEMANTICS, FILE_FLAG_RANDOM_ACCESS, FILE_FLAG_SESSION_AWARE,
                                      //   FILE_FLAG_SEQUENTIAL_SCAN, FILE_FLAG_WRITE_THROUGH } | {
                                      //   SECURITY_ANONYMOUS, SECURITY_CONTEXT_TRACKING, SECURITY_DELEGATION,
                                      //   SECURITY_EFFECTIVE_ONLY, SECURITY_IDENTIFICATION, SECURITY_IMPERSONATION }
        nullptr);                     // TemplateFile: A valid handle to a template file with the GENERIC_READ access right.
                                      // This parameter can be NULL.
    if (serialHandle == INVALID_HANDLE_VALUE)
    {
        dw = GetLastError();
        std::cout << "Cannot open port \"" << COM_PORT << "\", error code = " << std::setw(2) << std::hex << dw << std::dec << " = " << dw << std::endl;
        return 0;
    }

    // Set DCB values.
    DCB serialParams = { 0 };
    serialParams.DCBlength = sizeof(serialParams);
    GetCommState(serialHandle, &serialParams);
    LogDCB(serialParams, "defaults");
    bool DCB_changed = false;

    //serialParams.XonLim = 0x0800;
    //serialParams.XoffLim = 0x0200;

    serialParams.fDtrControl = DTR_CONTROL_ENABLE;
    //serialParams.fRtsControl = RTS_CONTROL_HANDSHAKE;

    //serialParams.XonChar = 0x11;
    //serialParams.XoffChar = 0x13;
    DCB_changed = true;

    if (DCB_changed)
    {
        try
        {
            SetLastError(0);
            SetCommState(serialHandle, &serialParams);
        }
        catch (...)
        {
            dw = GetLastError();
            std::cout << std::endl << "SetCommState error code = " << std::setw(2) << std::hex << dw << std::dec << std::endl;
        }
        GetCommState(serialHandle, &serialParams);
        LogDCB(serialParams, "changed");
    }

    // Reset buffers and raise DTR.
    EscapeCommFunction(serialHandle, CLRDTR);
    FlushFileBuffers(serialHandle);
    EscapeCommFunction(serialHandle, SETDTR);

    // Set timeouts
    COMMTIMEOUTS timeout = { 0 };
    timeout.ReadIntervalTimeout = 50;
    timeout.ReadTotalTimeoutConstant = 50;
    timeout.ReadTotalTimeoutMultiplier = 50;
    timeout.WriteTotalTimeoutConstant = 50;
    timeout.WriteTotalTimeoutMultiplier = 10;

    SetCommTimeouts(serialHandle, &timeout);

    OVERLAPPED overlapped;

    char buffer[BUFFER_SIZE];
    DWORD count;
    BOOL result;
    DWORD i;
    DWORD errors;
    COMSTAT comstat;

    for (int tries = 0; tries < 10;)
    {
        result = ReadFile(
            serialHandle,   // File
            &buffer,        // Buffer
            BUFFER_SIZE,    // NumberOfBytesToRead
            &count,         // NumberOfBytesRead
            &overlapped     // Overlapped
        );
        if (result)
        {
            tries = 0;
            for (i = 0; i < count; i++) OnRx(buffer[i]);
        }
        else
        {
            Sleep(100);
            tries++;
        }
        if (ClearCommError(serialHandle, &errors, &comstat))
        {
            std::cout << std::endl
                      << "errors code       = 0x" << std::hex << errors << std::dec << std::endl;
            std::cout << "comstat fCtsHole  = " << comstat.fCtsHold << std::endl;
            std::cout << "        fDsrHold  = " << comstat.fDsrHold << std::endl;
            std::cout << "        fRlsdHold = " << comstat.fRlsdHold << std::endl;
            std::cout << "        fXoffHold = " << comstat.fXoffHold << std::endl;
            std::cout << "        fXoffSent = " << comstat.fXoffSent << std::endl;
            std::cout << "        fEof      = " << comstat.fEof << std::endl;
            std::cout << "        fTxim     = " << comstat.fTxim << std::endl;
            std::cout << "        InQue     = " << comstat.cbInQue << std::endl;
            std::cout << "        OutQue    = " << comstat.cbOutQue << std::endl;
        }
        else
        {
            dw = GetLastError();
            std::cout << std::endl << "SetCommState error code = 0x" << std::hex << dw << std::dec << " = " << dw << std::endl;
        }
    }

    CloseHandle(serialHandle);

    std::cout << "Press Enter to continue..." << std::endl;
    std::cin.get();
    return 0;
}