SocketCAN select() and write() don't block

Question

I'm testing the CAN interface on an embedded device (SOC / ARM core / Linux) using SocketCAN, and I want to send data as fast as possible for testing, using efficient code.

I can open the CAN device ("can0") as a BSD socket, and send frames with "write". This all works well.

My desktop can obviously generate frames faster than the CAN transmission rate (I'm using 500000 bps). To send efficiently, I tried using a "select" on the socket file descriptor to wait for it to become ready, followed by the "write". However, the "select" seems to return immediately regardless of the state of the send buffer, and "write" also doesn't block. This means that when the buffer fills up, I get an error from "write" (return value -1), and errno is set to 105 ("No buffer space available").

This mean I have to wait an arbitrary amount of time, then try the write again, which seems very inefficient (polling!).

Here's my code (C, edited for brevity):

printf("CAN Data Generator\n");

int skt;      // CAN raw socket
struct sockaddr_can addr;
struct canfd_frame frame;

const int WAIT_TIME = 500;

// Create socket:
skt = socket(PF_CAN, SOCK_RAW, CAN_RAW);

// Get the index of the supplied interface name: 
unsigned int if_index = if_nametoindex(argv[1]);

// Bind CAN device to socket created above:
addr.can_family = AF_CAN;
addr.can_ifindex = if_index;
bind(skt, (struct sockaddr *)&addr, sizeof(addr));

// Generate example CAN data: 8 bytes; 0x11,0x22,0x33,...
// ...[Omitted]

// Send CAN frames:
fd_set fds;
const struct timeval timeout =  { .tv_sec=2, .tv_usec=0 };
struct timeval this_timeout;
int ret;
ssize_t bytes_writ;

while (1)
{
    // Use 'select' to wait for socket to be ready for writing:
    FD_ZERO(&fds);
    FD_SET(skt, &fds);
    this_timeout = timeout;
    ret = select(skt+1, NULL, &fds, NULL, &this_timeout);

    if (ret < 0)
    {
        printf("'select' error (%d)\n", errno);
        return 1;
    }
    else if (ret == 0)
    {
        // Timeout waiting for buffer to be free
        printf("ERROR - Timeout waiting for buffer to clear.\n");
        return 1;
    }
    else
    {
        if (FD_ISSET(skt, &fds))
        {
            // Ready to write!
            bytes_writ = write(skt, &frame, CAN_MTU);
            if (bytes_writ != CAN_MTU)
            {
                if (errno == 105)
                {
                    // Buffer full! 
                    printf("X"); fflush(stdout);
                    usleep(20);  // Wait for buffer to clear
                }
                else
                {
                    printf("FAIL - Error writing CAN frame (%d)\n", errno);
                    return 1;
                }
            }
            else
            {
                printf("."); fflush(stdout);
            }
        }
        else
        {
            printf("-"); fflush(stdout);
        }
    }
    usleep(WAIT_TIME);
}

When I set the per-frame WAIT_TIME to a high value (e.g. 500 uS) so that the buffer never fills, I see this output:

CAN Data Generator
...............................................................................
................................................................................
...etc

Which is good! At 500 uS I get 54% CAN bus utilisation (according to canbusload utility).

However, when I try a delay of 0 to max out my transmission rate, I see:

CAN Data Generator
................................................................................
............................................................X.XX..X.X.X.X.XXX.X.
X.XX..XX.XX.X.XX.X.XX.X.X.X.XX..X.X.X.XX..X.X.X.XX.X.XX...XX.X.X.X.X.XXX.X.XX.X.
X.X.XXX.X.XX.X.X.X.XXX.X.X.X.XX.X.X.X.X.XX..X..X.XX.X..XX.X.X.X.XX.X..X..X..X.X.
.X.X.XX.X.XX.X.X.X.X.X.XX.X.X.XXX.X.X.X.X..XX.....XXX..XX.X.X.X.XXX.X.XX.XX.XX.X
.X.X.XX.XX.XX.X.X.X.X.XX.X.X.X.X.XX.XX.X.XXX...XX.X.X.X.XX..X.XX.X.XX.X.X.X.X.X.

The initial dots "." show the buffer filling up; Once the buffer is full, "X" starts appearing meaning that the "write" call failed with error 105.

Tracing through the logic, this means the "select" must have returned and the "FD_ISSET(skt, &fds)" was true, although the buffer was full! (or did I miss something?).

The SockedCAN docs just say "Writing CAN frames can be done similarly, with the write(2) system call"

This post suggests using "select".

This post suggests that "write" won't block for CAN priority arbitration, but doesn't cover other circumstances.

So is "select" the right way to do it? Should my "write" block? What other options could I use to avoid polling?

Answer 1

After a quick look at canbusload:184, it seems that it computes efficiency (#data/#total bits on the bus).

On the other hand, according to this, max efficiency for CAN bus is around 57% for 8-byte frames, so you seem not to be far away from that 57%... I would say you are indeed flooding the bus.

When setting a 500uS delay, 500kbps bus bitrate, 8-byte frames, it gives you a (control+data) bitrate of 228kbps, which is lower than max bitrate of the CAN bus, so, no bottleneck here.

Also, since in this case only 1 socket is being monitored, you don't need pselect, really. All you can do with pselect and 1 socket can be done without pselect and using write.

(Disclamer: hereinafter, this is just guessing since I cannot test it right now, sorry.) As of why the behavior of pselect, think that the buffer could have byte semantics, so it tells you there is still room for more bytes (1 at least), not necessarily for more can_frames. So, when returning, pselect does not inform you can send the whole CAN frame. I guess you could solve this by using SIOCOUTQ and the max size of the Rx buffer SO_SNDBUF, but not sure if it works for CAN sockets (the nice thing would be to use SO_SNDLOWAT flags, but it is not changable in Linux's implementation).

So, to answer your questions:

1. Is "select" the right way to do it? Well, you can do it both ways, either (p)select or write, since you are only waiting for one file descriptor, there is no real difference.
2. Should my "write" block? It should if there is no single byte available in the send buffer.
3. What other options could I use to avoid polling? Maybe by ioctl'ing SIOCOUTQ and getsockopt'ing SO_SNDBUF and substracting... you will need to check this yourself. Alternatively, maybe you could set the send buffer size to a multiple of sizeof(can_frame) and see if it keeps you signaling when less than sizeof(can_frame) are available.

Anyhow, if you are interested in having a more precise timing, you could use a BCM socket. There, you can instruct the kernel to send a specific frame at a specific interval. Once set, the process run in kernel space, without any system call. In this way, user-kernel buffer problem is avoided. I would test different rates until canbusload shows no rise in bus utilization.

Answer 2

select and poll worked for me right with SocketCan. However, carefull configuration is require.

some background:

between user app and the HW, there are 2 buffers:

1. socket buffer, where its size (in bytes) is controlled by the setsockopt's SO_SNDBUF option
2. driver's qdisc, where its size (in packets) is controlled by the "ifconfig can0 txqueuelen 5" command. data path is: user app "write" command --> socket buffer -> driver's qdisc -> HW TX mailbox.

2 flow control points exist along this path:

1. when there is no free TX mailboxe, driver freeze driver's qdisc (__QUEUE_STATE_DRV_XOFF), to prevent more packets to be dequeued from driver's qdisc into HW. it will be un-freezed when TX mailbox is free (upon TX completion interrupt).
2. when socket buffer goes above half of its capacity, poll/select blocks, until socket buffer goes beyond half of its capacity.

now, assume that socket buffer has room for 20 packets, while driver's qdisc has room for 5 packets. lets assume also that HW have single TX mailbox.

1. poll/select let user write up to 10 packets.
2. those packets are moved down to socket buffer.
3. 5 of those packets continue and fill driver's qdisc.
4. driver dequeue 1st packet from driver's qdisc, put it into HW TX mailbox and freeze driver's qdisc (=no more dequeue). now there is room for 1 packet in driver's qdisc
5. 6th packet is moved down successfully from socket buffer to driver's qdisc.
6. 7th packet is moved down from socket buffer to driver's qdisc, but since there is no room - it is dropped and error 105 ("No buffer space available") is generated.

what is the solution? in the above assumptions, lets configure socket buffer for 8 packets. in this case, poll/select will block user app after 4 packets, ensuring that there is room in driver's qdisc for all of those 4 packets.

however, socket buffer is configured to bytes, not to packet. translation should be made as the following: each CAN packet occupy ~704 bytes at socket buffer (most of them for the socket structure). so, to configure socket buffer to 8 packet, the size in bytes should be 8*704:

int size = 8*704;
setsockopt(s, SOL_SOCKET, SO_SNDBUF, &size, sizeof(size));