I'm seeing a couple strange things with a pair of AF_UNIX sockets created by a call such as:
socketpair(AF_UNIX, SOCK_STREAM, 0, sfd);
Where sfd is an int[2] array for the file descriptors.
First, the default buffer size seems to be exactly 122K (124928 bytes), rather than anything from /proc/sys/net (such as wmem_default which is set to 128K). Does anyone know the cause of this strange buffer size?
Second, when writing small messages through the socket (8 bytes). I can only write 423 of them before the write blocks, which is only 8*423 = 3384 bytes, another odd size. The messages are acting as though they're taking up 295 + a little bytes each. What's the source of this overhead?
Running on RHEL6 (2.6.32, 64-bit)
I wrote a program to try different sizes of data to compare overhead costs:
#include <errno.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#define DATA_SIZE 4
void run(size_t size) {
int sfd[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, sfd) == -1) {
perror("error");
}
int sndbuf, sbsize = sizeof(sndbuf);
getsockopt(sfd[0], SOL_SOCKET, SO_SNDBUF, &sndbuf, (socklen_t*)&sbsize);
printf("Data Size: %zd\n", size);
char buff[size];
size_t wrote=0;
for (size_t ii=0; ii < 32768; ii++) {
if ((send(sfd[0], buff, size, MSG_DONTWAIT) == -1) && (errno == EAGAIN)) {
wrote = ii;
break;
}
}
printf("Wrote: %zd\n", wrote);
if (wrote != 0) {
int bpm = sndbuf/wrote;
int oh = bpm - size;
printf("Bytes/msg: %i\n", bpm);
printf("Overhead: %i\n", oh);
printf("\n");
}
close(sfd[0]); close(sfd[1]);
}
int main() {
int sfd[2];
socketpair(AF_UNIX, SOCK_STREAM, 0, sfd);
int sndbuf, sbsize = sizeof(sndbuf);
getsockopt(sfd[0], SOL_SOCKET, SO_SNDBUF, &sndbuf, (socklen_t*)&sbsize);
printf("Buffer Size: %i\n\n", sndbuf);
close(sfd[0]); close(sfd[1]);
for (size_t ii=4; ii <= 4096; ii *= 2) {
run(ii);
}
}
Which gives:
Buffer Size: 124928
Data Size: 4
Wrote: 423
Bytes/msg: 295
Overhead: 291
Data Size: 8
Wrote: 423
Bytes/msg: 295
Overhead: 287
Data Size: 16
Wrote: 423
Bytes/msg: 295
Overhead: 279
Data Size: 32
Wrote: 423
Bytes/msg: 295
Overhead: 263
Data Size: 64
Wrote: 423
Bytes/msg: 295
Overhead: 231
Data Size: 128
Wrote: 348
Bytes/msg: 358
Overhead: 230
Data Size: 256
Wrote: 256
Bytes/msg: 488
Overhead: 232
Data Size: 512
Wrote: 168
Bytes/msg: 743
Overhead: 231
Data Size: 1024
Wrote: 100
Bytes/msg: 1249
Overhead: 225
Data Size: 2048
Wrote: 55
Bytes/msg: 2271
Overhead: 223
Data Size: 4096
Wrote: 29
Bytes/msg: 4307
Overhead: 211
Versus using a pipe there's definitely a lot of overhead:
Data Size: 4
Wrote: 16384
Bytes/msg: 4
Overhead: 0
Data Size: 8
Wrote: 8192
Bytes/msg: 8
Overhead: 0
Data Size: 16
Wrote: 4096
Bytes/msg: 16
Overhead: 0
Data Size: 32
Wrote: 2048
Bytes/msg: 32
Overhead: 0
Data Size: 64
Wrote: 1024
Bytes/msg: 64
Overhead: 0
Data Size: 128
Wrote: 512
Bytes/msg: 128
Overhead: 0
Data Size: 256
Wrote: 256
Bytes/msg: 256
Overhead: 0
Data Size: 512
Wrote: 128
Bytes/msg: 512
Overhead: 0
Data Size: 1024
Wrote: 64
Bytes/msg: 1024
Overhead: 0
Data Size: 2048
Wrote: 32
Bytes/msg: 2048
Overhead: 0
Data Size: 4096
Wrote: 16
Bytes/msg: 4096
Overhead: 0
