14

C++ has the following function to receive bytes from socket, it can check for number of bytes available with the MSG_PEEK flag. With MSG_PEEK, the returned value of 'recv' is the number of bytes available in socket:

#include <sys/socket.h>
ssize_t recv(int socket, void *buffer, size_t length, int flags);

I need to get the number of bytes available in the socket without creating buffer (without allocating memory for buffer). Is it possible and how?

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
jondinham
  • 8,271
  • 17
  • 80
  • 137

4 Answers4

39

You're looking for is ioctl(fd,FIONREAD,&bytes_available) , and under windows ioctlsocket(socket,FIONREAD,&bytes_available).

Be warned though, the OS doesn't necessarily guarantee how much data it will buffer for you, so if you are waiting for very much data you are going to be better off reading in data as it comes in and storing it in your own buffer until you have everything you need to process something.

To do this, what is normally done is you simply read chunks at a time, such as

char buf[4096];
ssize_t bytes_read;
do {
     bytes_read = recv(socket, buf, sizeof(buf), 0);
     if (bytes_read > 0) {
         /* do something with buf, such as append it to a larger buffer or
          * process it */
     }
} while (bytes_read > 0);

And if you don't want to sit there waiting for data, you should look into select or epoll to determine when data is ready to be read or not, and the O_NONBLOCK flag for sockets is very handy if you want to ensure you never block on a recv.

hexist
  • 5,151
  • 26
  • 33
  • the incoming data to socket is supposed to be "ongoing stream of bytes" or a "series of tcp packages"? – jondinham Oct 20 '12 at 03:16
  • peeking with maximum number of bytes (handle-able by programme) will make a double job i think, coz when we actually deal with the data, we read the second time – jondinham Oct 20 '12 at 03:18
  • 1
    Correct, which is why the first method is preferable, although unless you are operating at Gbps speeds then you'll never notice it.. – hexist Oct 20 '12 at 03:21
  • 1
    Incoming data will trickle in. If the packets sent by the other end are small, they'll more than likely appear all at once, but it's by no means a guarantee. So it's an ongoing stream of bytes from your perspective as the reciever, and it's up to you to know when you've received all the bytes you are expecting. – hexist Oct 20 '12 at 03:24
  • You *can* know how much data is immediately available, which is what he is asking about. There aren't many valid uses of this information, but it does exist. – user207421 Oct 21 '12 at 22:36
  • 1
    Note that when using UDP, i.e. SOCK_DGRAM, you should read the send message at once as partial reads will discard the excess length; so in that case ioctl FIONREAD seems the way to go. – Klamer Schutte Dec 04 '14 at 10:40
  • 2
    @KlamerSchutte On SOCK_DGRAM you can use MSG_PEEK | MSG_TRUNK. Peek means the data is not discarded and trunk means the actual size of the message is returned even if larger than the buffer. Then simply pass in a 1 byte buffer and ignore it. – Goswin von Brederlow Dec 13 '18 at 14:50
3

On Windows, you can use the ioctlsocket() function with the FIONREAD flag to ask the socket how many bytes are available without needing to read/peek the actual bytes themselves. The value returned is the minimum number of bytes recv() can return without blocking. By the time you actually call recv(), more bytes may have arrived.

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
  • 1
    This answer is flat out wrong. It's quite clear in the documentation that `FIONREAD`, on Windows, does *not* return the number of bytes that are available from `recv` without blocking. On Windows, `recv` can pull data from lower layers (such as filters and queues with their own buffers) while `FIONREAD` does not do so and only checks the top layer. See [item 12](https://tangentsoft.net/wskfaq/articles/lame-list.html). – David Schwartz Jan 10 '17 at 20:18
  • Imagine if the data goes through a filter when `recv` is called and the filter has no reversal capability. How can `FIONREAD` possibly tell how many bytes `recv` could read without invoking that filter? And how can it invoke the filter if the filter has no reversal capability? This *cannot* work on Windows. (Worse, it *may* work when you try it because your system may not happen to have such a driver loaded. But when your code runs on other people's computers ... boom.) – David Schwartz Jan 10 '17 at 20:19
  • 2
    @DavidSchwartz: [per MSDN](https://msdn.microsoft.com/en-us/library/windows/desktop/ms738573.aspx): "*FIONREAD: Use to determine **the amount of data pending in the network's input buffer** that can be read from socket `s`... **`FIONREAD` returns the amount of data that can be read in a single call to the `recv` function**, which may not be the same as the total amount of data queued on the socket.*" – Remy Lebeau Jan 10 '17 at 21:28
  • @RemyLebeau Sadly, that's incorrect for the reasons I explained in my two comments. The `FIONREAD` call cannot do anything that is irreversible while `recv` can. So it may return a smaller number. Microsoft support of non-native socket operations is spotty in many ways and this is one of them. This is why it is highly recommended that software meant to run on Windows use the Windows APIs, not the ones meant to be POSIXy. – David Schwartz Jan 10 '17 at 22:18
  • @DavidSchwartz: I don't know anything about network filters, bu if you are reading the data anyway, what does it matter if it goes through a filter or not? The data is going in one direction and ends up in the socket's input buffer. I don't see why reversal logic is a limiting factor, unless maybe if `recv()` is being called with `MSG_PEEK`, but that is not common, and besides I would expect that to read the data through the filter and store it in the buffer anyway, and then just peek the buffer and leave the data there for a later non-peek read to remove it. – Remy Lebeau Jan 10 '17 at 22:25
  • @RemyLebeau Consider a compress/expand filter: Say fifty bytes are received from the socket in some compressed format and that each of them expand to enough data to fill the buffer that `recv` pulls from. So 49 of those bytes are still in some low-level buffer and the buffer `recv` pulls from is full. How can `FIONREAD` know how many bytes `recv` would return? It is impossible. The filter can't know what the later bytes expand to without expanding the earlier ones. While `recv` will refill the buffer when it's empty. This really does happen on Windows. – David Schwartz Jan 10 '17 at 22:30
  • @DavidSchwartz: if one compressed byte expands to fill the entire input buffer, I don't see why `FIONREAD` would not simply return the input buffer size. I don't expect it to account for later bytes until after the socket has to go back to the low level to read more data and it decompresses the next byte into the input buffer. `recv()` will not block if the input buffer has any final bytes in it, unless `MSG_WAITALL` is used – Remy Lebeau Jan 10 '17 at 22:38
  • @RemyLebeau Yes, exactly. While `recv` will be able to get all the bytes because as soon as the buffer is empty (copied into the buffer the user supplies in the call to `recv`), it attempts to refill it from lower layers. Thus `FIONREAD` *cannot* return as many bytes as `recv` might be able to because it cannot do anything irreversible like invoking the filter. So `FIOREAD` will return the X bytes in the highest level buffer, `recv` could return up to 50*X bytes by invoking the filter to refill the buffer from the received bytes in a lower-level buffer. – David Schwartz Jan 10 '17 at 22:39
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/132854/discussion-between-remy-lebeau-and-david-schwartz). – Remy Lebeau Jan 10 '17 at 22:41
3

Be careful when using FIONREAD! The problem with using ioctl(fd, FIONREAD, &available) is that it will always return the total number of bytes available for reading in the socket buffer on some systems.

This is no problem for STREAM sockets (TCP) but misleading for DATAGRAM sockets (UDP). As for datagram sockets read requests are capped to the size of the first datagram in the buffer and when reading less than she size of the first datagram, all unread bytes of that datagram are still discarded. So ideally you want to know only the size of the next datagram in the buffer.

E.g. on macOS/iOS it is documented that FIONREAD always returns the total amount (see comments about SO_NREAD). To only get the size of the next datagram (and total size for stream sockets), you can use the code below:

int available;
socklen_t optlen = sizeof(readable);
int err = getsockopt(soc, SOL_SOCKET, SO_NREAD, &available, &optlen);

On Linux FIONREAD is documented to only return the size of the next datagram for UDP sockets.

On Windows ioctlsocket(socket, FIONREAD, &available) is documented to always give the total size:

If the socket passed in the s parameter is message oriented (for example, type SOCK_DGRAM), FIONREAD returns the reports the total number of bytes available to read, not the size of the first datagram (message) queued on the socket.

Source: https://learn.microsoft.com/en-us/windows/win32/api/ws2spi/nc-ws2spi-lpwspioctl

I am unaware of a way how to get the size of the first datagram only on Windows.

Mecki
  • 125,244
  • 33
  • 244
  • 253
1
The short answer is : this cannot be done with MS-Windows WinSock2,
as I can discovered over the last week of trying. 

Glad to have finally found this post, which sheds some light on the issues I've been having, using latest Windows 10 Pro, version 20H2 Build 19042.867 (x86/x86_64) :

On a bound, disconnected UDP socket 'sk' (in Listening / Server mode):

1. Any attempt to use either ioctlsocket(sk, FIONREAD, &n_bytes)
   OR WsaIoctl with a shifted FIONREAD argument, though they succeed,
   and retern 0, after a call to select() returns > with that 
  'sk' FD bit set in the read FD set,
   and the ioctl call returns 0 (success), and n_bytes is > 0, 
   causes the socket sk to be in a state where any
   subsequent call to recv(), recvfrom(), or ReadFile() returns 
   SOCKET_ERROR with a WSAGetLastError() of :
   10045, Operation Not Supported, or ReadFile
   error 87, 'Invalid Parameter'.

Moreover, even worse:

2. Any attempt to use recv or recvfrom with the 'MSG_PEEK' msg_flags 
   parameter returns -1 and WSAGetLastError returns :
   10040 : 'A message sent on a datagram socket was larger than 
    the internal message buffer or some other network limit, 
    or the buffer used to receive a datagram into was smaller 
    than the datagram itself.
       ' .

   Yet for that socket I DID successfully call:
setsockopt(s, SOL_SOCKET, SO_RCVBUF, bufsz = 4096 , sizeof(bufsz) )
   and the UDP packet being received was of only 120 bytes in size.

   In short, with modern windows winsock2 ( winsock2.h / Ws2_32.dll) , 
   there appears to be absolutely no way to use any documented API
   to determine the number of bytes received on a bound UDP socket
   before calling recv() / recvfrom() in MSG_WAITALL blocking mode to
   actually receive the whole packet.

   If you do not call ioctlsocket() or WsaIoctl or 
   recv{,from}(...,MSG_PEEK,...) 
   before entering recv{,from}(...,MSG_WAITALL,...) ,
   then the recv{,from} succeeds.

   I am considering advising clients that they must install and run
   a Linux instance with MS Services for Linux under their windows
   installation , and developing some
   API to communicate with it from Windows, so that reliable 
   asynchronous UDP communication can be achieved - or does anyone 
   know of a good open source replacement for WinSock2 ? 

   I need access to a "C" library TCP+UDP/IP implementation for 
   modern Windows 10  that conforms to its own documentation, 
   unlike WinSock2 -  does anyone know of one ?
JVD
  • 645
  • 1
  • 7
  • 17