19

In a bash script, I try to read lines from standard input, using built-in read command after setting IFS=$'\n'. The lines are truncated at 4095 character limit if I paste input to the read. This limitation seems to come from reading from terminal, because this worked perfectly fine:

fill=
for i in $(seq 1 94); do fill="${fill}x"; done
for i in $(seq 1 100); do printf "%04d00$fill" $i; done | (read line; echo $line)

I experience the same behavior with Python script (did not accept longer than 4095 input from terminal, but accepted from pipe):

#!/usr/bin/python

from sys import stdin

line = stdin.readline()
print('%s' % line)

Even C program works the same, using read(2):

#include <stdio.h>
#include <unistd.h>

int main(void)
{
    char buf[32768];
    int sz = read(0, buf, sizeof(buf) - 1);
    buf[sz] = '\0';
    printf("READ LINE: [%s]\n", buf);
    return 0;
}

In all cases, I cannot enter longer than about 4095 characters. The input prompt stops accepting characters.

Question-1: Is there a way to interactively read from terminal longer than 4095 characters in Linux systems (at least Ubuntu 10.04 and 13.04)?

Question-2: Where does this limitation come from?

Systems affected: I noticed this limitation in Ubuntu 10.04/x86 and 13.04/x86, but Cygwin (recent version at least) does not truncate yet at over 10000 characters (did not test further since I need to get this script working in Ubuntu). Terminals used: Virtual Console and KDE konsole (Ubuntu 13.04) and gnome-terminal (Ubuntu 10.04).

Benjamin Loison
  • 3,782
  • 4
  • 16
  • 33
FooF
  • 4,323
  • 2
  • 31
  • 47

3 Answers3

12

Please refer to termios(3) manual page, under section "Canonical and noncanonical mode".

Typically, the terminal (standard input) is in canonical mode; in this mode the kernel will buffer the input line before returning the input to the application. The hard-coded limit for Linux (N_TTY_BUF_SIZE defined in ${linux_source_path}/include/linux/tty.h) is set to 4096 allowing input of 4095 characters not counting the ending new line. You can also have a look at file ${linux_source_path}/drivers/tty/n_tty.c, function n_tty_receive_buf_common() and the comment above that.

In noncanonical mode there will by default be no buffering by kernel and the read(2) system call returns immediately once a single character of input is returned (key is pressed). You can manipulate the terminal settings to read a specified amount of characters or set a time-out for non-canonical mode, but then too the hard-coded limit is 4095 per the termios(3) manual page (and the comment above the above mentioned n_tty_receive_buf_common()).

Bash read builtin command still works in non-canonical mode as can be demonstrated by the following:

IFS=$'\n'      # Allow spaces and other white spaces.
stty -icanon   # Disable canonical mode.
read line      # Now we can read without inhibitions set by terminal.
stty icanon    # Re-enable canonical mode (assuming it was enabled to begin with).

After this modification of adding stty -icanon you can paste longer than 4096 character string and read it successfully using bash built-in read command (I successfully tried longer than 10000 characters).

If you put this in a file, i.e. make it a script, you can use strace to see the system calls called, and you will see read(2) called multiple times, each time returning a single character when you type input to it.

FooF
  • 4,323
  • 2
  • 31
  • 47
  • There is a newer question in Unix & Linux stemming from somebody stumbling on the same limit, I borrowed the reference for the Linux kernel code from this great answer: https://unix.stackexchange.com/a/643783/19201 – FooF Apr 08 '21 at 09:33
-1

I do not have a workaround for you, but I can answer question 2. In linux PIPE_BUF is set to 4096 (in limits.h) If you do a write of more than 4096 to a pipe it will be truncated.

From /usr/include/linux/limits.h:

#ifndef _LINUX_LIMITS_H
#define _LINUX_LIMITS_H

#define NR_OPEN         1024

#define NGROUPS_MAX    65536    /* supplemental group IDs are available */
#define ARG_MAX       131072    /* # bytes of args + environ for exec() */
#define LINK_MAX         127    /* # links a file may have */
#define MAX_CANON        255    /* size of the canonical input queue */
#define MAX_INPUT        255    /* size of the type-ahead buffer */
#define NAME_MAX         255    /* # chars in a file name */
#define PATH_MAX        4096    /* # chars in a path name including nul */
#define PIPE_BUF        4096    /* # bytes in atomic write to a pipe */
#define XATTR_NAME_MAX   255    /* # chars in an extended attribute name */
#define XATTR_SIZE_MAX 65536    /* size of an extended attribute value (64k) */
#define XATTR_LIST_MAX 65536    /* size of extended attribute namelist (64k) */

#define RTSIG_MAX     32

#endif
Benjamin Loison
  • 3,782
  • 4
  • 16
  • 33
StianE
  • 3,055
  • 20
  • 20
  • Yes there is such a limitation for pipe, and actually it is relevant to notice that the non-interactive version uses pipes and by necessity there will be _several writes and reads_. But I think this limit should not affect terminal input (terminals are not pipes). – FooF Aug 02 '13 at 12:06
-3

The problem is definitely not the read() ; as it can read upto any valid integer value. The problem comes from the heap memory or the pipe size.. as they are the only possible limiting factors to the size..

Abhishek
  • 874
  • 2
  • 8
  • 23
  • No, the problem is the buffer size for the terminal canonical mode. Input line is 4096 (the last byte reserved for the new line). This is explained and demonstrated in the answer by the original questioner. He is saying the answer is not complete because he did not have yet time to write a C program to demonstrate this, and he could not point out where in the kernel this limit is defined. – FooF Aug 13 '13 at 10:00
  • That's what I am trying to say, the pipe buffer size is 4096. It can be checked using "ulimit -p" it will return 8 as answer, which means 8*512 bytes=4096. As each character is 1 byte, Hence it reads only 4095 bytes and the last character as newline as you said. – Abhishek Aug 14 '13 at 01:48
  • 1
    I think it is a different constant. As it happens 4096 is the default page size, explaining why the value happens to be the same for max pipe size and terminal input buffer in canonical mode. – FooF Aug 14 '13 at 14:20