3

I have been struggling with a pretty simple issue writing a little program in C.

Getting input (commands, arguments, flags to be executed) via fgets() works fine as long as the size of the input does not exceed 1024 bytes. After 1024 characters are typed, no more characters are accepted -- the prompt just stops. I assume reason for the problem doesn't lay in the fgets() parameters/configuration because otherwise it would at least take the input up to defined size instead of blocking.

How can I make fgets() accept lines as long as _SC_LINE_MAX (2048) bytes/chars?

John Bollinger
  • 160,171
  • 8
  • 81
  • 157
IpsumLorem
  • 39
  • 3
  • 5
    What OS/shell are you using? – Joseph Quinsey May 02 '16 at 19:53
  • 2
    It's most likely the limitations of the shell. Put the input in a file and use the shell's IO indirection mechanism to pipe the contents of the file to the stdin of the program. – R Sahu May 02 '16 at 20:01
  • 1
    If you're typing the long line, the trouble is the terminal driver on your system (but it is more generous than Mac OS X is). – Jonathan Leffler May 02 '16 at 20:14
  • 1
    Please share your code, even if it might be very simple. Then other persons can verify your findings – Stian Skjelstad May 02 '16 at 20:15
  • @StianSkjelstad: The code is probably as simple as: `#include ` and `int main(void) { char buffer[2048]; if (fgets(buffer, sizeof(buffer), stdin) != 0) printf("Data: [%s]\n", buffer); return 0; }`. Compile and run it. Then try to type more than ...256 or 1024…characters on a single line, and it won't work because the terminal has an upper limit on the length of line that it will buffer. Reading from a file with longer lines won't be a problem at all. As I said before, the problem is with the terminal driver (and wholly unrelated to the shell), regardless of what `_SC_LINE_MAX` says. – Jonathan Leffler May 02 '16 at 20:28
  • @RSahu: Why do you think the shell is a factor when the code is in a C program that has been run by the shell? – Jonathan Leffler May 02 '16 at 20:32
  • @JonathanLeffler, I should've been more precise. It's either the shell or the terminal. – R Sahu May 02 '16 at 20:35
  • 1
    @RSahu: Fair enough. Precise and concise do not always work hand-in-hand. I tested on Mac OS X 10.11.4 and the limit there is 1024 (1023 non-newlines plus a newline). It's odd: I'm fairly sure that previously I ran into that limit at more like 256 characters — but it is a while since I last explicitly tested it, and the problem normally manifests itself for me when trying to copy something (a hunk of JSON, for example) from a web site into a file. – Jonathan Leffler May 02 '16 at 20:54
  • You might be able to work around the problem by breaking the command into multiple lines, with a `\ ` immediately preceding the line break. `bash`, at any rate, will remove both `\ ` and newline, joining the lines. – John Bollinger May 02 '16 at 20:58
  • 1
    The related questions list [`fgets` is limited to 1024 bytes — what can I use instead](https://stackoverflow.com/questions/23461159/fgets-is-limited-to-1024-bytes-what-can-i-use-instead), where the user almost certainly ran into this problem. The answers there are not singularly helpful — basically, they're of the form "no, `fgets()` is not limited to 1024 bytes". – Jonathan Leffler May 02 '16 at 21:14
  • Put the information into a simple text file and feed it through `< yourdata.txt` into the program. – Jongware May 02 '16 at 21:18
  • @RSahu: No, it's not the shell. When you're running a user-written program, the shell isn't involved. – Keith Thompson May 02 '16 at 21:18
  • @KeithThompson, JonathanLeffler brought that up too. I should've been more precise. It's either the shell or the terminal. – R Sahu May 02 '16 at 21:20
  • @RSahu: Saying it's either the shell or the terminal implies that it might be the shell. In my opinion, there is little or no chance it could be the shell. Do you have reason to think it might be? – Keith Thompson May 02 '16 at 21:21
  • @KeithThompson, you are most likely right. I'm not 100% sure when the shell transfers control over to the terminal for IO. – R Sahu May 02 '16 at 21:25
  • Try running `stty cbreak ; ./your_program`. That should disable line editing (so backspace doesn't erase characters, for example), which means the tty driver doesn't need to buffer your input. Be sure to do the `stty` in the same shell command as your program; some shells reset tty settings at each prompt. You *might* need to do `stty cooked` after running the command. – Keith Thompson May 02 '16 at 21:38

1 Answers1

7

Terminal drivers limit the length of input lines

As I noted in the comments, the trouble is almost certainly that your terminal driver won't allow you to enter lines that are longer than 1024 bytes; it won't allocate more storage. That applies in 'canonical' mode; see Canonical vs non-canonical terminal input for more information about that.

There's a simple test program (now in the question). When I run it on Mac OS X 10.11.4, I can enter 1023 characters plus newline, or 1024 characters but no newline until I delete one.

For the record, the input string was:

12345678901234567890123456789X123456789Y123456789Z123456789A123456789B123456789C123456789D123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789s123456789t123456789u123456789v123456789w123456789x123456789y123456789z123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789u123456789v123456789w123456789x123456789y123456789z123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123456789r123456789s123456789t123456789u123456789v123456789w123456789x123456789y123456789z123456789a123456789b123456789c123456789d123456789e123456789f123456789g123456789h123456789i123456789j123456789k123456789l123456789m123456789n123456789o123456789p123456789q123

If you copy and paste that with the newline, it will probably be entered OK. If you add any more characters, you will probably get the terminal beeping at you.

There's another question (fgets() is limited to 1024 bytes — what can I use instead?) that almost certainly ran into the same problem — and really wasn't given much in the way of useful help.

If you use a library such as the GNU readline library, it can put the terminal into non-canonical mode and then can handle longer lines because the terminal driver doesn't wait until the newline is entered before sending the data to the program. Rebuilding the kernel with a bigger limit on the terminal input line length is nominally an option on Linux-like systems, but not a task for the casual programmer.

Meaning of LINE_MAX and related macros

Also note that _SC_LINE_MAX is the sysconf() code for determining LINE_MAX, which must be at least the value of {POSIX2_LINE_MAX} (minimum 2048), which is documented as being:

Unless otherwise noted, the maximum length, in bytes, of the input line of a utility (either standard input or another file), when the utility is described as processing text files. The length includes room for the trailing <newline>.

Note that a terminal is not a text file. This limit says that utilities such as grep must not mishandle lines that are 2048 bytes long, but it might get confused by longer lines (for example, because it reads 2048-byte chunks of a line, and does 'beginning of line' matches at the start of the second or subsequent chunks of a long line).

The rationale for POSIX notes:

{LINE_MAX}

This is a global limit that affects all utilities, unless otherwise noted. The {MAX_CANON} value from the System Interfaces volume of POSIX.1-2008 may further limit input lines from terminals. The {LINE_MAX} value was the subject of much debate and is a compromise between those who wished to have unlimited lines and those who understood that many historical utilities were written with fixed buffers. Frequently, utility writers selected the UNIX system constant BUFSIZ to allocate these buffers; therefore, some utilities were limited to 512 bytes for I/O lines, while others achieved 4096 bytes or greater.

It should be noted that {LINE_MAX} applies only to input line length; there is no requirement in POSIX.1-2008 that limits the length of output lines. Utilities such as awk, sed, and paste could theoretically construct lines longer than any of the input lines they received, depending on the options used or the instructions from the application. They are not required to truncate their output to {LINE_MAX}. It is the responsibility of the application to deal with this. If the output of one of those utilities is to be piped into another of the standard utilities, line length restrictions will have to be considered; the fold utility, among others, could be used to ensure that only reasonable line lengths reach utilities or applications.

And the {MAX_CANON} referenced is described as:

{MAX_CANON}

Maximum number of bytes in a terminal canonical input line.
Minimum Acceptable Value: {_POSIX_MAX_CANON}

And elsewhere (<limits.h>), the minimum acceptable value for _POSIX_MAX_CANON is 255.

Community
  • 1
  • 1
Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278