1

maybe a peculiar question, but

I came across an odd result when pasting through cat into a text file (cat > textfile and signaling EOF by Ctrl+D when done ) on ubuntu linux. It turns out the stdin line buffer prevents lines exceeding 4k (see Line Buffered Cat)

Neither the options of gnu cat (vs. posix cat) nor usage of stdbuf (stdbuf -i0 -o0 cat > textfile) resulted in files exceeding 4k.

Obviously I am doing something wrong, since cat does stdout these kind of files when created with nano or vim quite fine.

I would really like to keep using this convenient way of creating files on the fly or pipeing in to other commands from cli without that limitation.

Thank you

Version: cat (GNU coreutils) 8.28

Edit: it obviously is not cat itself but the way the stdin is buffered in the terminal, however vim and nano manage to switch it of (see also Disable buffering for stdin and stdout using setvbuf())

Summer-Sky
  • 463
  • 8
  • 21
  • 1
    I suspect the limitation has nothing to do with cat, but is an issue with the terminal buffer. – William Pursell Jun 23 '21 at 13:14
  • I suspected this too, thats why i tried this out with nano and vim both have the same terminal interface – Summer-Sky Jun 23 '21 at 13:22
  • 2
    `vim` doesn't use the terminal buffer. It puts the terminal in raw mode. I suspect nano does the same. – William Pursell Jun 23 '21 at 13:38
  • so the solution would be to somehow disable terminal buffer ... *sigh* i tried that – Summer-Sky Jun 23 '21 at 13:41
  • This might be a starting point to find out, which clipboard implementation you are using: https://unix.stackexchange.com/questions/254740/who-stores-copy-paste-buffers-in-x11 – ceving Jun 23 '21 at 13:48
  • @ceving the clipboard content is transferred to the terminal just fine ... it is only that the terminal is forcing a buffer ... https://stackoverflow.com/a/19395600/3623574 – Summer-Sky Jun 23 '21 at 13:54

1 Answers1

0

There is no buffer which clips lines.

Example:

$ { for i in {1..4096}; do printf x; done; echo y; } | cat > data

The file contains 4098 bytes: 4096 times x, one y and a new-line character.

$ stat -c %s data
4098

And it has exactly one line.

$ wc -l data
1 data
ceving
  • 21,900
  • 13
  • 104
  • 178
  • what does tell you that there is no flush after each print? – Summer-Sky Jun 23 '21 at 13:23
  • but good idea i have modified your script a little to verify that it is a correct assumption that cat does not use a buffer (as I assumed when i used it and ran into this pitfall) `z=""; for i in {1..4096}; do z="${z}x"; done; echo "${z}yz" | cat > /tmp/thisshouldwork.txt ; wc /tmp/thisshouldwork.txt` – Summer-Sky Jun 23 '21 at 13:33
  • 1
    @Summer-Sky Such a "flush" of pipes does not exist. It exists [`fsync`](https://stackoverflow.com/a/43188944/402322) but if works only for files and not for pipes. And a pipe buffer is bigger than 4k: https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer – ceving Jun 23 '21 at 13:34
  • not sure how that can be (every buffer should have a flush) but it is ruled out anyway ... However seeing that this cat in itself does NOT use a buffered std in the first place makes it even more peculiar – Summer-Sky Jun 23 '21 at 13:40
  • 1
    @Summer-Sky: In your case `cat` has a **tty** for input, in this answer it is a **pipe**. Seeing different buffering behavior is not surprising. – Ben Voigt Jun 23 '21 at 16:44
  • 1
    @ceving: Your test is invalid, you'd need to pass the data through a pseudo-tty in order to test `cat` in the same scenario OP has. – Ben Voigt Jun 23 '21 at 16:45
  • @BenVoigt you are also correct that somehow the tty is the issue here. using unbuffer for pipe uncoupling like shown here https://stackoverflow.com/questions/32910661/pretend-to-be-a-tty-in-bash-for-any-command#32981392 `z=""; for i in {1..4096}; do z="${z}x"; done; echo "${z}yzzzzyyyzzz" | unbuffer -p cat > /tmp/thisshouldwork.txt ; wc /tmp/thisshouldwork.txt` – Summer-Sky Jun 23 '21 at 17:16