3

Say that the ping command is running, and I type something on the terminal while ping is still running.

Now when ping terminates and bash gain back control, bash will print on the terminal what I typed while ping was running. This is a screenshot that shows what I mean:

enter image description here

How did bash get this information? I am sure it did not get it from stdin, because when I typed "I typed this while ping was running", I did not press Enter (and so stdin is empty).

So this data must have been stored in a "keystrokes buffer", and bash read from this buffer.

My question is, how does bash reads from this buffer (what function does it call...)?

paul
  • 131
  • 4
  • Bash didn't `read` from anything. Your *tty* `echos` your keystrokes (just the same as when you are typing a command) `ping` is running in a *subshell* and when that completes, and returns control to the parent, what you typed is there before you. Had you typed `echo foo` and hit `return` during `ping`, you would have `foo` on the line above your empty `PS1` prompt. (to be clear, there in an *input buffer* and on `return` bash does read your command from that buffer, but that has nothing to do with what you see after you type) – David C. Rankin May 21 '17 at 19:48
  • @David C. Rankin Yes, the *tty* echos my keystrokes, but the keystrokes that were echoed are mixed with the output of `ping`. The *"I typed this while ping was running"* that exists after *"paul@paul-laptop:~$"* is printed by `bash` (*tty* only echoes my keystrokes once not twice). – paul May 21 '17 at 19:52
  • You get the local echo, and the command remains in front of your `PS1` prompt until you hit `enter` despite it also being echoed to the terminal during `ping` (which you must have done a `ping -c x shomehost` rather than `ping somehost` and then `ctrl+c` which would have emptied the input buffer.) – David C. Rankin May 21 '17 at 19:55
  • 1
    @David C. Rankin If `bash` is not the one doing the echoing, then why when I used `sh`, no second echoing occurred? – paul May 21 '17 at 20:01
  • We may be splitting hairs here. You do have an input/command buffer. When you type, characters get echoed to the terminal, but they are also preserved for passing to the shell when you press `Enter` (or `Return`). After `ping` completes, bash displays the contents of what is waiting to be passed to the shell in front of your prompt. The echo during `ping` does affect what is awaiting a `Return`. If it did, you would have no clue what information was to be passed to the shell after you danced on the keyboard during `ping` (and pray it wasn't a joking `sudo rm -rf /`) – David C. Rankin May 21 '17 at 20:08
  • @DavidC.Rankin It isn't splitting hairs, there is a buffer and in your previous comment you said it wasn't used to populate the prompt, whereas in your latest comment you say it is... – 123 May 21 '17 at 20:11
  • No, I said bash wasn't reading from anything to output the characters in front of the prompt after ping completed. E.g. there is no `read` because the commands are not in `stdin`? What part of that is wrong? – David C. Rankin May 21 '17 at 20:13
  • @DavidC.Rankin You said `to be clear, there in an input buffer and on return bash does read your command from that buffer, but that has nothing to do with what you see after you type` and then `After ping completes, bash displays the contents of what is waiting to be passed to the shell in front of your prompt.` – 123 May 21 '17 at 20:14
  • We are splitting hairs here somewhere. When you type, your tty is reading the characters from the keyboard. Yes there is interaction between the shell and what is displayed, but on pressing keys, the characters are simply echoed to the terminal and remain waiting for passing to the shell on enter. What verbiage do you prefer? – David C. Rankin May 21 '17 at 20:16
  • I guess the way to put it is, when you type, your keystrokes are read and echoed by your terminal and a single-line of input awaits being passed to your shell on `return`, but it being there has nothing to do with a bash `read` it is a terminal operation waiting to be read by bash as a command [**3.1.1 Shell Operation**](https://www.gnu.org/software/bash/manual/html_node/Shell-Operation.html#Shell-Operation) (**1.** specifically) – David C. Rankin May 21 '17 at 20:25
  • @DavidC.Rankin You are the only person who mentioned bash `read`, reading from a buffer doesn't mean bash `read` was used. Also that page has nothing to do with this as OP is asking why the line is populated after ping, the page you linked is about executing the command. – 123 May 21 '17 at 20:29
  • `My question is, how does bash reads from this buffer (what function does it call...)?` – David C. Rankin May 21 '17 at 20:31
  • @DavidC.Rankin Yes, `reads from a buffer`, not uses `read`. – 123 May 21 '17 at 20:32
  • You know, this whole series of comments is related to the comment of "How does the text get there if it was echoed during ping" (see: "`(tty only echoes my keystrokes once not twice)`") Why you seem fixated on picking at the explanation for that is beyond me. That is why this discussion was in the **comments** and not posted as an **answer**. We are talking about two separate things, "**1)** how did it get there?" and "**2)** how do I read from what is holding it?" (which is the subject of the answer). – David C. Rankin May 21 '17 at 20:38
  • 1
    @DavidC.Rankin To answer 1), it gets there because `readline` puts it there. – zbw May 21 '17 at 20:42
  • Yes, thank you for your answer, the `readline` interface what is happening between the keypress and ultimately passing the argument list to the shell. I like the examples you posted. – David C. Rankin May 21 '17 at 20:45
  • @DavidC.Rankin No-one said `How does the text get there if it was echoed during ping`, why have you quoted it, I honestly don't even know what you are talking about anymore or what point you are trying to make. I never suggested you gave an answer but your comments at best are misleading. – 123 May 21 '17 at 20:57

3 Answers3

5

There is no special "read the terminal queue" mechanism going on in bash. It's just ordinary read and write system calls.

On Linux, the "tty device" is always buffered. If you type faster than the receiving program can read the input, the characters are not dropped into the bit bucket; they are placed in the input queue for the terminal device, from which they can be retrieved with a read system call.

The read call has the prototype ssize_t read(int fd, void *buf, size_t count);. When the terminal driver decides that the read call should return, it removes count bytes from the terminal's input queue and places them into buf. If the queue does not contain count bytes, then all of the bytes in the queue are read; if it contains more than count bytes, the remaining bytes remain in the queue for the next read.

count has no impact on the operation of the terminal itself; read(0, buf, 1) does not cause the read to return when one byte is available. It only limits the number of bytes placed into buf.

There are. however, a number of control settings which affect how read calls are handled. The most important one, in this context, is the ICANON flag. If this flag is set ("canonical mode"), a process waiting on a read system call on the tty will not be woken until an NL character is typed. (Actually, there are four characters which will wake the process up: NL, EOL, EOL2 and EOF.) In canonical mode, the kernel driver also handles some line editing characters, such as the ERASE character. (All of these characters are configurable through termios so that when I say "NL character", I mean "the character currently configured as NL. By default, the NL character is the "Enter" key and EOF is control-D.)

If ICANON is not set, the terminal is in non-canonical mode, and the VMIN and VTIME settings apply. VMIN is the minimum number of characters which must be present before the process is woken up; VTIME is the minimum amount of time which the terminal driver will wait for input before giving up and waking the process up. If both VMIN and VTIME are set, then the kernel driver will wait (indefinitely) for the first character, and will then wait up to VTIME for each successive character until VMIN characters are read. If neither VMIN nor VTIME is set, the read call will always return immediately.

Independent of the canonical mode setting, you can also configure the terminal's echoing behaviour. In the simplest configurations, you either set ECHO or not. If ECHO is set, printable characters are echoed by the terminal driver when they are typed. If ECHO is not set, the terminal driver does not echo characters. By default, ECHO is set.

Bash normally uses the readline library to handle terminal input. You can tell bash not to use readline, in which case the behaviour will be different; I'm only going to describe the normal case, with readline.

While bash is active and in control of the terminal, readline places the terminal into non-canonical mode and turns off echoing. It sets VMIN to 1 so read calls will return as soon as a single character is typed, and then goes into something like the following loop:

while (1) {
  ssize_t nread = read(0, buf, 1);
  if (nread && isprint(buf[0]))
    write(1, buf, 1);  /* Echo the character */
  else {
    /* Lots of complicated logic to handle other characters */
  }
}

Just before bash lets a child process execute, it resets the terminal by putting it back into canonical mode and turning echo on. Unless the command being executed changes the terminal mode, it stays like that until the command exits, at which point bash regains control, turns canonical mode and echoing off, and goes back into the readline loop.

So suppose that the command being executed does not change the terminal settings, and executes for some period of time (as in the ping example in your question). While ping is executing the terminal is in canonical mode, echoing is on, and no-one is reading from the terminal device. So anything you type will be placed into the terminal queue and echoed. "Anything" includes things which would normally terminate a read, like the Enter key. That's because there is no read to terminate.

When ping finally finishes, bash changes the terminal back to non-canonical mode and turns echoing off, so that characters you now type will not be echoed by the terminal driver. It then calls readline which starts up the loop above. However, at this point there is a bunch of stuff in the terminal queue, considerably more than the one character required by the VMIN setting. So the read call returns immediately with one character, and the write call in that loop then echoes the character which was just read. Of course, that character had already been echoed by the terminal driver, but that fact is not magically recorded in the character's binary encoding; it's just an ordinary character and so it gets echoed a second time. This continues until either the terminal queue is emptied or some character you previously typed requires readline's attention. If the terminal queue just contains some ordinary printable characters, they will all get echoed and the next read call will block until you type something.

rici
  • 234,347
  • 28
  • 237
  • 341
1

It's a thing with the readline library (here's a more approachable page about it).

You can see this through Python, which is compiled with readline support on most distros:

>>> import time
>>> time.sleep(5)
I am typing this during the sleep>>> I am typing this during the sleep

However, I also happened to have one without readline support:

>>> import time
>>> time.sleep(5)
I am typing this during the sleep>>>

(This can also be achieved through cat | python -i, because cat doesn't use readline, and python disables readline because its input is not a terminal.)

My guess as to what happens is:

  1. Readline disables buffering (probably something like this). This way, it can receive all the characters as they're typed.
  2. Readline disables echoing the characters to the terminal, and takes over control. (Here's a possible way.)
  3. Bash disables readline so echo can do its stuff.
  4. Echo ignores the characters, so they're still in stdin after being typed.
  5. Readline helpfully takes back control, disables buffering/echoing typed characters (too late for the ones already typed, though), and handles the characters from stdin.

TL;DR: They're automatically echoed once, and then again by the readline library.

Community
  • 1
  • 1
zbw
  • 922
  • 5
  • 13
  • *"Readline disables buffering (probably something like this but with stdin). This way, it can receive all the characters as they're typed."* do you mean the stdin buffering, or the buffering that the tty device (`/dev/tty1`) does? (if it disables the buffering of the tty device, so it's like it made the terminal in raw mode (which means that the keystrokes can be sent immediately without the user having to press **Enter**)). – paul May 21 '17 at 22:10
  • @paul It messes with some settings in `termios` -- it's a little too complex for me to figure out right now, but you can read all about it [here](http://git.savannah.gnu.org/cgit/readline.git/tree/rltty.c). A quick google search suggested that the best way to to something like that is through termios ([here](http://www.cs.uleth.ca/~holzmann/C/system/ttyraw.c)'s a simpler example). – zbw May 21 '17 at 22:35
  • I wrote an answer. – paul May 22 '17 at 02:14
  • `setvbuf` is only for output buffers, and virtual buffers are implemented entirely in the standard C library, which is an application layer. There is no equivalent for input streams. – rici May 22 '17 at 03:32
  • @rici Thanks! If you have more information, a better explanation, or any corrections, please suggest an edit! I'll admit that this is not my area of expertise, so please suggest an edit (or even make your own answer). – zbw May 22 '17 at 20:55
  • @rici I changed the link about disabling buffering to a different page that used termios to control it. – zbw May 22 '17 at 21:11
  • @zbw, well I tried to write an answer, which is quite a lot longer than yours (although I suppose that in the end it amounts to the same thing). The thing I was trying to emphasize is that "buffering" and "waking up" are completely orthogonal. (As you say, echoing is also orthogonal.) So characters typed are placed in the input queue (which is a kind of buffer, to be sure, but it cannot be turned off) until the read wakes up, at which point some number of characters are read. If you think of it just in terms of "buffering", you are likely to confuse yourself. – rici May 22 '17 at 21:46
0

This is what I think happens:

  1. When bash is running, it disables the tty device line buffering, it also disables the tty device echoing (now bash is the one that takes the responsibility of echoing everything it receives).
  2. Immediately before bash executes ping, bash re-enables the tty device line buffering and the tty device echoing. Then bash executes ping.
  3. Now I type"I typed this while ping was running" while ping is displaying its output. The tty device will echo back this string to the terminal window, but also this string is now buffered in the tty device.
  4. ping terminates and bash gains back control.
  5. bash re-disables the tty device line buffering and the tty device echoing. Disabling the tty device line buffering will cause the tty device line buffer to be flushed to bash, and bash will display it on the terminal window.
paul
  • 131
  • 4