1

I tried to write a little program to list a non existing directory and echo done, in a .sh file:

#!/bin/bash
ls notexist
echo 'done'

But my console outputs done on the first line, before the error message to list the nonexisting directory:

done
ls: notexist: No such file or directory

I don't think bash creates a thread automatically for each line of code, does it? I'm using terminal in macOS Big Sur.

Edit: I'm accessing terminal indirectly from the script package of the Atom text editor in macOS Big Sur. The error goes away if I run code directly in console via ./file.sh.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
Diego
  • 867
  • 7
  • 13
  • 1
    Bash won't do this _on its own_, but it can happen if your stdout and stderr aren't copies of the same fd (because when they aren't copies of the same file descriptor, the operating system doesn't guarantee anything about the order they'll be flushed in). – Charles Duffy Jun 16 '21 at 16:10
  • 1
    ...for example, if you run `exec > >(tee stdout.log) 2> >(tee stderr.log >&2)`, that'll create the conditions that allow this bug. Likewise, if the script this is in is run with its stdout and stderr being piped in different ways. – Charles Duffy Jun 16 '21 at 16:11
  • 2
    To be clear: The operations still _run_ in order; the problem the above creates one is the order in which output is written, which (when stdout and stderr aren't going to the same place) isn't guaranteed to be the same order things actually happened in. – Charles Duffy Jun 16 '21 at 16:12
  • 1
    Another way to cause a problem is to pipe standard output to `less`, in which case your script may be able to write its standard error before the standard output is read by, then written to the terminal, by `less`. – chepner Jun 16 '21 at 16:13
  • 1
    @chepner, ...though that would be reordering in the opposite direction; here, we have stdout showing up before stderr does. – Charles Duffy Jun 16 '21 at 16:15
  • 1
    Related: [redirect stdout and stderr to file and screen while preserving output ordering](https://stackoverflow.com/questions/54282108/redirect-stdout-and-stderr-to-file-and-stderr-to-screen-while-preserving-output) -- it's basically certain your system is set up to do this in a way that _doesn't_ preserve ordering. – Charles Duffy Jun 16 '21 at 16:17
  • 1
    @CharlesDuffy Oh, right. But the race condition says that it could happen the other way, too :) – chepner Jun 16 '21 at 16:17
  • Thanks to both of you! I'm running the code in the Atom text editor. I looked into the script package for how it runs bash. I tried running directly using ./file.sh and the output is as behaved. Sorry for not making that clear – Diego Jun 16 '21 at 16:19
  • 1
    @Diego, does Atom try to color stdout and stderr differently or otherwise distinguish between them? If so -- it can't do that without having them written to different FIFOs, but that means it doesn't have any way to be sure it's reading from those two FIFOs in the same order content was originally written. (If they're _not_ being handled differently, it's just sloppy to set it up this way, and Atom's authors can/should fix the bug). – Charles Duffy Jun 16 '21 at 16:21
  • @CharlesDuffy My output isn't colored, though they might made it toggleable. I'm not really sure how it works... – Diego Jun 16 '21 at 16:24
  • 1
    BTW, a quick way to test this, if your OS is Linux: `ls -l /proc/self/fd`. If you run it in a terminal, it should look like `1 -> /dev/pts/2`, `2 -> /dev/pts/2` -- meaning both stdout and stderr are going to the same PTY. If you run it in your script from Atom, I'll bet they'll be going two different places. (The `2` in `/dev/pty/2` is an arbitrary number, it'll be different for each concurrently-running terminal). – Charles Duffy Jun 16 '21 at 16:26
  • 1
    ...in Atom, it'll probably look like `1 -> 'pipe:[123]'` and `2 -> 'pipe:[234]'` -- what's important is whether or not the `123` and `234` numbers are identical. If they're the same, the problem is not Atom's fault; if they're different, it totally is, and you should report the bug to Atom's authors instead of talking to us about it here. – Charles Duffy Jun 16 '21 at 16:29
  • @CharlesDuffy using MacOS... seems that it doesn't use /proc folder. https://superuser.com/questions/631693/where-is-the-proc-folder-on-mac-os-x – Diego Jun 16 '21 at 16:33
  • 1
    Correct, that's why I said you could test that way "if your OS is Linux". I don't know, and don't care to know, if Apple has any equivalent. – Charles Duffy Jun 16 '21 at 16:33
  • 1
    That said, as a workaround... put `exec >&2` at the top of your script (just under the shebang) to make _everything_ be on stderr, or `exec 2>&1` at the top to make everything be on stdout. When content is all on the same stream, problems with the output from the streams being read out-of-order can't happen. – Charles Duffy Jun 16 '21 at 16:38
  • @CharlesDuffy that worked for me! I'm sure you diagnosed the source of my error. – Diego Jun 16 '21 at 16:45
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/233905/discussion-between-diego-and-charles-duffy). – Diego Jun 17 '21 at 18:36

1 Answers1

1

If we look at the source code to the Atom script plugin, the problem becomes clear:

It creates a BufferedProcess with separate stdout and stderr callbacks (using them, among other things, to determine whether any output has been written to each of these streams).

Implementing this requires stdout and stderr to be directed to different FIFOs. This means that, unlike a typical terminal where there's an absolute ordering of which content was written to the single FIFO shared by both stdout and stderr at the same time, there's no strict guarantee that content will be processed through those functions in the same order it was written.

As a workaround, you can exec 2>&1 into your script to put all content on stdout, or exec >&2 to put all content on stderr. Ideally, if the script plugin doesn't need to track the two streams separately, it would do this itself, and put a callback only on the single stream to which all content has been redirected.

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441