Output stdout and stderr to file and screen and stderr to file in a limited environment

Question

The problem could be probably fixed using mkfifo, but it doesn't exist on my QNAP. So, here is the description of problem and what I tried so far.

I have a function called activateLogs that restarts the script if writing logs to disk or both (screen and disk). Both option is the new functionality I would like to achieve.

exec 3<> "$logPath/$logFileName.log"
"$0" "${mainArgs[@]}" 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" 1>&3 &

This piece of code is the version that writes to disk. mainArgs contains all the arguments passed to the script and is defined out of this function. This solution came from https://stackoverflow.com/a/45426547/214898. It combines stderr & stdout in a file and still output stderr in another.

So, now, I would like to be able to keep this and add printing stderr & stdout to screen.

The accepted solution from the question linked above cannot be applied because the script is running using sh and mkfifo is not present.

Attempt #1

exec 3>&1
"$0" "${mainArgs[@]}" 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" 1>&3 &

--> to replace the code above and in a if branching (that already existed) I added the tee command.

local isFileDescriptor3Exist=$(command 2>/dev/null >&3 && echo "Y")

if [ "$isFileDescriptor3Exist" = "Y" ]; then
    tee -a "logs/123.log" &
    echo "Logs are configured"
else
    ### CODE ABOVE
fi

I have the screen, the error file, but the log file is empty.

Attempt #2 Now, no tee in the if branching above, but included in the relaunching command.

exec 3>&1
"$0" "${mainArgs[@]}" 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" 1>&3 3>&1 | tee -a "logs/123.log" &

Same result. I may understand in this one that the first tee not initially writing to the file descriptor #3, thus, the 3>&1 does nothing.

Attempt #3 (No more relaunching the script)

out="${TMPDIR:-/tmp}/out.$$"
err="${TMPDIR:-/tmp}/err.$$"
busybox mkfifo "$out" "$err"

trap 'rm "$out" "$err"' EXIT

tee -a "$logPath/$logFileName.log" &
tee -a "$logPath/$logFileName.err" < "$err" >&2 &
command >"$out" 2>"$err"

I am getting mkfifo: applet not found from busybox

Attempt #4 (No more relaunching the script)

out="${TMPDIR:-/tmp}/out.$$"
err="${TMPDIR:-/tmp}/err.$$"
python -c "import os; os.mkfifo(\"$out\")"
python -c "import os; os.mkfifo(\"$err\")"

trap 'rm "$out" "$err"' EXIT

tee -a "$logPath/$logFileName.log" &
tee -a "$logPath/$logFileName.err" < "$err" >&2 &
command >"$out" 2>"$err"

I have no logs (neither "real" logs" nor errors). The temporary files are deleted though. Moreover, the script never ends which is caused by trap.

Attempt #5

exec 3>&1
{ "$0" "${mainArgs[@]}" | tee -a "$logPath/$logFileName.log"; } 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" &
exit

That solution seemed promising, but now relaunching the script doesn't work because my code detects if currently executing and stop it. This is normal because it is executed in subprocess even though I use & at the end of the full line, but... (testing while writing). Replacing the terminator ; by & fixed it.

Attempt #6

I didn't realized it right away, but stdout & stderr were displayed to screen, stderr was written to a file, but only stdout was written to a file instead of both.

exec 3>&1
{ "$0" "${mainArgs[@]}" | tee -a "$logPath/$logFileName.log" & } 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" &
exit

Final version working

Now everything is written/displayed where it is supposed to be. See full code in the accepted answer.

{ "$0" "${mainArgs[@]}" 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" 1>&3 & } 3>&1 | tee -a "$logPath/$logFileName.log" &
exit

Incredible how my first attempt was so close to the solution.

Are you sure you don't have busybox, toybox, Python, Perl, or similar that can be used to create a fifo? — that other guy, Jan 19 '22 at 22:15
Sooooo does `echo > >(cat)` works? Because if it does... you basically have a fifo. And because you have `mainArgs[@]` arrays... what shell are you using? — KamilCuk, Jan 19 '22 at 22:25
@thatotherguy `-sh: mkfifo: command not found`. I presume the QNAP Linux distro has been built via busybox (and I think in the past I already seen that) and Perl is neither present, but Python is. Though, I would like to have a pure shell script solution. If it is not possible I will turn over other languages (probably only Python). — Master DJon, Jan 19 '22 at 23:22
@KamilCuk I think my previous comment answer your question, but in case of... The shell is `sh` and as said in the question description, the `mainArgs` variable it is something in define outside the function: in fact it is an array I build that contains `$@` which is also used in other places. — Master DJon, Jan 19 '22 at 23:24
@KamilCuk I skipped your first question. `echo > >(cat)` gives me `-sh: syntax error near unexpected token `>'` executed directly in command line. — Master DJon, Jan 19 '22 at 23:25
You can try `busybox mkfifo foo` or `python -c 'import os; os.mkfifo("foo")'` if a fifo would help, without rewriting anything else — that other guy, Jan 19 '22 at 23:26
@thatotherguy I'll give a try tomorrow at `busybox mkfifo foo` that seems to exist. I wasn't aware that building a distro using Busybox was containg the `busybox` software. Good to know! — Master DJon, Jan 19 '22 at 23:28
@thatotherguy OH! I thought about `buildroot` and not `busybox`. I thought you talked about the first. Finally, already heard of it, but never used it. — Master DJon, Jan 19 '22 at 23:32
`The shell is sh` `sh` is the name of the real super old [bourne shell](https://en.wikipedia.org/wiki/Bourne_shell) - it's not in use at least 20 years. Nowadays there are dash, ash, ksh, rc, bash - replacements of sh. And sh does not have arrays - `"${mainArgs[@]}"` would be an error. Shell from busybox - ash - also does not have arrays. — KamilCuk, Jan 20 '22 at 00:09
@KamilCuk OK! How would I know which shell I am using? I already tried printing `$SHELL` and other stuff, but it always gives me `sh` — Master DJon, Jan 20 '22 at 13:27
@thatotherguy You can look at my attempts #3 & #4. Unfortunately it still doesn't work. — Master DJon, Jan 20 '22 at 16:48
@MasterDJon It should be successfully creating the fifos, but you're not reading from `$out` so I imagine it's stuck waiting there. — that other guy, Jan 20 '22 at 17:57
@thatotherguy Adding `< "$out"` to the log tee command fixed the issue of "script not ending", but still nothing written in the files. — Master DJon, Jan 20 '22 at 18:44
@MasterDJon Works for me. Maybe your particular command is buffering since stdout/stderr is no longer a tty? Does it work with some other command like `echo foo`? — that other guy, Jan 20 '22 at 20:23
@thatotherguy See my final version in the question OR the full code in my answer below. — Master DJon, Jan 20 '22 at 20:25
@thatotherguy If you have any comments on the final version, everything is welcome. — Master DJon, Jan 20 '22 at 20:27

William Pursell · Answer 1 · 2022-01-20T19:43:30.827

1

Maybe I'm not reading the question correctly, but it seems that you could do something like:

#!/bin/sh


exec 3>&1
cmd (){
        echo stdout;
        echo stderr >&2;
}

stderr_outfile=outfile
if test -n "$log_stdout"; then
        stdout_outfile=$log_stdout
else
        stdout_outfile=/dev/null
fi

{ cmd | tee "$stdout_outfile"; } 2>&1 1>&3 | tee "$stderr_outfile"

edited Jan 20 '22 at 19:43

answered Jan 20 '22 at 17:16

William Pursell

204,365
48
270
300

I got it to work! See my **Final version** in the question. Maybe you could adjust your answer to reflect that and maybe add some details for beginners. In any case, I'll accept your answer. I am just waiting to get your feedback. – Master DJon Jan 20 '22 at 18:58
See my answer for full code. – Master DJon Jan 20 '22 at 19:23
It seems better to try to limit the actual invocation of the command to one place in the code. The additional `tee > /dev/null` is not particularly efficient, but probably not a significant issue. You could select `cat > /dev/null` easily enough. – William Pursell Jan 20 '22 at 19:48
If you look at the answer I added (the full code of the `activateLogs` function), you can see this part of the code can't be called twice because I verify if the file descriptor 3 already exists. – Master DJon Jan 20 '22 at 20:06
@MasterDJon I'm not concerned about the code being called twice, but about code cleanliness. It is better if you can refactor the code so that there is only one location in the script where the code is executed. Having it in multiple locations reduces readability. – William Pursell Jan 20 '22 at 20:09
Not sure about what you trying to tell me. Are you talking about the script relaunching (last `if`) that shall be merged using your version. But... how can I manage to have the two different options: DISK+SCREEN or DISK_ONLY? *Maybe best to comment under my answer, no?* – Master DJon Jan 20 '22 at 20:18
For example, you could do things like `cmd | tee | if ...; then tee /path; else cat; fi`. It's a bit kludgy since you get the mostly useless `cat`, and that particular example is arguably less readable. But this is a fairly academic point. – William Pursell Jan 20 '22 at 20:21
I understand your **cleanliness** argument. And I do believe in that more than efficiency (even though that latest is also important). But I don't really see how I could change it to be more readable without duplicating relaunching. – Master DJon Jan 20 '22 at 20:24
If you still have some time and you would like to solve another challenge: https://stackoverflow.com/questions/70836246/ – Master DJon Jan 24 '22 at 15:25
I declared victory too soon. I didn't realized it, but I lost something. The errors are no more in the LOG file. They are on SCREEN and in the ERR file. Being in the LOG file is really important to be able to replace the error in its context. – Master DJon Feb 20 '22 at 13:17
I fixed it couple of minutes after writing you the previous comment. Due to changes to important and that I wanted to explain the whole thing, I modified my answer and accepted mine. Though, I mentioned your name! – Master DJon Feb 20 '22 at 20:47

Master DJon · Accepted Answer · 2022-02-21T14:47:22.703

Finally, reaching the goal. I want to say that I have been inspired by @WilliamPursell's answer.

{ "$0" "${mainArgs[@]}" 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" 1>&3 & } 3>&1 | tee -a "$logPath/$logFileName.log" &

Explanation

Relaunch the script with...
Sending stderr (2>&1) to stdout and...
Sending stdout to a new file descriptor (1>&3)
Pipe it to tee which receives stderr to duplicate the errors in a file and to file descriptor #1 with...
Sending stdout to the new file descriptor (1>&3)...
And having & to ensure no blocking
Then grouping the previous commands using curly brackets.
Sending the grouped commands new file descriptor to stdout (3>&1)
Pipe it to tee which receives stdout that combines errors and normal output that write to file and display on screen
And having & to ensure no blocking

Full code of my activateLogs function for those interested. I also included the dependencies even though they could be inserted into the activateLogs function.

m=0
declare -a mainArgs
if [ ! "$#" = "0" ]; then
    for arg in "$@"; do
        mainArgs[$m]=$arg
        m=$(($m + 1))
    done
fi

function containsElement()
# $1 string to find
# $2 array to search in
# return 0 if there is a match, otherwise 1
{
  local e match="$1"
  shift
  for e; do [[ "$e" == "$match" ]] && return 0; done
  return 1
}

function hasMainArg()
# $1 string to find
# return 0 if there is a match, otherwise 1
{
    local match="$1"
    containsElement "$1" "${mainArgs[@]}"
    return $?
}

function activateLogs()
# $1 = logOutput: What is the output for logs: SCREEN, DISK, BOTH. Default is DISK. Optional parameter.
{
    local logOutput=$1
    if [ "$logOutput" != "SCREEN" ] && [ "$logOutput" != "BOTH" ]; then
        logOutput="DISK"
    fi
    
    if [ "$logOutput" = "SCREEN" ]; then
        echo "Logs will only be output to screen"
        return
    fi
    
    hasMainArg "--force-log"
    local forceLog=$?
        
    local isFileDescriptor3Exist=$(command 2>/dev/null >&3 && echo "Y")
    
    if [ "$isFileDescriptor3Exist" = "Y" ]; then
        echo "Logs are configured"
    elif [ "$forceLog" = "1" ] && ([ ! -t 1 ] || [ ! -t 2 ]); then
        # Use external file descriptor if they are set except if having "--force-log"
        echo "Logs are configured externally"
    else
        echo "Relaunching with logs files"
        local logPath="logs"
        if [ ! -d $logPath ]; then mkdir $logPath; fi
        
        local logFileName=$(basename "$0")"."$(date +%Y-%m-%d.%k-%M-%S)
    
        exec 4<> "$logPath/$logFileName.log" # File descriptor created only to get the underlying file in any output option
        if [ "$logOutput" = "DISK" ]; then
            # FROM: https://stackoverflow.com/a/45426547/214898
            exec 3<> "$logPath/$logFileName.log"
            "$0" "${mainArgs[@]}" 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" 1>&3 &
        else
            # FROM: https://stackoverflow.com/a/70790574/214898
            { "$0" "${mainArgs[@]}" 2>&1 1>&3 | tee -a "$logPath/$logFileName.err" 1>&3 & } 3>&1 | tee -a "$logPath/$logFileName.log" &
        fi
        
        exit        
    fi
}

#activateLogs "DISK"
#activateLogs "SCREEN"
activateLogs "BOTH"


echo "FIRST"
echo "ERROR" >&2
echo "LAST"
echo "LAST2"

Output stdout and stderr to file and screen and stderr to file in a limited environment

2 Answers2

Linked