How to write to and parse STDIN/STDOUT/temporary buffer?

Question

First I don't know if I talk about STDIN out STDOUT, but this is what I want to achieve :

There's a program that export database from distant server and send output as gzipped content.
I want to unzip the content, parse it.
If it's OK then import it, otherwise send an error message. I don't want to write to any temporary file on disk so I want to handle things directly from STD

someExportCommand > /dev/stdin #seems not work
#I want to write a message here
echo "Export database done"
cat /dev/stdin > gunzip > /dev/stdin
echo "Unzip done"

if [[ "$mycontentReadFromSTDIN" =* "password error" ]]; then
    echo "error"
    exit 1
fi

#I want to echo that we begin impor"
echo "Import begin"
cat /dev/stdin | mysql -u root db
#I want to echo that import finished
echo "Import finish"

The challenge here is not to write to a physical file. It's easier if it's the case but I want to do the hard way. Is it possible and how?

Answer 1

A literal implementation of what you're asking for (not a good idea, but doing exactly what you asked) might look like the following:

This is a bad idea for several reasons:

• If a database is large enough to be bothering with, trying to fit it in memory, and especially in a shell variable is a bad idea.
• In order to fit binary data into a shell variable, it needs to be encoded (as with base64, or uunencode, or other tools). This makes it even larger than it was before, and also adds performance overhead

...however, the bad-idea code, as requested:

#!/usr/bin/env bash

set -o pipefail # if any part of a command fails, count the whole thing a failure

if exportOutputB64=$(someExportCommand | base64); then
  echo "Export database done" >&2
else
  echo "Export database reports failure" >&2
  exit 1
fi

if exportOutputDecompressedB64=$(base64 --decode <<<"$exportOutputB64" | gunzip -c | base64); then
  echo "Decompress of export done" >&2
else
  echo "Decompress of export failed" >&2
  exit 1
fi
unset exportOutputB64

if grep -q 'password error' < <(base64 --decode <<<"$exportOutputDecompressedB64"); then
  echo "Export contains password error; considering it a failure" >&2
  exit 1
fi

echo "Import begin"
mysql -u root db < <(base64 --decode <<<"$exportOutputDecompressedB64")

---

If I were writing this myself, I'd just set up a pipeline that processes the whole thing in-place, and uses pipefail to ensure that errors in early stages are detected:

set -o pipefail
someExportCommand | gunzip -c | mysql -u root db

The important thing about a pipeline is that all parts of it run at the same time. Thus, someExportCommand is still running when mysql -u root db starts. Consequently, there's no need for a large buffer anywhere (in memory, or on disk) to store your database contents.

Answer 2

The requirement to not use a temporary file seems extremely misdirected; but you can avoid it by reading into a shell variable, or perhaps an array.

Any error message is likely to be on stderr, not stdin. But you should examine the program's exit status instead of looking for whether it prints an error message.

#!/bin/bash
result=$(someExportCommand) || exit

At this point, the script will have exited if there was a failure; and otherwise, result contains its output.

Now, similarly to error messages, status messages, too, should be printed to standard error, not standard output. A common convention is also to include the name of the script in the message.

echo "$0: Import begin" >&2

Now, pass the variable to mysql.

mysql -u root db <<<"$result"

Notice that the <<<"here string" syntax is a Bash feature; you can't use it with /bin/sh. If you need the script to be portable to sh, the standard solution is still to use a pipe;

printf '%s\n' "$result" | mysql -u root db

Finally, print the status message to stderr again.

echo "$0: Import finished" >&2

Using a shell variable for a long string is not particularly efficient or elegant; capturing the output into a temporary file is definitely the recommended approach.