Bash - case statement always enters default

Question

I am writing a Bash script to parse a CSV file (values separated by ; character) and extract some arguments. Depending on the current argument read from the file a specific string is appended to a variable. However, the case statement always enters the default state *) and I can't figure out why.

I have no problems reading the .csv file. Already did that and displayed the output of the arguments read from the file. Everything works perfectly. The problem is that the case statement is not processed as expected.

So, the problem is not reading the .csv file but the processing of the arguments in the case statement.

This is my code:

while IFS=";" read -r arg
do
  case ${arg} in
    "valueX")
      var+="blabla"
      ;;
    "valueY")
      var+="blublu"
      ;;
    *)
      echo -e "Argument ${arg} not supported"
      exit 1
      ;;
  esac
done < filename

Assuming "valueX" is the current argument read from the file. Somehow the script always outputs:

Argument "valueX" not supported

Apparently, the argument (here: "valueX") read from the file is correct, but the script won't enter the corresponding state. Instead, it always enters the default state, no matter what value ${arg} holds.

[EDIT] I thought it would be a good idea to ask the question more generally, but it turns out to be confusing. So here is the full bash script and .csv file:

Script:

#!/bin/bash

# style reset
STYLE_RESET='\e[0m'
# Red foreground color
FOREGROUND_RED='\e[31m'
# Green foreground color
FOREGROUND_GREEN='\e[32m'
# Blue foreground color
FOREGROUND_BLUE='\e[34m'
# Red background color
BACKGROUND_RED='\e[41m'
# Green background color
BACKGROUND_GREEN='\e[42m'
# Blue background color
BACKGROUND_BLUE='\e[44m'

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"


usage()
{
    echo "ToDo"
    exit 1
}
# --------------------------------------- #
# --- Checking Command Line Arguments --- #
# --------------------------------------- #
# Supported command line arguments:
#   -h|--help
#   -a|--address    IP of SSH server for remote VMAF
#   -u|--user       User for SSH server login
#   -d|--doe        DoE worksheet exported as CSV UTF-8 file
PARAMS=""
while (( "$#" )); do
    case "$1" in
        -h|--help)
            usage
            shift
            ;;
        -u|--user)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then # Check length of argument and first character of argument != '-'
                SSH_USER=$2
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        -a|--address)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then # Check length of argument and first character of argument != '-'
                SSH_IP=$2
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        -d|--doe)
            if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then # Check length of argument and first character of argument != '-'
                DOE_FILE=$2
                shift 2
            else
                echo "Error: Argument for $1 is missing" >&2
                exit 1
            fi
            ;;
        -*|--*=) # unsupported flags
            echo "Error: Unsupported flag $1" >&2
            exit 1
            ;;
        *) # DEFAULT
            PARAMS="${PARAMS} $1" # preserve positional arguments
            shift
            ;;
    esac
done
# set positional arguments in their proper place
eval set -- "${PARAMS}"


# ---------------------- #
# --- Processing DoE --- #
# ---------------------- #
echo -e "${BACKGROUND_BLUE}Processing DoE specified in file ${DOE_FILE}:${STYLE_RESET}"
echo -e "${BACKGROUND_BLUE}Configuring Video source for GStreamer pipeline...${STYLE_RESET}"
GSTPIPE_SRC="gst-launch-1.0 -e "
run=1
while IFS=";" read -r motion bitrate_in bitrate_out twopass iframe quantI quantP quantB mvbuffer cabac vbv
do
    echo -e "\n\n${BACKGROUND_BLUE}Setting #${run}:${STYLE_RESET}"
    echo -e "${BACKGROUND_BLUE}${motion} ${bitrate_in} ${bitrate_out} ${twopass} ${iframe} ${quantI} ${quantP} ${quantB} ${mvbuffer} ${cabac} ${vbv}${STYLE_RESET}"
    echo -e "\n${BACKGROUND_BLUE}Generating GStreamer pipelines...${STYLE_RESET}"

    case ${motion} in
        "low")
            GSTPIPE_SRC+="videotestsrc pattern=colors num-buffers=300 ! " # -> no motion content
            case ${bitrate_in} in   # -> bitrate of video source (width*height*framerate)
                "low") # -> 640x480
                    GSTPIPE_SRC+="'video/x-raw, width=(int)640, height=(int)480, framerate=(fraction)30/1, format=(string)I420' "
                    width=640
                    height=480
                    fps=30
                    ;;
                "high") # -> 3840x2160
                    GSTPIPE_SRC+="'video/x-raw, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, format=(string)I420' "
                    width=3840
                    height=2160
                    fps=30
                    ;;
                *)
                    echo -e "\n\n${BACKGROUND_RED}Input bitrate ${bitrate_in} not supported${STYLE_RESET}"
                    echo -e "Use low, or high instead"
                    exit 1
                    ;;
            esac
            ;;
        "high")
            GSTPIPE_SRC+="filesrc location=${SCRIPT_DIR}/extensive.mp4 " # -> high motion content
            case ${bitrate_in} in   # -> bitrate of video source (width*height*framerate)
                "low") # -> 640x480
                    GSTPIPE_SRC+="blocksize=460800 ! " # blocksize=width*height*bytesPerPixel (I420->12bit->bytesPerPixel=1.5)
                    GSTPIPE_SRC+="'video/x-raw, width=(int)640, height=(int)480, framerate=(fraction)30/1, format=(string)I420' "
                    width=640
                    height=480
                    fps=30
                    ;;
                "high") # -> 3840x2160
                    GSTPIPE_SRC+="blocksize=12441600 ! " # blocksize=width*height*bytesPerPixel (I420->12bit->bytesPerPixel=1.5)
                    GSTPIPE_SRC+="'video/x-raw, width=(int)3840, height=(int)2160, framerate=(fraction)30/1, format=(string)I420' "
                    width=3840
                    height=2160
                    fps=30
                    ;;
                *)
                    echo -e "\n\n${BACKGROUND_RED}Input bitrate ${bitrate_in} not supported${STYLE_RESET}"
                    echo -e "Use low, or high instead"
                    exit 1
                    ;;
            esac
            ;;
        *)
            echo -e "${BACKGROUND_RED}Argument ${motion} for DoE factor 'motion' not supported${STYLE_RESET}"
            echo -e "Use low, or high instead"
            exit 1
            ;;
    esac

    GSTPIPE_ENC=$GSTPIPE_SRC
    GSTPIPE_REF=$GSTPIPE_SRC

    GSTPIPE_REF+="! y4menc ! filesink location=${SCRIPT_DIR}/reference${fps}fps_run${run}.y4m"

    GSTPIPE_ENC+="! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2h264enc "
    GSTPIPE_ENC+="bitrate=${bitrate_out} EnableTwopassCBR=${twopass} "
    case ${iframe} in
        "low")
            GSTPIPE_ENC+="iframeinterval=20 SliceIntraRefreshInterval=10 "
            ;;
        "high")
            GSTPIPE_ENC+="iframeinterval=120 SliceIntraRefreshInterval=60 "
            ;;
        *)
            echo -e "${BACKGROUND_RED}Argument ${motion} for DoE factor iframe is not supported${STYLE_RESET}"
            echo -e "Use low, or high instead"
            exit 1
            ;;
    esac
    # The range of B frames does not take effect if the number of B frames is 0. (https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/accelerated_gstreamer.html#wwpID0E0YX0HA)
    GSTPIPE_ENC+="quant-i-frames=${quantI} quant-p-frames=${quantP} quant-b-frames=${quantB} num-b-frames=1 EnableMVBufferMeta=${mvbuffer} cabac-entropy-coding=${cabac} "
    GSTPIPE_ENC+="! nvv4l2decoder ! nvvidconv ! 'video/x-raw' ! y4menc ! filesink location=${SCRIPT_DIR}/distorted${fps}fps_run${run}.y4m"
    echo -e "${BACKGROUND_BLUE}Distorted Video:${STYLE_RESET}"
    echo -e "${FOREGROUND_BLUE}${GSTPIPE_ENC}${STYLE_RESET}"
    echo -e "${BACKGROUND_BLUE}Reference Video:${STYLE_RESET}"
    echo -e "${FOREGROUND_BLUE}${GSTPIPE_REF}${STYLE_RESET}"

    # --- Launching GStreamer pipelines (surpress detailed output) --- #
    echo -e "${BACKGROUND_BLUE}Launching GStreamer pipeline for encoded video (distorted):${STYLE_RESET}"
    eval "${GSTPIPE_ENC[@]}" #> /dev/null
    echo -e "${BACKGROUND_BLUE}Launching GStreamer pipeline for uncompressed video (reference):${STYLE_RESET}"
    eval "${GSTPIPE_REF[@]}" #> /dev/null

    # --- Create and Check Remote Directories --- #
    echo -e "\n${BACKGROUND_BLUE}Video transfer to remote machine:${STYLE_RESET}"
    SSH_DIR_ADD="v4l2h264/motion_${motion}/bitrate_${bitrate_in}/"
    SSH_DIR="/home/${SSH_USER}/metrics/${SSH_DIR_ADD}"  # Create variable holding path of directory for both:
                                                    #   1.) reference.y4m and distorted.y4m videos
                                                    #   2.) remote VMAF
    ssh ${SSH_USER}@${SSH_IP} "test -d ${SSH_DIR}" < /dev/null # see https://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash
    if [ $? -ne 0 ]; then # Directory does not exist
        echo -e "${BACKGROUND_BLUE}Creating remote directory for run #${run}: ${SSH_DIR}...${STYLE_RESET}"
        ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${SSH_DIR}" < /dev/null # see https://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash
    else # Directory already exists
        echo -e "${BACKGROUND_BLUE}Remote directory ${SSH_DIR} already exists${STYLE_RESET}"
    fi

    # --- Transfer Video Files --- #
    echo -e "${BACKGROUND_BLUE}Transfering reference and distorted videos of run #${run}:${STYLE_RESET}"
    scp ${SCRIPT_DIR}/distorted${fps}fps_run${run}.y4m ${SSH_USER}@${SSH_IP}:${SSH_DIR} < /dev/null # see https://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash
    scp ${SCRIPT_DIR}/reference${fps}fps_run${run}.y4m ${SSH_USER}@${SSH_IP}:${SSH_DIR} < /dev/null # see https://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash

    # --- Run VMAF on Remote Machine --- #
    echo -e "\n${BACKGROUND_BLUE}Running VMAF metric for DoE run #${run} on remote machine...${STYLE_RESET}"
    ssh ${SSH_USER}@${SSH_IP} "vmaf -r ${SSH_DIR}/reference${fps}fps_run${run}.y4m -d ${SSH_DIR}/distorted${fps}fps_run${run}.y4m -w ${width} -h ${height} -p 420 -b 12 -o ${SSH_DIR}/log${fps}fps_run${run}.xml --threads 8" < /dev/null # see https://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash
    echo -e "${BACKGROUND_BLUE}VMAF metric for DoE run #${run} finished.${STYLE_RESET}"

    # --- Remove Videos on Remote Machine (Laptop) --- #
    echo -e "\n${BACKGROUND_BLUE}Removing videos from remote machine${STYLE_RESET}"
    ssh ${SSH_USER}@${SSH_IP} "rm ${SSH_DIR}/distorted${fps}fps_run${run}.y4m ${SSH_DIR}/reference${fps}fps_run${run}.y4m" < /dev/null # see https://stackoverflow.com/questions/9393038/ssh-breaks-out-of-while-loop-in-bash

    # --- Remove videos on local machine (SPU) --- #
    echo -e "\n${BACKGROUND_BLUE}Removing videos from local machine${STYLE_RESET}"
    rm distorted${fps}fps_run${run}.y4m reference${fps}fps_run${run}.y4m

    ((run++))

done < <(cut -d ";" -f5,6,7,8,9,10,11,12,13,14,15,16 ${DOE_FILE} | tail -n +2) # read from the second line of the file (no header) and only read the columns specified with -f
((run--))

exit 0

.csv file:

"StdOrder";"RunOrder";"CenterPt";"Blocks";"motion";"bitrate_in";"bitrate_out";"twopass";"iframe";"quantI";"quantP";"quantB";"mvbuffer";"cabac"
1;1;1;1;"low";"low";200000;"false";"low";0;0;0;"true";"true"
6;2;1;1;"high";"low";80000000;"false";"low";51;0;51;"true";"false"
8;3;1;1;"high";"high";80000000;"false";"high";0;0;0;"false";"true"
2;4;1;1;"high";"low";200000;"false";"high";0;51;51;"false";"false"
3;5;1;1;"low";"high";200000;"false";"high";51;0;51;"false";"false"
7;6;1;1;"low";"high";80000000;"false";"low";0;51;51;"true";"false"
10;7;1;1;"high";"low";200000;"true";"high";51;0;0;"true";"false"
9;8;1;1;"low";"low";200000;"true";"low";51;51;51;"false";"true"
4;9;1;1;"high";"high";200000;"false";"low";51;51;0;"true";"true"
13;10;1;1;"low";"low";80000000;"true";"high";0;0;51;"true";"true"
5;11;1;1;"low";"low";80000000;"false";"high";51;51;0;"false";"true"
12;12;1;1;"high";"high";200000;"true";"low";0;0;51;"false";"true"
15;13;1;1;"low";"high";80000000;"true";"low";51;0;0;"false";"false"
14;14;1;1;"high";"low";80000000;"true";"low";0;51;0;"false";"false"
16;15;1;1;"high";"high";80000000;"true";"high";51;51;51;"true";"true"
11;16;1;1;"low";"high";200000;"true";"high";0;51;0;"true";"false"

I believe you need to show the content of `.csv` file as well which is being parsed to the `while` loop... — User123, Aug 11 '21 at 10:41
The root cause for many inexplicable behaviors is that you used a DOS editor and it added carriage returns to your file where you don't expect or indeed are able to see any. Possible duplicate of [Are shell scripts sensitive to encoding and line endings?](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings) — tripleee, Aug 11 '21 at 10:45
@User123 I have no problems reading the .csv file. I have the correct arguments in my variable `${arg}` but the case statement won't evaluate them correctly. — Bruno Kempf, Aug 11 '21 at 11:00
But you are forcing us to believe that the file contains what you say it contains, while many of the symptoms would be typical for when that was not really the case after all. Again, please provide a [mre] with the actual data. For one thing, the file cannot be delimited at all for your script to work. — tripleee, Aug 11 '21 at 11:01
@tripleee That is also not the problem. Same problem no matter what line endings are used. — Bruno Kempf, Aug 11 '21 at 11:01
@tripleee I provide the .csv file, but as mentioned reading the values is not the problem. When I enter the default state I display the value of `${arg}` and it actually holds the value it is supposed to, but the case won't evaluate it accordingly. I have no idea why. — Bruno Kempf, Aug 11 '21 at 11:06
Thanks for the update, but none of the lines in your data contain `valueX`. Again, for the script to work, the line must only contain a single field. The trivial explanation is that you expect `arg` to contain only one field, but it contains all of them. — tripleee, Aug 11 '21 at 11:07
@tripleee Thank for your advice. I thought it would be helpful to post a short version. As it turns out to be confusing, I posted the complete bash script and .csv file. As i mentioned, reading the file is not the problem. Evaluating the read arguments in the case-statement is the problem. Without the case (just echoing the read arguments) everything works fine. — Bruno Kempf, Aug 11 '21 at 11:17
The idea to provide a _minimal_ example was certainly correct, but it needs to be _reproducible,_ too. — tripleee, Aug 11 '21 at 11:19
@tripleee: If a CR would sneak into the variable, I think the output would nor be _Argument "valueX" not supported_, but something like _" not supportedX" not supported_ — user1934428, Aug 11 '21 at 11:29
@BrunoKempf : The main problem is an incorrect parsing of the CSV-file. According to the definition of a CSV file, a field with the value `foo` could also be represented quoted, i.e. by `"foo"`. Both denote the same value in CSV terms. This allows to have fields which have inside them quotes as a value, or multi-line fields. However, when you read the CSV file, an entry `foo` becomes the value `foo` (3 characters) in your variable, while a CSV entry `"foo"` would become the value `"foo"` (5 characters). You have to decode the CSV fields after reading. — user1934428, Aug 11 '21 at 11:34

score 3 · Accepted Answer · answered Aug 11 '21 at 11:07

3

Apparently, the argument (here: "valueX") read from the file is correct

Since you consider the quotes being part of the argument to be correct, in order to match them with a case pattern, you have to escape the pattern quotes:

  case ${arg} in
    \"valueX\")

Interestingly, this appears to be a Bash (documentation) error, as that says only:

Each pattern undergoes tilde expansion, parameter expansion, command substitution, and arithmetic expansion.

It doesn't tell that the pattern undergoes quote removal.

answered Aug 11 '21 at 11:07

Armali

18,255
14
57
171

3

Spot on - with the updated code, it's clear that this was the actual problem here. – tripleee Aug 11 '21 at 11:17
1

You have eagle eye vision! Well spotted! – kvantour Aug 11 '21 at 11:28
1

Thanks Armali this fixed it! @tripleee Sorry for the confusion. Next time I will provide a better (reproducible) example. – Bruno Kempf Aug 11 '21 at 11:32
This is not a *documentation error*. Pattern matching itself entails consumption of quoting characters, listing quote removal among expansions a pattern undergoes would imply that two rounds of quote removal are performed, which is.. incorrect. – oguz ismail Aug 11 '21 at 15:44
@oguz ismail - So you call the Bash documentation of [`case`](https://www.gnu.org/software/bash/manual/html_node/Conditional-Constructs.html), which says _The_ word _undergoes tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal …_ incorrect - that does not sound credible. – Armali Aug 11 '21 at 16:02
No, the *word* does undergo quote removal, as it is not interpreted as a pattern. That's irrelevant. What I'm saying is, the second part of your answer is wrong. And I suggest removing it. – oguz ismail Aug 11 '21 at 16:37
1

It is provably true that the `case` patterns undergo a removal of quotes, even if you deny to name it "quote removal" and invent the term "consumption of quoting characters" for the process. I maintain that there is a documentation error, as the description of [Pattern Matching](https://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html) does not mention that it _itself entails consumption of quoting characters_ - on the contrary, it says _Any character that appears in a pattern, other than the special pattern characters described below, matches itself_… – Armali Aug 11 '21 at 21:23
1

… and quotes are not among those special characters. If you have further insights, it would be valuable if you could back them up with relevant quotes from the documentation. – Armali Aug 11 '21 at 21:23
If there is a documentation error, it's in the pattern matching section, not the part you quoted. According to the manual, [quote removal is performed after filename expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Expansions.html#Shell-Expansions), and filename expansion uses the same pattern matching mechanism as the case command. However, quoting characters in filename patterns do not match themselves either (e.g. `ls 'a'*` lists `abc` but not `'abc`). So the *removal* of quoting characters we're discussing here is a step in pattern matching, and not **the** quote removal – oguz ismail Aug 17 '21 at 18:50
.. that comes after all word expansions, and the two needs to be differentiated. – oguz ismail Aug 17 '21 at 18:51
Another difference between the two processes, and perhaps the most important one, is that [quoting characters that result from word expansions are not affected by **the** quote removal](https://www.gnu.org/software/bash/manual/html_node/Quote-Removal.html), but pattern matching does consume them. For example, in a `case` command, given `p='a\*'`, the pattern `$p` matches the string `a*`, but it does not match itself (see [online demo](https://ideone.com/dtfPy2)). – oguz ismail Aug 17 '21 at 19:50
What you try to stylize as **the** quote removal is not the only occurrence of the term in the documentation - if you read your own [comment](https://stackoverflow.com/questions/68740361/bash-case-statement-always-enters-default/68740818?noredirect=1#comment121493558_68740818), you'll find another one. So, your argument is incorrectly based on the assumption that there were only the singular quote removal which is performed after filename expansion, even if you insist on calling certain quote removals "removal of quoting characters". – Armali Aug 17 '21 at 19:56
All instances of the term quote removal in the manual refer to the same thing. Anyway, why don't you report this *documentation bug* at bug-bash@gnu.org then? – oguz ismail Aug 18 '21 at 04:02
1

Your example with `p='a\*'` is interesting at first glance, but replacing the backslash with `"` shows that you are mistaken to think _pattern matching does consume them_ [_quoting characters that result from word expansions_]. The exceptional example with backslash follows the documented pattern matching behavior: _A backslash escapes the following character; the escaping backslash is discarded when matching._ - As to _why don't_ I _report this documentation bug at bug-bash@gnu.org_ - I can't because I'm too busy discussing with you. – Armali Aug 18 '21 at 15:58
1

I arrived at this SO question after going slightly bonkers trying to figure out why _pattern_ text was apparently going through the quote removal stage even though the man page did not document it. Glad I'm not alone! – Ti Strga Apr 28 '22 at 19:30

score 1 · Answer 2 · answered Aug 11 '21 at 11:15

None of your sample data contains valueX, but assuming you were looking for e.g. "high" in the fifth field, you need to ask the shell to split into at least six fields;

while IFS=";" read -r _first _second _third _fourth arg _rest
do
  case ${arg} in
    '"high"')
      var+="blabla"
      ;;
    '"low"')
      var+="blublu"
      ;;
    *)
      echo -e "Argument ${arg} not supported"
      exit 1
      ;;
  esac
done < filename

Probably a better solution here is to use Awk, though.

awk -F ';' '$5 !~ /"(high|low)"/ { print "Argument " $5 " is not supported" }' filename

Without the rest of your script, it's hard to tell in which direction to continue this; very often, if you are using Awk anyway, it makes sense to implement the rest of the logic in Awk, too.

The funky quoting is definitely something I picked up from @armali's answer. — tripleee, Aug 11 '21 at 11:18

Bash - case statement always enters default

2 Answers2