Bash parse csv file dynamically

Question

I changed it to this now:

Parse_csv() {
  # Read the first row of the CSV file
  IFS=',' read -ra headers <<< "$(head -n 1 "$1")"

  # Declare an associative array to store the variables

  # Iterate through the headers and dynamically create variables

  element_number=0
  total_elements=${#headers[@]}
  echo "total elements $total_elements"
  # Read and process subsequent rows
  tail -n +2 "$1" | while IFS=',' read -ra values; do
    for value in "${values[@]}"; do
    
    #echo "element Number" $element_number
    #echo "current header" "${headers[element_number]}"
    
    #echo "value $value"
    if [ $element_number == $total_elements ];
    then
    echo " This is the end of elements"
    #eval "${headers[$element_number]}=$value"
    element_number=0
    else
    
    echo " element number $element_number"
    element_number=$((element_number+1))
    fi
    done


  
  done


}

It is printing correctly but the eval is not happening. Also, last field since the csv file doesn't have a carriage return it is ignoring the last field.

Answer 1

The question is unclear. If it is a rewrite of a previous question, please make it self-contained.

To “parse” a CSV file in Bash while turning its headers into array variable names and storing each column as an array, one can use “pointers” in Bash, i.e. the declare -n construct.

The entire approach has major drawbacks though:

0. A function modifies global variable state instead of merely providing output and leaving it up tho the caller to (not) process it. This is hard to debug and bad for code isolation and reusability.
1. Changing a script’s variables based on input data is not a great idea. If you read a CSV column called PATH, for example, it becomes obvious why this is not the most secure way of data processing.
2. Unfortunately, Bash doesn’t have a (straightforward) equivalent of a struct (other than declare -A) such that it could be nested in another array. There goes the dream of per-row CSV data representation. Storing each column in a separate array makes many inconsistent states representable, such as column arrays with different sizes.

With all that↑ said, let’s read the CSV without unnecessary subshells, redirections, expensive external processes (head, tail) and without trying to assign variables “behind a pipe”, which won’t have any effect in the local (fork() parent) shell:

#!/bin/bash

parse_csv() {
  local -a headers line
  local header
  local -i index
  IFS=, read -ra headers
  for header in "${headers[@]}"; do
    declare -ag "${header}=()"
  done
  while IFS=, read -ra line; do
    for index in "${!line[@]}"; do
      local -n column="${headers[index]}"
      column+=("${line[index]}")
    done
  done
}

And of course we should test it using a bit of (valid) data. As noted above, handling of invalid data (i.e. correct error reporting) would be a challenge with this setup, so the whole example assumes (and contains) valid inputs only:

#!/bin/bash
set -euo pipefail

parse_csv <<- BLAH
a,b,c,d,e
1,2,3,4,5
f,g,h,i,j
blah,foo,meh,bar,oops
BLAH

for column in {a..e}; do
  declare -n array="$column"
  printf '%s\n' "${array[*]@A}"
done

Here’s the output from the above, showing that each column is now (indeed) a Bash array:

declare -a a=([0]="1" [1]="f" [2]="blah")
declare -a b=([0]="2" [1]="g" [2]="foo")
declare -a c=([0]="3" [1]="h" [2]="meh")
declare -a d=([0]="4" [1]="i" [2]="bar")
declare -a e=([0]="5" [1]="j" [2]="oops")

Answer 2

Thanks for all the input. This was an adventure The way I was able to achieve this is using two functions leveraging awk like suggested by @barmar

print_variables() {
    declare -A vars="${1#*=}"
    local end_of_line=$2
    for key in "${!vars[@]}"; do
        echo "Field: $key Value: ${vars[$key]}"
    done
    if [[ $end_of_line -eq 1 ]]; then
        echo "End of line reached"
        act_on_csv_data
        # Call another function here if you wish
    fi
}

parse_csv() {
    local filename=$1
    awk -F ',' '
        NR==1 {
            for(i=1; i<=NF; i++) {
                header[i]=$i;
            }
        } 
        NR>1 {
            gsub(/\r$/, "", $NF);
            printf "("
            for(i=1; i<=NF; i++) {
                if(length($i) > 0) {
                    printf "[\"%s\"]=\"%s\"", header[i], $i;
                    if(i != NF) {
                        printf ", "
                    }
                }
            }
            printf ")\nEOL\n"
        }
    ' "$filename" | {
    vars=""
    while read -r line; do
        if [[ $line != "EOL" ]]; then
            vars="declare -A vars=$line"
        else
            print_variables "$vars" 1
            vars=""
        fi
    done
    }
}