extracting paths from second column in csv files using bash?

Question

I am trying to extract the second line of information from a csv file that has the following format:

1234,//data/sub-1234/ses-Baseline
1235,//data/sub-1235/ses-followup
1236,//data/sub-1234/ses-01
1235,//data/sub-1235/ses-02

I have the following script that should work, and it prints out the correct paths when asked. However, it always prints out "Path not found", which points to a problem with the csv file.

#!/bin/bash

csv_file="input.csv"

# Skip the first line 
sed '1d' "$csv_file" | while IFS=, read -r col1 col2 rest; do
  if [[ ! -e "$col2" ]]; then
    echo "Path not found: $col2"
  fi
done

where the output for a file that exists is still:

Path not found: //data/sub-1235/ses-02

Thanks so much.

score 0 · Answer 1 · answered Mar 29 '23 at 11:34

0

found a way around the csv problem:

#!/bin/bash

input_file="input.txt"

while read -r line; do
    path=$(echo "$line" | cut -f2)
    if ! ls "$path"; then
        echo "Path not found: $path"
    fi
done < "$input_file"

answered Mar 29 '23 at 11:34

Maria

73
6

`if ! ls "$path"; then` would produce a presumably undesirable error output from `ls` if the file didn't exist. You could mask that by `if ! ls "$path" 2>/dev/null` but you don't need `ls` here anyway as your original `-e` will work fine once the input issue is fixed, nor do you need a pipe to `cut` for every input line as your original `IFS=,` will work fine. – Ed Morton Mar 29 '23 at 12:54

score 0 · Answer 2 · answered Mar 29 '23 at 12:47

You probably have DOS line endings in your input, see Why does my tool output overwrite itself and how do I fix it?.

Try this (I tidied up a couple of other things too):

#!/usr/bin/env bash

csv_file="input.csv"

# Skip the first line 
while IFS=, read -r _ path; do
  if [[ ! -e "$path" ]]; then
    printf 'Path not found: %s\n' "$path"
  fi
done < <(tail -n +2 "$csv_file" | tr -d '\r')

The above assumes any \rs in your input are undesirable.

extracting paths from second column in csv files using bash?

2 Answers2