How to only concatenate files with same identifier using bash script?

Question

I have a directory with files, some have the same ID, which is given in the first part of the file name before the first underscore (always). e.g.:

S100_R1.txt
S100_R2.txt
S111_1_R1.txt
S111_R1.txt
S111_R2.txt
S333_R1.txt 

I want to concatenate those identical IDs (and if possible placing the original files in another dir, e.g. output:

original files (folder)
S100_merged.txt
S111_merged.txt
S333_R1.txt

Small note: I imaging that perhaps a solution would be to place all files which will be processed by the code in a new directory and than in a second step move the files with the appended "merged" back to the original dir or something like this...

I am extremely new to bash scripting, so I really can't produce this code. I am use to R language and I can think how it should be but can't write it.

My pitiful attempt is something like this:

while IFS= read -r -d '' id; do
    cat *"$id" > "./${id%.txt}_grouped.txt"
done < <(printf '%s\0' *.txt | cut -zd_ -f1- | sort -uz)

or this:

for ((k=100;k<400;k=k+1)); 
do
IDList= echo "S${k}_S*.txt" | awk -F'[_.]' '{$1}'
while [ IDList${k} == IDList${k+n} ]; do
        cat IDList${k}_S*.txt IDList${k+n}_S*.txt S${k}_S*.txt S${k}_S*.txt >cat/S${k}_merged.txt &; 
  done

Sometimes there are only one version of the file (e.g. S333_R1.txt) sometime two (S100*), three (S111*) or more of the same.

I am prepared for harsh critique for this question because I am so far from a solution, but if someone would be willing to help me out I would greatly appreciate it!

Answer 1

while read $fil;
do
  if [[ "$(find . -maxdepth 1 -name $line"_*.txt" | wc -l)" -gt "1" ]]
  then
      cat $line_*.txt >> "$line_merged.txt"
  fi 
done <<< "$(for i in *_*.txt;do echo $i;done | awk -F_ '{ print $1 }')"

Search for files with _.txt and run the output into awk, printing the strings before "_". Run this through a while loop. Check if the number of files for each prefix pattern is greater than 1 using find and if it is, cat the files with that prefix pattern into a merged file.

Answer 2

A plain bash loop with preprocessing:

# first get the list of files
find . -type f |
# then extract the prefix
sed 's@./\([^_]*\)_@\1\t&@' |
# then in a loop merge the files
while IFS=$'\t' read prefix file; do
     cat "$file" >> "${prefix}_merged.txt"
done

That script is iterative - one file at a time. To detect if there is one file of specific prefix, we have to look at all files at a time. So first an awk script to join list of filenames with common prefix:

find . -type f |    # maybe `sort |` ?
# join filenames with common prefix
awk '{ 
      f=$0;                            # remember the file path
      gsub(/.*\//,"");gsub(/_.*/,"");  # extract prefix from filepath and store it in $0
      a[$0]=a[$0]" "f                  # Join path with leading space in associative array indexed with prefix
    }
    # Output prefix and filanames separated by spaces.
    # TBH a tab would be a better separator..
    END{for (i in a) print i a[i]}
' |
# Read input separated by spaces into a bash array
while IFS=' ' read -ra files; do
     #first array element is the prefix
    prefix=${files[0]}
    unset files[0]
    # rest is the files
    case "${#files[@]}" in
    0) echo super error; ;;
    # one file - preserve the filename
    1) cat "${files[@]}" > "$outdir"/"${files[1]}"; ;;
    # more files - do a _merged.txt suffix
    *) cat "${files[@]}" > "$outdir"/"${prefix}_merged.txt"; ;;
    esac
done

Tested on repl.

---

IDList= echo "S${k}_S*.txt"

Executes the command echo with the environment variable IDList exported and set to empty with one argument equal to S<insert value of k here>_S*.txt.

Filename expansion (ie. * -> list of files) is not executed inside " double quotes.

To assign a result of execution into a variable, use command substitution var=$( something seomthing | seomthing )

IDList${k+n}_S*.txt

The ${var+pattern} is a variable expansion that does not add two variables together. It uses pattern when var is set and does nothing when var is unset. See shell parameter expansion and this my answer on ${var-pattern}, but it's similar.

To add two numbers use arithemtic expansion $((k + n)).

awk -F'[_.]' '{$1}'

$1 is just invalid here. To print a line, print it {print %1}.

Remember to check your scripts with http://shellcheck.net

Answer 3

for id in $(ls | grep -Po '^[^_]+' | uniq) ; do
    if [ $(ls ${id}_*.txt 2> /dev/null | wc -l) -gt 1 ] ; then
        cat ${id}_*.txt > _${id}_merged.txt
        mv ${id}_*.txt folder
    fi
done

for f in _*_merged.txt ; do
    mv ${f} ${f:1}
done

Answer 4

A pure bash way below. It uses only globs (no need for external commands like ls or find for this question) to enumerate filenames and an associative array (which is supported by bash since the version 4.0) in order to compute frequencies of ids. Parsing ls output to list files is questionable in bash. You may consider reading ParsingLs.

#!/bin/bash

backupdir=original_files # The directory to move the original files
declare -A count # Associative array to hold id counts

# If it is assumed that the backup directory exists prior to call, then
# drop the line below
mkdir "$backupdir" || exit

for file in [^_]*_*; do ((++count[${file%%_*}])); done
for id in "${!count[@]}"; do
    if ((count[$id] > 1)); then
        mv "$id"_* "$backupdir"
        cat "$backupdir/$id"_* > "$id"_merged.txt
    fi
done