How can I use the concatenate command to automate the merging of several files?

Question

I need to merge two sets of files together. I don't mind what language is used as long as it works. For example: I have a folder with both sets of files

N-4A-2A_S135_L001_R1_001.fastq.gz
N-4A-2A_S135_L001_R2_001.fastq.gz
N-4A-2A_S135_L001_R1_001-2.fastq.gz
N-4A-2A_S135_L001_R2_001-2.fastq.gz

N-4-Bmp-1_S20_L001_R1_001.fastq.gz
N-4-Bmp-1_S20_L001_R2_001.fastq.gz
N-4-Bmp-1_S20_L001_R1_001-2.fastq.gz
N-4-Bmp-1_S20_L001_R2_001-2.fastq.gz

I need them merged like this:

N-4A-2A-S135-L001_R1_001.fastq.gz + N-4A-2A-S135-L001_R1_001-2.fastq.gz > N-4A-2A-S135-L001_R1_001-cat.fastq.gz
N-4A-2A-S135-L001_R2_001.fastq.gz + N-4A-2A-S135-L001_R2_001-2.fastq.gz > N-4A-2A-S135-L001_R2_001-cat.fastq.gz
N-4-Bmp-1_S20_L001_R1_001.fastq.gz + N-4-Bmp-1_S20_L001_R1_001-2.fastq.gz > N-4-Bmp-1_S20_L001_R1_001-cat.fastq.gz
N-4-Bmp-1_S20_L001_R2_001.fastq.gz + N-4-Bmp-1_S20_L001_R2_001-2.fastq.gz > N-4-Bmp-1_S20_L001_R2_001-cat.fastq.gz

I have a total of ~180 files so any way to automate this would be amazing. They all follow the same format of:

(sample name) + "R1_001" or "R2_001" + "" or "-2" + .fastq.gz

Answer 1

According to example you provided, your file names look sortable. If this is not the case, this will not work and this answer will be invalid.

Idea of the following snippet is really simple - sort all the filenames and then join adjacent tuples.

This should work in bash:

#!/bin/bash
files=( $( find . -type f | sort -r) )
files_num=${#files[@]}
for (( i=0 ; i < ${files_num} ; i=i+2 )); do
  f1="${files[i]}"
  f2="${files[i+1]}"
  fname=$(basename -s '.gz' "$f1")".cat.gz"
  echo "$f1 and $f2 -> $fname"
  cat "$f1" "$f2" > "$fname"
done

or simpler version in ZShell (zsh)

#!/bin/zsh
files=( $( find . -type f | sort -r) )
for f1 f2 in $files; do
  fname=$(basename -s '.gz' "$f1")".cat.gz"
  echo "$f1 and $f2 -> $fname"
  cat "$f1" "$f2" > "$fname"
done

Answer 2

You did say any language, so ... I prefer perl

#!/usr/bin/perl
# concatenate files

master(@ARGV);
exit(0);

# master -- master control
sub master
{
    my(@argv) = @_;
    my($tail,@tails);
    local(%tails);

    # scan argv for -opt
    while (1) {
        my($opt) = $argv[0];
        last unless (defined($opt));

        last unless ($opt =~ s/^-/opt_/);
        $$opt = 1;

        shift(@argv);
    }

    # load up the current directory
    @tails = dirload(".");

    foreach $tail (@tails) {
        $tails{$tail} = 0;
    }

    # look for the base name of a file
    foreach $tail (@tails) {
        docat($tail)
            if ($tail =~ /R[12]_001\.fastq\.gz$/);
    }

    # default mode is "dry run"
    unless ($opt_go) {
        printf("\n");
        printf("rerun with -go to actually do it\n");
    }
}

# docat -- process a pairing
sub docat
{
    my($base) = @_;
    my($tail);
    my($out);
    my($cmd);
    my($code);

    # all commands are joining just two files

    $tail = $base;
    $tail =~ s/\.fastq/-2.fastq/;

    # to an output file
    $out = $base;
    $out =~ s/\.fastq/-cat.fastq/;

    $cmd = "cat $base $tail > $out";

    if ($opt_v) {
        printf("\n");
        printf("IN1: %s\n",$base);
        printf("IN2: %s\n",$tail);
        printf("OUT: %s\n",$out);
    }
    else {
        printf("%s\n",$cmd);
    }

    die("docat: duplicate IN1\n")
        if ($tails{$base});
    $tails{$base} = 1;

    die("docat: duplicate IN2\n")
        if ($tails{$tail});
    $tails{$tail} = 1;

    die("docat: duplicate OUT\n")
        if ($tails{$out});
    $tails{$out} = 1;

    {
        last unless ($opt_go);

        # execute the command and get error code
        system($cmd);
        $code = $? >> 8;

        exit($code) if ($code);
    }
}

# dirload -- get list of files in a directory
sub dirload
{
    my($dir) = @_;
    my($xf);
    my($tail);
    my(@tails);

    # open the directory
    opendir($xf,$dir) or
        die("dirload: unable to open '$dir' -- $!\n");

    # get list of files in the directory excluding "." and ".."
    while ($tail = readdir($xf)) {
        next if ($tail eq ".");
        next if ($tail eq "..");
        push(@tails,$tail);
    }

    closedir($xf);

    @tails = sort(@tails);

    @tails;
}

---

Here's the program output with the -v option:

IN1: N-4-Bmp-1_S20_L001_R1_001.fastq.gz
IN2: N-4-Bmp-1_S20_L001_R1_001-2.fastq.gz
OUT: N-4-Bmp-1_S20_L001_R1_001-cat.fastq.gz

IN1: N-4-Bmp-1_S20_L001_R2_001.fastq.gz
IN2: N-4-Bmp-1_S20_L001_R2_001-2.fastq.gz
OUT: N-4-Bmp-1_S20_L001_R2_001-cat.fastq.gz

IN1: N-4A-2A_S135_L001_R1_001.fastq.gz
IN2: N-4A-2A_S135_L001_R1_001-2.fastq.gz
OUT: N-4A-2A_S135_L001_R1_001-cat.fastq.gz

IN1: N-4A-2A_S135_L001_R2_001.fastq.gz
IN2: N-4A-2A_S135_L001_R2_001-2.fastq.gz
OUT: N-4A-2A_S135_L001_R2_001-cat.fastq.gz

rerun with -go to actually do it