Concatenate multiple log files into one single file to automate debug package generation

Question

Here are the files in my folder:

messages        
messages.1      
messages.2      
messages.3      
messages.4      
messages.5      
messages.6      
messages.7

I want to merge this files in one single file(e.g. log.txt) so that it is in this way:

messages.7 -> messages.6 -> messages.5 -> messages.4 -> messages.3 -> messages.2 -> messages.1 -> messages

I was manually executing cat commands i.e.

cat messages.7 >> log.txt
cat messages.6 >> log.txt
cat messages.5 >> log.txt
cat messages.4 >> log.txt
cat messages.3 >> log.txt
cat messages.2 >> log.txt
cat messages.1 >> log.txt
cat messages >> log.txt

I want to write a bash script which can handle any number of files.

FYI: messages is file with latest logs and messages.7 is the oldest.

Also, I do not want to reverse the content of the file. Number of files are dynamic and follows the same numbering patterns.

Check this out https://stackoverflow.com/questions/2150614/concatenating-multiple-text-files-into-a-single-file-in-bash?rq=2 — Maxwell D. Dorliea, Jun 05 '23 at 23:59
Actually, to do this manually, you can do it in a single statement: `cat messages{.{7..1},} >total_messages`. — user1934428, Jun 06 '23 at 05:46

Renaud Pacalet · Answer 1 · 2023-06-06T07:02:17.643

There are 3 main difficulties:

List only the files you want, no more: use the extglob and nullglob bash options.
List them in a portable way (ls is not recommended for scripting): use regular pathname expansion.
Sort them in reverse numeric order of extension: use sort.

You can try:

#!/usr/bin/env bash

shopt -s extglob nullglob
m=( messages?(.+([0-9])) )
printf '%s\n' "${m[@]}" | sort -t. -rnk2 | xargs cat >> log.txt

With the extglob option of bash messages?(.+([0-9])) expands as messages (if the file exists) and all existing messages.n files where n is a string of one or more digits. We store the file names in array m, print its content and pipe to sort with the -t. (use . as field separator) and -rnk2 (reverse, numeric, second field) options.

The nullglob bash option takes care of the case where none of the files exist (without nullglob the pathname expansion produces the messages?(.+([0-9])) pattern itself, which causes an error because no such file exists).

score 1 · Accepted Answer · answered Jun 06 '23 at 01:03

The key point of the answer is to use a properly crafted sort command for file names. For the given filename pattern messages[.N] the command would be:

sort -t . -k 2 -n -r

-t . -k 2 allows to sort by the numeric suffix, -n actually instructs sort to make 2 < 10, -r reverses the order of the output.

So you can use something like this to check the order of files:

ls -1 messages* | sort -t . -k 2 -n -r

... and to actually get the concatenated output in 'total_messages':

ls -1 messages* | sort -t . -k 2 -n -r | xargs cat > total_messages

Perhaps it should be noted that this would also include a, say, file `messages.zip`. I guess you can restrict the glob pattern to include only those files with numeric extension, by using extended globbing. — user1934428, Jun 06 '23 at 05:49

score 1 · Answer 3 · answered Jun 06 '23 at 18:58

1

Strange that 7 is the oldest, but anyway....

cat `ls -r messages*` >total_messages

or I might suggest using timestamps...

cat `ls -t messages*` >total_messages

answered Jun 06 '23 at 18:58

xpusostomos

1,309
11
15

Concatenate multiple log files into one single file to automate debug package generation

3 Answers3