Bash script: cannot sort file just created in same statement?

Question

Here is the code I am using in my script:

cat log.txt | egrep -o "[0-9]{1,3}" >new.txt | sort new.txt | uniq >uniq.txt

I am essentially trying to find numbers within a file and list all of those numbers in a new file, then sort the new file, then create another file of only the unique numbers. This is for a school project and the professor said all 4 of the commands should be completed in a single line of code using pipes. The code does work, but not the first time. I can run the script and it gives the error below:

sort: cannot read: new.txt: no such file or directory

It does still create new.txt and uniq.txt though as I can see them when I ls. If I run the script again it will now work. If I remove the files and run it again it will once again give the error until I run it a second time. The last part of the script is supposed to delete those files which led me to discover this issue.

Any help would be greatly appreciated.

Answer 1

Your syntax is incorrect. Using a tool like https://www.shellcheck.net will help you with this type of error.

Perhaps something like this will work:

grep -Eo "[0-9]{1,3}" log.txt | sort | uniq > uniq.txt

Edit

To keep the intermediate file ("new.txt") you can use the tee command:

grep -Eo "[0-9]{1,3}" log.txt | tee new.txt | sort | uniq > uniq.txt

Answer 2

The problem is that you're trying to pass the data from egrep to sort in two different (conflicting) ways: with a pipe, and via the file new.txt. You can use either one of these, but you can't use both at once.

Here's the command you're using:

cat log.txt | egrep -o "[0-9]{1,3}" >new.txt | sort new.txt | uniq >uniq.txt

The >new.txt part sends the output from egrep into the file new.txt, instead of through the pipe (|) to sort. Similarly, you told sort to read from new.txt instead of from its standard input (the pipe coming from egrep). This sounds like it might work, but the problem is that in a pipeline like this, all of the commands run at once, rather than one at a time. Apparently, sort is trying to read from new.txt before it's even been created (unless you run it twice, in which case sort finds the leftover file from the first time).

You could separate the sections so they run in sequence:

cat log.txt | egrep -o "[0-9]{1,3}" >new.txt
sort new.txt | uniq >uniq.txt

Here, there are two separate pipelines, and the second doesn't run until the first one has finished, so there's no timing problem. But your prof wants pipes all the way, so use the other solution: remove all of the bits relating to new.txt (I won't give the full solution, since this is homework).

BTW, you don't actually need cat here, since egrep can read directly from files just fine, either by naming it or with an input redirect:

egrep -o "[0-9]{1,3}" log.txt ...
egrep -o "[0-9]{1,3}" <log.txt ...

Using cat to pipe a file to something that could've just read it directly is a classic Useless Use of cat. Some people consider this wasteful, others consider it good style. There's some arguments (from both sides) in this question.