How to output any lines from a file, excluding those found in another file?

Question

I need to output all lines in file1.txt, except those that are found in file2.txt, by matching the entirely lines.

E.g., file1.txt:

Cats eat fish.
Mice eat cheese.
Eagles can fly.
Mountains are tall.

E.g., file2.txt:

Cats eat fish.
Birds can fly.
Trees are tall.

E.g., output:

Mice eat cheese.
Eagles fly.
Mountains are tall.

I have used the following command:

grep -v -x -f file1.txt file2.txt

This appears to work, however, when the files are of certain lengths, it often reports grep: memory exhausted, so I need an alternative that will not create this memory problem.

The order of the lines is important, so they should not be sorted.
Any tool typically found within a default Linux install will be acceptable.

How can I output the lines of file1.txt, except those found in file2.txt, without encountering a memory problem?

Shouldn't you use `grep -vxf f2 f1` instead? (First f2, then f1). It works fine to me. — fedorqui, Mar 21 '14 at 12:37

score 3 · Answer 1 · edited May 23 '17 at 12:28

3

Try:

grep -Fxvf file2.txt file1.txt

References: find difference between two text files with one item per line

edited May 23 '17 at 12:28

Community

1
1

answered Mar 21 '14 at 12:50

Danilo Muñoz

623
1
7
17

In my tests so far, it seems to work. How does the addition of `-F` stop the memory problem? – Village Mar 21 '14 at 13:51

score 1 · Answer 2 · answered Mar 21 '14 at 12:48

try:

rm -f out.txt && while read -r line; do echo "checking if line $line exists in file2.txt"; if `grep -Fxq "$line" file2.txt`; then echo "$line exists in other for"; else echo "$line" >> out.txt; fi; done < file.txt

Explaination:

this deletes the output file (in case of continuous use...), then it checks line by line if one exists in the other.

As a bash file it's clearer:

rm out.txt
while read -r line
do
    echo "checking if line $line exists in file2.txt"
    if `grep -Fxq "$line" file2.txt`
    then
        echo "$line exists in other file"
    else 
        echo "$line" >> out.txt
    fi
done < file.txt

And the obvious generalization:

while read -r line
do
    echo "checking if line $line exists in $2"
    if `grep -Fxq "$line" $2`
    then
        echo "$line exists in $2"
    else 
        echo "$line" >> $3
    fi
done < $1

Where the first and second arguments would be the files and the outut file would be the third argument

How to output any lines from a file, excluding those found in another file?

2 Answers2