diff a directory recursively, ignoring all binary files

Question

Working on a Fedora Constantine box. I am looking to diff two directories recursively to check for source changes. Due to the setup of the project (prior to my own engagement with said project! sigh), the directories contain both source and binaries, as well as large binary datasets. While diffing eventually works on these directories, it would take perhaps twenty seconds if I could ignore the binary files.

As far as I understand, diff does not have an 'ignore binary file' mode, but does have an ignore argument which will ignore regular expression within a file. I don't know what to write there to ignore binary files, regardless of extension.

I'm using the following command, but it does not ignore binary files. Does anyone know how to modify this command to do this?

diff -rq dir1 dir2

Answer 1

Kind of cheating but here's what I used:

diff -r dir1/ dir2/ | sed '/Binary\ files\ /d' >outputfile

This recursively compares dir1 to dir2, sed removes the lines for binary files(begins with "Binary files "), then it's redirected to the outputfile.

Answer 2

Maybe use grep -I (which is equivalent to grep --binary-files=without-match) as a filter to sort out binary files.

dir1='folder-1'
dir2='folder-2'
IFS=$'\n'
for file in $(grep -Ilsr -m 1 '.' "$dir1"); do
   diff -q "$file" "${file/${dir1}/${dir2}}"
done

Answer 3

I came to this (old) question looking for something similar (Config files on a legacy production server compared to default apache installation). Following @fearlesstost's suggestion in the comments, git is sufficiently lightweight and fast that it's probably more straightforward than any of the above suggestions. Copy version1 to a new directory. Then do:

git init
git add .
git commit -m 'Version 1'

Now delete all the files from version 1 in this directory and copy version 2 into the directory. Now do:

git add .
git commit -m 'Version 2'
git show

This will show you Git's version of all the differences between the first commit and the second. For binary files it will just say that they differ. Alternatively, you could create a branch for each version and try to merge them using git's merge tools.

Answer 4

If the names of the binary files in your project follow a specific pattern (*.o, *.so, ...) as they usually do, you can put those patterns in a file and specify it using -X (hyphen X).

Contents of my exclude_file

*.o
*.so
*.git

Command:

diff -X exclude_file -r . other_tree > my_diff_file

---

UPDATE:

-x can be used instead of -X, to specify exclusion patterns on the command line rather than in a file:

diff -r -x *.o -x *.so -x *.git dir1 dir2

Answer 5

Well, as a crude sort of check, you could ignore files that match /\0/.

Answer 6

Use a combination of find and the file command. This requires you to do some research on the output of the file command in your directory; below I'm assuming that the files you want to diff is reported as ascii. OR, use grep -v to filter out the binary files.

#!/bin/bash

dir1=/path/to/first/folder
dir2=/path/to/second/folder

cd $dir1
files=$(find . -type f -print | xargs file | grep ASCII | cut -d: -f1)

for i in $files;
do
    echo diffing $i ---- $dir2/$i
    diff -q $i $dir2/$i
done

Since you probably know the names of the huge binaries, place them in a hash-array and only do the diff when a file is not in the hash,something like this:

#!/bin/bash

dir1=/path/to/first/directory
dir2=/path/to/second/directory

content_dir1=$(mktemp)
content_dir2=$(mktemp)

$(cd $dir1 && find . -type f -print > $content_dir1)
$(cd $dir2 && find . -type f -print > $content_dir2)

echo Files that only exist in one of the paths
echo -----------------------------------------
diff $content_dir1 $content_dir2    

#Files 2 Ignore
declare -A F2I
F2I=( [sqlite3]=1 [binfile2]=1 )

while read f;
do
    b=$(basename $f)
    if ! [[ ${F2I[$b]} ]]; then
        diff $dir1/$f $dir2/$f
    fi
done < $content_dir1