1

I'm trying to compare the contents of two files in a bash script.

local_file=$(cat my_local_file.txt)
remote_file=$(curl -s "http://example.com/remote-file.txt")

if [ local_file == remote_file ]; then
  echo "Files are the same"
else
  echo "Files are different. Here is the diff:"
  diff <(echo "$local_file") <(echo "$remote_file")
fi

When I run the script, I see that I have a syntax error:

./bin/check_files.sh: line 8: syntax error near unexpected token `('
./bin/check_files.sh: line 8: `  diff <(echo "$local_file") <(echo "$remote_file")'

What am I doing wrong? How can I display a diff of these two strings from a bash script?

codeforester
  • 39,467
  • 16
  • 112
  • 140
Andrew
  • 227,796
  • 193
  • 515
  • 708
  • 1
    It's probably a better idea to download the file locally, rather than hold it in memory in Bash. In particular `$(...)` [doesn't preserve trailing newlines or NUL characters](https://stackoverflow.com/a/22607352/113632), so the contents of `local_file` and `remote_file` could potentially not reflect what's on-disk. – dimo414 Jul 10 '18 at 20:35
  • @dimo414 Does `diff` deal with NUL characters? I suspect not. – Barmar Jul 10 '18 at 21:49
  • @Barmar I can't say universally, since there are different implementations, but I'd consider any `diff` program that fails to do so broken. GNU diff does indeed work with NUL characters, try `diff <(printf 'foo\0bar') <(printf 'foo\0\0bar')`. – dimo414 Jul 11 '18 at 00:20
  • @dimo414 I don't think POSIX requires utilities that process text files to deal with embedded NUL characters. Many GNU utilities try to go beyond the requirements, but it's not something I would generally expect. – Barmar Jul 11 '18 at 04:28
  • @Barmar I'm not sure if this is a canonical source, but it looks like [POSIX `diff`](https://www.unix.com/man-page/posix/1posix/diff/) (see "Diff Binary Output Format") explicitly supports binary files. – dimo414 Jul 11 '18 at 05:32
  • @dimo414 Which basically says that if they're not text files, it becomes more like `cmp`. – Barmar Jul 11 '18 at 14:26
  • Sure - but it doesn't silently discard `NUL`s, which is what Bash does and what you originally objected to. – dimo414 Jul 12 '18 at 07:21

3 Answers3

5

Process substitution is a bash feature, which is usually not available in /bin/sh which is meant to be POSIX compatible.

Make sure to use the following shebang line if you want to run the script as an executable:

#!/bin/bash

instead of

#!/bin/sh

or use

bash script.sh

instead of

sh script.sh

if you run it like that

To make the script work with POSIX conform shells I would just download the file and compare it against the local file. Remove the downloaded file after the diff.

hek2mgl
  • 152,036
  • 28
  • 249
  • 266
  • 3
    Process substitution isn't even available in bash if it's run under the name "sh"; therefore, you *must* explicitly use "bash" instead of "sh" (preferably via the shebang, but running the script with `bash scriptname` would also work). – Gordon Davisson Jul 10 '18 at 20:34
2

In addition to the <(command) (process substitution) syntax issue, your code if [ local_file == remote_file ] compares the literal strings local_file and remote_file, rather than the content of the variables. You need $local_file and $remote_file to compare the contents. Need to enclose them in double quotes to prevent word splitting issues.

You could do this:

#!/bin/bash

local_file=$(< my_local_file.txt) # this is more efficient than $(cat file)
remote_file=$(curl -s "http://example.com/remote-file.txt")

if [ "$local_file" = "$remote_file" ]; then
  echo "Files are the same"
else
  echo "Files are different. Here is the diff:"
  diff <(printf '%s' "$local_file") <(printf '%s' "$remote_file")
fi

As stated by @dimo414, the limitation here is that the command substitution $(...) removes trailing newlines and that would cause a problem. So, it is better to download the remote file and compare it with the local file:

local_file=my_local_file.txt
curl -s "http://example.com/remote-file.txt" -o remote_file

if diff=$(diff -- "$local_file" remote_file); then
  echo "Files are the same"
else
  echo "Files are different. Here is the diff:"
  printf '%s' "$diff"
fi
codeforester
  • 39,467
  • 16
  • 112
  • 140
  • 2
    diff's return value tells you if the inputs match, so you could get by without the cmp call. Something like this should work and be slightly quicker. `local_file=my_local_file.txt curl -s "http://example.com/remote-file.txt" -o remote_file OUTPUT=$( diff -- "$local_file" "$remote_file" ) if [ $? ]; echo "Files are the same" else echo "Files are different. Here is the diff:" echo "$OUTPUT" fi` – Ryan Prescott Jul 10 '18 at 20:49
  • 1
    In your initial description and code, `if [ "$a" = "$b" ]; then` compares the strings also? curl would download to the file called `remote_file`, so the cmp line should be `if cmp -s -- "$local_file" "remote_file"; then` – Ryan Prescott Jul 10 '18 at 21:10
  • 1
    Thanks @RyanPrescott. Updated the answer. – codeforester Jul 10 '18 at 21:12
-1

You can also use the following command:

cmp -b "File_1.txt" "File_2.txt"
latika bhurani
  • 80
  • 7