Perform diff on two directories by excluding hidden files using Python

Question

I am trying to write a Python script that performs diff -r on two directories. I would like to exclude hidden files on the directories.

Here is what I have.

source = "/Users/abc/1/"
target = "/Users/abc/2/"
bashCommand = 'diff -x ".*" -r ' + source + ' ' + target
# Below does not work either
# bashCommand = "diff -x '.*' -r " + source + ' ' + target

import subprocess

process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
if output:
    print "Directories do not match: " + output
else:
    print "Directories match"

I know that I should use -x '.*' to ignore dot files. I looked at this SO post. But it didn't help. How should I write this line?

bashCommand = 'diff -x ".*" -r ' + source + ' ' + target

Edit 1: I also tried this and it does not work either

pat = "-x \".*\""
bashCommand = "diff " + pat + " -r " + source + " " + target
print bashCommand

When I print the output and run the command manually it works as expected. However the Python script is not producing the desired result

$python BashCommand.py
diff -x ".*" -r /Users/abc/1/ /Users/abc/2/
Directories do not match: Only in /Users/abc/1/: .a


$diff -x ".*" -r /Users/abc/1/ /Users/abc/2/
$

Peter Gibson · Accepted Answer · 2018-01-09T23:14:29.487

In bash, single quotes and double quotes mean different things. From Difference between single and double quotes in Bash

Enclosing characters in single quotes (') preserves the literal value of each character within the quotes.

Whereas for double quotes:

The special parameters * and @ have special meaning when in double quotes (see Shell Parameter Expansion).

So your ".*" is getting expanded before being passed diff. Try switching the quotes

bashCommand = "diff -x '.*' -r " + source + ' ' + target

EDIT

Popen doesn't normally use a shell to execute your command (unless you pass shell=True) so you don't need actually need to escape the pattern at all:

>>> subprocess.Popen(['diff', '-x', "'.*'", '-r', 'a', 'b'])
<subprocess.Popen object at 0x10c53cb50>
>>> Only in a: .dot

>>> subprocess.Popen(['diff', '-x', '.*', '-r', 'a', 'b'])
<subprocess.Popen object at 0x10c53cb10>

It does not work either. When I pass a directory that has dot files, I get the dot files in diff's output. — SyncMaster, Jan 09 '18 at 23:05

Perform diff on two directories by excluding hidden files using Python

1 Answers1