0

I am trying to delete all duplicate files from my computer using a Python script

import os
import hashlib

# Set the directory you want to search for duplicate files
dir_path = "/Users/ebbyrandall"

# Create a dictionary to store the files and their corresponding hashes
files_dict = {}

# Walk through the directory and calculate the hash of each file
for root, dirs, files in os.walk(dir_path):
    for file in files:
        file_path = os.path.join(root, file)
        with open(file_path, "rb") as f:
            file_data = f.read()
            file_hash = hashlib.md5(file_data).hexdigest()

            # If the file hash is not in the dictionary, add it
            # Otherwise, it's a duplicate and we should delete it
            if file_hash not in files_dict:
                files_dict[file_hash] = file_path
            else:
                os.remove(file_path)

I get the following error message:

Traceback (most recent call last):
  File "deleteDuplicateFiles.py", line 14, in <module>
    with open(file_path, "rb") as f:
IOError: [Errno 2] No such file or directory: '/Users/ebbyrandall/myPython/env/bin/python3'
edwrand
  • 21
  • 2
  • If you try to delete a file that's supposed to be in a directory that doesn't exist, you get this error. – Grismar Dec 06 '22 at 21:59
  • so to "fix" this, should I create an error exception and prompt the script to continue despite not finding a specific file? – edwrand Dec 06 '22 at 22:03
  • See the answer below, I've tried to explain what can go on, and how to avoid it. – Grismar Dec 06 '22 at 22:12
  • Context: Is `myPython` a venv? Debug: What is the output of shell command `ls -l /Users/ebbyrandall/myPython/env/bin/python3`? – hc_dev Dec 06 '22 at 22:44
  • Be very careful doing things like this -- just because two files are identical doesn't mean it's safe to delete one. Some obvious examples: lock files (generally all empty, their significance is that they exist) and .gitignore files (multiple projects with identical ignore rules is normal, delete one and suddenly things that project is tracking things it shouldn't be). – Gordon Davisson Dec 07 '22 at 01:27

2 Answers2

0

Let's say you have this situation:

folder1/file.txt

And then run this:

import os

for folder in ['folder1/file.txt', 'folder1/other.txt', 'folder2/file.txt']:
    try:
        os.remove(folder)
        print('file from {folder} removed')
    except Exception as e:
        print('problem removing file from folder{folder}:')
        print(e)

The result is:

file from {folder} removed
problem removing file from folder{folder}:
[WinError 2] The system cannot find the file specified: 'folder1/other.txt'
problem removing file from folder{folder}:
[WinError 3] The system cannot find the path specified: 'folder2/file.txt'

This makes sense because in the first case, the file was there and could be removed. In the second case, the file didn't exist (error 2) and in the third case the folder didn't exist (error 3).

However, in your case, you were actually trying to open a folder as a file, or to open a file in a folder that doesn't exist:

try:
    with open('folder', 'rb') as f:
        print('File "folder" opened')
except Exception as e:
    print('problem opening file "folder"')
    print(e)

try:
    with open('folder2/file.txt', 'rb') as f:
        print('File "folder2/file.txt" opened')
except Exception as e:
    print('problem opening file "folder2/file.txt"')
    print(e)

Result:

problem opening file "folder"
[Errno 2] No such file or directory: 'folder'
problem opening file "folder2/file.txt"
[Errno 2] No such file or directory: 'folder2/file.txt'

The solution would be to distinguish files from folders and only to open files. Also, as the examples show, you can catch other errors (for example being unable to open a file that is locked by some other program) with a try .. except .. block.

Grismar
  • 27,561
  • 4
  • 31
  • 54
0

Issues with symbolic links

Assume your file name /Users/ebbyrandall/myPython/env/bin/python3 is a symbolic link because folder myPython is a virtual environment created with venv.

Reproduce the error

Creating a virtual environment

python3 -m venv ./my_venv                                                                                                       cd my_venv                                                                                                                      ls -l my_venv/bin/python3

shows a symbolic link: lrwxrwxrwx my_venv/bin/python3 -> /usr/bin/python3

Then trying to read from this file name:

import os

root = '/Users/hc_dev'
file = 'my_env/bin/python3'
file_path = os.path.join(root, file)
with open(file_path, "rb") as f:
     file_data = f.read()
     print(file_data)

outputs your error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '/Users/hc_dev/my_env/bin/python3'

Debug and test the type and existence

According to How to check if file is a symlink in Python? this file can be evaluated with pathlib as follows:

  • is not a link
  • is not a symlink
  • does not exist
  • is not a regular file
os.path.islink(file_path)
# False

from pathlib import Path

Path(file_path).is_symlink()
# False
Path(file_path).exists()
# False
Path(file_path).is_file()
# False
hc_dev
  • 8,389
  • 1
  • 26
  • 38