0

I have a python script that uses Pillow to open all png/jpg/jpeg files in a folder, copy some of their meta data (file name, size, width, height, pixels, size, etc) into a new object and push that object into a list called imageMetaData. I then traverse that list to compare every image to every other image to try and delete duplicate images (I have amassed a TON of duplicates to the point where of my 6000 images at least 1500 may be duplicates)

With a small size of images (~1500 is the biggest i have done successfully) it works fine! but when trying to run on my folder that has 6100 files it does not successfully create the imageMetaData list successfully and actually prints:

zsh: killed     python3 remove-duplicates.py

I have looked into this and it seems to be running out of ram. But it seems like my RAM should be enough to hold a list of ~6000 objects where each object has about 8 fields.

My function is below:

from PIL import Image
from os import listdir

mypath = 'my-path-to-folder/remove-dupes/'
initialLocation = 'my-folder-of-photos'
directoryList = listdir(mypath + initialLocation)

def loadObjects():
    myObjects = []
    if len(directoryList) > 1:
        for x in range(len(directoryList)):
            if ('jp' in directoryList[x].lower() or 'png' in directoryList[x].lower()):
                i = Image.open(mypath + initialLocation + '/' + directoryList[x])
                width, height = i.size
                pixels = i.load()
                i.close()
                myObjects.append({
                    'name': directoryList[x],
                    'width': width,
                    'height': height,
                    'pixels': pixels,
                    'size': os.stat(mypath + initialLocation + '/' + directoryList[x]).st_size,
                    'biggest': directoryList[x],
                    'index': x
                })
    return myObjects

As can be seen, the image is opened, loaded, and closed (correctly?) so i dont think I am leaving anything hanging. Any ideas as to why this is being killed or possibly how to get more details into why it was killed?

skyleguy
  • 979
  • 3
  • 19
  • 35
  • Doesn't each `pixels` contain a representation of the entire image? If so, that could consume quite a bit of memory. – Noah Dec 31 '20 at 20:12
  • yes right after posting i did consider this. what i do with those pixel arrays is take the current image's pixel array and check twenty random pixels to each image after it. if they are all the same i consider the image a duplicate. if one doesnt match then i know they are not the same image. i could cut back on memory usage considerably if i just pluck twenty random pixels from each object from the beginning. Will definitely work on this. seems like the correct way to go anyway – skyleguy Dec 31 '20 at 20:14
  • Comparing hashes is a typical way to avoid that problem – Noah Dec 31 '20 at 20:16
  • definitely unfamiliar with hashing in python. is it as easy as calling some hash function with the pixels array? and storing that instead? seems like using hash(pixels) is not working successfully with my test folder of 30 images, where it is supposed to cut out 9 duplicates but after hash cuts out 0 – skyleguy Dec 31 '20 at 20:27
  • 1
    Have a read here... https://stackoverflow.com/a/54053080/2836621 and here... https://stackoverflow.com/a/50118123/2836621 and here... https://stackoverflow.com/a/46246137/2836621 – Mark Setchell Dec 31 '20 at 21:04
  • excellent Mark, I like the ImageMagick approach for sure, i will try that out for sure! – skyleguy Dec 31 '20 at 21:11

1 Answers1

0

while it wasnt too pretty looking i did use Noah's suggestion to cut down on pushing the entire pixel array of each image into my new object by selecting 20 pixels using the height and width of the image. Unfortunately using hash(pixels) seemed to be returning a unique value even when the pixel arrays should have been the same? Would need to look into why that would be before attempting to cut out this logic and use hash values.

def getRandomPixels(pixels, width, height):
    randomPixels = []
    randomPixels.append(pixels[0,0])
    randomPixels.append(pixels[width - 1, height - 1])
    randomPixels.append(pixels[0, height - 1])
    randomPixels.append(pixels[width - 1, 0])
    randomPixels.append(pixels[width/2, height/2])
    randomPixels.append(pixels[0, height/2])
    randomPixels.append(pixels[width/2, 0])
    randomPixels.append(pixels[width - 1, height/2])
    randomPixels.append(pixels[width/2, height - 1])
    randomPixels.append(pixels[width/4, height/4])
    randomPixels.append(pixels[0, height/4])
    randomPixels.append(pixels[width/4, 0])
    randomPixels.append(pixels[width - 1, height/4])
    randomPixels.append(pixels[width/4, height - 1])
    randomPixels.append(pixels[width/8, height/8])
    randomPixels.append(pixels[0, height/8])
    randomPixels.append(pixels[width/8, 0])
    randomPixels.append(pixels[width - 1, height/8])
    randomPixels.append(pixels[width/8, height - 1])
    randomPixels.append(pixels[width/2, height/8])
    return randomPixels

I chose to select the pixels like this because the images in the folder could be different dimensions and this ensures that a picture of 1024 x 2048 will always compare itself to another picture of the same dimension by looking at the same pixels. it also selects from portions of the image that are further apart and less likely to be similar unless the picture is a duplicate.

with this new method i was able to run through 4975 images and load them in 618 seconds, and then the checking of duplicates took only 61 seconds due to the pixel array i was previously running through being 20x20 instead of height x width! Thanks Noah!

skyleguy
  • 979
  • 3
  • 19
  • 35