Faster way to check can image be open?

Question

I've got 200GB folders with images and some of them can't be opened. I want to find these images and delete them from their folder.

I tried Python code like this:

for image in all_image:   
   try: # open image
   except: # delete image

And it's too slow. How I can do it faster?

How I can parallelize this code?

import PIL
import os
import cv2
from PIL import ImageFile
from tqdm import tqdm
from pathlib import Path
import pandas as pd


def create_df(data_path):
    data = pd.DataFrame()
    folder_namee = [i for i in data_root.iterdir() if i.is_dir()]
    files = [j for i in sku_dirs for j in i.glob('*.jpg')]
    data['path'] = [str(i) for i in files]
    data['label'] = [i.parts[-2] for i in files]
    return data

if __name__ == "__main__":
    root = Path('some_path')
    data_root = root / 'dataset'
    df = create_df(data_root)


    for i, row in tqdm(df.iterrows()):
        try:
            img = PIL.Image.open(row.path)
        except Exception:
            print(row.path)
            print(row)
            if os.path.exists(row.path):
                os.remove(row.path)

@Alderven thank you for comment. Can you say how i can parallelize this code? — Igor Terekhin, Nov 02 '21 at 05:10
Please use the internationally agreed SI units. https://en.wikipedia.org/wiki/Gigabit `Gb` = gigabits. `GB` = gigabytes. — Mark Setchell, Nov 02 '21 at 08:02
check other libraries than PIL, [here is comparison of opening and reading data](https://stackoverflow.com/a/57717879/4601890) (so not the same as your case, but similar) — dankal444, Nov 02 '21 at 12:07
You might be able to go considerably faster if you can identify if there is a common issue (or two). Maybe you could print just the errors when they occur and see if there are just a couple of common problems... like truncated files, or unsupported compression or something. — Mark Setchell, Nov 03 '21 at 07:42

score 1 · Answer 1 · answered Nov 02 '21 at 07:21

You can use multiprocessing to parallelize processes. Example:

import os
from PIL import Image
from multiprocessing.pool import ThreadPool

IMAGE_EXT = ('.jpg', '.jpeg', '.png', '.gif')


def check_image(image_path):
    try:
        Image.open(image_path)
        print(f'Image is OK: {image_path}')
    except:
        os.remove(image_path)
        print(f'Image deleted: {image_path}')


def delete_broken_images(root_dir):
    pool = ThreadPool(processes=10)
    for subdir, dirs, files in os.walk(root_dir):
        for file in files:
            if file.endswith(IMAGE_EXT):
                image_path = os.path.join(subdir, file)
                pool.apply_async(check_image, (image_path,)).get()


delete_broken_images(r'c:\so\69805310\images')

When you try to open image you use PIL and not cv2, why that? cv2 is a bit faster as I know — Igor Terekhin, Nov 03 '21 at 04:31
I didn't know about that. You can switch to `cv2` if you'd like. — Alderven, Nov 03 '21 at 06:06

Faster way to check can image be open?

1 Answers1