2

I am generating patches from high resolution images. Below script worked fine in my local PC but when i shifted all images to server and generate patches it gives error in between.

Script to generate 32x32 patches

import os.path
import matplotlib.pyplot as plt
import os
import random
import numpy as np
import sys
from PIL import Image
from PIL import ImageFilter

cwd = os.getcwd()
print(cwd)

ori_data_dir = '/home2/multilab/Khawar/Image_Batches/Batch3/'

print(ori_data_dir)
output_path = '/home2/multilab/Khawar/Image_Batches/com_patches/'

imgNames = os.listdir(ori_data_dir)

patch_size = [32, 32]

sample_num_for_each_seq = 20
sample_idx = 0

for im_name in imgNames:

    seq_path = ori_data_dir + im_name

    img = Image.open(seq_path)
    im_height = img.height
    im_width = img.width

    img_arr = np.array(img)

    if len(img_arr.shape) < 3:
        continue

    sample_num_for_each_seq = int(im_height * im_width / patch_size[0] / patch_size[1]) * 4

    for sample_id in range(sample_num_for_each_seq):
        random_resize_factor = random.random() * 0.4 + 0.6  #random 0.6 - 1.0 resize
        crop_size = [round(patch_size[0] / random_resize_factor), round(patch_size[1] / random_resize_factor)]

        random_crop_x1 = 0 + int(random.random() * (im_width - crop_size[1] - 2))
        random_crop_y1 = 0 + int(random.random() * (im_height - crop_size[0] - 2))
        random_crop_x2 = random_crop_x1 + crop_size[1]
        random_crop_y2 = random_crop_y1 + crop_size[0]

        random_box = (random_crop_x1, random_crop_y1, random_crop_x2, random_crop_y2)

        sample_array = None

        randomCropPatch = img.crop(random_box)
        randomCropPatch = randomCropPatch.resize(patch_size, Image.BICUBIC)

        sample_out_path = output_path + im_name[0:len(im_name) - 4] + "_%04d.png" % (sample_id)
        randomCropPatch.save(sample_out_path)

    sample_idx += 1

    if sample_idx % 10 == 0:
        # break
        print(sample_idx)

Error

 (base) multilab@CTN-0824171555:~/Khawar$ python3 

generatePatchFlickr.py
/home2/multilab/Khawar
/home2/multilab/Khawar/Image_Batches/Batch3/
Traceback (most recent call last):
  File "generatePatchFlickr.py", line 57, in <module>
    randomCropPatch.save(sample_out_path)
  File "/home2/multilab/anaconda3/lib/python3.7/site-packages/PIL/Image.py", line 1947, in save
    fp = builtins.open(filename, "w+b")
OSError: [Errno 28] No space left on device: '/home2/multilab/Khawar/Image_Batches/com_patches/5420908975_858440e4b5_b_0000.png'
(base) multilab@CTN-0824171555:~/Khawar$

output of "df -i"

Filesystem                       Inodes      IUsed       IFree IUse% Mounted on
overlay                        16179200    2569364    13609836   16% /
tmpfs                          49334187         25    49334162    1% /dev
sf-app:/icc                     3260416    1871505     1388911   58% /APP2
/dev/sda1                      16179200    2569364    13609836   16% /engrid/ensh
sf-home:/cloudhome/multilab 67075440640 6596266286 60479174354   10% /home2/multilab
tmpfs                          49334187          1    49334186    1% /dev/shm
tmpfs                          49334187         16    49334171    1% /sys/fs/cgroup
tmpfs                          49334187          7    49334180    1% /proc/driver/nvidia
devtmpfs                       49330051        622    49329429    1% /dev/nvidia0
tmpfs                          49334187          1    49334186    1% /proc/scsi
tmpfs                          49334187          1    49334186    1% /sys/firmware
tmpfs                          49334187         74    49334113    1% /run
tmpfs                          49334187          2    49334185    1% /run/lock

output of "df -Bm -h"

Filesystem                   Size  Used Avail Use% Mounted on
overlay                      243G   63G  168G  28% /
tmpfs                         64M     0   64M   0% /dev
sf-app:/icc                  100G   58G   43G  58% /APP2
/dev/sda1                    243G   63G  168G  28% /engrid/ensh
sf-home:/cloudhome/multilab  4.9T   71G  4.9T   2% /home2/multilab
tmpfs                        189G     0  189G   0% /dev/shm
tmpfs                        189G     0  189G   0% /sys/fs/cgroup
tmpfs                        189G   12K  189G   1% /proc/driver/nvidia
devtmpfs                     189G     0  189G   0% /dev/nvidia0
tmpfs                        189G     0  189G   0% /proc/scsi
tmpfs                        189G     0  189G   0% /sys/firmware
tmpfs                        189G   44K  189G   1% /run
tmpfs                        5.0M     0  5.0M   0% /run/lock
Khawar Islam
  • 2,556
  • 2
  • 34
  • 56
  • 2
    what's the question? – Derek Eden Sep 22 '20 at 12:34
  • 3
    The error message is pretty self-explanatory: there's no more storage space left, so you need to buy more space or lower the quality of the images – ForceBru Sep 22 '20 at 12:36
  • Does this answer your question? [Python causing: IOError: \[Errno 28\] No space left on device: '../results/32766.html' on disk with lots of space](https://stackoverflow.com/questions/6998083/python-causing-ioerror-errno-28-no-space-left-on-device-results-32766-h) – Jao Sep 22 '20 at 12:36
  • How to solve this error?. I have 5TB space.@Jao No because my problem is little different – Khawar Islam Sep 22 '20 at 12:46
  • @KhawarIslam The error appear immediatly or after some time the program is running? – Carlo Zanocco Sep 22 '20 at 13:20
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/221873/discussion-between-khawar-islam-and-carlo-zanocco). – Khawar Islam Sep 22 '20 at 13:21
  • @KhawarIslam did you ever find out what was causing this problem? – someone_somewhere Apr 01 '22 at 10:02
  • @someone_somewhere No, if you increase RAM and processing power with SSD then maybe you can run 50% of the overall process. – Khawar Islam Apr 05 '22 at 02:01
  • @KhawarIslam thanks for the reply. For anyone else who runs into this error, I managed to get around it by dividing the generated images into subfolders 100k images in each – someone_somewhere Apr 06 '22 at 06:19

0 Answers0