1

I am trying to process some satellite photo with the size over 1G. I code in python, and I have tried in these ways.

image = np.array(io.imread(each_dir))
image = np.array(TiffImagePlugin.open(each_dir))
image = np.array(Image.open(each_dir))

Only io.read works with photos with size of 200Mb, when the size is larger than 1G, there's an error MemoryError: Unable to allocate 17.1 GiB for an array with shape (156105, 39136, 3) and data type uint8

I have changed my RAM from 16g to 32g, but this doesn't work. So I 'd like to ask is there a way to read the imgae by lines but not read the whole image in a time ? Thank you very much.

louise
  • 13
  • 2

2 Answers2

2

I would suggest you tile your rather large TIFF outside of Python before you start. Drawing heavily on John's (@jcupitt) answer here, you can do that with vips.

I created a 39100x156100 TIFF in my Terminal, and tiled it using vips' "DeepZoom Save" feature:

vips dzsave big.tif outdir --depth one --tile-size 8192 --overlap 0 --suffix .tif

It took 38s and 2GB of RAM, so it is pretty frugal. Below is a list of the tiles it produced and their metadata.

Results

0_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
0_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
0_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
1_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
1_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
2_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
2_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_0.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_10.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_11.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_12.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.010u 0:00.000
3_13.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_14.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_15.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_16.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_17.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_18.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_19.tif TIFF 8192x452 8192x452+0+0 8-bit sRGB 10.594MiB 0.000u 0:00.000
3_1.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_2.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_3.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_4.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_5.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_6.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_7.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_8.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
3_9.tif TIFF 8192x8192 8192x8192+0+0 8-bit sRGB 192.001MiB 0.000u 0:00.000
4_0.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_10.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_11.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_12.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_13.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_14.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_15.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_16.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_17.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_18.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_19.tif TIFF 6332x452 6332x452+0+0 8-bit sRGB 8.18867MiB 0.000u 0:00.000
4_1.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_2.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_3.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_4.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_5.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_6.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_7.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_8.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000
4_9.tif TIFF 6332x8192 6332x8192+0+0 8-bit sRGB 148.407MiB 0.000u 0:00.000

Another, completely different approach, might be to use the stream command which is part of the ImageMagick suite - also outside of Python. So, I made a 50,000x50,000 pixel RGB TIFF called big.tif and then streamed off the bottom 1/5 (i.e. rows 40,000 onwards) into an RGB888 raw file like this:

magick stream -map rgb -storage-type char -extract 50000x10000+0+40000 big.tif bottom.dat

That command runs in around 18s on my Mac. You can then read that into a Numpy array in Python like this:

import numpy as np

# Load RGB888 file and reshape back to its correct shape
im = np.fromfile('bottom.dat', dtype=np.uint8).reshape((10000,50000,3))

In case you are unfamiliar with ImageMagick's geometry specifications, the 50000x10000+0+40000 part in the command above means "extract a 50,000 pixel wide by 10,000 pixel tall region whose top-left corner is 0 pixels in from the left edge and 40,000 pixels down from the top".

Mark Setchell
  • 191,897
  • 31
  • 273
  • 432
  • Thank you very much. I tried the first solution of `vips` and this works for me. The only difference is that I got many warnings like `(vips:3844): VIPS-WARNING **: 21:11:55.984: Unknown field with tag 33550 (0x830e) encountered`. – louise Dec 23 '21 at 13:32
  • Tag 33550 is a specific tag for GeoTIFF files that `vips` doesn't know, so it has been ignored. There's a searchable list of tags here https://www.awaresystems.be/imaging/tiff/tifftags/search.html?q=33550&Submit=Find+Tags – Mark Setchell Dec 23 '21 at 14:28
  • Yes, I am processing GeoTIFF files. By searching on the website you provide, I found warnings similiar to `Tag 33550` are relative to the extra information in the GeoTIFF files. Thank you very much. – louise Dec 23 '21 at 14:46
1

You can use pyvips to read out parts of a tiled TIFF. For example:

import pyvips

image = pyvips.Image.new_from_file("huge-tiled-image.tiff")
tile = image.crop(100, 110, 1000, 1000)
tile = image.crop(8000, 6500, 1000, 1000)
...

Where the four numbers are left, top, width and height in pixels. It's very fast, you can read out any area you like, it'll only decode the parts of the image that you ask for, and it keeps a cache of recently decompressed pixels.

You can do further processing in pyvips, or render the pixels to an array and hand them on to numpy, see:

https://libvips.github.io/pyvips/intro.html#numpy-and-pil

jcupitt
  • 10,213
  • 2
  • 23
  • 39
  • Thank you very much. I found `conda` installation didn't work, I installed it with `pip`. This works good. – louise Dec 27 '21 at 08:24