1

When I tried to open and read a certain geo tiff image with gdal and rasterio, I can read and do things like img.meta and img.descriptions. But when I tried to do img.read() with rasterio or img.GetRasterBand(1).ReadAsArray() with gdal the kernel always died after a certain runtime. It's not happening to all geo tiff images but some. Could anyone help me? Thanks!

Python version: 3.9

System: Mac Big Sur Version 11.3.1

Raster information:

File size: 400 MB

Band number: 3

Coordinate reference system: EPSG:26917

Metadata: {'driver': 'GTiff', 'dtype': 'uint8', 'nodata': 255.0, 'width': 580655, 'height': 444631, 'count': 3, 'crs': CRS.from_epsg(26917), 'transform': Affine(0.08000000000000004, 0.0, 607455.9245999996, 0.0, -0.080000000000001, 4859850.802499999)}

Raster description: (None, None, None)

Geotransform : | 0.08, 0.00, 607455.92| | 0.00,-0.08, 4859850.80| | 0.00, 0.00, 1.00|

# with rasterio
img = rasterio.open('certain_tiff_file.tif')
metadata = img.meta
print('Metadata: {metadata}\n'.format(metadata=metadata))
# kernel died if run the line below
full_img = img.read()

# with gdal
img = gdal.Open('certain_tiff_file.tif')
img_band1 = img.GetRasterBand(1)
img_band2 = img.GetRasterBand(2)
img_band3 = img.GetRasterBand(3)
# kernel died if run the line below
array = img_band1.ReadAsArray()
Chloe
  • 71
  • 1
  • 7
  • cannot say for sure with the level of information your're providing - but sounds like you're trying to read a big tiff file and run out of memory. What's the shape & datatype of the raster? – Val Aug 05 '21 at 14:12
  • Thank you for helping me! I have included more raster information in the question. – Chloe Aug 05 '21 at 23:37
  • Your raster is very large - when you're trying to read into memory, rasterio is trying to create a numpy array with dtype `uint8` and the shape `(580655, 444631)` which would have a size of ~240 GB in memory, very likely more than you have which leads to the kernel thankfully dying – Val Aug 06 '21 at 07:11
  • Got it! And the memory you mean is buffer memory not the size of the disk right? – Chloe Aug 06 '21 at 20:26
  • Yes, the Random-access memory (RAM) not the harddrive – Val Aug 08 '21 at 15:40

1 Answers1

0

I had the same problem when reading large .tiff files.

Following what @Val said in the comments I checked for how much free RAM memory I had as described here:

import psutil
psutil.virtual_memory()

And indeed my issue was that I was running out of RAM. You may try to use del arr once you're done with some array and you don't need to use that anymore to clean a bit of memory. Might be worth looking into gc.collect as well.

guin0x
  • 337
  • 2
  • 10