ImageMagick crop huge image

Question

I am trying to create tiles from a huge image say 40000x40000

i found a script on line for imagemagick he crops the tiles. it works fine on small images like say 10000x5000

once i get any bigger it ends up using to much memory and the computer dies.

I have added the limit options but they dont seem to take affect

i have the monitor in there but it does not help as the script just slows down and locksup the machine

it seems to just goble up like 50gig of swap disk then kill the machine

i think the problem is that as it crops each tile it keeps them in memory. What i think i needs is for it to write each tile to disk as it creates it not store them all up in memory.

here is the script so far

 #!/bin/bash
 file=$1
 function tile() {
 convert -monitor -limit memory 2GiB -limit map 2GiB -limit area 2GB $file -scale ${s}%x -crop 256x256 \
 -set filename:tile "%[fx:page.x/256]_%[fx:page.y/256]" \
 +repage +adjoin "${file%.*}_${s}_%[filename:tile].png"
 }
 s=100
 tile
 s=50
 tile

Answer 1

After a lot more digging and some help from the guys on the ImageMagick forum I managed to get it working.

The trick to getting it working is the .mpc format. Since this is the native image format used by ImageMagick it does not need to convert the initial image, it just cuts out the piece that it needs. This is the case with the second script I setup.

Lets say you have a 50000x50000 .tif image called myLargeImg.tif. First convert it to the native image format using the following command:

 convert -monitor -limit area 2mb myLargeImg.tif myLargeImg.mpc

Then, run the bellow bash script that will create the tiles. Create a file named tiler.sh in the same folder as the mpc image and put the below script:

 #!/bin/bash
 src=$1
 width=`identify -format %w $src`
 limit=$[$width / 256]
 echo "count = $limit * $limit = "$((limit * limit))" tiles"
 limit=$((limit-1))
 for x in `seq 0 $limit`; do
   for y in `seq 0 $limit`; do
     tile=tile-$x-$y.png
     echo -n $tile
     w=$((x * 256))
     h=$((y * 256))
     convert -debug cache -monitor $src -crop 256x256+$w+$h $tile
   done
 done

In your console/terminal run the below command and watch the tiles appear one at at time into your folder.

 sh ./tiler.sh myLargeImg.mpc

Answer 2

libvips has an operator that can do exactly what you want very quickly. There's a chapter in the docs introducing dzsave and explaining how it works.

It can also do it in relatively little memory: I regularly process 200,000 x 200,000 pixel slide images using less than 1GB of memory.

See this answer, but briefly:

$ time convert -crop 512x512 +repage huge.tif x/image_out_%d.tif
real    0m5.623s
user    0m2.060s
sys     0m2.148s
$ time vips dzsave huge.tif x --depth one --tile-size 512 --overlap 0 --suffix .tif
real    0m1.643s
user    0m1.668s
sys     0m1.000s

Answer 3

You may try to use gdal_translate utility from GDAL project. Don't get scared off by the "geospatial" in the project name. GDAL is an advanced library for access and processing of raster data from various formats. It is dedicated to geospatial users, but it can be used to process regular images as well, without any problems.

Here is simple script to generate 256x256 pixel tiles from large in.tif file of dimensions 40000x40000 pixels:

#!/bin/bash
width=40000
height=40000
y=0
while [ $y -lt $height ]
do
   x=0
   while [ $x -lt $width ]
   do
      outtif=t_${y}_$x.tif
      gdal_translate -srcwin $x $y 256 256 in.tif $outtif
      let x=$x+256
   done
   let y=$y+256
done

GDAL binaries are available for most Unix-like systems as well as Windows are downloadable.

Answer 4

ImageMagick is simply not made for this kind of task. In situations like yours I recommend using the VIPS library and the associated frontend Nip2

VIPS has been designed specifically to deal with very large images.

http://www.vips.ecs.soton.ac.uk/index.php?title=VIPS