Broken results on batch convert with ImageMagick command line on Linux

Question

This is my IM command:

/usr/bin/convert 
'src.tif'
  -limit memory 0
  -limit map 0
  -limit file 0
  -alpha transparent
  -clip
  -alpha opaque
  -resize 800x600
  'end.png'
2>&1

So this will remove the white background of my TIFF by clipping the path that is given in the file. It will be resized and saved as transparent PNG.

I got no errors from IM running this.

But if I run this command with PHP to execute it on about 13000 files - I sometimes get these errors:

sh: line 1: 25065 Killed                  /usr/bin/convert \
            'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' \
             -limit memory 0 -limit map 0 -limit file 0 -alpha transparent \
             -clip -alpha opaque -resize 800x600 \
            'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1

sh: line 1: 25702 Killed                  /usr/bin/convert \
            'public_html/source_files/XXXX123/XXXX123/XXXX123.tif' \
             -limit memory 0 -limit map 0 -limit file 0 -alpha transparent \
             -clip -alpha opaque -resize 800x600 \
            'public_html/converted/XXXX123/XXXX123/XXXX123_web.png' 2>&1

But the bigger problem is: Some of the pictures are broken. Below is a "bad" image on the left, a "good" image on the right (ondrag/on a dark background you see the problem better):

On running the command manually the result was ok. Only on running this PHP loop script will provide broken results. ( PHP loop script )

I run the script this way: php55 run.php. A simple loop with find as shell script provides same results.

So I searched, asked in the IM discourse server and run this procedure on 2 machines with different distribution (Debian Wheezy, Ubuntu Server 14.04)

Note/EDIT 1: Running the command in the terminal with the same file provides a perfect result.

EDIT 2: Added example TIFF file here

Answer 1

I'm not sure if this is an answer. For now it is pure speculation. So here goes...

By setting the limits to a 0 value, you are basically telling ImageMagick: "Your resources are not limited at all. You do not need to care for any limits."

• What if didn't set any limit? Remove all -limit ... 0 parts from your command. In this case ImageMagick would use its built-in defaults, or the otherise defined settings (which may be contained in the policy.xml file of your IM installation, or through various environment variables). You can query the current limits of your system with the following command:

  identify -list resource
  

  On my system, I get these values:

  File       Area     Memory     Map       Disk  Thread  Throttle        Time
  ---------------------------------------------------------------------------
   192    4.295GB       2GiB    4GiB  unlimited       1         0   unlimited
  

• What if you did set these limits to a reasonable value, that matches your system's really available resources? Assuming you have: 8 GByte of RAM, 50 GByte of free disk space and plenty of free inodes on your disk volume. Then try to set it like this: -limit disk 10GB -limit memory 3GB -limit map 6GB.

---

ImageMagick resource management

For all its processing and intermediate steps, ImageMagick needs access to an intermediate pixel cache memory/storage, before it can deliver the final result.

This need for pixel cache storage can be satisfied by different resources:

• heap memory,
• anonymous memory map,
• disk-based memory map,
• direct disk.

ImageMagick makes use of all these resources progressively:

• Once heap memory is exhausted, it stores pixels in an anonymous map.
• Once the anonymous memory map is exhausted, it creates the pixel cache on disk and attempts to memory-map it.
• Once memory-map memory is exhausted, it simply uses standard disk I/O.

Disk storage is cheap but very slow too: it is in the order of 3 magnitudes (a thousand times) slower than memory. Some speed improvements (up to 5 times) can be obtained by using memory mapping to the disk-based cache.

ImageMagick is aware of various ways to control the amount of these resources:

1. Built-in default values. These limits are: 768 files, 3GB of image area, 1.5GiB memory, 3GiB memory map, and 18.45EB of disk space.

2. policy.xml config file. Please look up what's in your own policy.xml file. Use convert -list policy to find the location of this file first. Then use cat /some/path/policy.xml to see its contents. (The file uses an XML syntax. Don't forget: anything enclosed in <!-- and --> is a comment!) It also contains comments explaining various details. The policy.xml can define much more things than just the available limit resources. Settings in policy.xml take precedence over the built-in default values if they are defined there.

3. Environment variables. Here is a list of environment variables which can limit IM resources: MAGICK_AREA_LIMIT (image area limits), MAGICK_DISK_LIMIT (disk space limit), MAGICK_FILE_LIMIT (maximum no. of open files limit), MAGICK_MEMORY_LIMIT (heap memory limit), MAGICK_MAP_LIMIT (memory map limit), MAGICK_THREAD_LIMIT (maximum no. of threads limit) and MAGICK_TIME_LIMIT (maximum elapsed time in seconds). These environment variables, if set, take precedence over the policy.xml config file.

4. -limit <name> <value> settings on command line. The following <names> are recognized:

   • width (maximum width of an image). When limit is exceeded, exception is thrown and processing stops.
   • height (maximum height of an image). When limit is exceeded, exception is thrown and processing stops.
   • area (maximum number of bytes for any single image to reside in pixel cache memory). When limit is exceeded, automagical caching to disk (possibly memory-mapped) sets in.
   • memory (maximum memory allocated for the pixel cache from anonymous mapped memory or heap).
   • map (maximum amount for memory map allocated for pixel cache).
   • disk (maximum amount of disk space permitted for use by pixel cache). When limit is exceeded, pixel cache is not created and a fatal exception is thrown.
   • files (maximum number of open pixel cache files). When limit is exceeded, all subsequent pixels cached to disk are closed and reopened on demand.
   • thread (maximum number of threads which can in parallel).
   • time (maximum time in seconds a process is permitted to execute). When this limit is exeeded, an exception is thrown and processing stops.

   The -limit setting on a command line takes precendence and overrides all other settings.