3

When I run a specific ImageMagick convert command (to produce an animated GIF) inside a Singularity container it is consistently giving an memory error:

convert-im6.q16: cache resources exhausted `foo.gif' @ error/cache.c/OpenPixelCache/4083.
convert-im6.q16: memory allocation failed `foo.gif' @ error/quantize.c/AssignImageColors/496.

How can I debug this?

I'm not getting any memory issues outside of Singularity which gives me the impression that either Singularity is artificially limiting my available memory or ImageMagick is configured poorly inside of Singularity. I am not seeing memory issues for any of my other applications running inside Singularity which makes me think it's an ImageMagick issue.

Here are some details about my system:

$ singularity --version
singularity version 3.7.1-1.el8

Inside the container:

Singularity> convert --version
Version: ImageMagick 6.9.10-23 Q16 x86_64 20190101 https://imagemagick.org
Copyright: © 1999-2019 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP 
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png tiff webp wmf x xml zlib

and

Singularity> convert -list resource
Resource limits:
  Width: 16KP
  Height: 16KP
  List length: 18.446744EP
  Area: 128MP
  Memory: 256MiB
  Map: 512MiB
  Disk: 1GiB
  File: 768
  Thread: 256
  Throttle: 0
  Time: unlimited

The Memory: 256MiB stands out as especially low. I should have at least 2GiB. How does ImageMagick configure these limits and how can I change them?

sligocki
  • 6,246
  • 5
  • 38
  • 47
  • Have a read here... https://stackoverflow.com/a/31388904/2836621 – Mark Setchell Feb 16 '21 at 21:18
  • Thanks for the pointer, Mark. I tried `convert -limit memory 2GiB -list resource`, but it's still listing `Memory: 256MiB`. It seems like something very strange is happening inside the container ... – sligocki Feb 17 '21 at 15:52
  • Oh, I see, I can use `-limit` to reduce the memory limit, but not to increase it ... – sligocki Feb 17 '21 at 15:53
  • Can you try `identify -list configure | grep CONFIGURE_PATH` which should help you find your `policy.xml` file that may be restricting you. – Mark Setchell Feb 17 '21 at 15:57

1 Answers1

5

The problem turned out to be with ImageMagick's system-wide policy.xml that was installed in my container. Updating that file with more generous "memory" and "disk" values fixed this problem.

You can find the location of your system's policy.xml file using the command convert -list policy (Hat tip to Kurt Pfeifle's answer which clued me in on this). For me it was at /etc/ImageMagick-6/policy.xml. You can edit that file (if you have root access). In the end I decided just to delete the file since I don't want my system from restricting my use at all inside the container.

You can set limits via the command line, say convert -limit memory 2GiB ... or environment variables (See Kurt Pfeifle's answer for details). However, this method does not allow expanding larger than system-wide limits set in policy.xml because this policy file is meant to be a way for system administrators to forcibly limit users. Therefore, the only way to remedy this is to update/remove the system-wide policy.

sligocki
  • 6,246
  • 5
  • 38
  • 47