6

I have an old code base that makes extensive use of GD (no, switching to imagemagick is not an option). It's been running for several years, through several versions. However, when I run it in my current development environment, I'm running into a mysterious gd-png error: cannot allocate image data errors when calling imagecreatefrompng(). The PNG file I'm using is the same as I've been using, so I know it works.

Current setup:

Ansible-provisioned vagrant box  
Ubuntu 14.04  
PHP 5.5.9  
GD 2.1.1  
libPNG 1.2.50

PHP's memory limit at the time the script is run is 650M, though it's ultimately the kernel itself that ends up killing the script, and changing PHP's memory limit doesn't seem to have an effect.

The image dimensions are 7200x6600 and are about 500KiB on disk. This hasn't been a problem in my other environments and is only newly-occurring in my development environment. Unfortunately, I don't have access to the other environments anymore to do a comparison, though the setup was similar in the last working one -- Ubuntu 14.04, PHP 5.5, sufficient memory allocations.

What could be happening in this setup that wasn't happening in my previous setups? How can I fix this?

kamal
  • 27
  • 4
Shauna
  • 9,495
  • 2
  • 37
  • 54
  • 1
    For which lower resolution does it work properly? Similar problem (the only answer was connected with memory limit): https://github.com/claviska/SimpleImage/issues/81 You may try to calculate the memory required by `$required_memory = Round($width * $height * $size['bits']);` – Pawel Gumiela Aug 28 '15 at 14:18
  • If you create a blank script with one line `var_dump(imagecreatefrompng($file))` what would you get? – Yang Aug 28 '15 at 14:29
  • What does `ulimit -v` report? How much memory is available for the VM? – aergistal Aug 28 '15 at 15:29
  • You mentioned that `it's ultimately the kernel itself that ends up killing the script`, so are you sure that you have enough memory on the system? If it is really the kernel that kills the script and you have this logged in `dmesg`, then you should increase the memory of the server. – VolenD Aug 28 '15 at 22:55
  • 7200x6600 considering 1 byte for each component (rgb+a) is about 190M uncompressed. The kernel usually kill processes only when is out of memory – Alex Aug 31 '15 at 10:12
  • It never works or just sometime? Could be a [memory fragmentation](http://stackoverflow.com/a/4863812/2498790) problem? – Alex Aug 31 '15 at 10:21

5 Answers5

4

I was browsing a bit through the PHP 5.5.9 source to try and find your specific error string "cannot allocate image data", but the closest I could find is "gd-png error: cannot allocate gdImage struct". The bundled version of GD for PHP 5.5.9 (all the way up to 7.0.0) is version 2.0.35, so it seems you're looking at a custom build(?). Bugging the PHP guys might not help here.

Looking at the GD source (2.1.2, can't seem to find 2.1.1), the only place this error occurs is: https://github.com/libgd/libgd/blob/master/src/gd_png.c#L435

image_data = (png_bytep) gdMalloc (rowbytes * height);
if (!image_data) {
    gd_error("gd-png error: cannot allocate image data\n");
    png_destroy_read_struct (&png_ptr, &info_ptr, NULL);
    if (im) {
        gdImageDestroy(im);
    }
    if (palette_allocated) {
        gdFree (palette);
    }
    return NULL;
}

Where gdMalloc is:

https://github.com/libgd/libgd/blob/GD-2.1/src/gdhelpers.c#L73

 void *
 gdMalloc (size_t size)
 {
      return malloc (size);
 }

I'm afraid this is as far as my detective work goes. As far as I can tell the bundled 2.0.35 GD versions in PHP also just use malloc, so at first glance there's no real difference there. I also tried to find some equivalent piece of code in the bundled versions, but so far I haven't found it and it seems to be here:

https://github.com/php/php-src/blob/PHP-5.5.9/ext/gd/libgd/gd_png.c#L323

 image_data = (png_bytep) safe_emalloc(rowbytes, height, 0);

I can't seem to find safe_emalloc, but it seems the old PHP bundled versions did use a different memory allocation here than the version your environment uses. Perhaps your best bet is to check with the GD devs?. To avoid a merry goose chase, I think trying another environment is your best bet -after confirming they are indeed using a non-strandard GD version.

After some more PHP source safari, it seems safe_emalloc is used throughout all extensions, so I guess (guess mind you) that this is the preferred way to allocate memory for PHP extensions (looks like it). If your environment is in fact using an unaltered GD 2.1.1, it's likely it is ignoring any PHP memory settings/limits. You might be able to find some other way of specifying the ('stand-alone') GD memory limit?

Edit: Looking at the official libgd faq, near the bottom it states that the PHP extension should indeed respect the memory limit and it specifies that an 8000x8000 pixel image would take about 256MB, so your 650MB limit should really be enough (unless you're working with multiple copies in the same run?) and the fact that it's not working corroborates that something fishy is going on with the memory allocation.

Fasermaler
  • 943
  • 6
  • 14
  • `so it seems you're looking at a custom build(?)` Nope, stock php5-gd package in Ubuntu 14.04 repos. – Shauna Aug 29 '15 at 01:29
  • @Shauna That's odd, checking the Ubuntu 14.04 source at http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/php5/trusty/view/head:/ext/gd/libgd/gd.h shows that it must be using 2.0.35 as well. If you don't think this is going in the right direction, I won't argue any further, but it's curious how you ended up with GD 2.1.1. A non-standard environment would be the best explanation for non-standard behavior, after all. – Fasermaler Aug 30 '15 at 19:13
  • It seems the version reporting might be a mistake. `phpinfo()` (where I got the info in my question) says GD's version is 2.1.1 (which went final release in January), but `apt-cache policy libgd3` says it's 2.1.0-3 (which is what Ubuntu ships with). – Shauna Aug 31 '15 at 12:43
  • So, another interesting find - the script doing the task sets the memory_limit to 650M, but the VM was configured to 512M, which is clearly a problem. What makes this odd is that this is still consistent with the old VM (both settings were the same on it, too), which never failed. – Shauna Aug 31 '15 at 13:07
3

If I was you I would do the following:

  1. Send messages to the php-devel list and see if it is a known issue. You'll also want to search their bug tracker. http://php.net/mailing-lists.php and https://bugs.php.net/
  2. Use GD command line utilities to see if you can process the same image with the same version with cmd line stuff. This is trying to determine if it's a problem with GD and the image or if it is an issue with the PHP-gd lib or some combination.

1.a Create a PHP CLI program to resize the image and see if it works

  1. Figure out exactly what happens when you call the php function in the underlying (probably c) code. This will involve downloading the source to the php-gd module and looking through it.
  2. Code a minimal c application that does the same thing as the underlying PHP-gd library and see if you can reproduce the error

Something in there should tell you exactly what is going on, though it will take a bit of work. You should be able to get all the tools/sources for your environment. The beauty of open source, right?

The other option would be to try different versions of these applications in your environment and see if you find one that works. This is the "shotgun" approach and not the best, IMO.

mikeb
  • 10,578
  • 7
  • 62
  • 120
0

From my experience:

  • Try a file with a different row size. I had issues with another image library that did not read images longer than 3000px/row. It was otherwise not restricted in absolute size.
  • You have quite an image there, if this is in RGBA in memory, you end up with 180M uncompressed image data, 140M on RGB, still 45M as 8Bit. You are sure, your process limits will not be exceeded when this is loaded?
thst
  • 4,592
  • 1
  • 26
  • 40
  • I have been using the same image for several months on a puppet-provisioned vm running the same os, with the same version of php, and the same memory limit with no issue, which is what puzzles me. – Shauna Aug 29 '15 at 01:35
0

You used the same image with the same libraries, it means libgd and libpng have no restrictions about the memory size (or at least not one for the size you're using), the problem may be php but you are sure the memory limit is 650M (if your php installation is using suhosin, you may want to check suhosin.memory_limit)

You tell the kernel is killing the script (but I can't find why you are telling so, is there a dmesg log about?) but the kernel kill a process only when is out of memory.

One problem could be memory fragmentation / contiguous alloc, nowadays it should be more a kernel space problem than user space, but the size your script is allocating is not an every day size. Fragmentation could be a problem if your script is running on a server with a long uptime with a lot of process allocating and freeing memory, if is a virtual machine that you just start up and the script fail at the first launch, then the problem isn't fragmentation.

You can do a simple test, command and code below, it use the gcc compiler to create (in the directory where you launch the command) a a.out executable which allocate and set to 0 a memory chunk with size 191 x 1024 x 1024 (about ~190M)

echo '#include <stdlib.h>\n#include <string.h>\n#define SIZ 191*1024*1024\nint main() {void *p=malloc(SIZ);memset(p,0,SIZ);return (p==NULL?1:0);}' \
| gcc -O2 -xc -

run the executable with:

./a.out

If the problem is the system memory the kernel should kill the executable (I'm not sure 100%, the kernel have some policy about which process to kill when there's no memory, to be checked)

You should see the kill message though, with dmesg perhaps.

(I think this shouldn't happen, instead the malloc should just fail)

If the malloc succeed, the executable can allocate enough memory, it should return 0, if it fail should return 1

to verify the return code just use

./a.out 
echo $? 

(don't execute any other command in the middle)

If the malloc fail you probably just need to add physical or virtual memory to the system.

Keep in mind that ~190M is just the memory for one uncompressed image, depending on how php, libgd and libpng are working the memory for the whole process may be even twice (or more).

I did a simple test and profiled the memory usage, the peak with an image 7600x2200 (1.5M on disk) on my system is about ~342M, here's the test:

<?php
$im     = imagecreatefrompng("input.png");
$string = "Sinker sucker socks pants";
$orange = imagecolorallocate($im, 220, 210, 60);
$px     = (imagesx($im) - 7.5 * strlen($string)) / 2;
imagestring($im, 3, $px, 9, $string, $orange);
imagepng($im);
imagedestroy($im);

I tried xhprof at first but it returned values too low.

So I tried a simple script memusg

memusg /usr/bin/php -f test.php >output.png

(the test work btw)

Community
  • 1
  • 1
Alex
  • 3,264
  • 1
  • 25
  • 40
0

I met this problem yesterday and fixed it today.

Yesterday's env:

  1. php-7.0.12
  2. libpng-1.6.26
  3. libgd-2.1.1

This suite will crash when I resize a png image. After memory check, I thought it might be a bug in the latest php or libpng. So I changed the env to this:

  1. php-5.6.27
  2. libpng-1.2.56
  3. libgd-2.1.1

I changed php and libpng to mature versions which are used longtime.

Recompile and re-install these - it works well for png and jpeg.

m02ph3u5
  • 3,022
  • 7
  • 38
  • 51
yangyu
  • 1