PHP x86 Memory Limit

Question

I have a file that I need to base64 encode that is 418 MB but I keep getting memory limit errors. I've set the memory_limit=2048M in my php.ini but I still cannot read it into memory as a base64 string with an x86 CLI version of PHP.

The PHP Script

<?php

echo "\r\n";
echo "\r\nMemory Usage\r\n" . memory_get_usage(false) . " Bytes\r\nReal " . memory_get_usage(true) . " Bytes\r\n";
echo "\r\nPeak Memory Usage\r\n" . memory_get_peak_usage(false). " Bytes\r\nReal " . memory_get_peak_usage(true) . " Bytes\r\n";
echo "\r\nMemory Limit\r\n" . ini_get('memory_limit') . "\r\n";

$file = file_get_contents('C:\Path\To\File');

echo "\r\n";
echo "\r\nMemory Usage\r\n" . memory_get_usage(false) . " Bytes\r\nReal " . memory_get_usage(true) . " Bytes\r\n";
echo "\r\nPeak Memory Usage\r\n" . memory_get_peak_usage(false). " Bytes\r\nReal " . memory_get_peak_usage(true) . " Bytes\r\n";
echo "\r\nMemory Limit\r\n" . ini_get('memory_limit') . "\r\n";

$encodedFile = base64_encode($file);

echo "\r\n";
echo "\r\nMemory Usage\r\n" . memory_get_usage(false) . " Bytes\r\nReal " . memory_get_usage(true) . " Bytes\r\n";
echo "\r\nPeak Memory Usage\r\n" . memory_get_peak_usage(false). " Bytes\r\nReal " . memory_get_peak_usage(true) . " Bytes\r\n";
echo "\r\nMemory Limit\r\n" . ini_get('memory_limit') . "\r\n";

?>

Running that script produces the following.

PS C:\> php .\MemoryIssueTest.php


Memory Usage
382184 Bytes
Real 2097152 Bytes

Peak Memory Usage
422256 Bytes
Real 2097152 Bytes

Memory Limit
2048M


Memory Usage
447075680 Bytes
Real 448790528 Bytes

Peak Memory Usage
447084280 Bytes
Real 448790528 Bytes

Memory Limit
2048M

VirtualAlloc() failed: [0x00000008] Not enough memory resources are available to process this command.


VirtualAlloc() failed: [0x00000008] Not enough memory resources are available to process this command.

PHP Fatal error:  Out of memory (allocated 448790528) (tried to allocate 593015064 bytes) in C:\MemoryIssueTest.php on line 14
PHP Stack trace:
PHP   1. {main}() C:\MemoryIssueTest.php:0
PHP   2. base64_encode() C:\MemoryIssueTest.php:14

Fatal error: Out of memory (allocated 448790528) (tried to allocate 593015064 bytes) in C:\MemoryIssueTest.php on line 14

Call Stack:
    0.4046     382152   1. {main}() C:\MemoryIssueTest.php:0
   33.4669  447075680   2. base64_encode() C:\MemoryIssueTest.php:14

After upgrading to a 64 bit version of PHP I get the following results from the same script.

PS C:\> php .\MemoryIssueTest.php


Memory Usage
402088 Bytes
Real 2097152 Bytes

Peak Memory Usage
444160 Bytes
Real 2097152 Bytes

Memory Limit
2048M


Memory Usage
440804144 Bytes
Real 442499072 Bytes

Peak Memory Usage
440812824 Bytes
Real 442499072 Bytes

Memory Limit
2048M


Memory Usage
1025909576 Bytes
Real 1027604480 Bytes

Peak Memory Usage
1025909760 Bytes
Real 1027604480 Bytes

Memory Limit
2048M

From my understanding, x86 applications should be able to consume more memory than 1GB. Is the x86 version of php hard coded to only allow less than 1GB? If not, why would I get the PHP Fatal error: Out of memory error when I only need to consume 1GB but I have a 2GB limit?

Does the x86 version run correctly if you set `memory_limit` to `-1` to let it use whatever it needs? — Dave, Mar 07 '19 at 16:19
What are you going to do with the file once you base64encode it? — Stephanie Temple, Mar 07 '19 at 18:51

EPB · Accepted Answer · 2019-03-07T18:20:24.700

Theoretically, you should be able to allocate up to 2GB of memory with an x86 process on Windows. However, there are other constraints on how much you can allocate. Critically: memory fragmentation in your available address space. Since base64_encode allocates its needed memory all at once, you'll need enough contiguous address space to encode the entire string. You're more likely to have enough with a 64-bit address space, which is why the 64-bit version of PHP runs just fine.

For a little more information in this answer: Java maximum memory on Windows XP

Addendum: In my testing, I was able to allocate only slightly more than yours, but smaller address allocations let me get closer to that 2GB limit.

Addendum 2: If you absolutely need to use the 32-bit version and are encoding to a file, you can work around this somewhat by encoding in some multiple of 3 bytes as then you can concatenate them.

// Example, from memory string to file.
$current_position = 0;
$length = strlen($file);
$outputfile = fopen('testoutput.txt', 'w');
while($current_position < $length) {
  fwrite($outputfile, base64_encode(substr($file, $current_position, 60000)));
  $current_position += 60000;
}
fclose($outputfile);

Or going all the way, from file to file, which allocates very little memory:

$inputfile = fopen('test.mp4', 'rb');
$outputfile = fopen('testoutput2.txt', 'w');
while( ($str = fread($inputfile, 61440)) !== false ) {
  if($str === '') {
    break;
  }
  fwrite($outputfile, base64_encode($str));
  $current_position += 61440;
}
fclose($outputfile);

score 0 · Answer 2 · answered Mar 07 '19 at 18:13

0

Check that you have sufficient memory. Changing a limit and changing the amount of addressable memory (64bit) wont matter if you only have <1gb free system memory (or other virtual memory limitations like container memory limits)

answered Mar 07 '19 at 18:13

Mike Metully

1

My machine has 16GB of RAM and according to resmon was only using about 3GB of it at the time of testing. – Matthew Goheen Mar 07 '19 at 18:16

PHP x86 Memory Limit

2 Answers2