Java 32 bit memory threshold on 64 bit Windows 10

Question

I've got some software that requires around 1500m of memory to run correctly. I force this amount at the start using xms and xmx to make sure it has the memory it needs. I used to run it on Windows XP or Windows 7 and it ran fine. After switching to Windows 10 I suddenly find I don't have enough memory to run it anymore. After researching how memory in Java works I came across the following information on this page:

http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#gc_heap_32bit

Due to various additional constraints such as available swap, kernel address space usage, memory fragmentation, and VM overhead, in practice the limit can be much lower. On most modern 32-bit Windows systems the maximum heap size will range from 1.4G to 1.6G.

It sounds like I've been running on the edge of what 32bit Java is capable of. Testing the limits of my machine it seems my current max is about 1.1g. At the moment switching to 64bit may not be on the table, so I'm trying to figure out how to get back the roughly .5g I've lost from my max memory threshold. I've tried researching all of the things listed in the FAQ that could affect it.

• available swap: This appears to be the page file or virtual memory. Apparently increasing this can sometimes help. Sometimes when I increase this I see a temporary improvement after a restart, but the fix never seems to last if I restart again.

• kernel address space usage: This seems like it’s the memory ceiling for the operating system. Nothing we can do the change this right?

• memory fragmentation: Supposedly Windows 10 is actually better at handling this than previous versions of Windows. So this shouldn’t be a factor.

• VM overhead: Seems like it’s referring to RAM usage from other virtual machines. Other than Java I don’t think I have anything else running that counts as a virtual machine.

Is there any solid way to identify and correct what could be causing the difference in performance between versions of Windows? It's super frustrating because I have more physical memory now than ever, but the only RAM that matters is the RAM JVM is allowed to use, which is now somehow less.

Update

I hear you. I know I can fix it easily switching to 64 bit; my question is how I can fix it without doing that. Isn't it helpful to understand how JVM determines it's max heap for scholarly reasons as well as practical? I doubt there are many apps that hit the upper limit of 64 bit Java, but wouldn't Java behave the same way if there were?

Answer 1

I wasn't able to figure out why the memory was different between systems, but I did learn of a LARGEADDRESSAWARE flag I could set on the Java.exe that would allow it to have up to 1G more memory.

Out of several tools people were typically using to set the flag I found this custom coded python script that attempts to set it on any .exe file you hand it. Using that I was able to feed it the Java.exe and now everything works.

It would appear that setting this flag is not typically recommended, but no side effects so far. I'll use this until I can properly switch everything to 64 bit.

https://github.com/pyinstaller/pyinstaller/issues/1288

# See http://en.wikibooks.org/wiki/X86_Disassembly/Windows_Executable_Files#PE_Header
# and https://msdn.microsoft.com/en-us/library/ms680349%28v=vs.85%29.aspx

import struct

IMAGE_FILE_LARGE_ADDRESS_AWARE = 0x0020
PE_HEADER_OFFSET = 60
CHARACTERISTICS_OFFSET = 18

def set_large_address_aware(filename):
    f = open(filename, 'rb+')
    # Check for MZ Header
    if f.read(2) != b'MZ':
        print('Not MZ')
        return False
    # Get PE header location
    f.seek(PE_HEADER_OFFSET)
    pe_header_loc, = struct.unpack('i', f.read(4))
    # Get PE header, check it
    f.seek(pe_header_loc)
    if f.read(4) != b'PE\0\0':
        print('error in PE header')
        return False
    # Get Characteristics, check if IMAGE_FILE_LARGE_ADDRESS_AWARE bit is set
    charac_offset = pe_header_loc + 4 + CHARACTERISTICS_OFFSET
    f.seek(charac_offset)
    bits, = struct.unpack('h', f.read(2))
    if (bits & IMAGE_FILE_LARGE_ADDRESS_AWARE) == IMAGE_FILE_LARGE_ADDRESS_AWARE:
        return True
    else:
        print('large_address_aware is NOT set - will try to set')
        f.seek(charac_offset)
        bytes = struct.pack('h', (bits | IMAGE_FILE_LARGE_ADDRESS_AWARE))
        f.write(bytes)
    f.close()
    return False

if set_large_address_aware('file.exe'):
    print('large_address_aware is set')
else:
    print('large_address_aware was NOT set')