8

My program can allocate suddenly a lot of RAM based on usage. I want to limit the RAM it can take from the system.

I saw here: Limit RAM usage to python program

But it works only for Unix. Any solution for Windows?

BestR
  • 669
  • 2
  • 6
  • 17
  • Tried combining it with resource.RLIM_INFINITY? https://docs.python.org/3/library/resource.html#resource.RLIM_INFINITY – Sharan Arumugam Mar 01 '19 at 17:05
  • Rather than limiting the amount of memory you are able to utilise, I think it better for you to do the actions you want in a more efficient way that is not so consuming on the systems resources. Ramping up hardware is the worst kind of scalability. – Swift Mar 01 '19 at 17:44
  • @SharanArumugam As I said, resource is only unix library – BestR Mar 02 '19 at 09:11
  • @Swift its other party program I cant control it – BestR Mar 02 '19 at 09:12
  • @eryksun how to do it? – BestR Mar 02 '19 at 09:12

2 Answers2

6

A Job object supports limiting the committed memory of a process. In Python, we can implement this via PyWin32 or ctypes.

Note that prior to Windows 8 a process can only be in a single Job. A couple of common cases where this is a concern include the py.exe launcher (the default association for .py files), which runs python.exe in a Job, and the Task Scheduler service, which runs each task in a Job.

PyWin32 Example

import sys
import warnings

import winerror
import win32api
import win32job

g_hjob = None

def create_job(job_name='', breakaway='silent'):
    hjob = win32job.CreateJobObject(None, job_name)
    if breakaway:
        info = win32job.QueryInformationJobObject(hjob,
                    win32job.JobObjectExtendedLimitInformation)
        if breakaway == 'silent':
            info['BasicLimitInformation']['LimitFlags'] |= (
                win32job.JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK)
        else:
            info['BasicLimitInformation']['LimitFlags'] |= (
                win32job.JOB_OBJECT_LIMIT_BREAKAWAY_OK)
        win32job.SetInformationJobObject(hjob,
            win32job.JobObjectExtendedLimitInformation, info)
    return hjob

def assign_job(hjob):
    global g_hjob
    hprocess = win32api.GetCurrentProcess()
    try:
        win32job.AssignProcessToJobObject(hjob, hprocess)
        g_hjob = hjob
    except win32job.error as e:
        if (e.winerror != winerror.ERROR_ACCESS_DENIED or
            sys.getwindowsversion() >= (6, 2) or
            not win32job.IsProcessInJob(hprocess, None)):
            raise
        warnings.warn('The process is already in a job. Nested jobs are not '
            'supported prior to Windows 8.')

def limit_memory(memory_limit):
    if g_hjob is None:
        return
    info = win32job.QueryInformationJobObject(g_hjob,
                win32job.JobObjectExtendedLimitInformation)
    info['ProcessMemoryLimit'] = memory_limit
    info['BasicLimitInformation']['LimitFlags'] |= (
        win32job.JOB_OBJECT_LIMIT_PROCESS_MEMORY)
    win32job.SetInformationJobObject(g_hjob,
        win32job.JobObjectExtendedLimitInformation, info)

def main():
    assign_job(create_job())
    memory_limit = 100 * 1024 * 1024 # 100 MiB
    limit_memory(memory_limit)
    try:
        bytearray(memory_limit)
    except MemoryError:
        print('Success: available memory is limited.')
    else:
        print('Failure: available memory is not limited.')
    return 0

if __name__ == '__main__':
    sys.exit(main())
Eryk Sun
  • 33,190
  • 5
  • 92
  • 111
  • Thank you for this answer! However, I still don't understand how to use this code, say, I have a function run_pipeline() for which I want to set the RAM limit, where do I put run_pipeline() in this code? – Lilianna Jul 03 '20 at 09:46
  • 1
    @Lilianna, the process is assigned to a Job object via `assign_job(create_job())`. Then `limit_memory(memory_limit)` configures a memory limit that applies to all processes in the job. Subsequently, calling your `run_pipeline` function will be subject to the memory limit. Note that by default the job is created in silent breakaway mode, so only the current process will be in the job, i.e. you don't have to worry about child processes being affected by this limit. – Eryk Sun Jul 03 '20 at 11:39
  • any idea for this error: `Assertion failed: error not defined [8] (C:\ci\zeromq_1599741307349\work\src\ip.cpp:481)` I got this while running above code – Talha Anwar Mar 15 '21 at 06:45
2

I had about the same problem as the OP, except that I wanted to limit the amount of physical RAM used, not virtual. @eryksun's answer and limit_memory() function work great, but limit the total amount of allocatable memory (virtual memory). Here is an additional function to his/her code that limits the amount of physical memory (the "working set"). 

def limit_working_set(memory_limit):
    if g_hjob is None:
        return
    info = win32job.QueryInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation)
    info['MinimumWorkingSetSize'] = 50 * 4096  # default minimum value
    info['MaximumWorkingSetSize'] = memory_limit
    info['LimitFlags'] = (win32job.JOB_OBJECT_LIMIT_WORKINGSET)
    win32job.SetInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation, info)
cxxl
  • 4,939
  • 3
  • 31
  • 52