3

I am trying to understand few important OS concepts (for simplicity, lets stick to Linux Kernel). Assume I run this in kernel mode , perhaps adding these lines (either caseA or caseB not both) into source code of some system call.

# Assume __malloc() here is a simple heap memory manager
void consume_heap_forever(void)
    {

      for (;;)
        (void) __malloc(PAGE_SIZE);         
    }

Case A: The above consumes heap in a loop. I will first start consuming memory and things will go normal. After a high enough consumption, what begins to happen (before a crash) ? I know that kernel space is within reserved chunk in process address space. Will I crash at point when I cross the stack portion the kernel uses? Or will this expand that reservation (and perhaps consume whole of virtual memory)? 

# Vanilla Factorial logic
int factorial(int value)
    {
        if (value == 0)
           return 1;
         return value * factorial(value-1)
    }

Case B: I am aware that the kernel has a fixed (and small) amount of stack reserved for it. So perhaps when I give a value big enough -- I will run out of that predefined stack space. What kind of crash happens here? Will I cross into the heap section of kernel?

Tsyvarev
  • 60,011
  • 17
  • 110
  • 153
visweshn92
  • 301
  • 1
  • 3
  • 13

1 Answers1

0

In your example A, your application would loop forever. At some point malloc will be unable to map pages to the logical address space and will return 0.

In your example B: Each process has its own kernel mode stack (usually, there is one shared interrupt stack).

It's likely that you'd eventually hit a guard page at the end of the stack and get an access violation. You're not going to run over the kernel's memory pool.

user3344003
  • 20,574
  • 3
  • 26
  • 62
  • For case A, will I expand my kernel space while looping? I.e lets say initially kernel took 1GB of 4 GB in VA. Now will it expand further than that? – visweshn92 Feb 14 '17 at 21:17
  • Calling malloc in kernel mode will not expand the system area. If it works at all, it would expand the user area. – user3344003 Feb 15 '17 at 05:56
  • I meant further expand into "user area" only. Since, it expands into user area for this process. Will every other process get to see the same expanded kernel space? Because kernel space is generally same across processes. Expanding in one process should make it expand in others as well right? – visweshn92 Feb 15 '17 at 13:54
  • If malloc actually worked in kernel mode, its effects would only be seen by the process that executed. It does not expand the shared, system space. – user3344003 Feb 15 '17 at 16:12
  • Thanks! You clarified important points for me. Please consider updating the answer with these details. Thanks again – visweshn92 Feb 16 '17 at 13:06