why update big array makes "change_protection" kernel call dominating CPU?

Question

here are the codes gcc test.c -std=c99

#include <stdio.h>
#include <stdlib.h>
void main() {
        size_t size = (long) 40 * 1024 * 1024 * 1024;
        int* buffer = malloc(size * sizeof(int));
        for (size_t i = 0; i < size; i++) {
                buffer[i] = 1;
        }
        printf("hello\n");
        for (size_t i = 0; i < size; i++) {
                buffer[i] = 2;
        }
        printf("hello\n");
}

160G ram allocated in one go, and traversed twice

the first loop runs happily

however the program kinda stuck inside the 2nd loop

with perf top showing this

Samples: 7M of event 'cpu-clock', Event count (approx.): 14127849698
 74.95%  [kernel]             [k] change_protection
 23.52%  [kernel]             [k] vm_normal_page
  0.40%  [kernel]             [k] _raw_spin_lock
  0.34%  [kernel]             [k] _raw_spin_unlock

with top showing this

top - 10:52:36 up 55 min,  4 users,  load average: 1.16, 1.18, 1.04
Tasks: 240 total,   2 running, 238 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  3.1%sy,  0.0%ni, 96.9%id,  0.0%wa,  0.0%hi,  0.0%s
Mem:  251913536k total, 170229472k used, 81684064k free,    27820k b
Swap:        0k total,        0k used,        0k free,   352816k cac

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12825 dapeng    20   0  160g 160g  376 R 100.0 66.6  30:38.55 a.out

and the best part is now this program doesn't answer to kill command any more

gcc version gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)

server AWS EC2 r3.8xlarge

Operating System Amazon Linux AMI release 2014.09

This is related question to Why does `change_protection` hog CPU while loading a large amount of data into RAM?

Answer 1

So what happens is:

1. you malloc code. That means the kernel maps pages into your program's space; for that, the pages don't even have to be available, its more of a "notify me once you actually access that memory" situation.
2. your first loop sequentially access the pages. Thus, the kernel frequently has to actually assign real memory to your formerly unused pages. Probably, the AWS hypervisor is fast at providing that memory from its pool
3. you access these pages again -- but as bad luck might have it, the pages at the start of your 160GB area might be "far" away now, and getting them back close to the CPU your code is running on might be slow. Remember, you're running on a VM on something resembling a NUMA machine.

EDIT the fact that it doesn't react to SIGTERM etc isn't surprising -- there's no explicit context switch happening and no signal checking done in the for loops, so there's no place that your program could react to signals.