0

Please help me understand ... I am using a very memory-intensive algorithm (LDA on Mahout). Because it requires huge amounts of memory, I used a 5 node Hadoop cluser with 24GB of RAM each.

However, when running the algorithm, I get this error: Container [pid=14275,containerID=container_1458474415429_0004_01_000012] is running beyond virtual memory limits. Current usage: 265.0 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.

Help me understand this ... How does Hadoop allocate memory? How should I set up the environment to use the maximum possible memory available? I believed Hadoop will just break up the tasks into smaller ones, therefore each of them using less memory - however, that seems naive now.

neuromouse
  • 921
  • 1
  • 12
  • 32
  • Check this: http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits/21008262#21008262 – PetrosP Mar 21 '16 at 10:15
  • Thanks. Where should I change this settings - in mapred-site.xml on the namenode, or across whole cluster? – neuromouse Mar 21 '16 at 10:32

0 Answers0