JVM Out of Memory inspite of giving high Heap Space

Question

I created a test application in Amazon cloud using a small image which is capable of producing 800 threads.The heap space assigned to the JVM is 1024M. The application ran successfully and i had no problem.

I created another large image with 7.5 gb Ram and produced 1000 threads. the heap space assigned to the JVM is 2048M. The application failed saying out of memory error.

Inspite of having a higher configuration why JVM is giving this kind of error?

Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8808972f>] :jbd:journal_stop+0x1e2/0x1ee
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80216452>] __generic_file_aio_write_nolock+0x375/0x3e8
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80291a5c>] autoremove_wake_function+0x0/0x2e
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:27 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  22   Cold: hi:   62, btch:  15 usd:  60
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 158   Cold: hi:   62, btch:  15 usd:  55
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1835342 inactive:7 dirty:180 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2194 slab:11116 mapped:317 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:8776kB min:11144kB low:13928kB high:16716kB active:7341496kB inactive:28kB present:7764880kB pages_scanned:264302704 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 52*4kB 4*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 8768kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025d98b>] __wait_on_bit_lock+0x5b/0x66
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8023fe2e>] __lock_page+0x5e/0x64
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  22   Cold: hi:   62, btch:  15 usd:  60
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 158   Cold: hi:   62, btch:  15 usd:  55
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1835374 inactive:7 dirty:180 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2194 slab:11116 mapped:317 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:8776kB min:11144kB low:13928kB high:16716kB active:7341496kB inactive:28kB present:7764880kB pages_scanned:264523454 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 52*4kB 4*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 8768kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 55853 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Out of memory: kill process 5555 (java) score 4378836 or a child
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Killed process 5556 (java)
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025d98b>] __wait_on_bit_lock+0x5b/0x66
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8023fe2e>] __lock_page+0x5e/0x64
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  22   Cold: hi:   62, btch:  15 usd:  60
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 158   Cold: hi:   62, btch:  15 usd:  55
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1835407 inactive:7 dirty:180 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2194 slab:11116 mapped:317 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:8776kB min:11144kB low:13928kB high:16716kB active:7341628kB inactive:28kB present:7764880kB pages_scanned:264550577 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 52*4kB 4*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 8768kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1929 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8808972f>] :jbd:journal_stop+0x1e2/0x1ee
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80216452>] __generic_file_aio_write_nolock+0x375/0x3e8
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80291a5c>] autoremove_wake_function+0x0/0x2e
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 154   Cold: hi:   62, btch:  15 usd:  47
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 182   Cold: hi:   62, btch:  15 usd:  48
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1886171 inactive:12 dirty:2 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2781 slab:10912 mapped:5 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:11124kB min:11144kB low:13928kB high:16716kB active:7544684kB inactive:48kB present:7764880kB pages_scanned:178561830 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 9*4kB 232*8kB 43*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 11124kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1939 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8808972f>] :jbd:journal_stop+0x1e2/0x1ee
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80216452>] __generic_file_aio_write_nolock+0x375/0x3e8
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80222c57>] __up_read+0x19/0x7f
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 154   Cold: hi:   62, btch:  15 usd:  47
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 182   Cold: hi:   62, btch:  15 usd:  48
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1886171 inactive:12 dirty:2 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2781 slab:10912 mapped:5 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:11124kB min:11144kB low:13928kB high:16716kB active:7544684kB inactive:48kB present:7764880kB pages_scanned:178800160 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 9*4kB 232*8kB 43*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 11124kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1924 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8808972f>] :jbd:journal_stop+0x1e2/0x1ee
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80216452>] __generic_file_aio_write_nolock+0x375/0x3e8
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80291a5c>] autoremove_wake_function+0x0/0x2e
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 154   Cold: hi:   62, btch:  15 usd:  47
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 182   Cold: hi:   62, btch:  15 usd:  48
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1886171 inactive:12 dirty:2 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2781 slab:10912 mapped:5 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:11124kB min:11144kB low:13928kB high:16716kB active:7544684kB inactive:48kB present:7764880kB pages_scanned:178896858 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 9*4kB 232*8kB 43*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 11124kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1920 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8808972f>] :jbd:journal_stop+0x1e2/0x1ee
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80216452>] __generic_file_aio_write_nolock+0x375/0x3e8
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80222c57>] __up_read+0x19/0x7f
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 154   Cold: hi:   62, btch:  15 usd:  47
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 182   Cold: hi:   62, btch:  15 usd:  48
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1886171 inactive:12 dirty:2 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2781 slab:10912 mapped:5 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:11124kB min:11144kB low:13928kB high:16716kB active:7544684kB inactive:48kB present:7764880kB pages_scanned:179113524 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 9*4kB 232*8kB 43*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 11124kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1934 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025d98b>] __wait_on_bit_lock+0x5b/0x66
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8023fe2e>] __lock_page+0x5e/0x64
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 154   Cold: hi:   62, btch:  15 usd:  47
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 182   Cold: hi:   62, btch:  15 usd:  48
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1886211 inactive:12 dirty:2 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2781 slab:10912 mapped:5 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:11124kB min:11144kB low:13928kB high:16716kB active:7544844kB inactive:48kB present:7764880kB pages_scanned:179739273 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 9*4kB 232*8kB 43*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 11124kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1857 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8808972f>] :jbd:journal_stop+0x1e2/0x1ee
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80216452>] __generic_file_aio_write_nolock+0x375/0x3e8
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80222c57>] __up_read+0x19/0x7f
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Mem-info:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA per-cpu:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 154   Cold: hi:   62, btch:  15 usd:  47
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 182   Cold: hi:   62, btch:  15 usd:  48
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Active:1886279 inactive:0 dirty:2 writeback:0 unstable:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  free:2781 slab:10912 mapped:5 pagetables:8823 bounce:0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA free:11124kB min:11144kB low:13928kB high:16716kB active:7545116kB inactive:0kB present:7764880kB pages_scanned:180180710 all_unreclaimable? yes
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: lowmem_reserve[]: 0 0 0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: DMA: 9*4kB 232*8kB 43*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 2*4096kB = 11124kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap  = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Total swap = 0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Free swap:            0kB
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1968128 pages of RAM
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 38914 reserved pages
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 1843 pages shared
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 0 pages swap cached
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: printk: 485 messages suppressed.
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: java invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: 
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel: Call Trace:
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802afc1f>] out_of_memory+0x6d/0x1c4
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8020edc1>] __alloc_pages+0x23e/0x2c7
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff802123fc>] __do_page_cache_readahead+0xf4/0x258
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8023dd5c>] lock_timer_base+0x26/0x4b
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80235881>] find_extend_vma+0x16/0x59
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8029594a>] get_futex_key+0x47/0x10c
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80222c57>] __up_read+0x19/0x7f
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8021397f>] filemap_nopage+0x140/0x332
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80208ef3>] __handle_mm_fault+0x442/0x114a
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff80261a33>] do_page_fault+0xf23/0x128d
Nov  2 07:35:31 274535-12-31-39-0E-B6-12 kernel:  [<ffffffff8025f447>] error_exit+0x0/0x79

There are many possible cases for an OutOfMemoryError, what is the exact error message you get? I assume this is a 64-bit JVM. — Peter Lawrey, Nov 08 '11 at 18:10
Please, give us the exact error message (see [this](http://stackoverflow.com/questions/6244055)) and maybe a stack trace. The problem might be too many native threads created. — Tomasz Nurkiewicz, Nov 08 '11 at 18:10

score 2 · Answer 1 · edited May 23 '17 at 10:34

This got me curious about what resources Java thread might use other then the stack's memory. It seems Java uses the OS as the scheduler and if the OS cannot spawn the thread, Java will report it as an OutOfMemoryException.

http://www.talkingtree.com/blog/index.cfm/2005/3/11/NewNativeThread

The post is pretty old, but it's consistent with your situation... You can try the suggestions on this question and see if it helps :

Maximum number of threads per process in Linux?

JVM Out of Memory inspite of giving high Heap Space

1 Answers1