2

I am confused by hadoop namenode memory problem.

  1. when namenode memory usage is higher than a certain percentage (say 75%), reading and writing hdfs files through hadoop api will fail (for example, call some open() will throw exception), what is the reason? Does anyone has the same thing? PS.This time the namenode disk io is not high, the CPU is relatively idle.

  2. what determines namenode'QPS (Query Per Second) ?

Thanks very much!

Ani Menon
  • 27,209
  • 16
  • 105
  • 126
jun zhou
  • 133
  • 1
  • 2
  • 6

1 Answers1

1

Since the namenode is basically just a RPC Server managing a HashMap with the blocks, you have two major memory problems:

  1. Java HashMap is quite costly, its collision resolution (seperate chaining algorithm) is costly as well, because it stores the collided elements in a linked list.
  2. The RPC Server needs threads to handle requests- Hadoop ships with his own RPC framework and you can configure this with the dfs.namenode.service.handler.count for the datanodes it is default set to 10. Or you can configure this dfs.namenode.handler.count for other clients, like MapReduce jobs, JobClients that want to run a job. When a request comes in and it want to create a new handler, it go may out of memory (new Threads are also allocating a good chunk of stack space, maybe you need to increase this).

So these are the reasons why your namenode needs so much memory.

What determines namenode'QPS (Query Per Second) ?

I haven't benchmarked it yet, so I can't give you very good tips on that. Certainly fine tuning the handler counts higher than the number of tasks that can be run in parallel + speculative execution. Depending on how you submit your jobs, you have to fine tune the other property as well.

Of course you should give the namenode always enough memory, so it has headroom to not fall into full garbage collection cycles.

Thomas Jungblut
  • 20,854
  • 6
  • 68
  • 91
  • Thanks, I set dfs.namenode.handler.count = 8000, according to your answer, will it cost namnode 8000 * 2M (assume a thread cost 2M)?The another thing, why reading or writing hdfs file failed when namenode memory usage > 75%? I can not think a reason – jun zhou Nov 02 '12 at 10:30
  • You can't say that generally, but is uses at least 1mb stack, 2mb is a good upper bound I believe. – Thomas Jungblut Nov 02 '12 at 10:32
  • I write a program which needs to get many files from hdfs, when namonode memory usage > 75%, it will find that many files can not be got from hdfs – jun zhou Nov 02 '12 at 10:34