Hadoop CDH5 Maximum Network Performance on 10 node using 1 MapReduce TaskTracker

Question

I'm trying to get maximum network performance with CDH5 Cloudera with 10 nodes with 1 MapReduce TaskTracker but can't seem to get the traffic to produce the traffic I need. I have 40G nic cards and currently only getting 10G at the most with is really low.

When using 1 MapReduce TaskTracker how can I produce better performance and network throughput?

Running -- 10 nodes RedHat 6.6 Current Nic driver CDH5 Kernel: 3.18.1

I've decommissioned 9 MapReduce TaskTracker nodes but still not getting good results.

My question was cut in (Stackoverflow) half so hope this is enough information or just ask. — Queasy, Mar 16 '15 at 16:04
Why are you only running one TaskTracker? You should run a TaskTracker on every host running a DataNode. — Jeremy Beard, Mar 18 '15 at 02:43

score 0 · Accepted Answer · answered Mar 19 '15 at 21:09

I finally figured out that you really don't need a task tracker on data nodes for network dfsio test so I decommissioned all datanode "task trackers" and put the task tracker on the name node and now I have both write/tx and read/rx traffic now which is what I wanted for network throughput data.

Hadoop CDH5 Maximum Network Performance on 10 node using 1 MapReduce TaskTracker

1 Answers1