Linux tool for MPI calls caused bottleneck analysis

Question

Good morning!

I am looking for a LINUX tool that can help to analyze bottlenecks caused by MPI communication calls.

The code (C++) is highly parallel and run on many computational nodes, connected by a fast network.

It doesn't use GPU, only use CPU for computations.

What I need is to find if/when it happens that other MPI processes spend time waiting for information from other MPI processes. This could be caused e.g. by different nodes hardware.

I am currently NOT trying to profile say single core code efficiency, I'm only interested in the bottlenecks caused by MPI communication calls. In other words, trying to analyze/improve performance scaling for large number of cores/nodes.

Thank you very much.

You should try to use https://software.intel.com/en-us/intel-vtune-amplifier-xe or https://www.allinea.com/products/map — Mahmoud Fayez, Mar 26 '17 at 18:45
However both of them are not for free and most likely you will find one of them installed on your system — Mahmoud Fayez, Mar 26 '17 at 18:46
Thank you, I will ask cluster administrator whether we have any of these tools installed. — strider, Mar 26 '17 at 18:58

score 0 · Answer 1 · answered Mar 26 '17 at 19:31

http://www.scalasca.org/ and http://www.mcs.anl.gov/research/projects/perfvis/ are tools targeting MPI performance analysis.

They were recommended to me during MPI training sessions i have followed once, but i have never tried them myself.

I was told also about TotalView http://www.roguewave.com/products-services/totalview but it's not free.

Hope this helps.

Linux tool for MPI calls caused bottleneck analysis

1 Answers1