2

I built a Rocks Cluster for my school project, which is matrix multiplication, with one frontend and 5 other computers which are nodes. Over MPI I send them partions of matrix which they use for multiplication and then they send data back. Command which I run is:

mpirun -hostfile myhostfile ./myprogram

where myhostfile is a file of names of nodes and their slots(thread) numbers. My program is working and I'm trying to analize it now.

My question is how can i see the work of each nodes core/processor working on his task, are the all processors working, is there some kind of overload? I tried to install Vampir profiler and Intels Vtune Amplifierbut but I have some problems attaching them to my program with this command above (other comands dont allow me to run my programs on all threads of a node). All that i have accomplished (to see my nodes working good besides Ganglia) is to login to a node from the frontend and with the command "top" I could see when my program is executing by the number of threads and almost 100% CPU usage on each thread.

naveen-rn
  • 98
  • 1
  • 6
warkovich
  • 21
  • 2
  • Can you give us some details on the MPI implementation details? And, also are you looking for some memory footprint details? – naveen-rn Jan 24 '16 at 05:57
  • Hi naveen-m, I am looking for something like that, just to see nodes own detailed work, in terms of memory usage, and cpu usage on cores, here is the code i used [link](https://www.daniweb.com/programming/software-development/code/334470/matrix-multiplication-using-mpi-parallel-programming-approach) – warkovich Jan 24 '16 at 18:19
  • Did you have a chance to try other profilers? I would suggest to use Allinea(DDT), or TAU. Allinea has 30 trial and it should be straight forward to configure and use...Allinea can provide the best usage memory footprint details. TAU is also a similar tool, with almost similar properties. https://www.cs.uoregon.edu/research/tau/about.php – naveen-rn Jan 25 '16 at 07:20
  • meanwhile, let me look at the mpich documentation and try to figure if there are any env variables to print the memory details!! – naveen-rn Jan 25 '16 at 07:22

1 Answers1

1

Take a look at mpstat

With no params it will show aggregated load for all cores

mpstat -P ALL shows load for each core

This will give you realtime stats for your nodes:

watch pdsh -w compute-01-[01-10] mpstat

(use your compute nodes names)

dimm
  • 100
  • 1
  • 5