0

I am running some data intensive program using Numpy. Interestingly, I found the runtime performance of the program differs drastically on my laptop (400%CPU ubuntu) and our server (100% centos) using exactly the same code. The hardware on our server is actually better than my machine.

The server is much slower. How can make the server use 400% CPU? Can anyone provide some ideas on this?

I checked the server's running tasks. There is only a hadoop process which only uses 100%CPU. And it is 32 cores....

Roger
  • 161
  • 1
  • 9
  • Perhaps https://stackoverflow.com/questions/11963148/sharing-numpy-arrays-in-python-multiprocessing-pool?rq=1 – Ignacio Vergara Kausel May 31 '17 at 08:40
  • @IgnacioVergaraKausel thanks, but this is not what I mean. I did not use multithread module explicitly in my code – Roger May 31 '17 at 08:47
  • Presumably you saying that the server is slower. There will be many differences between the two systems, the OS will only be one of them. How many other processes are running on the server compared to your laptop? What are they all doing? Is there disc-drive contention? Where are the system bottlenecks? – cdarke May 31 '17 at 08:51
  • One important aspect on *our big server* is its NUMA architecture. Each memory access to memory not directly connected to the socket the process is running on leads to a latency in the order of magnitude of factor 5. This problem is intensified if (1) there are many small memory access and (2) the OS moves permanently the workers away from its local memory in order to balance the load. – Chickenmarkus May 31 '17 at 09:00
  • This cannot be answered without more input. Too many things can be involved here: other tasks on the server competing for memory, disk IO or processor, NAS usage instead of a local disk, etc. – Serge Ballesta May 31 '17 at 09:09

0 Answers0