3

I am currently doing some I/O intensive load-testing using python. All my program does is to send HTTP requests as fast as possible to my target server.

To manage this, I use up to 20 threads as I'm essentially bound to I/O and remote server limitations.

According to 'top', CPython uses a peak of 130% CPU on my dual core computer.

How is that possible ? I thought the GIL prevented this ? Or is it the way Linux 'counts' the resources consumed by each applications ?

yadutaf
  • 6,840
  • 1
  • 37
  • 48
  • I don't know about python, but I've certainly frequently seen the linux System Monitor reporting CPU usage of > 100%, all the way up to 180%. No idea why however. – Vala Jul 23 '12 at 15:19
  • The GIL is released on I/O calls. – Martijn Pieters Jul 23 '12 at 15:19
  • I know the total 'available percentage' is 100*'Core count'. So that > 100% implies that your application runs on multiple cores. But I did not knew that the GIL was released on I/O tasks. Still, I did not know HTTP lib was multithreaded. – yadutaf Jul 23 '12 at 15:36

3 Answers3

15

100 percent in top refer to a single core. On a dual-core machine, you have up to 200 per cent available.

A single single-threaded process can only make use of a single core, so it is limited to 100 percent. Since your process has several threads, nothing is stopping it from making use of both cores.

The GIL only prevents pure-Python code from being executed concurrently. Many library calls (including most I/O stuff) release the GIL, so no problem here as well. Contrary to much of the FUD on the internet, the GIL rarely reduces real-world performance, and if it does, there are usually better solutions to the problem than using threads.

Sven Marnach
  • 574,206
  • 118
  • 941
  • 841
  • You're mixing up processors and cores! – Danilo Bargen Jul 23 '12 at 15:21
  • Isn't that a very odd way of defining the CPU usage percentage? Any idea how this works with hyperthreading? My computer has 4 hyperthreaded cores, but I never see a reported usage percentage above ~180%. – Vala Jul 23 '12 at 15:22
  • With the differences in AMD/Intel Cores/Processors, superscalar, hyperthreading, Linux threads/processes, Windows threads/processes and scheduling. I challenge anyone to keep track of exactly 'how concurrent will my code be?' – lynks Jul 23 '12 at 15:22
1

That is possible in situations when used C-extension library call releases GIL and does some further processing in the background.

Alex
  • 2,837
  • 5
  • 25
  • 27
0

If you find this irritating, set your preferences (specifically, the preferences of your System Monitor or equivalent tool) to enable "Solaris Mode," which calculates CPU% as a proportion of total processing power, not the proportion of a single core's processing power.

algorowara
  • 1,700
  • 1
  • 15
  • 15
  • It's not. I was wondering how python could use more than one core at a time but this was because I misunderstood the role of the GIL. – yadutaf Jul 23 '12 at 18:44