4

What is the best way to avoid enormous CPU usage when writing while True loops which could do nothing in some circumstances? It seems like time.sleep(0) hurt CPU:

import time

msg = []

# in main thread I will add messages to msg list sometimes
msg.append('some_msg')

# in some other thread I will parse them like this:
while True:
  for msg in msgs:
    # do some operations
  time.sleep(0) # hurt CPU in case of len(msg) == 0

Yeah, I can use time.sleep(1) instead of it but is it really the most correct way to do it in Python?

msw
  • 42,753
  • 9
  • 87
  • 112
FrozenHeart
  • 19,844
  • 33
  • 126
  • 242
  • 3
    Why are you doing this? – dursk Oct 25 '15 at 13:38
  • 3
    `time.sleep(0)` makes no sense here as it doesn't do anything useful, it just calls a whole bunch of platform-dependent code (thus, wasting resources) and that's all. – ForceBru Oct 25 '15 at 13:42
  • Does pass not suffice? – Robin Gertenbach Oct 25 '15 at 13:46
  • @Robin Gertenbach No – FrozenHeart Oct 25 '15 at 13:49
  • I'm still not sure what's wrong with `time.sleep(x)` but did you know you can supply floating point seconds? Like `time.sleep(0.5)`? – President James K. Polk Oct 25 '15 at 13:49
  • @JamesKPolk The specified code will use CPU as much as possible when `len(msgs) == 0` – FrozenHeart Oct 25 '15 at 13:50
  • 3
    The specified link doesn't contain any real answer to this question, please reopen my question – FrozenHeart Oct 25 '15 at 13:52
  • @FrozenHeart: agreed, i have voted to reopen. Also see [this](http://stackoverflow.com/questions/3221314/asynchronous-programming-in-python) SO question for one way of addressing the problem in a IO context. – President James K. Polk Oct 25 '15 at 14:01
  • 3
    The reason to call sleep() in this context is to allow other threads to run while your message loop waits for a message. You should set the sleep time to a value you can live with. If your app needs to respond to a message within one tenth of a second, the use sleep(0.05) and release the cpu to other processes while waiting for a msg to arrive. – cstrutton Oct 25 '15 at 14:06
  • @Markus Meskanen http://stackoverflow.com/questions/3727420/significance-of-sleep0 – FrozenHeart Oct 25 '15 at 14:06
  • Maybe `time.sleep(0.01)` or `time.sleep(0.001)`? – Uri Oct 25 '15 at 14:13
  • @Uri Maybe, but I'm asking about the most correct way to do it – FrozenHeart Oct 25 '15 at 14:14
  • 2
    The 'correct' way to do this depends on what/how your `msgs` variable is actually added to/deleted from. When you have 'processed' a message, is it discarded? You need to provide more information on your specific problem before we can help you. – Tom Dalton Oct 25 '15 at 14:43
  • @Tom Dalton "When you have 'processed' a message, is it discarded?" How will it help you? – FrozenHeart Oct 25 '15 at 15:09
  • @FrozenHeart Tom asked the question he did because your code as written is an infinite loop around code that doesn't appear to ever increase or decrease the work needed. Put another way, your question is horribly under-specified and without more clarification is useless *and* unanswerable. – msw Oct 25 '15 at 18:24

2 Answers2

2

Its just a classical producer consumer problem, and you must be seeking a better implementation than you did.

In your case if you are using a positive value for sleep then you have already done saving CPU for other processes to also use it since high CPU usage is good (see Note at the bottom for clarity).

If you are looking for optimization and better performance then you need to tweak your code specific to your application needs, here are two suggestions.

  1. You might use signaling in threads (also call inter-thread communication) using wait and notify calls if you can convert msg to a thread-safe queue; this way you can overcome arbitrary waiting.

  2. If your processing on the msg items (that you perform in for loop) is IO hungry then you can try splitting in independent threads or you can upgrade your code to do async processing.

Note: If you have a high CPU usage then it means you are best using your computing resources, but it can lead to deprived reliability down the line and starvation of resources for other process which might be bad hence try putting priority on your threads/process as per your needs are.

anand
  • 1,506
  • 14
  • 28
  • Polling with sleep is almost always wrong and in this case a virtual certainty. Since the code consumes nothing and the OP is explicitly refusing to clarify, there is no valid answer. That's what you get with bad questions. (±0). – msw Oct 25 '15 at 18:18
  • 1
    @msw yes polling with sleep is brutal way but its simple and lot of people use it, this why I have suggested to use event loop and thread-safe queue. – anand Oct 25 '15 at 18:29
  • 1
    @msw further as it has been mentioned its a non-blocking calls thats being used then using `event-loop` is not giving to give any better throughput it will just prevent starvation, my best bet would be to use `wait` and `notify` so that when `msgs` ends up CPU can do something better. – anand Oct 25 '15 at 18:38
  • Agreed. I'll read this answer as "if the OP had asked better, this is a good exposition of asynchronous methods". – msw Oct 26 '15 at 03:36
0

You are using the wrong datatype. Lists are not made for inter-thread communication. Use queues. The get-method is blocking.

Daniel
  • 42,087
  • 4
  • 55
  • 81