Why is Idle Python Thread Consuming upto 90% of CPU?

Question

This is my first threading program. I'm facing a strange issue here. I'm building a simple scheduler like application in Django, where function names (to be executed periodically) will be stored in Django Model along with their next execution time.
A management command is executed to start a thread which runs continuously to check if execution of any function is due, if yes, a new thread is started for executing this function. This way, separate thread for each function is created (at least, that's the idea!).

class Command(BaseCommand):

    def __init__(self):
        super(Command, self).__init__()
        self.lock = None

    def handle(self, *args, **kwargs):
        self.lock = threading.RLock()
        t1 = threading.Thread(target=self.cron_thread)
        t1.start()
        t1.join()

    def cron_thread(self):
        while True:
            # Fetch only Active records
            scheduled_actions = Scheduler.objects.filter(active=True)
            for scheduled_action in scheduled_actions:
                # check if execution is due
                if scheduled_action.next_execution_time == datetime.now():
                    # creating a new thread
                    function_thread = threading.Thread(target=eval(scheduled_action.function_name), args=[self.lock])
                    function_thread.start()
                    function_thread.join()
                    scheduled_action.next_execution_time = local_timezone.localize(datetime.now() + relativedelta(minutes=scheduled_action.interval))
                    scheduled_action.run_now = False
                    scheduled_action.save()

    def somefunction(self):
        self.lock.acquire()
        # function body
        self.lock.release()

The command that I have created to start execution of whole program is: python3 manage.py runcrons-debit

As soon as I execute this command, I can see in the htop results that two process are running and consuming almost 80% of CPU, as shown in the following image: View Image Please note here that no scheduler records are active yet.

When scheduler records are made active, and when the function actually runs, the processes displayed in htop increase to three and CPU usage decreases drastically to 0.0%. As shown in the following image: View Image

There are two things here that I can't understand,

once the execution of function is over and there's no function being executed the htop result goes back to two process consuming almost 80-90% of CPU. Why are idle threads here consuming this much of CPU?

Also, when no function is being executed why are there still two process being displayed? I can understand one of them is of the command itself but what is causing the second process to be created?

Chances are very low that your condition `scheduled_action.next_execution_time == datetime.now()` would ever be `True` (`datetime.now()` is a date-time object, it includes hours, minutes, seconds and even microseconds). — julienc, Jan 25 '19 at 14:49
because `while True: do stuff` ? uses computational power? use a sleep of 30s , 1 min, 10min, depending on the resolution you need this to have. Better: use a scheduler and let each thread that needs it restart itself in x time .. see f.e. https://stackoverflow.com/questions/2398661/schedule-a-repeating-event-in-python-3 — Patrick Artner, Jan 25 '19 at 14:49

score 4 · Accepted Answer · answered Jan 25 '19 at 15:04

cron_thread has an infinite loop. This loop first retrieves scheduled actions, then loops over them. For each action, if the action is scheduled for the exact current time, the action is executed.

If no actions are scheduled, the loop will simply keep retrieving the scheduled actions, over and over again. If there is an action, it will check whether now is the time to execute it. Here's another problem: datetime.datetime.now() has very high precision (nearest microsecond), so the chances that it'll match the scheduled time for an action are very low. This means your loop will retrieve all scheduled actions, loop over all the actions, and then go back to the top.

On the off-chance that a scheduled action's time does match the current time, that action will be executed, then the inner loop moves onto the next action. When it's looped through all the actions, it will go back to the top and retrieve all actions once again.

Basically, your program is constantly comparing any scheduled actions to the current time. That takes processing power. A better way to execute these actions would be to check the time for each new action as it's added to the list of tasks, calculate the necessary delay until that action needs to be executed, and then set a timer to execute that action after the necessary delay (time.sleep in a thread, after calls in tkinter, that sort of thing).

That's a great suggestion! For now, I've hard-coded the time duration for time.sleep() as 1 minute, as one of my scheduler is having interval of 2 mins and another one of 3 mins. CPU usage % went down to almost 0.0% after using sleep(). Thanks a lot! — Abhishek Dalvi, Jan 28 '19 at 07:45

Why is Idle Python Thread Consuming upto 90% of CPU?

1 Answers1