2

I have written a code to help accumulate some data over a constant time interval from a non-constant data set. It is working as expected on my current computer which is a 6 core processor (Intel i5-8400), and consumes ~25% of CPU usage. The problem is that now I am trying to rerun the same code on another computer which has 40 cores (Intel Xeon E5-2630 v4), and as soon as it starts the below function, it spikes to 55% CPU usage, while I was expecting it to be closer to 5%.

This has become an issue because I am sharing this computer with other people and it is hindering their work. Doing a comparison of the time it takes to run the script between the two computers, it seems to be negligible so I am not sure if it is actually using all of the processors (which makes sense, since I have no idea of how to multi thread loops!)

I have previously tried to find an answer, and the closest answer I could find was that it may be due to a while loop (Python script too much cpu usage) but I don't think that is applicable to my case... Most other questions were related to how to increase cpu usage, which is the opposite of my problem.

def accumulate_value (table_data_1,i_time_step):

    # table_data_1 formatted as a dataframe with columns names ['value','date_time']

    table_data_1.reset_index(drop=True,inplace=True)

    table_data_1["delta_time"] = table_data_1["date_time"].diff(periods=1).dt.seconds/60
    table_data_1["delta_time_aux"] = 0
    table_data_1["value_aux"] = 0
    table_data_1["datetime_stamp"] = pd.to_datetime(0)


    # Start loop

    for aux_counter_1 in table_data_1.index.values.tolist():

        table_data_1_aux1 = table_data_1.loc[aux_counter_1:,:]
        table_data_3 = table_data_1_aux1.loc[table_data_1_aux1.index[table_data_1_aux1.loc[:,'date_time'] - table_data_1_aux1.loc[aux_counter_1,'date_time'] <= datetime.timedelta(minutes=i_time_step)],:]


        if len(table_data_3.index) > 1 :

            table_data_1.iloc[aux_counter_1,3] = datetime.timedelta.total_seconds(table_data_3.iloc[-1,1] - table_data_3.iloc[0,1])/60
            if table_data_1.iloc[aux_counter_1,3] >= i_time_step:
                table_data_1.iloc[aux_counter_1,4] = table_data_3.loc[:,'value'].sum() - table_data_3.iloc[-1,0]
            else:
                table_data_1.iloc[aux_counter_1,4] = table_data_3.loc[:,'value'].sum()
            table_data_1.iloc[aux_counter_1,5] = table_data_3.iloc[-1,1]

        elif len(table_data_3.index) == 1 :
            table_data_1.iloc[aux_counter_1,3] = 0
            table_data_1.iloc[aux_counter_1,4] = table_data_3.loc[:,'value'].sum()
            table_data_1.iloc[aux_counter_1,5] = table_data_3.iloc[-1,1]
        else:
                print(table_data_3)


    table_data_1["year_stamp"] = table_data_1["datetime_stamp"].dt.year

    table_data_2 = table_data_2

    return table_data_2
eyllanesc
  • 235,170
  • 19
  • 170
  • 241
William TL
  • 23
  • 3
  • Is it 55% usage of all cores or just the active core? – Polkaguy6000 Jan 09 '19 at 04:15
  • 1
    Upon a closer check of task manager, I have noticed that under the Performance tab, it only shows 1 graph (https://imgur.com/a/3tv58at) rather than a graph for each processor. This has confused me a bit! EDIT: A closer inspection at resource monitor, I am able to clarify, yes, it is the CPU total. It seems to be using all cores except for "Service CPU Usage" – William TL Jan 09 '19 at 04:27

1 Answers1

1

I think the issue is hat numpy is creating a large number of threads, assuming it is the only process running on the machine.

See Limit number of threads in numpy for how to limit the number of threads used.

cco
  • 5,873
  • 1
  • 16
  • 21
  • Thanks so much! It fixed my issue. Out of curiosity, do you know why this behaviour occurs? (It was running as expected in my machine) – William TL Jan 10 '19 at 06:17
  • Numpy uses multiple threads internally to run as fast as possible. I think the assumption (by default) is that there aren't other users, so it should use as much CPU as is available, or at least a large fraction of what is available. – cco Jan 10 '19 at 21:02