3

I'm trying to divide a task between all cores of a CPU. I'm doing this basically as the code below. I have, for example, a grid that has 80 elements and I have 8 cores in my CPU. For each element of the grid, I need to do a calculation, for this case, It will be 10 calculations for each core. The code is running, but It is not having the development I'm wanting, the max usage of CPU I'm having is 50%, but most of the time It stays at 25-30%. How can I make this code runs faster? How can I get more usage of a CPU?

from multiprocessing import Process, cpu_count

def initialize(info):
    my_object = my_class(info)
    grid = my_object.get_grid()
    core_amount = cpu_count()
    var_aux, start_index, end_index = 0, 0, 0
    processes = []
    while var_aux <= core_amount:
        # these if/elif is just to determinate the start and end index of the calculation for each core job
        if var_aux < core_amount:
            relation = floor(len(grid)/core_amount)
            start_index = relation * var_aux
            end_index = relation * (var_aux + 1)
        elif var_aux == core_amount:
            start_index = floor(len(grid)/core_amount) * var_aux
            end_index = len(grid)
        if start_index != end_index:
          processes.append(Process(target = partial_calculation, args=(my_object, grid, start_index, end_index )))
          processes[var_aux].start()
        var_aux += 1
    for process in processes:
        process.join()
    return my_object

def partial_calculation(my_class_object, grid, start_index, end_index): #this is the method that is called in parallel
    while start_index < end_index:
        my_class_object.make_calculation(grid[start_index]) #Here is the calculation of a grid state
        start_index += 1

if __name__ == '__main__':
    info = something()
    my_object = initialize(info)
    my_object.use_it_after_calculation_is_done()

I hope I was clear, thanks!

João Paulo
  • 6,300
  • 4
  • 51
  • 80
  • 1
    I don't know if this is the case, but maybe ```cpu_count()``` is not aware of hyperthreading. Therefore, you may have only 4 physical processor, but 8 cores in total, 4 being virtual and being disregarded. – Lucas Virgili Jul 11 '14 at 20:48
  • cpu_count in my case here is returning all the cores rightly. The yield of the code above is almost the same if I just call once `my_class_object.make_calculation(grid[position])` normally, for all grids states. – João Paulo Jul 11 '14 at 20:56
  • 2
    OK. In your while loop, you are never going to into the elif, as you are only in the while loop if ```var_aux < core_amout```. Hence, it will never be equal to it. – Lucas Virgili Jul 11 '14 at 20:59
  • this is for the cases when the division len(grid)/core_amount is not exact. And i forgot to put <=. thanks! – João Paulo Jul 11 '14 at 21:06
  • 2
    Ok, now you get into the ```elif```. Now, in your function ```partial_calculation``` you use ```start``` when I suppose it should be ```start_index``` right? – Lucas Virgili Jul 11 '14 at 21:12
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/57182/discussion-between-lucas-virgili-and-joao-paulo-oliveira-fernandes). – Lucas Virgili Jul 11 '14 at 21:15
  • 2
    is your initialize method returning the value you expect? because i dont think that should work. objects are not shared between processes, and any modification to an object in one process should not affect the object in another process. – pseudonym117 Jul 11 '14 at 21:17
  • there is a process of randomness involved, but I believe they are returning right values​​, let me make a modification here to try to check ... – João Paulo Jul 11 '14 at 21:23
  • Its returning wrong values!! – João Paulo Jul 11 '14 at 21:28
  • Despite being returning wrong values​​, the calculation is done normally. it only returns wrong values​​. I can fix it, but fixing it wont wont change the fact of cpu is being little used. – João Paulo Jul 11 '14 at 21:34
  • @JoãoPauloOliveiraFernandes How heavyweight is the calculation being done in `make_calculation`? If that method doesn't take too long to finish, it may never get th CPU usage of the process running it up to 100%. – dano Jul 11 '14 at 21:36
  • the make_calculation method is called updateState: I uploaded the code snippet here: [link](http://codeviewer.org/view/code:41b8) – João Paulo Jul 11 '14 at 21:41
  • an average of 70% for me is already good... but it is at 30% – João Paulo Jul 11 '14 at 21:42
  • here is a picture of the task manager: [img](http://s15.postimg.org/y9em6sky3/cores.png) – João Paulo Jul 11 '14 at 21:55
  • 1
    yeah i dont understand what i assume is spanish, so replying in case you didnt sort out at least the wrong value part of this - you will have to do some sort of interprocess communication (e.g. http://stackoverflow.com/questions/6920858/interprocess-communication-in-python) – pseudonym117 Jul 14 '14 at 06:14

0 Answers0