0

I have some code that needs to work through a huge set of matrices (all 5 by 10 binary matrices in reduced row echelon form, with no zero rows) and either accept or reject each matrix depending on whether it satisfies some conditions. Since there's a lot to get through, I'm trying to use multiprocessing to speed things up. Here's roughly what my code currently looks like:

import multiprocessing as mp
import numpy as np

def check_valid(matrix):
    # Perform some checks and things
    if all_checks_passed:
        return matrix.copy()
    return None

subgroups = []

with mp.Pool() as pool:
    subgroups_iter = pool.imap(
        check_valid,
        get_rref_matrices(5),
        chunksize=1000
    )

    for item in subgroups_iter:
        if item is not None:
            subgroups.append(item)

get_rref_matrices is a generator function that recursively finds all the rref matrices (I'm not sure if this function is causing any issues). The full code for this function is here, if that's of interest.

When I run the program, it seems to be very slow (hardly any faster than a single process) and the CPU usage is only about 10%. I've previously run code that has maxed out my CPU, so I'm stumped as to why this code isn't running faster.

MrLatinNerd
  • 85
  • 9
  • 5
    Have you profiled your code? I'm thinking there could be a bottleneck of some kind, and that the CPU is not the bottleneck. Or maybe it's that some shared resources are locking out all other threads? Just a few guesses. – Random Davis Sep 08 '21 at 21:25
  • 1
    Have you tried a smaller chunk size? 1000 seems a lot. Also, what's the approximate total number of matrices? – AKX Sep 08 '21 at 21:26
  • 1
    See [How can you profile a Python script?](https://stackoverflow.com/questions/582336/how-can-you-profile-a-python-script) – martineau Sep 08 '21 at 21:30
  • 1
    In particular `get_rref_matrices` seems like a likely candidate to be a bottleneck as it's a generator and not a precomputed list. – Kyle Parsons Sep 08 '21 at 21:36
  • 1
    Yeah, the only work you seem to be passing to the worker processes is `matrix.copy()`, almost certainly, that is not the bottleneck – juanpa.arrivillaga Sep 08 '21 at 21:45
  • @RandomDavis I just tried profiling the code, but I keep getting this error: `AttributeError: Can't get attribute 'check_valid' on ` – MrLatinNerd Sep 08 '21 at 22:09
  • 2
    @MrLatinNerd does [this](https://stackoverflow.com/questions/41385708/multiprocessing-example-giving-attributeerror) help with that error? Also I meant that you could just profile the single-process version since that'd be easier I'm sure. – Random Davis Sep 08 '21 at 22:31
  • If your generator yielded a 1000 or fewer items, for example, with a *chunksize* value of 1000, you would in effect not be multiprocessing at all; one process in the pool would be grabbing all the submitted items as a single chunk. – Booboo Sep 10 '21 at 14:25

0 Answers0