Optimizing function

Question

I have a python script but there is a function inside that kills my process:

def my_function(data):
    data = np.array(data).flatten()
    reshaped_data = np.reshape(data, (len(data), 1))
    return reshaped_data / reshaped_data.transpose()

Any ideas how to make it efficient?

Technical Details:

Number of logical CPUs: 64
Number of physical CPUs: 32
Current CPU frequency: 3103.227812499999 Mhz
Max CPU frequency: 0.0 Mhz
Total memory: 249.02 GB
Available memory: 215.45 GB

You mean that a function you use, breaks the execution? Does it produce any error? — LoukasPap, Jun 27 '23 at 13:36
what is it that you want to achieve with reshaped_data / reshaped_data.transpose() ? — user2489139, Jun 27 '23 at 13:42
Does it run OOM? Then please provide some details like input array dimensions and your RAM size - depending on those, different solutions may apply. — STerliakov, Jun 27 '23 at 13:42
@SUTerliakov-supportsstrike dataset is quite big. A few million lines across a few thousand users — Beck, Jun 27 '23 at 13:50
thats the problem, when your input data is size n, by broadcasting it gets size n*n, as long as size of data is in order of 10000, its okay, but beyond, that breaks your kernel, its getting just too big. thats the problem. What you can do about it depends on what you do with that broadcasted data. — user2489139, Jun 27 '23 at 13:53
@user2489139 hm? This is element-wise division, no matrix multiplication, why should the dimensions grow to `n^2`? — STerliakov, Jun 27 '23 at 13:56
@SUTerliakov-supportsstrike because of the different dimensions of A and B the data broadcasts into n*n, see https://numpy.org/doc/stable/user/basics.broadcasting.html figure 4 — user2489139, Jun 27 '23 at 13:58
Ough. Sorry, @user2489139 is correct, such array won't really fit any RAM for several M rows. There are ways to batch (generators, on-demand loading, etc.), but then you'll need to explain how this data is used - if you strictly need an array of such size, you'll have to rewrite other program parts as well. — STerliakov, Jun 27 '23 at 14:07
I doubt you really *need* to generate or even to compute this thousand of billion generated numbers. Even if you could generate it, it would be very slow to compute (with a very low entropy). Please explain the purpose of doing such a thing so we can help you. — Jérôme Richard, Jun 27 '23 at 16:50

score 0 · Answer 1 · answered Jun 28 '23 at 05:45

0

I have found a way around it. Consider that the function is being used in a bigger function as so:

def my_function(data):
    data = np.array(data).flatten()
    reshaped_data = np.reshape(data, (len(data), 1))
    return reshaped_data / reshaped_data.transpose()

d1 = my_function(data)
res = d1 - np.triu(d1, k=0) - np.tril(d1, k=-16)

Which I replaced with:

n = len(data)
sum_of_ratios = 0
count = 0
for i in range(n):
    for j in range(i+1, min(i + 16, n)):
        sum_of_ratios += data[j] / data[i]
        count += 1

Any other ideas are welcomed

answered Jun 28 '23 at 05:45

Beck

115
6

You could also try to replace `flatten()` by `ravel()`. Since `flatten()` always returns a copy it might be more efficient to use ravel ([see this question](https://stackoverflow.com/questions/28930465/what-is-the-difference-between-flatten-and-ravel-functions-in-numpy)), – Peter234 Jun 28 '23 at 06:12
But I m not using `flatten()` anymore – Beck Jun 28 '23 at 07:44
Yes, it's a suggestion for your first approach (or the next time). – Peter234 Jun 28 '23 at 10:46
Is `sum_of_ratios` what you want to compute or just an example? This is not mention in the question so this is surprising to see this in an answer... Besides, the proposed version should be very slow (and far from being optimal starting from `count += 1` which is not useful here since you know the size of the range ahead of time). – Jérôme Richard Jun 28 '23 at 16:58
@JérômeRichard Yes, `sum_or_ratios` is what I want to compute – Beck Jun 29 '23 at 09:57

Optimizing function

1 Answers1