0

I have slow running python-3 script and upon profiling it, I have found the bottleneck to be the numpy array function. After reading up some documentation about it (https://numpy.org/neps/nep-0018-array-function-protocol.html), it seems that we can disable this functionality with an environment variable. So in my script, I do the following:

os.environ["NUMPY_EXPERIMENTAL_ARRAY_FUNCTION"]="0"

However, my profiling output (with cProfile) still shows calls for this array function:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
1321842/822748 1684.424    0.001 1938.861    0.002 {built-in method numpy.core._multiarray_umath.implement_array_function}

Am I missing something in terms of disabling this functionality? Do I need to use a different environment variable? My understanding is that if this functionality is disabled, no underlying calls to the array function would be made. Please let me know in case there is another workaround available to disable this numpy functionality.

I am using python 3.6.9 with numpy v 1.18.1.

Thank you.

ForceBru
  • 43,482
  • 10
  • 63
  • 98
D.prd
  • 605
  • 1
  • 7
  • 21
  • I think it's not related to the `__array_function__` protocol: your code is just generating a lot of arrays – ForceBru Jun 09 '20 at 19:16
  • That function is not the bottleneck. It's just that most of your numpy function calls pass through it, so its called often. – hpaulj Jun 09 '20 at 19:38
  • See https://stackoverflow.com/questions/61983372/is-built-in-method-numpy-core-multiarray-umath-implement-array-function-a-per, https://stackoverflow.com/questions/58909525/what-is-numpy-core-multiarray-umath-implement-array-function-and-why-it-costs-l/58909765#58909765 – hpaulj Jun 09 '20 at 19:42
  • Are you sure your `pandas` and `numpy` code is making optimal use of the compiled whole-array capabilities of these packages? Lots of iteration and `pandas` `apply` will slow your code down. – hpaulj Jun 09 '20 at 19:50
  • thank you for your responses. I don't use pandas apply in my code and yes I do have quite a lot of arrays so I wanted to disable the array functionality which is one of the recommendations to try with numpy 1.17/18. Is there any other guideline on how I could optimize the usage of this array function which implicitly gets called in numpy? – D.prd Jun 11 '20 at 17:35

0 Answers0