Stack Overflow
Stack Overflow

Questions tagged [numexpr]

Numexpr is a fast numerical expression evaluator for NumPy.

Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like 3*a+4*b) are accelerated and use less memory than doing the same calculation in Python.

In addition, its multi-threaded capabilities can make use of all your cores, which may accelerate computations, most specially if they are not memory-bounded (e.g. those using transcendental functions).

Last but not least, numexpr can make use of Intel's VML (Vector Math Library, normally integrated in its Math Kernel Library, or MKL). This allows further acceleration of transcendent expressions.

Numexpr User Guide lists the functionalities supported by it alongwith other useful info related to it.

94 questions
64
votes
3 answers

Parallelizing a Numpy vector operation

Let's use, for example, numpy.sin() The following code will return the value of the sine for each value of the array a: import numpy a = numpy.arange( 1000000 ) result = numpy.sin( a ) But my machine has 32 cores, so I'd like to make use of them.…
user1475412
  • 1,659
  • 2
  • 22
  • 30
21
votes
1 answer

numexpr.evaluate("a+b",out=a)

Is it safe in python numexpr to assign values to the same array you are operating on to avoid creating a temporary array? From the description of memory usage on the project homepage it looks okay, but without diving into the source code, that is…
David
  • 1,391
  • 11
  • 22
17
votes
5 answers

Evaluating a mathematical expression (function) for a large number of input values fast

The following questions Evaluating a mathematical expression in a string Equation parsing in Python Safe way to parse user-supplied mathematical formula in Python Evaluate math equations from unsafe user input in Python and their respective…
s-m-e
  • 3,433
  • 2
  • 34
  • 71
11
votes
2 answers

Parallel in-place sort for numpy arrays

I often need to sort large numpy arrays (few billion elements), which became a bottleneck of my code. I am looking for a way to parallelize it. Are there any parallel implementations for the ndarray.sort() function? Numexpr module provides parallel…
Maxim Imakaev
  • 1,435
  • 2
  • 13
  • 26
10
votes
2 answers

Optimization and speedup of a mathematical function in python

The purpose of this mathematical function is to compute a distance between two (or more) protein structures using dihedral angles: It is very useful in structural biology, for example. And I already code this function in python using numpy, but the…
NoExiT
  • 103
  • 9
8
votes
1 answer

Why " NumExpr defaulting to 8 threads. " warning message shown in python?

I am trying to use the lux library in python to get visualization recommendations. It shows warnings like NumExpr defaulting to 8 threads.. import pandas as pd import numpy as np import opendatasets as od pip install lux-api import lux import…
Akash Kumar
  • 540
  • 2
  • 4
  • 15
7
votes
1 answer

Why is Pandas.eval() with numexpr so slow?

Test code: import numpy as np import pandas as pd COUNT = 1000000 df = pd.DataFrame({ 'y': np.random.normal(0, 1, COUNT), 'z': np.random.gamma(50, 1, COUNT), }) %timeit df.y[(10 < df.z) & (df.z < 50)].mean() %timeit df.y.values[(10 <…
John Zwinck
  • 239,568
  • 38
  • 324
  • 436
7
votes
0 answers

numexpr malicious attack safety

Is numexpr safe again malicious attack? I'm considering using it in a web application, to evaluate user input text. I've also considered using PLY, ASTEVAL and Pyparsing.
user744629
  • 1,961
  • 1
  • 18
  • 27
6
votes
1 answer

Boosting the runtime of NumPy Code with NumExpr: An analysis

Since NumPy doesn't make use of multiple cores, I'm learning to speed up NumPy code with NumExpr since it has very good support for multithreading. Below is an example that I'm working with: # input array to work with x = np.linspace(-1, 1, 1e7) #…
kmario23
  • 57,311
  • 13
  • 161
  • 150
6
votes
1 answer

numba guvectorize target='parallel' slower than target='cpu'

I've been attempting to optimize a piece of python code that involves large multi-dimensional array calculations. I am getting counterintuitive results with numba. I am running on an MBP, mid 2015, 2.5 GHz i7 quadcore, OS 10.10.5, python 2.7.11. …
Brian Pollack
  • 198
  • 3
  • 12
6
votes
1 answer

Numexpr: How to use "local_dict" and "global_dict"?

I have been experimenting and trying to learn the Numexpr package. Examples on how to use it are sparse at best. Can someone please give me a quick example on how to use the "local_dict" and "global_dict" arguments?
Noob Saibot
  • 4,573
  • 10
  • 36
  • 60
5
votes
1 answer

How does parakeet differ from Numba? Because I didn't see any improvements on some NumPy expressions

I am wondering if anyone knows some of the key differences between the parakeet and the Numba jit? I am curious, because I was comparing Numexpr to Numba and parakeet, and for this particular expression (which I expected to perform very very well on…
user2489252
4
votes
2 answers

Is it possible to sum over multiple axis in numexpr?

I am trying to do something like the following: import numexpr as ne a = np.random.rand(10, 1) b = np.random.rand(1, 10) ne.NumExpr('sum(sum(a*b, 1), 0)').run(a, b) # <- error: reduction operations must occur last ne.NumExpr('sum(a*b, [1,…
evan54
  • 3,585
  • 5
  • 34
  • 61
4
votes
4 answers

Numpy sum of operator results without allocating an unnecessary array

I have two numpy boolean arrays (a and b). I need to find how many of their elements are equal. Currently, I do len(a) - (a ^ b).sum(), but the xor operation creates an entirely new numpy array, as I understand. How do I efficiently implement this…
Ponkadoodle
  • 5,777
  • 5
  • 38
  • 62
4
votes
1 answer

Capping a sub-expression in numexpr

How do I efficiently express the following using numexpr? z = min(x-y, 1.0) / (x+y) Here, x and y are some large NumPy arrays of the same shape. In other words, I am trying to cap x-y to 1.0 before dividing it by x+y. I would like to do this using…
NPE
  • 486,780
  • 108
  • 951
  • 1,012
1
2 3 4 5 6 7