0

I have a memmap for reading binary data (> 1000 files of >> 1000 records each)

    f  = np.memmap('data.dat', dtype= myFormat, mode= 'r')

and know how to count records satisfying a condition depending on a single field (named 'Status' in myFormat):

    eval = f['Status'] & 0x20 == 0x20
    nNotEval = np.count_nonzero(eval == False)

Another field is used conditionally on eval:

    v = calibration * f['V'][eval]

Now, how to combine two conditions element-wise? The following failes:

    unexpected = (f['Status'] & 0x02 == 0) and (v > 15.)

Python protests "The truth value of an array with more than one element is ambiguous" and suggests "Use a.any() or a.all()", but that is not what I need for

    nUnexpected = np.count_nonzero(unexpected)

Do I have to iterate? If so, how is the syntax? Something like for (a,b) in ...?

Rainald62
  • 706
  • 12
  • 19

1 Answers1

1

This isn't a memmap issue, or even count_nonzero one. It's about doing the compound boolean operations:

unexpected = (f['Status'] & 0x02 == 0) and (v > 15.)

This should work better:

unexpected = (f['Status'] & 0x02 == 0) & (v > 15.)

and is a Python scalar operation (short circuiting). But your expressions produce boolean arrays. You need to use & to join them.

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() python (duplicate?)

Community
  • 1
  • 1
hpaulj
  • 221,503
  • 14
  • 230
  • 353
  • Thank you. & works for me. And I should have searched SO for the verbatim error message. – Rainald62 Mar 21 '17 at 00:37
  • Sorry, I accepted the answer before proper testing. & has worked for one case, but now I get the following error message `ValueError: operands could not be broadcast together with shapes (682,) (674,)`. – Rainald62 Mar 21 '17 at 11:01
  • 682 is the `len` of `f['Status'] & 0x02 == 0`, of `eval`, and of `f['V']`. The length becomes 674 in the operation `f['V'][eval]`. – Rainald62 Mar 21 '17 at 11:23
  • I postponed that operation (needed for further evaluation) and found it works: `eval = f['Status'] & 0x20 == 0x20; v = calibration * f['V']; unexpected = (f['Status'] & 0x02 == 0) & (v > 15.); v = v[eval]`. – Rainald62 Mar 21 '17 at 11:36