-1

I have a number of numpy arrays which I've generated in Python 3.4 scripts and saved, i.e. 

import numpy as np
np.save('array1.npy')

I seem to be running into slight issues trying to use these in Python2.7 in terms of performance (maybe more). Is there a difference?

EDIT: The numpy arrays are multidimensional, with around 1e8 elements. Scripts I run in Python2.7 with .npy files created in Python3.4 take forever/run endlessly. I suspect there are compatibility issues.

ShanZhengYang
  • 16,511
  • 49
  • 132
  • 234
  • 1
    `I seem to be running into slight issues trying to use these in Python2.7 in terms of performance ` - can you show this in a code snippet? – cel Dec 09 '15 at 01:12
  • Please be more specific. What kind of an array is it? How large? What sort of "performance issues"? – ali_m Dec 09 '15 at 11:54
  • @ali_m These are arrays shaped `(999, 1000, 1000)`. Dimensions like that. `dtype('float64')`. As for the behavior, if I run these Python3 arrays in Python2.7 scripts, they sometimes "stall", i.e. they do not run to completion – ShanZhengYang Dec 10 '15 at 21:56
  • Each of those arrays will be about 8GB in size. Are you sure you're not just running out of memory? – ali_m Dec 10 '15 at 22:10
  • It would help if you could show *where* your scripts are stalling - is it literally when you are reading in the arrays, or do problems occur later on when you are processing them? Can you give an example? – ali_m Dec 10 '15 at 22:15
  • @ali_m They run in Python3 though. I have 16GB available, so it moves. I break the arrays up, and execute function calls on individual elements, iterating through the arrays with a for statement. It runs in Python3.4...It doesn't in Python2.7 – ShanZhengYang Dec 10 '15 at 22:18
  • Are both Python versions 64 bit? Are there any version differences between numpy, or any of the other libraries you are using? Without seeing any code, all I can do is guess at possible causes. – ali_m Dec 10 '15 at 22:23
  • In general Python2.7 has no "issues" with 64 bit compatibility. If you have problems setting it up then you should ask a separate question (if you do, then please be more specific!) – ali_m Dec 10 '15 at 23:07
  • I don't really know what you're referring to. Like I said, you will have to ask another question. – ali_m Dec 10 '15 at 23:37

1 Answers1

0

In a (now deleted) comment above, it turned out that the OP was comparing a 32 bit version of Python2.7 against a 64 bit version of Python3.4. This is almost certainly the reason for the "performance issues" alluded to in the question.

A (999, 1000, 1000) float64 array, such as the OP is using, will be approximately 8GB in size. Although the OP has 16GB of RAM, a 32 bit process will not be able to address more than 4GB 3GB of memory. Consequently it must either crash, or start swapping and become extremely slow.

Community
  • 1
  • 1
ali_m
  • 71,714
  • 23
  • 223
  • 298
  • Why would it start swapping? The process is running into an address space limit, not a physical memory limit. – user2357112 Dec 10 '15 at 23:13
  • @user2357112 You may be right - I must admit that I've never before hit a process-level limit on address space. As far as I can see, only one of two things could happen - either the process will fail to allocate enough memory for the array and crash immediately, or it will start swapping. The latter scenario seemed more consistent with what the OP was describing. In any case, I'm convinced that the root cause of the problem is the bitness of the Python process. Feel free to set me straight by writing a better answer :-) – ali_m Dec 10 '15 at 23:24