1

*update - added this code to show what pivYears is

pivYears = [] for i in range(1990,2011): year = str(i) varAdd = 'XXX_'+year pivYears.append(varAdd)

So I have two lines of code below.

One that works successfully with Python 2.7.5 and Pandas 0.12.0

dfp = pd.pivot_table(df, values=pivYears, rows=['partid'],cols=['finestflag_OV'], aggfunc=np.sum)

And this one that does not work successfully with Python 2.7.6 and Pandas 0.13.1. I'm getting error refering to 

malloc: *** mach_vm_map …. error code = 3

here's the code:

dfp = pd.pivot_table(df, values=pivYears, index=['partid'],columns=['finestflag_OV'], aggfunc=np.sum)

It could be that I am misunderstanding pivot vs. pivot_table and/or how it works in 0.13.1.

Lastly, since I've been using 0.12.0 for a while and am comfortable with it, is it possible to downgrade from 0.13.1 to 0.12.0 on my new machine? (Also, can I go back in Python versions so all my machines have the same version of Python and Pandas?)

Edits for questions: Both OS's are OS X 10.9.2.

Here's a subset of 9 rows of data in the CSV. https://github.com/nygeog/python/blob/master/pandas/pivot_sample.csv

The actual file has ~1.7 million rows. Which works fine on machine with 0.12.0.

Thanks, Danny

UPDATE

So I used Enthought Canopy 1.4.0 (64 bit) and it can process this data but it does yield the following error message:

(1564174, 79)/Users/danielmsheehan/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/io/parsers.py:1070: DtypeWarning: Columns (38,41,43,52,66,67) have mixed types. Specify dtype option on import or set low_memory=False. data = self._reader.read(nrows) /Users/danielmsheehan/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/io/parsers.py:1070: DtypeWarning: Columns (43,66,67) have mixed types. Specify dtype option on import or set low_memory=False. data = self._reader.read(nrows)

This all still works fine in Pandas 0.12.0 on my other machine. But I can't really figure out why it works in Enthought but not on my machine through Sublime text 2.

GIS Danny
  • 173
  • 1
  • 7
  • 1
    What operating system are you using? – Phillip Cloud Apr 29 '14 at 13:31
  • Have a look [here](http://stackoverflow.com/q/20970417/564538). Looks like you might be running out of memory. – Phillip Cloud Apr 29 '14 at 13:47
  • Strange though, that it's not occurring on v0.12.0 – Phillip Cloud Apr 29 '14 at 13:50
  • you can post a frame that is equivalent of your original? (e.g. the code to do so) – Jeff Apr 29 '14 at 13:59
  • What is `pivYears`? And you don't have the issue with the sample dataframe? (only with the full one?) – joris Apr 29 '14 at 20:52
  • Sorry, added the code that creates pivYears. It worked with a smaller dataset (~400 MB) but didn't work with the larger ones. I have 32 GB of RAM. on the Python 2.7.6 and Pandas 0.13.1 so I wouldn't think it should have a prob with the size. – GIS Danny Apr 30 '14 at 21:08
  • Oh and here's a subset of 9 rows of data in the CSV. https://github.com/nygeog/python/blob/master/pandas/pivot_sample.csv – GIS Danny Apr 30 '14 at 21:09

0 Answers0