Memory error on large Shapefile in Python

Question

import shapefile
data = shapefile.Reader("data_file.shp")
shapes = data.shapes()

My problem is that getting the shapes from the Shapefile reader gives me an exception MemoryError when using Pyshp.

The .shp file is quite large, at 1.2 gB. But I am using ony 3% of my machine's 32gB, so I don't understand it.

Is there any other approach that I can take? Can process the file in chunks in Python? Or use some tool to spilt the file into chinks, then process each of them individually?

score 3 · Answer 1 · edited May 23 '17 at 11:51

3

Quoting from this answer by thomas:

The MemoryError exception that you are seeing is the direct result of running out of available RAM. This could be caused by either the 2GB per program limit imposed by Windows (32bit programs), or lack of available RAM on your computer. (This link is to a previous question). You should be able to extend the 2GB by using 64bit copy of Python, provided you are using a 64bit copy of windows.

So try a 64bit copy of Python or provide more detail about your platform and Python versions.

edited May 23 '17 at 11:51

Community

1
1

answered Sep 22 '16 at 12:57

nick_gabpe

5,113
5
29
38

Despite possibly being right, this is more a comment than an answer... One line and half that is a question? – Paulo Scardine Sep 22 '16 at 12:58
1

I haven't enough rating to comment but i still want to help – nick_gabpe Sep 22 '16 at 13:00
Nick, you might be right. How do I tell? when I run the interperter, I see `Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:32:19) [MSC v.1500 32 bit (Intel)] on win32` – Mawg says reinstate Monica Sep 22 '16 at 13:06
This question gave me the answer, and, yes, I only have 32 bit (d'oh!). thanks, Nick, and welcome to the site hwere we are not supposed to say "thanks" :-/ See http://stackoverflow.com/questions/1405913/how-do-i-determine-if-my-python-shell-is-executing-in-32bit-or-64bit-mode-on-os – Mawg says reinstate Monica Sep 22 '16 at 13:08
@nick_gabpe Sorry if i sounded like a jerk, I did upvote but still think the answer deserves some improvement. – Paulo Scardine Sep 22 '16 at 17:11
@paulo-scardine feel free to improve it. – nick_gabpe Sep 22 '16 at 17:16

score 1 · Accepted Answer · answered Sep 22 '16 at 21:57

Although I haven't been able to test it, Pyshp should be able to read it regardless of the file size or memory limits. Creating the Reader instance doesn't load the entire file, only the header information.

It seems the problem here is that you used the shapes() method, which reads all shape information into memory at once. This usually isn't a problem, but it is with files this big. As a general rule you should instead use the iterShapes() method which reads each shape one by one.

import shapefile
data = shapefile.Reader("data_file.shp")
for shape in data.iterShapes():
    # do something...

I Can conform that the problem was not with the creation of the object, but with the `shapes()` method. However, the problem was that I was using 32 bit Python, which can only address 32gB RAM. When I installed the 64 bot version, the problem went away. However, if the `iterShapes()` method only loads a single shape into memory at a time then, of course, I will use that — Mawg says reinstate Monica, Sep 23 '16 at 13:08

Memory error on large Shapefile in Python

2 Answers2