0

Having an issue in Python and not sure where to start to debug a memory issue. Using the suggestions from answers I have made the changes in code and commented the lines which were previously in place:

user3224907
  • 758
  • 2
  • 15
  • 40

2 Answers2

1

Your result is becoming too large. Python cannot allocate enough memory to add to it.

More than this is impossible to say without the source code and an explanation of what you're trying to accomplish.

Roland Smith
  • 42,427
  • 3
  • 64
  • 94
1

You keep building up results until it is too large to fit into memory.

The simplest, most immediate fix for this is to turn getORMClaims() into a generator, yielding one result at a time, instead of the fully formed list of all results. Get rid of results, and every time you would have appended something to it, yield that something instead.

In your case, you would replace results.append(labels) with yield labels; and replace results.append(result) with yield result.

So, it's not hard to get rid of the memory usage problem within getORMClaims(), but then you may have to also alter whatever is calling it, so that it can handle receiving one "row" at a time instead of a list of all the rows.

For example, if the calling code looks something like

for row in foo.getORMClaims():
    writer.writerow(row)

then you should be fine, because the loop is already just using one row at a time. But if it looks more like

reportData = foo.getORMClaims()

then you have to figure out how to get rid of reportData and just consume the output of getORMClaims() directly in a loop.

If you have never heard of yield or the term generator before, then this may be a bit confusing, and you will just have to buckle down and read about them. One resource is this comprehensive Stack Overflow answer to the question "What does the yield keyword do in Python?".

Community
  • 1
  • 1
John Y
  • 14,123
  • 2
  • 48
  • 72
  • According to your answer, I believe all I need to do is replace with `yield` since the calling program has a for loop. Please correct me if I am wrong. – user3224907 May 23 '16 at 20:15
  • No, you have to use `getORMClaims()` *directly* in the `for` loop. Your code assigns the result to a variable. Your calling code (according to your latest edit) looks like my `reportData = foo.getORMClaims()` example, just you have `ocs` instead of `reportData` and `vr` instead of `foo`. – John Y May 23 '16 at 20:27