6

I have some piece of python code which generates a MemoryError after a while. I know that it consumes a lot of memory. So, I decided to put the code within a try/except block so that the skeleton looks like the following:

while True:

      while True:

            try:
            #---- do some stuff

            except MemoryError as err:
                   print(err)
                   break

So, my idea is to break out of the first while-loop if a MemoryError occurs and since I have an outer while-loop, it will start the program again.

It seems that it works for the moment but I am not sure. After a while, it stops again and I need to restart the program again. Does somebody know a better solution so that the program can run after the MemoryError again?

MSeifert
  • 145,886
  • 38
  • 333
  • 352
aminakoy
  • 413
  • 1
  • 5
  • 13
  • Just to be clear, you ask what to do when `MemoryError` occurs? – MaLiN2223 Feb 05 '17 at 21:36
  • yes. Or better: How to continue the try-block after MemoryError occurs? – aminakoy Feb 05 '17 at 21:37
  • Possibly related: [How can I explicitly free memory in Python?](http://stackoverflow.com/a/1316799/190597) – unutbu Feb 05 '17 at 21:45
  • What kinds of data structures are you using? Just standard lists and dictionaries, or are you using some third party package like `numpy` or `pandas`. Memory error when trying to create and use very large `numpy` arrays comes up periodically on SO. – hpaulj Feb 05 '17 at 21:53
  • In another post you say you are learning Python. A beginner shouldn't be getting memory errors unless they are doing something very wrong. Don't try to `except` out of the error. Figure out what is using memory uncontrollably. – hpaulj Feb 05 '17 at 21:56
  • @hpaulj We cannot tell for sure but OP might be raising this exception by himself. – MaLiN2223 Feb 05 '17 at 21:57

3 Answers3

6

Note that Python only throws the MemoryError when it realizes it will overuse the memory beforehand. If it happens by accident (or "unnoticed" by Python) then you're out of luck. The documentation already mentions this:

MemoryError

Raised when an operation runs out of memory but the situation may still be rescued (by deleting some objects). The associated value is a string indicating what kind of (internal) operation ran out of memory. Note that because of the underlying memory management architecture (C’s malloc() function), the interpreter may not always be able to completely recover from this situation; it nevertheless raises an exception so that a stack traceback can be printed, in case a run-away program was the cause.

So if there is nothing to rescue or if the interpreter can't recover there is no MemoryError.

A good approach would require knowing what you're doing and how. In a majority of cases generators (see for example PEP 289 on generator expressions) or map-reduce approaches can save you a lot of memory. These might be applicable here as well.

Community
  • 1
  • 1
MSeifert
  • 145,886
  • 38
  • 333
  • 352
  • the code in the try-block, opens a file, writes some stuff into it & delete it. And this happens in a endless while-loop. – aminakoy Feb 05 '17 at 21:46
  • and? How does that create a MemoryError? What stuff? Why delete the files? Do you close file handles? – MSeifert Feb 05 '17 at 21:51
  • That only answers parts of the questions. Maybe you could [edit] your question to show a bit more of your actual code? – MSeifert Feb 05 '17 at 23:03
  • Thanks for this answer. I had noticed that python is very inconsistent in throwing MemoryError. Sometimes it does throw and error and many other times it does not. – Fardin Abdi Jan 29 '19 at 01:12
2

It is hard to assess what to do without knowing what do you do inside this try but I will try.

Frstrly, regarding continuing the try-except block. I am afraid you cannot do this.

So short answer is: you cannot go back to try block to place where exception occured, you can go to first line of try

What you can do:

I usually handle my exceptions like the following. Create while True loop as such:

while True:
    try:
        #some code here
    except SomeException:
        continue

Now you can continue to try after exception occured.

Second way (but not reccomended) is to embedd your code using some function and recursively execute it.

def foo():
    try:
        while True:
            foo2()
    except StopIteration:
        #end code

def foo2():
    while True:
        try:
            #some code here
        except MemoryError:
            foo2()
            raise StopIteration()

However this is very DANGEROUS however if your memory is being exceeded not many times (<1000) this can be okay way to go if you need to do something before while True however you need to watch solution closely.

MaLiN2223
  • 1,290
  • 3
  • 19
  • 40
  • Thanks, for your answer. In the try-block, the program opens a file, write some data to it and deletes it after a while. – aminakoy Feb 05 '17 at 21:48
  • @aminakoy I do not see any other solution to your problem. How about posting your code where you do file operations in another question and maybe we can help you create program so it will not raise MemoryException? – MaLiN2223 Feb 05 '17 at 21:52
0

Your asking a very ambiguous question, because you're not plotting the scenario that fills the memory.

What do you do when memory fills? You empty it! The good practice is that you have to be aware of how much memory your script consumes.

  • If you can make your program break down to more steps and consume less memory, then do that! If you

  • If you can empty some lists/arrays and continue execution, then do it!

  • If you believe your program will consume so much memory, then if a MemoryError occures, and there's no programmatic way around it, then the system of the user of your program does not meet the requirements of using your program, and crashing with a memory error is OK!

The Quantum Physicist
  • 24,987
  • 19
  • 103
  • 189