2

I wrote a recursive python function. After 30min - 1 hour it cancel the process because it ran out of memory. If I look in the task manager I also see that too much ram memory is used. If I restart the function everything is fine again. How can I avoid this problem and let the program "run forever"?

Flex Texmex
  • 1,074
  • 2
  • 11
  • 23
  • 1
    Programs won't run forever until they have the resources to. It sounds like a bug in your implementation rather than a limitation of Python. Attaching the source can help. – Carlos H Romano Aug 19 '15 at 01:03
  • 1
    Can you post the recursive function or an approximation of it? And the input used? You may be experiencing a Stack Overflow. – Michael R Aug 19 '15 at 01:04
  • 1
    Write an algorithm that doesn't run out of memory .. remember that Python does not support TCO, so recursion is only good for a 'bound depth'. If this is the issue, write it in whole (or in parts) without recursion, possibly using trampolines, etc. – user2864740 Aug 19 '15 at 01:04
  • Python has a default recursion limit of 1000; how long is each call taking if it runs for 30-60 minutes, and isn't even hitting the recursion limit? Please include your code. – Cyphase Aug 19 '15 at 01:14
  • If you are hitting a recusion limit then its necessary to avoid that by recoding possibly by eliminating recursion altogether such as with a while loop as suggested by John La Rooy on http://stackoverflow.com/questions/13591970/does-python-optimize-tail-recursion. –  Aug 19 '15 at 01:28

1 Answers1

0

As a rule of thumb, you should not use recursive calls in long-running programs. Of course there are exceptions...

Recursive functions are typically used where the application indicates managing something which calls for that functionality: Eg. traversing a tree structure, or looking up something in a database until a certain level of detail is reached.

Long-running programs rarely are of that class... Even scraping pages with a robot always have a point of reference (when the 'scrape' of one site is finished), where the recursion has to start from scratch. It is sometimes difficult to diagnose the problem (I found, in the logs of my website, that one of the better-known robots got stuck on an infinite loop on a bad link. Beware of infinite loops!).

Enlarging Python's stack is rarely the solution... Add lines to your code to check that the recursion depth returns to 0 at known intervals in your application. Or add a depth counter...

jcoppens
  • 5,306
  • 6
  • 27
  • 47