I read on internet similar problem, but none of the answer could help me. I have a function that for each line of data (data have around 2'000'000 rows) do something and then recall the same function with different parameter depending on what it has done. The issue is that after a while I get this error in the terminal: 'Fatal Python error: Cannot recover from stack overflow.'

It seams that the most frequent mistake causing this error is infinite loop, but I controlled and have no infinite loop. Hence, to me the issue it that 'sys.getrecursionlimit()' is set to 3000, which means that after 3000 call of the same function it will give me the error.

The first thing, is that I dont understand the difference between 'Fatal Python error: Cannot recover from stack overflow.' in the terminal, or a 'RecursionError: maximum recursion depth exceeded in comparison' in the jupyternotebook. Indeed, to me it can come from the same mistake (e.g. infinite loop).

When replacing my function by a simple one called 'test_', I have the following code:

import sys
print(sys.getrecursionlimit())
def test_(x,t):
    x = x+1
    if x<t:
        test_(x=x,t=t)
print(test_(0,2971)) # output: None
print(test_(0,2972)) # RecursionError: maximum recursion depth exceeded in comparison
  
--------------------------------------------------------------------------- RecursionError                            Traceback (most recent call
  last)  in ()
        9 print(test_(0,2971))
  ---> 10 print(test_(0,2972))
  
 in test_(x, t)
        5     x = x+1
        6     if x 7         test_(x=x,t=t)
        9 print(test_(0,2971))
  
... last 1 frames repeated, from the frame below ...
  
 in test_(x, t)
        5     x = x+1
        6     if x 7         test_(x=x,t=t)
        9 print(test_(0,2971))
  
RecursionError: maximum recursion depth exceeded in comparison
To overcome this issue I adapted the function without loosing the 'continuity of the run', so that I can use batches:
for i in np.arange(0,9000,2000):
    test_(i,i+2000)
Would someone have nicer solution? Also, in general its a bad idea to do recursive function when we know that we have a lot of iteration to do? Also does anyone knows how I could print the recursiondeepth at each loop?
I am working on a Linux virtual environement, with jupyter notebook, on python 3.6 with anaconda.
                Can you instead return from your function with the current state, and let the caller call it again? Then you won't exhaust the stack.
– Arndt Jonasson
                May 24 '18 at 11:51
                @miki, I tested your code with the batch approach and it worked for me. Maybe the recursion limit is different where you run the for loop?
– Elad
                May 24 '18 at 11:59
                Thank you for your answer Jonasson. I will try this, but are we sure that the error comes from calling in an 'embeded' manner to many times the fonction? Because, if this would be the case, then the 'batch correction' should already correct the error, isn't?
– miki
                May 24 '18 at 11:59
                Thank you @Elad! yes it also worked for me, but in the case of my true function (and not this test one) I am keep getting the error... so thats why I would like to print the recursiondepth at each iteration to understand it better.
– miki
                May 24 '18 at 12:02
Please check this question (it worked for me):
How do I get the current depth of the Python interpreter stack?
your code based on that answer:
import sys
import inspect
print(sys.getrecursionlimit())
def test_(x,t):
    print len(inspect.stack())
    x = x+1
    if x<t:
        test_(x=x,t=t)
print(test_(0,7))
output:
                Thank you very much @Elad for your help. Thanks to this, I advance in the resolution of my problem! Now I will use it to understand the main issue, keeping you inform if solved the main problem. Also, a small thing by curiosity, I dont understand why their is the error before this number can not reached 3000... would you have any idea?
– miki
                May 24 '18 at 12:40
                the same inspect module can help you understand also that, just "print inspect.stack()" you will see that python executes a few functions before it start executing your code.
– Elad
                May 24 '18 at 12:45
                this seems to explain why it starts at 21, but to me it does not explains why it can not reached 3000... perhaps I miss something? Also do you know why for the same error sometimes it print in the notebook: 'RecursionError: maximum recursion depth exceeded in comparison', and sometimes in the terminal 'Fatal Python error: Cannot recover from stack overflow.'? That would be of great help too. Thank you very much for all your answers!
– miki
                May 24 '18 at 12:51
                @miki, I believe it does reach to 3000 when the exception happens, in your tests x start from 0 instead of the current recursion length (which very much depends where you run your script) another issue is that when you capture the exception it already backtracked a few stacks so you won't see 3000 just a number that is close to it. about the second issue you described I can not help you.
– Elad
                May 24 '18 at 13:21
                if you add to the function 'test_' the following line: '    else: print(len(inspect.stack()))', and run 'test_(0,2962)' it will finish and print 2983, while if you run test_(0,2963), it will give the RecursionError. Hence, it is not finishing at 3000 nor at 2963 but a 2983 (i.e. nothing to do with the fact that I started at 0, I guess). But it might be due to your second explanation :). Thank you for all!
– miki
                May 24 '18 at 13:28
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
                        
Related
                            
4782Calling an external command from Python
1246Extracting extension from filename in Python
1804Why is reading lines from stdin much slower in C++ than Python?
1676How to remove a key from a Python dictionary?
1135fatal error: Python.h: No such file or directory
0Python - MergeSort Recursion Error
0Yield in recursion multiplayer
1Python Recursive Call
2Python: using recursion to see if one string is a rotation of another
site design / logo © 2020 Stack Exchange Inc; user contributions licensed under cc by-sa 4.0
                            with attribution required.                    rev 2020.4.3.36492