0

I have the following python function to extract unique object names in an AWS S3 bucket, and it passed the test runs in python:

###############################
# aws_get_session_name_fuc.py #
###############################
def get_session_names(s3, BUCKET, session_path):
    session_names = []
    unique_session_names = []
    n = 2
    session_archives = s3.Bucket(BUCKET).objects.filter(Prefix = session_path)

    # Extract object names from path
    for i in session_archives:
        temp_str = i.key.split('/')
        session_names.append(temp_str[n])

    # obtain unique object names
    for i in session_names:
        if i not in unique_session_names:
            unique_session_names.append(i)
    return unique_session_names

'''
# Test
test_extract = get_session_names(s3, BUCKET, session_path)
test_extract
Out[19]: 
['',
 'testSession01',
 'testSession02',
 'testSession03']
'''

When I run the above script using reticulate::source_python("aws_get_session_name_fuc.py") in R, I would get the following error:

> existing_session = get_session_names(s3, BUCKET, session_path)
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  RecursionError: maximum recursion depth exceeded

I tried to increase the recursion limit in the python script using sys.setrecursionlimit(), it would either trigger the same error when the value is not "large enough", or crash the R session if the value was "too large":

  • Recursion limit value not "large enough": enter image description here

  • Recursion limit value "too large": enter image description here

So, I'm trying to understand:

  1. Why does the function pass in Python, but fail when referenced in R?
  2. Is there a better way to resolve this?

EDIT-1:

I managed to resolve this problem by converting the defined function into script, and then using reticulate::py_run_file("python/aws_get_session_name_script.py") directly to obtain the extraction, write results to file, and then load back into R.

Y Ming
  • 27
  • 7
  • May be this will help https://stackoverflow.com/questions/3323001/what-is-the-maximum-recursion-depth-in-python-and-how-to-increase-it – Niranjan Poudel Oct 27 '22 at 19:52
  • @niranjan-poudel Increasing the recursion limit doesn't work for my scenario, unfortunately. – Y Ming Oct 28 '22 at 09:13

0 Answers0