How to increase PSAW Max Retries

Question

I keep receiving the following error when trying to collect a large data from pushshift.io using PSAW.

Exception: Unable to connect to pushshift.io. Max retries exceeded.

How can I increase the "max retries" so that this won't happen?

my_reddit_submissions=api.search_submissions(before=int(end_epoch.timestamp()),
                                            post_hint='image',
                                            filter=['id','full_link','title', 'url', 'subreddit', 'author_fullname'],
                                            limit = frequency
                                            )


for submission_x in my_reddit_submissions:
           data_new=data_new.append(submission_x.d_, ignore_index=True)

BTW, my code works fine till a point...

score 1 · Accepted Answer · answered Jan 21 '22 at 17:14

You Should take a look at this question [Might help] : Max retries exceeded with URL in requests

This Exception Is Raised When The Server Actively refuses to communicate with you. This May happen if you request too many times to the server in a short period of time.

To OverCome This, you should wait for a few seconds before retrying

Here is an example :

import time
import requests

with requests.Session() as session:
    while 1: # Infinite Loop used to send infinite requests to the server without waiting time
        try:
            response = session.get("https://www.example.com")           
        except requests.exceptions.ConnectionError:
            response.status_code = "Connection Refused By The Server"
            time.sleep(2) # Sleeping For 2 seconds to resolve the server overload error
        print(response.status_code)

How to increase PSAW Max Retries

1 Answers1