Pytrends ResponseErrore code 429 for a search list of many words

Question

I am trying to get google search volume index using Pytrends. However, when I use a large set of words, I get the following error:

TooManyRequestsError: The request failed: Google returned a response with code 429

There are similar posts on this issue (for example here). However, none of the solutions in this step work in my case. I thought maybe there is a way to introduce a certain stop in time after several searches? (in total I have 12000 words). When I run the code with a smaller number of words in a list, the code works fine. Below is the code:

from pytrends.request import TrendReq
import pandas as pd 

pytrends = TrendReq()

terms_df = pd.read_csv('search_words.csv', sep=',', low_memory=False)
terms = list(terms_df['words'])


def scrape_google(term):
    pytrends.build_payload(term, cat=0, timeframe='all', geo='US', gprop='')
    trends = pytrends.interest_over_time()
    try:
        trends = trends.drop(columns=['isPartial'])
    except:
        pass

    return trends

def get_trends(terms):
    for i in range(0,len(terms)):
        if i == 0:
            trends = scrape_google(terms[i:i+1])
        else:
            trends = pd.concat([trends, scrape_google(terms[i:i+1])], axis=1)
    return trends

trends = get_trends(terms)

Alternative way:

from pytrends.request import TrendReq
import pandas as pd

pytrends = TrendReq(hl='en-US', tz=360, timeout=(10, 25))


pytrends.build_payload(terms, cat=0, timeframe='all', geo='US', gprop='')
trends = pytrends.interest_over_time()
trends = trends.drop(columns=['isPartial'])

Here I dont know how to make a loop that will go over a list (terms) and take just 5 words (maximum in pytrends) and that will concatenate pd DF into one.

You're doing 2 words per request (and you're also sending every word twice). Why aren't you doing fewer requests, but with more words in each request? — Tim Roberts, Nov 03 '22 at 20:56
Thank you for your reply. Since pytrends dont provide the actual search volume, just relative, I was comparing to all the words. However, I see your points and decided to make search within a word (in terms of relative volume). The problem is that I can insert a list of a max length = 5, can you please help me with the loop that will plug 5 words into pytrends.build_payload in each iteration and concatenate pd DF into one? I updated the question. — Alberto Alvarez, Nov 03 '22 at 22:05

score 0 · Accepted Answer · answered Nov 03 '22 at 22:07

Here's how you would do 5 at a time. The key is to run your loop in increments of 5, not increments of 1.

def get_trends(terms):
    trends = scrape_google(terms[0:5])
    for i in range(5,len(terms),5):
        trends = pd.concat([trends, scrape_google(terms[i:i+5])], axis=1)
    return trends

Pytrends ResponseErrore code 429 for a search list of many words

1 Answers1