Can't find extisting html element when using Beautifulsoup and requests

Question

I'm facing a problem when I want to scrape https://www.encyclopedia.com/gsearch?q=World+War+2 I want to find the a element with a class of gs-title ((The first clickable link in the result rows that aren't a adsense)) This is my code:

import requests
from bs4 import BeautifulSoup

def scrape_encyclopedia(product_name):
    ### SETUP ###
    URL_raw = 'https://www.encyclopedia.com/gsearch?q=' + product_name
    URL = URL_raw.strip().replace(" ", "+")
    ## gets the html from the url
    try:
        page = requests.get(URL)
    except:
        print("Could not find URL..")

    ## a way to come around scrape blocking
    soup = BeautifulSoup(page.content, 'html.parser')

    print(soup.find("a", {"class": "gs-title"})

scrape_encyclopedia('world war 2')

Output:

None

I think I know the problem but I can't find a sutable solution. The problem that i'm facing is that the site loads without the results, then my scraper scrapes the website and misses the result information. Then something loads the results. Do anyone have any ideas of a solution? Thx!

Please share the relevant part of the HTML source code. Did you check that the content is not dynamically generated? — AMC, Mar 14 '20 at 01:03

score 0 · Answer 1 · answered Mar 14 '20 at 00:07

The webpage uses a internal API starting with https://cse.google.com/cse/element/v1?rsz=filtered_cse&num=10&hl=en&... if you try accessing it directly without valid (time sensitive) tokens it gives ... "Unauthorized access to internal API...

I can scrape it with Selenium. First install Selenium:

pip3 install selenium

Then get a driver https://sites.google.com/a/chromium.org/chromedriver/downloads

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.common.exceptions import ElementClickInterceptedException

driver = webdriver.Chrome() # on Windows give a path to the driver


def scrape_encyclopedia(product_name):
    url_raw = 'https://www.encyclopedia.com/gsearch?q=' + product_name
    url = url_raw.strip().replace(" ", "+")
    driver.get(url)
    driver.implicitly_wait(10)
    # if you want to accept cookies not really necessary
    try:
        driver.find_element_by_xpath("//a[text() = 'Accept All']").click()
    except ElementClickInterceptedException:
        pass

    soup = BeautifulSoup(driver.page_source, 'html.parser')
    for a in soup.find_all("a", {"class": "gs-title"}):
        print(a)


search_string = 'world war 2'
print(search_string)
scrape_encyclopedia(search_string)

search_string = 'world war 1'
print(search_string)
scrape_encyclopedia(search_string)

driver.quit()

You can easily adapt it to click through the pagination if required.

Thanks alot with your idea but there's a problem with using Selenium that I didn't mention and I'm sorry for that. I'm working on a website where I need to scrape other websites (in this case Encyclopedia.com). This is why selenium doesn't work tho selenium is a client based library and can't be used as a server based operator. Thanks again for your answer but unfortunaty this isn't gonna work. — Pilhamre, Mar 14 '20 at 12:22

αԋɱҽԃ αмєяιcαη · Answer 2 · 2020-03-14T14:42:01.997

Basically the website is loads via JavaScript where it's rendering the content dynamically.

You can use Selenium approach:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options


options = Options()
options.add_argument('--headless')
driver = webdriver.Firefox(options=options)

driver.get("https://www.encyclopedia.com/gsearch?q=World+War+2")

anch = driver.find_element_by_css_selector(
    "a.gs-title").get_attribute("href")

print(anch)

driver.quit()

Output:

https://www.encyclopedia.com/history/modern-europe/wars-and-battles/world-war-ii

Also you can call the API directly which the data fetched from.

import requests
import json

r = requests.get("https://cse.google.com/cse/element/v1?rsz=filtered_cse&num=10&hl=en&source=gcsc&gss=.com&cselibv=8b2252448421acb3&cx=partner-pub-8594935838850960:3418779728&q=World%20War%202&safe=off&cse_tok=AJvRUv13ayi48SPrxK5VHwyxIZ3T:1584145209532&exp=csqr,cc&callback=google.search.cse.api4230")


start = r.text.find('{')
end = r.text.rfind('}') + 1

data = json.loads(r.text[start:end])

print(json.dumps(data, indent=4))

Output:

{
    "cursor": {
        "currentPageIndex": 0,
        "estimatedResultCount": "65700",
        "moreResultsUrl": "http://www.google.com/cse?oe=utf8&ie=utf8&source=uds&q=World+War+2&safe=off&cx=partner-pub-8594935838850960:3418779728&start=0",
        "resultCount": "65,700",
        "searchResultTime": "0.26",
        "pages": [
            {
                "label": 1,
                "start": "0"
            },
            {
                "label": 2,
                "start": "10"
            },
            {
                "label": 3,
                "start": "20"
            },
            {
                "label": 4,
                "start": "30"
            },
            {
                "label": 5,
                "start": "40"
            },
            {
                "label": 6,
                "start": "50"
            },
            {
                "label": 7,
                "start": "60"
            },
            {
                "label": 8,
                "start": "70"
            },
            {
                "label": 9,
                "start": "80"
            },
            {
                "label": 10,
                "start": "90"
            }
        ]
    },
    "context": {
        "title": "Encyclopedia New",
        "total_results": "405"
    },
    "results": [
        {
            "cacheUrl": "http://www.google.com/search?q=cache:xwjCrpEtBmYJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/history/modern-europe/wars-and-battles/world-war-ii&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjAAegQIABAB&usg=AOvVaw3nnixoLPrvEniK8qfc-iYZ",
            "content": "<b>World War II</b> [1] \u00b7 JUDSON KNIGHT The Second <b>World War</b> [<b>2</b>] was history&#39;s \nlargest and most significant armed conflict.",
            "contentNoFormatting": "World War II [1] \u00b7 JUDSON KNIGHT The Second World War [2] was history's \nlargest and most significant armed conflict.",
            "title": "<b>World War Ii</b> | Encyclopedia.com",
            "titleNoFormatting": "World War Ii | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/history/modern-europe/<b>war</b>s.../<b>world</b>-<b>war</b>-<b>ii</b>",
            "unescapedUrl": "https://www.encyclopedia.com/history/modern-europe/wars-and-battles/world-war-ii",
            "url": "https://www.encyclopedia.com/history/modern-europe/wars-and-battles/world-war-ii",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "cseImage": {
                    "src": "https://www.encyclopedia.com/sites/default/files/styles/slide_thumb/public/5/2799765.jpg?itok=W8cmnWvZ"
                },
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "Encyclopedia.com \u2013 Online dictionary and encyclopedia with pictures, facts, and videos. Get information and homework help with millions of articles in our FREE, online library.",
                    "title": "World War Ii | Encyclopedia.com"
                },
                "cseThumbnail": {
                    "src": "https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQIu86aFr2mtKeCRuvCRoRT37033dO-G3zzQGiUh8F9i6K3OtAwQnltW3k",
                    "width": "225",
                    "height": "163"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:Wnck_kBuXhcJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/education/news-and-education-magazines/world-war-ii-mobilization-1939-1943&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjABegQIARAB&usg=AOvVaw2aXxTnEOlymsJi5X9CFLT6",
            "content": "Germany&#39;s invasion of Poland in September 1939 quickly led Britain and France \nto declare war on Germany. <b>World War II</b> had officially begun. Both countries had\n&nbsp;...",
            "contentNoFormatting": "Germany's invasion of Poland in September 1939 quickly led Britain and France \nto declare war on Germany. World War II had officially begun. Both countries had\n\u00a0...",
            "title": "<b>World War II</b> Mobilization 1939-1943 | Encyclopedia.com",
            "titleNoFormatting": "World War II Mobilization 1939-1943 | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/.../<b>world</b>-<b>war</b>-<b>ii</b>-mobilization-1939-1943",
            "unescapedUrl": "https://www.encyclopedia.com/education/news-and-education-magazines/world-war-ii-mobilization-1939-1943",
            "url": "https://www.encyclopedia.com/education/news-and-education-magazines/world-war-ii-mobilization-1939-1943",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "World War II Mobilization 1939-1943IntroductionIssue SummaryContributing ForcesPerspectivesImpactNotable PeoplePrimary SourcesSuggested Research TopicsBibliography",
                    "title": "World War II Mobilization 1939-1943 | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:uEm2mxyKCL4J:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/journalism-world-war-ii&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjACegQIAhAB&usg=AOvVaw3yydvNI1_gYT_0MuMEwhKe",
            "content": "JOURNALISM, WORLD WAR IIBy all measures, war reporting came of age \nbetween 1939 and 1945. The global conflagration of <b>World War II</b> elicited a \nmassive&nbsp;...",
            "contentNoFormatting": "JOURNALISM, WORLD WAR IIBy all measures, war reporting came of age \nbetween 1939 and 1945. The global conflagration of World War II elicited a \nmassive\u00a0...",
            "title": "Journalism, <b>World War II</b> | Encyclopedia.com",
            "titleNoFormatting": "Journalism, World War II | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/defense/energy.../journalism-<b>world</b>-<b>war</b>-<b>ii</b>",
            "unescapedUrl": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/journalism-world-war-ii",
            "url": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/journalism-world-war-ii",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "JOURNALISM, WORLD WAR IIBy all measures, war reporting came of age between 1939 and 1945. The global conflagration of World War II elicited a massive response from the free press around the world. In countries under repressive regimes, print journalism and radio continued as well, but their effectiveness was compromised by rigid central control and the dictates of party propaganda.",
                    "title": "Journalism, World War II | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:biM0zkSF1CQJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/civil-liberties-world-war-ii&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjADegQIAxAB&usg=AOvVaw0ReRrUBKDaPVZ7KglD9Vr-",
            "content": "CIVIL LIBERTIES, <b>WORLD WAR</b> IIThe federal government, in response to \nperiods of insecurity and conflict, sometimes restricts civil liberties in an effort to&nbsp;...",
            "contentNoFormatting": "CIVIL LIBERTIES, WORLD WAR IIThe federal government, in response to \nperiods of insecurity and conflict, sometimes restricts civil liberties in an effort to\u00a0...",
            "title": "Civil Liberties, <b>World War II</b> | Encyclopedia.com",
            "titleNoFormatting": "Civil Liberties, World War II | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/defense/energy.../civil-liberties-<b>world</b>-<b>war</b>-<b>ii</b>",
            "unescapedUrl": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/civil-liberties-world-war-ii",
            "url": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/civil-liberties-world-war-ii",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "CIVIL LIBERTIES, WORLD WAR IIThe federal government, in response to periods of insecurity and conflict, sometimes restricts civil liberties in an effort to maintain national security. U.S.",
                    "title": "Civil Liberties, World War II | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:MLIsVfbEvJwJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/financing-world-war-ii&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjAEegQIBBAB&usg=AOvVaw0ixq969zNEgseSoulGcugO",
            "content": "FINANCING, <b>WORLD WAR</b> IIWorld War <b>II</b> was the most expensive war in \nAmerican history, exceeding all other conflicts in economic impact. Nearly forty \nmillion&nbsp;...",
            "contentNoFormatting": "FINANCING, WORLD WAR IIWorld War II was the most expensive war in \nAmerican history, exceeding all other conflicts in economic impact. Nearly forty \nmillion\u00a0...",
            "title": "Financing, <b>World War II</b> | Encyclopedia.com",
            "titleNoFormatting": "Financing, World War II | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/defense/energy.../financing-<b>world</b>-<b>war</b>-<b>ii</b>",
            "unescapedUrl": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/financing-world-war-ii",
            "url": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/financing-world-war-ii",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "FINANCING, WORLD WAR IIWorld War II was the most expensive war in American history, exceeding all other conflicts in economic impact. Nearly forty million Americans paid income taxes for the first time, and an elaborate price control system touched the life of every consumer. To sell war bonds, the U.S. government made direct and frequent contact with more than 90 percent of the American population. The U.S.",
                    "title": "Financing, World War II | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:p_dClMaK4SIJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/history/educational-magazines/world-war-ii-nurses&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjAFegQIBRAB&usg=AOvVaw3H_f5fyXR8CHoEJ_TRgUbm",
            "content": "During <b>World War II</b> (1939-45) nearly sixty thousand American nurses served in \nthe Army Nurse Corps (ANC). Whether stationed in Europe or in the Pacific, they\n&nbsp;...",
            "contentNoFormatting": "During World War II (1939-45) nearly sixty thousand American nurses served in \nthe Army Nurse Corps (ANC). Whether stationed in Europe or in the Pacific, they\n\u00a0...",
            "title": "<b>World War II</b> Nurses | Encyclopedia.com",
            "titleNoFormatting": "World War II Nurses | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/history/educational.../<b>world</b>-<b>war</b>-<b>ii</b>-nurses",
            "unescapedUrl": "https://www.encyclopedia.com/history/educational-magazines/world-war-ii-nurses",
            "url": "https://www.encyclopedia.com/history/educational-magazines/world-war-ii-nurses",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "World War II NursesExcerpt from No Time for Fear: Voices of American Military Nurses in World War II  Edited by Diane Burke Fessler Published in 1996",
                    "title": "World War II Nurses | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:xRbIUkA2dnQJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/labor-world-war-ii&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjAGegQIBhAB&usg=AOvVaw3guMbNwOQGzvUEdFxmI5iL",
            "content": "LABOR, <b>WORLD WAR</b> IIIn the eight years prior to the attack on Pearl Harbor, the \nAmerican labor movement had experienced the greatest growth and most&nbsp;...",
            "contentNoFormatting": "LABOR, WORLD WAR IIIn the eight years prior to the attack on Pearl Harbor, the \nAmerican labor movement had experienced the greatest growth and most\u00a0...",
            "title": "Labor, <b>World War II</b> | Encyclopedia.com",
            "titleNoFormatting": "Labor, World War II | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/defense/energy...and.../labor-<b>world</b>-<b>war</b>-<b>ii</b>",
            "unescapedUrl": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/labor-world-war-ii",
            "url": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/labor-world-war-ii",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "LABOR, WORLD WAR IIIn the eight years prior to the attack on Pearl Harbor, the American labor movement had experienced the greatest growth and most dramatic changes in its history. The number of workers in trade unions had increased from under three million at the beginning of 1933 to over ten million by the end of 1941.",
                    "title": "Labor, World War II | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:bAzs3H2i4gIJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/economics/encyclopedias-almanacs-transcripts-and-maps/world-war-ii-and-ending-depression&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjAHegQIBxAB&usg=AOvVaw35G5M_qQ-CgwmyoReaesyC",
            "content": "<b>WORLD WAR II</b> AND THE ENDING OF THE DEPRESSIONWorld War II had a \nprofound and multifaceted impact on the American economy. Most obviously, it&nbsp;...",
            "contentNoFormatting": "WORLD WAR II AND THE ENDING OF THE DEPRESSIONWorld War II had a \nprofound and multifaceted impact on the American economy. Most obviously, it\u00a0...",
            "title": "<b>World War II</b> and the Ending of the Depression | Encyclopedia.com",
            "titleNoFormatting": "World War II and the Ending of the Depression | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/.../<b>world</b>-<b>war</b>-<b>ii</b>-and-ending-depression",
            "unescapedUrl": "https://www.encyclopedia.com/economics/encyclopedias-almanacs-transcripts-and-maps/world-war-ii-and-ending-depression",
            "url": "https://www.encyclopedia.com/economics/encyclopedias-almanacs-transcripts-and-maps/world-war-ii-and-ending-depression",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "WORLD WAR II AND THE ENDING OF THE DEPRESSIONWorld War II had a profound and multifaceted impact on the American economy. Most obviously, it lifted the nation out of the Great Depression of the 1930s.",
                    "title": "World War II and the Ending of the Depression | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:393EXCjRNrAJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/religion-world-war-ii&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjAIegQICBAB&usg=AOvVaw39rjhe20cM5_B7ed78IfJs",
            "content": "RELIGION, <b>WORLD WAR</b> IIThe devastation of <b>World War</b> I loomed in the \nbackground for Americans when war began in Europe in September 1939 and \nwhen&nbsp;...",
            "contentNoFormatting": "RELIGION, WORLD WAR IIThe devastation of World War I loomed in the \nbackground for Americans when war began in Europe in September 1939 and \nwhen\u00a0...",
            "title": "Religion, <b>World War II</b> | Encyclopedia.com",
            "titleNoFormatting": "Religion, World War II | Encyclopedia.com",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/defense/energy.../religion-<b>world</b>-<b>war</b>-<b>ii</b>",
            "unescapedUrl": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/religion-world-war-ii",
            "url": "https://www.encyclopedia.com/defense/energy-government-and-defense-magazines/religion-world-war-ii",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "RELIGION, WORLD WAR IIThe devastation of World War I loomed in the background for Americans when war began in Europe in September 1939 and when the United States moved to assist Britain prior to entering the war in December 1941.",
                    "title": "Religion, World War II | Encyclopedia.com"
                }
            }
        },
        {
            "cacheUrl": "http://www.google.com/search?q=cache:xnBuVKnkgRQJ:www.encyclopedia.com",
            "clicktrackUrl": "https://www.google.com/url?client=internal-element-cse&cx=partner-pub-8594935838850960:3418779728&q=https://www.encyclopedia.com/politics/encyclopedias-almanacs-transcripts-and-maps/french-underground-during-world-war-ii-communication-and-codes&sa=U&ved=2ahUKEwiqspyT2pjoAhWtAGMBHRu8CJAQFjAJegQICRAB&usg=AOvVaw0RVv2DbFZm-BZcVoF6givP",
            "content": "French Underground during <b>World War II</b>, Communication and Codes \u00b7 \nADRIENNE WILMOTH LERNER Source for information on French Underground \nDuring&nbsp;...",
            "contentNoFormatting": "French Underground during World War II, Communication and Codes \u00b7 \nADRIENNE WILMOTH LERNER Source for information on French Underground \nDuring\u00a0...",
            "title": "French Underground During <b>World War II</b>, Communication and Codes",
            "titleNoFormatting": "French Underground During World War II, Communication and Codes",
            "formattedUrl": "https://<b>www</b>.encyclopedia.com/.../french-underground-during-<b>world</b>-<b>war</b>-<b>ii</b>- communication-and-codes",
            "unescapedUrl": "https://www.encyclopedia.com/politics/encyclopedias-almanacs-transcripts-and-maps/french-underground-during-world-war-ii-communication-and-codes",
            "url": "https://www.encyclopedia.com/politics/encyclopedias-almanacs-transcripts-and-maps/french-underground-during-world-war-ii-communication-and-codes",
            "visibleUrl": "www.encyclopedia.com",
            "richSnippet": {
                "metatags": {
                    "handheldfriendly": "true",
                    "viewport": "width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=0",
                    "mobileoptimized": "width",
                    "abstract": "French Underground during World War II, Communication and Codes\n\n\n\n\n\n\u2588 ADRIENNE WILMOTH LERNER",
                    "title": "French Underground During World War II, Communication and Codes | Encyclopedia.com"
                }
            }
        }
    ],
    "findMoreOnGoogle": {
        "url": "https://www.google.com/search?client=ms-google-coop&q=World+War+2&cx=partner-pub-8594935838850960:3418779728"
    }
}

Google CSE is custom search engine which is stable and no needs to care about it's requirement to call it.

Note: you can get track the API call via Developer-Tools within your browser and navigating to Network section. here you can view all requests done.

Can't find extisting html element when using Beautifulsoup and requests

2 Answers2