1

I need help with the scrape so I included the page source I am trying to scrape.

# Enter the search parameter
search_bar = driver.find_element_by_name('txtQuickSearch')
search_param = input('Insert Search Parameter Here:') # Replace with your desired search parameter
search_bar.send_keys(search_param)
search_bar.send_keys(Keys.RETURN)

sleep(30)

# Get the URL of the search results page
search_results_url = driver.current_url

# Use try/except to handle errors
try:
# Create a BeautifulSoup object from the search results page HTML
soup = BeautifulSoup(driver.page_source, 'html.parser')

    # Find the main data table by class name
    table = soup.find('table', {'class': 'testBlack'})
    
    # Use if/else to check if the table was found
    if table is not None:
        # Create a list to store the table data
        table_data = []
        
        # Loop over each table row and cell, and add the cell text to the table data list
        for row in table.find_all('tr'):
            row_data = []
            for cell in row.find_all('td'):
                row_data.append(cell.text)
            table_data.append(row_data)
    
        # Print the table data
        print(table_data)
    else:
        print('Error: Could not find table')

I need help with the scrape portion: Here is a snippet of the page source code:

html style="height: 100%">
  <head id="Head1" x-frame-options="sameorigin">
  <title> eLEAD </title>
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <script type="text/javascript" src="bam.nr-data.net/1/NRJS-f91177a87c587e19a33?
Pawel Kam
  • 1,684
  • 3
  • 14
  • 30
Victoria
  • 21
  • 3
  • 1
    I need help with the scrape portion: Here is a snippet of the page source code: html style="height: 100%"> eLEAD – Victoria Mar 09 '23 at 20:33

2 Answers2

1

Find the element to hover over

hover_element = driver.find_element_by_css_selector('div.fixed4-inner-container:nth-child(1)') # replace with your actual selector

Find the element to click on after hovering

click_element = driver.find_element_by_css_selector('div.fixed4-inner-container:nth-child(1)') # replace with your actual selector

Create an ActionChains object and move the cursor to the hover element

action = ActionChains(driver)
action.move_to_element(hover_element)

Perform a click on the click element after the hover

action.click(click_element)

Execute the action sequence

action.perform()

action = ActionChains(driver)
action.key_down(Keys.CONTROL)
action.send_keys('c')
action.key_up(Keys.CONTROL)
action.perform()


df = pd.read_clipboard()
df.reset_index(inplace=True)

df.to_csv('filename.csv', index=False)
driver.quit()

This is how I did it for a similar table but for google sheets, essentially clicking on the select all function, copying and reading the paste into a dataframe.

Victoria
  • 21
  • 3
0

It looks like the content of the page is generated using javascript. BeautifulSoup is not able to run the javascript before scraping the page. There is a really good explanation here if you like.

Robert Jamison
  • 104
  • 6