1

I am trying to make a webscraper in order to have a user search for characters, comics and other things related to their search from https://www.marvel.com/search

however, no matter what I do I can't figure out how to get the search function to work. I have two codes on my github, below is my most recent attempt that I found from stackoverflow.

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

user_input = input("character: ")
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome("C:\Program Files (x86)\chromedriver.exe")
driver.get("https://www.marvel.com/search")

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.nav-icon.gcom-icon-search"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#searchString"))).send_keys(user_input + Keys.RETURN)

https://github.com/chrismc0723/WebScrapers/blob/MCU/MCU.py https://github.com/chrismc0723/WebScrapers/blob/MCU/MCU2.py

Christopher McClure
  • 49
  • 6

1 Answers1

0

Try this:

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

user_input = input("character: ")
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome("D:\Downloads\chromedriver\chromedriver.exe")
driver.get("https://www.marvel.com/search")

search_input_xpath = "//input[@placeholder='Search']"
search_input = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, search_input_xpath)))
search_input.send_keys(user_input)

first_item_in_auto_suggest_area_xpath = "//div[contains(@id,'react-autowhatever')]/ul"
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, first_item_in_auto_suggest_area_xpath)))
search_input.send_keys(Keys.ENTER)

Notes:

  1. Pay attention that in the code you never use options variable.
  2. Running the code you may see warning DeprecationWarning: executable_path has been deprecated, please pass in a Service object. To fix it - follow suggestions from this answer.
Andrey Kotov
  • 1,344
  • 2
  • 14
  • 26
  • 1
    Tried your code and it worked. I did see a solution that had XPATH stuff but I wasn't fully sure on it because the websites I seen in other solutions were structured differently, or so it seemed. but that worked. – Christopher McClure Jun 25 '22 at 21:30
  • 1
    Also, yeah it showed the DeprecationWarning error, ill look at the suggestion you linked and ill post here if I make some progress with the code. thank you so much – Christopher McClure Jun 25 '22 at 21:31
  • I am following a techwithtim video. I am now trying to get it where it leads to getting the href link and name but on his example theres a h1 line which for me there isnt its mainly div lines. I have this code in this link, https://paste.pythondiscord.com/onumefocap and get this error: divs = section.find_elements_by_tag_name("div") AttributeError: 'WebElement' object has no attribute 'find_elements_by_tag_name' – Christopher McClure Jun 25 '22 at 22:33
  • Use `driver.find_elements` and search all elements from the root of the page. [Code is here.](https://pastebin.com/s2wY7hAR) – Andrey Kotov Jun 25 '22 at 23:35
  • the code you provided worked for getting the names to display. it's awesome how it all works. Going to try implement getting the href links to be printed along side them – Christopher McClure Jun 26 '22 at 01:52