i'm trying to create a list in python of all links from a page that contain a certain string. For example I'd like all links that contain "New York Rangers @" from this page https://www.stubhub.com/new-york-rangers-tickets/performer/2764/ .
Thanks for all the help - sorry if this is a dumb question but couldn't find it anywhere.
The data is embedded within the page inside the <srcipt> tag. You can use this example to parse the data (with re and json modules):
import re
import json
import requests
headers = {'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0'}
url = 'https://www.stubhub.com/new-york-rangers-tickets/performer/2764/'
txt = requests.get(url, headers=headers).text
data = json.loads(re.search(r'window.__INITIAL_STATE__\s*=\s*(.*})<', txt)[1])
# print(json.dumps(data, indent=4)) # <-- uncomment to see all data (prices, dates, etc.)
for event in data['EVENT_SEO_LIST']['events']:
if 'PARKING PASSES ONLY' in event['name']:
continue
print('{:<45} {}'.format(event['name'], 'https://www.stubhub.com/' + event['webURI']))
Prints:
Detroit Red Wings at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-1-31-2020/event/104217508/
New York Rangers at Detroit Red Wings https://www.stubhub.com/detroit-red-wings-tickets-detroit-red-wings-detroit-little-caesars-arena-2-1-2020/event/104215245/
Dallas Stars at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-3-2020/event/104212773/
Toronto Maple Leafs at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-5-2020/event/104215469/
Buffalo Sabres at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-7-2020/event/104217518/
Los Angeles Kings at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-9-2020/event/104214839/
New York Rangers at Winnipeg Jets Tickets (Replica Hall of Fame Banner Giveaway) https://www.stubhub.com/winnipeg-jets-tickets-winnipeg-bell-mts-place-2-11-2020/event/104212882/
New York Rangers at Minnesota Wild https://www.stubhub.com/minnesota-wild-tickets-minnesota-wild-saint-paul-xcel-energy-center-2-13-2020/event/104216234/
New York Rangers at Columbus Blue Jackets https://www.stubhub.com/columbus-blue-jackets-tickets-columbus-blue-jackets-columbus-nationwide-arena-2-14-2020/event/104212942/
Boston Bruins at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-16-2020/event/104217520/
New York Rangers at Chicago Blackhawks https://www.stubhub.com/chicago-blackhawks-tickets-chicago-blackhawks-chicago-united-center-2-19-2020/event/104213910/
New York Rangers at Carolina Hurricanes https://www.stubhub.com/carolina-hurricanes-tickets-carolina-hurricanes-raleigh-pnc-arena-2-21-2020/event/104212812/
San Jose Sharks at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-22-2020/event/104217524/
New York Rangers at New York Islanders https://www.stubhub.com/new-york-islanders-tickets-new-york-islanders-uniondale-nycb-live-home-of-the-nassau-veterans-memorial-coliseum-2-25-2020/event/104354662/
New York Rangers at Montreal Canadiens https://www.stubhub.com/montreal-canadiens-tickets-montreal-bell-centre-2-27-2020/event/104215418/
New York Rangers at Philadelphia Flyers https://www.stubhub.com/philadelphia-flyers-tickets-philadelphia-flyers-philadelphia-wells-fargo-center-philadelphia-2-28-2020/event/104212712/
Philadelphia Flyers at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-1-2020/event/104215027/
St. Louis Blues at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-3-2020/event/104217528/
Washington Capitals at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-5-2020/event/104215030/
New Jersey Devils at New York Rangers https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-7-2020/event/104215474/
New York Rangers at Dallas Stars https://www.stubhub.com/dallas-stars-tickets-dallas-stars-dallas-american-airlines-center-3-10-2020/event/104214902/
Well, first you need to get the content of the webpage you want to search the links. I highly recommend using requests, a simple HTTP library for Python:
import requests
response = request.get(https://www.stubhub.com/new-york-rangers-tickets/performer/2764/)
This specific URL for some reason requires a User-Agent header, so you should send one on the request:
url = 'https://www.stubhub.com/new-york-rangers-tickets/performer/2764/'
user_agent = 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0'
response = requests.get(url, headers={'User-Agent':user_agent})
Then you can start analyzing the content of the page using beautifulsoup4. You can use the method find_all passing a compiled regular expression as the text parameter to find all a tags that contains a certain text:
from bs4 import BeautifulSoup
import re
soup = BeautifulSoup(response.content, "html.parser")
rangers_anchor_tags = soup.find_all("a", text=re.compile(r".*\bNew York Rangers at\b.*")
urls = [anchor["href"] for anchor in rangers_anchor_tags]
urls, then, would be a list of URLs that the respective inner text of the anchor tag contains the string in question.
Using Selenium you won't need beautifulsoup and to create a list of all links i.e. href attributes from the page https://www.stubhub.com/new-york-rangers-tickets/performer/2764/ that contains the text New York Rangers you need to to induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following Locator Strategy:
Using XPATH:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# configuring the driver for optimum results
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.stubhub.com/new-york-rangers-tickets/performer/2764/")
# just one line of code
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[./div[contains(., 'New York Rangers')]]")))])
Console Output:
['https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-1-31-2020/event/104217508/', 'https://www.stubhub.com/detroit-red-wings-tickets-detroit-red-wings-detroit-little-caesars-arena-2-1-2020/event/104215245/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-3-2020/event/104212773/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-5-2020/event/104215469/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-7-2020/event/104217518/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-9-2020/event/104214839/', 'https://www.stubhub.com/winnipeg-jets-tickets-winnipeg-bell-mts-place-2-11-2020/event/104212882/', 'https://www.stubhub.com/minnesota-wild-tickets-minnesota-wild-saint-paul-xcel-energy-center-2-13-2020/event/104216234/', 'https://www.stubhub.com/columbus-blue-jackets-tickets-columbus-blue-jackets-columbus-nationwide-arena-2-14-2020/event/104212942/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-16-2020/event/104217520/', 'https://www.stubhub.com/chicago-blackhawks-tickets-chicago-blackhawks-chicago-united-center-2-19-2020/event/104213910/', 'https://www.stubhub.com/carolina-hurricanes-tickets-carolina-hurricanes-raleigh-pnc-arena-2-21-2020/event/104212812/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-2-22-2020/event/104217524/', 'https://www.stubhub.com/new-york-islanders-tickets-new-york-islanders-uniondale-nycb-live-home-of-the-nassau-veterans-memorial-coliseum-2-25-2020/event/104354662/', 'https://www.stubhub.com/montreal-canadiens-tickets-montreal-bell-centre-2-27-2020/event/104215418/', 'https://www.stubhub.com/philadelphia-flyers-tickets-philadelphia-flyers-philadelphia-wells-fargo-center-philadelphia-2-28-2020/event/104212712/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-1-2020/event/104215027/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-3-2020/event/104217528/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-5-2020/event/104215030/', 'https://www.stubhub.com/new-york-rangers-tickets-new-york-rangers-new-york-madison-square-garden-3-7-2020/event/104215474/']