Web-scraping slow but not sure why

Question

I have a lot of web scraping to do so I switched to a headless browser hoping that would make things faster, but it didn't improve the speed by much.

I looked at this stack overflow post but I don't understand the answer someone wrote Is Selenium slow, or is my code wrong?

here is my slow code:

# followed this tutorial https://medium.com/@stevennatera/web-scraping-with-selenium-and-chrome-canary-on-macos-fc2eff723f9e
from selenium import webdriver
options = webdriver.ChromeOptions()
options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'
options.add_argument('window-size=800x841')
options.add_argument('headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://poshmark.com/search?')
xpath='//input[@id="user-search-box"]'
searchBox=driver.find_element_by_xpath(xpath)

brand="anthropology"

style="headband"

searchBox.send_keys(' '.join([brand,style]))

from selenium.webdriver.common.keys import Keys
#EQUIValent of hitting enter key
searchBox.send_keys(Keys.ENTER)




url=driver.current_url
print(url)
import requests
response=requests.get(url)
print(response)


print(response.text)
# using beautiful soup to grab the listins:






#______________________________


#print(response)
html=response.content
from bs4 import BeautifulSoup
from urllib.parse import urljoin



#print(html)
soup=BeautifulSoup(html,'html.parser')

#'a' as in links or anchore tags
anchore_tags=soup.find_all('a')


#print(x)




# finding the hyper links
#href is the hyperlink
hyper_links=[link.get("href") for link in soup.find_all("a")]
#print(hyper_links)

                        #(Better visual link this )
                        #href is the hyperlink
                        # for link in soup.find_all("a"):
                        #
                        #     print(link.get("href"))

clothing_listings=set([listing for listing in hyper_links if listing and "listing" in listing]) #  if the element and the word listing is in the element (becuase there could be a hyperlink that is NONE whcich is why we need the and )
# turning the list into a set because some of them are repeated
print(len(clothing_listings))
print(set(clothing_listings))
print(len(set(clothing_listings)))

#for somereason a link that is called unlike is showing up so im geting rid of those
clothing_listings=set([listing for listing in hyper_links if listing and "unlike" in listing]) #  if the element and the word listing is in the element (becuase there could be a hyperlink that is NONE whcich is why we need the and )
print(len(clothing_listings))# this is the correct size of the amount of clothing items by that search





driver.quit()

Why is it taking so long to scrape things?

Answer 1

You're using requests to fetch the URL. So, why not use it to accomplish the entire task. The part where you use selenium seems redundant. You merely open the link using it, and then use requests to fetch the resulting URL. All you have to do is pass appropriate headers, which you can gather by viewing the network tab of developer tools in Chrome or Firefox.

rh = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9',
    'referer': 'https://poshmark.com/search?',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'
}

Modify the URL to search for a specific term:

query = 'anthropology headband'
url = 'https://poshmark.com/search?query={}&type=listings&department=Women'.format(query)

And then, use BeautifulSoup. Also, you can narrow down the links you scrape by using any attribute that's specific to the ones you want. In your case, it's the class attribute of covershot-con.

r = requests.get(url, headers = rh)
soup = BeautifulSoup(r.content, 'lxml')

links = soup.find_all('a', {'class': 'covershot-con'})

Here's the result:

for i in links:
    print(i['href'])

/listing/Anthro-Beaded-Headband-5a78fb899a9455e90aef438e
/listing/NWT-ANTHROPOLOGIE-Twisted-Vines-Crystal-Headband-5abbfb4a07003ad2dc58142f
/listing/Anthropologie-Nicole-Co-White-Floral-Headband-59dea5adeaf0302a5600bc41
/listing/NWT-ANTHROPOLOGIE-Namrata-Spring-Blossom-Headband-5ab5509d72769b52ba31829e
.
.
.
/listing/Anthropologie-By-Lilla-Spiky-Blue-Headband-59064f2ffbf6f90bfb01b854
/listing/Anthropologie-Beaded-Headband-5ab2cfe79d20f01a73ab0ddb
/listing/Anthropologie-Floral-Hawaiian-Headband-59d09eb941b4e0e1710871ec

Edit (Tips):

1. Use selenium as a last resort (when all other methods fail). Like @Gilles Quenot says, selenium is not for fast execution of web requests.

2. Learn how to work with the requests library (using headers, passing data, etc.). Their documentation page is more than enough to get started. It'll suffice for most scraping tasks, and it's fast.

3. Even for pages that require JS execution, you can get by with requests if you can figure out how to execute JS part using a library like js2py.