0

also after that how to handle "load more" in end to extract all links. as there are total 317 links in the page.

from bs4 import BeautifulSoup
import requests
import time

r = requests.get('https://www.tradeindia.com/manufacturers/a3-paper.html')
time.sleep(5)
soup = BeautifulSoup(r.text,'lxml')

for div in soup.find_all('div',class_='company-name'):
    links = div.find('a')
    print(links['href'])

please someone help me to find the best way to extract all 317 links in the page.

  • Your understanding of "all" is different from the webserver's understanding of "all". Do you have JavaScript enabled in the selenium driver? The items > 20 are fetched lazy via ajax. If you don't scroll down it won't work either. – Daniel W. Jun 19 '20 at 14:11
  • how to do with selenium driver –  Jun 19 '20 at 16:45
  • Use the search. Enabling JavaScript in Selenium: https://stackoverflow.com/questions/55480924/how-to-enable-javascript-in-selenium-webdriver-chrome-using-python – Daniel W. Jun 19 '20 at 17:51
  • Scrolling down a page using Selenium: https://stackoverflow.com/questions/12293158/page-scroll-up-or-down-in-selenium-webdriver-selenium-2-using-java – Daniel W. Jun 19 '20 at 17:52

0 Answers0