0

i want to scrape company name links, but dont know how to solve "load more" button to scroll down the page.

    from bs4 import BeautifulSoup
    import requests
    import time

    r = requests.get('https://www.tradeindia.com/manufacturers/a3-paper.html')
    time.sleep(5)
    soup = BeautifulSoup(r.text,'lxml')

    for div in soup.find_all('div',class_='company-name'):
        links = div.find('a')
        print(links['href'])

please help someone to solve this issue.

Nitesh Rao
  • 1
  • 2
    Does this answer your question? [crawl site that has infinite scrolling using python](https://stackoverflow.com/questions/22702277/crawl-site-that-has-infinite-scrolling-using-python) – Lucan Jun 25 '20 at 11:20
  • @Lucan No. i see that in network XHR tab, link is changing by clicking on load more. but that also not working. – Nitesh Rao Jun 25 '20 at 17:08
  • Hi there dear Lucan - thank you for this advice!!! – zero Jun 26 '20 at 11:10

0 Answers0