1

Hi I want to scrape https://www.tiktok.com/trending?lang=en and I have to wait to load videos on it. Then I want to scrape video data.

import requests
from bs4 import BeautifulSoup as bs


page = requests.get('https://www.tiktok.com/trending?lang=en', timeout=5)
soup = bs(page.text, 'html.parser')
video_list = soup.find_all('a')
print(video_list)

The main problem is tiktok takes some time to load video list of that page. So you can check the url first and help me. I checked with selenium with phantomjs but not working. Thanks for your help.

Martin Gergov
  • 1,556
  • 4
  • 20
  • 29
liming
  • 232
  • 2
  • 4
  • 18
  • the "time" it takes is from Javascript running in the background and grabbing the data. This helps the webpage load gradually and continuously while you browse it. Translating that to ```requests``` may be a bit difficult but it can be done with ```Selenium``` using either a [custom wait](https://selenium-python.readthedocs.io/waits.html) for the elements to appear or just a static wait like ```sleep(5)```. – Xosrov Dec 24 '19 at 19:48

0 Answers0