-1

I try to web scrape data from this site: https://price.metal.com/Rare-Earth (rare earth material prices) with Python/BeautifulSoup. The question is not about Python, but the HTML source code of the website. When I am using the "Inspect Element" function of Firefox I am getting the needed data/HTML tag I am looking for: 

<div class="td-border-item frame-text-over-flow" title="294,000 ~ 299,000"> 294,000 ~ 299,000

But the problem is that when I view the Page Source of https://price.metal.com/Rare-Earth I cannot find the above element with

So I am curious if there is another HTML file which contains the information that I am looking for?

mplungjan
  • 169,008
  • 28
  • 173
  • 236
Mia
  • 1
  • 1
  • Yes. There are AJAX calls to `https://price.metal.com/api/get_product_lists?t=1582288095974&second_level_name=rare+earth` which has `"newest_price":{"Average":296500,"Highs":299000,"Low":294000,` – mplungjan Feb 21 '20 at 12:29
  • The data has likely been loaded and inserted via JavaScript. What resources actually get loaded, should be easy enough to figure out using your browser dev tools, network panel. – CBroe Feb 21 '20 at 12:29

1 Answers1

0

This element does not contain the value because it was loaded by asynchronization mechanism (Ajax appending HTML value after that page loaded or binding data for example).

I suggest you combine it with Selenium Python parallelly, inspect the tag and get value by Selenium. You will get it.

https://selenium-python.readthedocs.io/

Duc Nguyen
  • 168
  • 2
  • 11