How to scrape text of specific cell from website using BeautifulSoup

Question

I've been trying to scrape text from a website for the past hour and have made no progress, simply because I have very little knowledge on how to actually use BSoup.

def select_ticker():
    url = "https://www.barchart.com/stocks/performance/gap/gap-up?screener=nasdaq"

    r = requests.get(url)
    html = r.text
    soup = BeautifulSoup(html)


    find = soup.findAll('td, {"data-ng-if:"row.blankRow"}')

    print(find)

I'm going to this website and trying to get the first symbol from the table. Right now that symbol is BFBG

I know this should be extremely easy for someone who actually knows what they're doing with BSoup but I don't understand searching for things and this website doesn't make it easy to search either.

I appreciate your time and thanks for the help!

Possible duplicate of [scrape html generated by javascript with python](https://stackoverflow.com/questions/2148493/scrape-html-generated-by-javascript-with-python) — bobrobbob, Jun 27 '18 at 18:36
it doesn't work because the page is generated in javascript. use selenium — bobrobbob, Jun 27 '18 at 18:38

score 1 · Accepted Answer · answered Jun 27 '18 at 19:48

Actually, you cannot scrap the first symbol from the html get request. You need to fetch the json.

import urllib3
import json
http = urllib3.PoolManager()
r = http.request('GET', 'https://core-api.barchart.com/v1/quotes/get?lists=stocks.gaps.up.nasdaq&orderDir=desc&fields=symbol,symbolName,lastPrice,priceChange,gapUp,highPrice,lowPrice,volume,tradeTime,symbolCode,symbolType,hasOptions&orderBy=gapUp&meta=field.shortName,field.type,field.description&hasOptions=true&page=1&limit=100&raw=1')
print(json.loads(r.data)['data'][0]['symbol'])

And there you got the first symbol.

With the Json you can also find every information you probably want to scrap.

Here is how you can usually find those Jsons :

Going into the console, network tab, xhr tab and reload the page. If there are a lot of ressources fetched, you can also filter by the name of the domain ! :)

However, this syntax is wrong: soup.findAll('td, {"data-ng-if:"row.blankRow"}')

you need to give a dictionnary to the find_all method according to BS4 doc https://www.crummy.com/software/BeautifulSoup/bs4/doc/#find-all

soup.find_all('td', {'data-ng-if':'row.blankRow'})

Hope this helps

How to scrape text of specific cell from website using BeautifulSoup

1 Answers1

Linked