3

The problem

I have the following Page01.htm

<!DOCTYPE html><html lang="it-IT"><head>    <meta charset="utf-8">    <meta http-equiv="X-UA-Compatible" content="IE=Edge">    <head><title>Title here</title></head>
<body>
<script id="TargetID" type="application/json"><![CDATA[
{ "name":"Kate", "age":22, "city":"Boston"}
]]>
</script><script id=“AnotherID” type="application/json"><![CDATA[{ "name":"John", "age":31, "city":"New York"}]]>
</script>
</body></html>

and I want to extract the informations inside the JSON between the the script tags with ID=TargetID.

What I've done

I wrote the following Python 3.6 code:

from bs4 import BeautifulSoup
import codecs

page_path="/Users/me/Page01.htm"

page = codecs.open(page_path, "r", "utf-8")

soup = BeautifulSoup(page.read(), "lxml")
vegas = soup.find_all(id="TargetID")

invalid_tags = ['script']
soup = BeautifulSoup(str(vegas),"lxml")
for tag in invalid_tags: 
    for match in soup.findAll(tag):
        match.replaceWithChildren()

JsonZ = str(soup)

Now, if I look inside vegas variable I can see 

[<script id="TargetID" type="application/json"><![CDATA[ {
> "name":"Kate", "age":22, "city":"Boston"} ]]> </script>]

but if I try to remove the script tags (using this answer script), I get the following JsonZ variable

'<html><body><p>[&lt;![CDATA[\n{ "name":"Kate", "age":22, "city":"Boston"}\n]]&gt;\n]</p></body></html>'

that have no script tags but have another 3 tags (<html><body><p>) completely unuseful. My target is to get the following string { "name":"Kate", "age":22, "city":"Boston"} to load with Python JSON modules.

Nicolaesse
  • 2,554
  • 12
  • 46
  • 71

2 Answers2

4

BeautifulSoup will take practically anything give it and attempt to transform that into a complete page of HTML. That's why you received '<html><body> ...'. Usually this is a good thing in that the HTML can be pretty badly formed yet BeautifulSoup will still process it.

In your case, one way of extracting that json would be like this.

>>> import bs4
>>> page = bs4.BeautifulSoup(open('Page01.htm').read(), 'lxml')
>>> first_script = page.select('#TargetID')[0].text
>>> first_script 
'<![CDATA[\n{ "name":"Kate", "age":22, "city":"Boston"}\n]]>\n'
>>> content = first_script[first_script.find('{'): 1+first_script.rfind('}')]
>>> content
'{ "name":"Kate", "age":22, "city":"Boston"}'

Once you have this you can turn it into a Python dictionary, like this.

>>> import json
>>> d = json.loads(content)
>>> d['name']
'Kate'
>>> d['age']
22
>>> d['city']
'Boston'
Bill Bell
  • 21,021
  • 5
  • 43
  • 58
2

It can be made much simpler:

from bs4 import BeautifulSoup
soup = BeautifulSoup(open("/Users/me/Page01.htm", encoding='utf-8'), "html.parser")
result = soup.find('script', type='application/json', id='TargetID').text
# Workaround to get CDATA content (It seems that it can't be done with bs):
result = result.replace("<![CDATA[", "").replace("]]>", "").strip()
Francisco Jiménez Cabrera
  • 318
  • 2
  • 8
  • Thanks, it works very well and it is a very good code! I mark the @Bill Bell answer as "acceprance" only because there is an explanation on why BeautifulSoup add the three tags listed above. Thank you again. – Nicolaesse Jan 29 '18 at 23:21