Python find JSON in HTML and get select values

Question

Gah. I've spent a fair amount of time trying to find how to do this both properly and even hackishly, and I am just stumped. I have 2500+ HTML files that I've downloaded from a site, and I need only to extract a limited amount of information from any given page: the title of the talk described by the page (so I can collate this data with a giant CSV we already have), and then the event at which a given talk was given, and the date on which the talk was published.

The HTML for these pages is sprawling and filled with <script> elements. I want only the one that is followed by a q. The line that starts this block looks like this:

<script>q("talkPage.init", {

What follows is quite a bit of data. I need only the three items that look like this:

"event":"TEDGlobal 2005",
"filmed":1120694400,
"published":1158019860,

Luckily, "filmed" and "published" only occur once in this large block, but "event" occurs several times. It's always the same, so I don't care which of these any script grabs.

My thought was to use BeautifulSoup to find the <script>q element and then pass that onto the json module to parse, but I cannot figure out hot to tell soup to grab the <script> element followed by a q -- classes and ids are easy. Followed by ... not so much.

To begin working on the JSON portion, I've created a text file which has only the contents of the <script>q element in it, but I confess that getting the json module to load this is not working terribly well.

The code I have for the experiment first loads the text file with the JSON block I'm interested in, and then tries to decode it so I can do other things with it:

import json

text = open('dawkins_script_element.txt', 'r').read()
data = json.loads(text)

But clearly the JSON decoder doesn't like what I have, because it throws a ValueError: Expecting value: line 1 column 1 (char 0). Bah!

Here's what the first three lines of this script element looks like:

<script>q("talkPage.init", {
"el": "[data-talk-page]",
"__INITIAL_DATA__":

And that is where I am at the current moment. Any light that can be shed on either the soup or the json to get this done would be much appreciated.

Answer 1

Without knowing the full context, here's a poor man's attempt:

Assuming your html looks something like this:

<script>foo</script>
<script>bar</script>
<script>q("talkPage.init",{
"foo1":"bar1",
"event":"TEDGlobal 2005",
"filmed":1120694400,
"published":1158019860,
"foo2":"bar2"
})</script>
<script>q("talkPage.init",{
"foo1":"bar1",
"event":"foobar",
"filmed":123,
"published":456,
"foo2":"bar2"
})</script>
<script>foo</script>
<script>bar</script>

You can code like this:

res = requests.get(url) # your link here
soup = bs4.BeautifulSoup(res.content)
my_list = [i.string.lstrip('q("talkPage.init", ').rstrip(')') for i in soup.select('script') if i.string and i.string.startswith('q')]

# my_list should now be filled with all the json text that is from a <script> tag followed by a 'q'
# note that I lstrip and rstrip on the script based no your sample (assuming there's a closing bracket), but if the convention is different you'll need to update that accordingly.

#...#
my_jsons = []
for json_string in my_list:
    my_jsons.append(json.loads(json_string))

# parse your my_jsons however you want.

Then you can start interpreting the jsons:

print(my_jsons[0]['event'])
print(my_jsons[0]['filmed'])
print(my_jsons[0]['published'])

# Output:
# TEDGlobal 2005
# 1120694400
# 1158019860

There's a lot of assumptions here. That's assuming all your texts within <script>q elements will always be starting with q("talkPage.init", and ending with a ). Also it's assuming the texts returned follow the json format for your next stage of parsing. I'm also assuming you have knowledge of how to parse the json results.

Answer 2

You can use a regex expression to match the part you want.

import re
# Filters the script-tag all the way to end ')' of q.
scipt_tag = re.findall(r'<script>q\((?s:.+)\)', t)
json_content = re.search(r'(?<=q\()(?s:.+)\)', script_tag[0]).group()
json_content = json_content[:-1]  # Strip last ')'

To find the stuff you need you can either use pythons json library to parse it or match the last things with what you want. Since filmed and published are unique and event doesn't differ (as I understood?)

import json
json_content = json.loads(json_content)
json_content['event']  # or whatever

OR

def get_val(a):
re.search('r(?<=' + a + r'\": )(.+)').group(0)

The latter needs to be filtered a bit to remove trailing ]" and preceding "[, or what not you want from it.

I've heard beautifulsoup is also a good library for matching html-stuff, but im not so familiar with it.

Answer 3

Here's the script I ended up using, with real thanks to both @Idlehands and @Three. To reach into the weird single-quoted JSON, I took the entire JSON element and read it into a list, split on commas. It's a hack, but it mostly works.

def get_metadata(the_file):

    # Load the modules we need
    from bs4 import BeautifulSoup
    import json
    import re
    from datetime import datetime

    # Read the file, load it into BS, then grab section we want
    text = the_file.read()
    soup = BeautifulSoup(text, "html5lib")
    my_list = [i.string.lstrip('q("talkPage.init", {\n\t"el": "[data-talk-page]",\n\t "__INITIAL_DATA__":')
               .rstrip('})')
               for i in soup.select('script') 
               if i.string and i.string.startswith('q')]

    # Read first layer of JSON and get out those elements we want
    pre_json = '{"' + "".join(my_list)
    my_json = json.loads(pre_json)
    slug = my_json['slug']
    vcount = my_json['viewed_count']
    event = my_json['event']

    # Read second layer of JSON and get out listed elements:
    properties = "filmed,published" # No spaces between terms!
    talks_listed = str(my_json['talks']).split(",")
    regex_list = [".*("+i+").*" for i in properties.split(",")]
    matches = []
    for e in regex_list:
        filtered = filter(re.compile(e).match, talks_listed)
        indexed = "".join(filtered).split(":")[1]
        matches.append(indexed)
    filmed = datetime.utcfromtimestamp(float(matches[0])).strftime('%Y-%m-%d')
    # published = datetime.utcfromtimestamp(float(matches[1])).strftime('%Y-%m-%d')
    return slug, vcount, event, filmed, #published