How can I convert the html code with irregular columns into a nested json file?

Question

I have the following HTML code. I want to convert the HTML code below:

<div class="company_data__list">

 <div class="company_data__row"><div class="company_data__head">Name</div><div class="company_data__data">ABC Company<br/>Subtitle</div></div>
 <div class="company_data__row"><div class="company_data__head">Capital</div><div class="company_data__data">230000</div></div>
 <div class="company_data__row"><div class="company_data__head">Total</div><div class="company_data__data">103</div></div>

 <div class="company_data__row"><div class="company_data__head">Name</div><div class="company_data__data">XYZ Company<br/>Subtitle</div> 
 <div class="company_data__row"><div class="company_data__head">Total</div><div class="company_data__data">10</div></div>

 <div class="company_data__row"><div class="company_data__head">Name</div><div class="company_data__data">CAT Company<br/>Subtitle</div></div>
 <div class="company_data__row"><div class="company_data__head">Capital</div><div class="company_data__data">430000</div></div>
 <div class="company_data__row"><div class="company_data__head">Total</div><div class="company_data__data">10233</div></div>
 <div class="company_data__row"><div class="company_data__head">URL</div><div class="company_data__data">www.abc.com</div></div>

</div>

into a Json file which looks like this:

{ id: '1',
  data:{
    name: 'ABC CAT Company',
    capital: '230000',
    total:'103'
  },
  id:'2',
  data: {
    name: 'XYZ CAT Company',
    total:'10'
  },
  id:'3',
  data: {
    name: 'CAT Company',
    capital: '430000',
    total:'10',
    url:'www.abc.com'
  },


}

I'm using python3, bs4, re (Regular Expression) I'm having trouble with matching the html head rows with data rows since it does not have a specific #id to differentiate with.

I think you are asking something like this https://stackoverflow.com/questions/34504050/how-to-convert-selected-html-to-json — Adam Strauss, Oct 29 '19 at 05:43

score 1 · Answer 1 · answered Oct 29 '19 at 06:42

The below code can help you.

html_text = 'your_html_text'    
splits = re.split('\n\n', html_text)[1:]
output = []
for i, text in enumerate(splits):
    matches = [re.search(r'company_data__head">(\w+).*company_data__data">([\w\s\.]+)', m) for m in text.split('\n')]
    attrs = {}
    for match in matches:
        if match:
            key = match.group(1)
            value = match.group(2)
            attrs[key] = value
    if attrs:
        output.append({'id': i + 1, 'data': attrs})

Output

[{'id': 1, 'data': {'Name': 'ABC Company', 'Capital': '230000', 'Total': '103'}}, {'id': 2, 'data': {'Name': 'XYZ Company', 'Total': '10'}}, {'id': 3, 'data': {'Name': 'CAT Company', 'Capital': '430000', 'Total': '10233', 'URL': 'www.abc.com'}}]

Please let me know if it helps and approve it as the answer. Thank — Karmveer Singh, Oct 29 '19 at 06:46

score 0 · Answer 2 · answered Oct 29 '19 at 06:45

you can get text from divs with class="company_data__row", using get_text() in beautiful soup. Use a seperator eg:separator='<>', so that u can split data using that seperator later.

<div class="company_data__list">

 <div class="company_data__row"><div class="company_data__head">Name</div><div class="company_data__data">ABC Company<br/>Subtitle</div></div>  <div class="company_data__row"><div class="company_data__head">Capital</div><div class="company_data__data">230000</div></div>  <div class="company_data__row"><div class="company_data__head">Total</div><div class="company_data__data">103</div></div>

 <div class="company_data__row"><div class="company_data__head">Name</div><div class="company_data__data">XYZ Company<br/>Subtitle</div></div>   <div class="company_data__row"><div class="company_data__head">Total</div><div class="company_data__data">10</div></div>

 <div class="company_data__row"><div class="company_data__head">Name</div><div class="company_data__data">CAT Company<br/>Subtitle</div></div>  <div class="company_data__row"><div class="company_data__head">Capital</div><div class="company_data__data">430000</div></div>  <div class="company_data__row"><div class="company_data__head">Total</div><div class="company_data__data">10233</div></div>  <div class="company_data__row"><div class="company_data__head">URL</div><div class="company_data__data">www.abc.com</div></div>

</div>

code:

from bs4 import BeautifulSoup 
import urllib.request
url = 'yout_html_address.html'                                                             
htmlpage = urllib.request.urlopen(url) 
page = BeautifulSoup(htmlpage, "lxml")

data_dict ={}
id=0

data = page.find_all('div',{'class': 'company_data__row'})
last = len(data)
for i,item in enumerate(data):
    text = item.get_text(separator='<>',strip = True)
    if i < last-1:
        if 'Name' in text:
            if id>0:
                data_dict[id] = details_dict
            id+=1

            details_dict ={}
            details_dict[text.split('<>')[0]] = text.split('<>')[1]
        else:
            details_dict[text.split('<>')[0]] = text.split('<>')[1]
    else:
        data_dict[id] = details_dict
print(data_dict)

output:

{
    1:
        {
            'Name': 'ABC Company', 
            'Capital': '230000', 
            'Total': '103'
        }, 
    2: 
        {
            'Name': 'XYZ Company', 
            'Total': '10'
        }, 
    3: 
        {
            'Name': 'CAT Company', 
            'Capital': '430000', 
            'Total': '10233'
        }
}

How can I convert the html code with irregular columns into a nested json file?

2 Answers2