Python: pulling bold text and the text that follows

Question

Using the below html I would like to pull 2 bits of data out and add them into a list in python. each bold text his a horse name and following that is the comments.

<div id="ANALYSIS" class="tabContent tabSelected">A weak handicap that looked wide open.
  <br>
  <br> <b class="black">LADY MAKFI</b> showed vastly improved form to shed her maiden tag on this seasonal debut for a new yard. The filly offered little for Tony Martin last year, but did show some ability on her debut and is evidently capable when fresh.
  She saw it out well and it´ll be interesting to see how she copes with a rise.
  <br>
  <br> <b class="black">Weardiditallgorong</b> went down fighting over this longer trip and probably improved again on her last-time-out second at Bath. This was her best effort yet on the AW.
  <br>
  <br> <b class="black">Chauvelin</b>, in second-time blinkers, turned in his most encouraging effort for some time and is certainly well treated on his best form.
  <br>
  <br> <b class="black">Happy Jack</b> not for the first time travelled easily until making heavy weather of it when asked for his effort. [David Orton]
  <br>
  <br>
  <div id="resultRaceReport" class="hide"></div>
</div>

from the above output i would like it to look like the following

[LADY MAKFI, showed vastly improved form to shed her maiden tag on this seasonal debut for a new yard. The filly offered little for Tony Martin last year, but did show some ability on her debut and is evidently capable when fresh. She saw it out well and it´ll be interesting to see how she copes with a rise.]

[Weardiditallgorong, went down fighting over this longer trip and probably improved again on her last-time-out second at Bath. This was her best effort yet on the AW.]

[Chauvelin, in second-time blinkers, turned in his most encouraging effort for some time and is certainly well treated on his best form.]

[Happy Jack, not for the first time travelled easily until making heavy weather of it when asked for his effort. [David Orton]]

but im just not sure how to get the desired output (more the logic behind it)

I currently use lxml to scrape content and would need to match the bold (horses name) against my table so I can add the comments (text after the bold) to my database

Possible duplicate of [Parsing HTML using Python](http://stackoverflow.com/questions/11709079/parsing-html-using-python) — OneCricketeer, Jun 10 '16 at 08:26
@emma perkins, I presume you are using lxml as per your previous question? — Padraic Cunningham, Jun 10 '16 at 08:27
apology yes i am (i will add that into the question) - this is more the logic on doing it rather then how to — emma perkins, Jun 10 '16 at 08:28
@PadraicCunningham - sorry to miss lead you - this question is not about scraping itself but more the logic on how i would scrape the above — emma perkins, Jun 10 '16 at 08:30
yes :) - this will allow me to do an if statement on my other lists — emma perkins, Jun 10 '16 at 08:43

Padraic Cunningham · Accepted Answer · 2016-06-10T09:06:53.050

using lxml:

h = """<div id="ANALYSIS" class="tabContent tabSelected">A weak handicap that looked wide open.<br><br> <b class="black">LADY MAKFI</b> showed vastly improved form to shed her maiden tag on this seasonal debut for a new yard. The filly offered little for Tony Martin last year, but did show some ability on her debut and is evidently capable when fresh. She saw it out well and it´ll be interesting to see how she copes with a rise.<br><br> <b class="black">Weardiditallgorong</b> went down fighting over this longer trip and probably improved again on her last-time-out second at Bath. This was her best effort yet on the AW.<br><br> <b class="black">Chauvelin</b>, in second-time blinkers, turned in his most encouraging effort for some time and is certainly well treated on his best form.<br><br> <b class="black">Happy Jack</b> not for the first time travelled easily until making heavy weather of it when asked for his effort. [David Orton]<br><br> <div id="resultRaceReport" class="hide"></div></div>"""

from lxml import html

x = html.fromstring(h)

div = x.xpath("//*[@id='ANALYSIS']")[0]

# find bold tags by class name
for b in div.xpath(".//b[@class='black']"):
    # get bold text
    print(b.text)
    # get text between current bold up to next br tag.
    print(b.xpath("./following::text()[1]"))

Would give you:

LADY MAKFI
[u' showed vastly improved form to shed her maiden tag on this seasonal debut for a new yard. The filly offered little for Tony Martin last year, but did show some ability on her debut and is evidently capable when fresh. She saw it out well and it\xc2\xb4ll be interesting to see how she copes with a rise.']
Weardiditallgorong
[' went down fighting over this longer trip and probably improved again on her last-time-out second at Bath. This was her best effort yet on the AW.']
Chauvelin
[', in second-time blinkers, turned in his most encouraging effort for some time and is certainly well treated on his best form.']
Happy Jack
[' not for the first time travelled easily until making heavy weather of it when asked for his effort. [David Orton]']

If you want it all in a single list exactly as posted:

from lxml import html

x = html.fromstring(h)
div = x.xpath("//*[@id='ANALYSIS']")[0]
out = [b.text + "," +  b.xpath("./following::text()[1]")[0].lstrip(",") for b in div.xpath(".//b[@class='black']")]

Which gives you:

[u'LADY MAKFI, showed vastly improved form to shed her maiden tag on this seasonal debut for a new yard. The filly offered little for Tony Martin last year, but did show some ability on her debut and is evidently capable when fresh. She saw it out well and it\xc2\xb4ll be interesting to see how she copes with a rise.',
 'Weardiditallgorong, went down fighting over this longer trip and probably improved again on her last-time-out second at Bath. This was her best effort yet on the AW.',
 'Chauvelin, in second-time blinkers, turned in his most encouraging effort for some time and is certainly well treated on his best form.',
 'Happy Jack, not for the first time travelled easily until making heavy weather of it when asked for his effort. [David Orton]']

No prob, we can actually simplify the xpath to just get the first following text after each bold tag. Are you doing some analysis on the data? — Padraic Cunningham, Jun 10 '16 at 09:07
yes - so im collecting results from past horse races ... then doing analysis on them for betting :) so each comment on the horse need to go inot my database and match with that horse — emma perkins, Jun 10 '16 at 09:09
I did something similar a couple of years ago, unfortunately I did quite crack a winning formula, you will have lots of scraping to do after this week after Royal Ascot ;) — Padraic Cunningham, Jun 10 '16 at 09:14
So now i need to some how add then into a loop and do if x = horse name then add comment to database — emma perkins, Jun 10 '16 at 09:14
my own selections have been going well .... just need to cut it down alittle :D — emma perkins, Jun 10 '16 at 09:15
I used to lay more than back, it is harder than you would think to pick a loser in a race! Good luck :) — Padraic Cunningham, Jun 10 '16 at 09:36

score 1 · Answer 2 · answered Jun 10 '16 at 09:27

I prefer Beautiful Soup's api over using lxml directly. I can avoid xpath entirely and just write python.

import bs4 
soup = bs4.BeautifulSoup(document, 'lxml')
[b.text + b.next_sibling.rstrip() for b in soup.find_all('b')]

output:

['LADY MAKFI showed vastly improved form to shed her maiden tag on this seasonal debut for a new yard. The filly offered little for Tony Martin last year, but did show some ability on her debut and is evidently capable when fresh.\n  She saw it out well and it´ll be interesting to see how she copes with a rise.',
 'Weardiditallgorong went down fighting over this longer trip and probably improved again on her last-time-out second at Bath. This was her best effort yet on the AW.',
 'Chauvelin, in second-time blinkers, turned in his most encouraging effort for some time and is certainly well treated on his best form.',
 'Happy Jack not for the first time travelled easily until making heavy weather of it when asked for his effort. [David Orton]']

Python: pulling bold text and the text that follows

2 Answers2