Beautiful Soup is getting XML data incomplete

Question

I'm using Python3.4 and Beautiful Soup 4 to get some data of a RSS XML feed. Everything seems to work fine, but sometimes it behaves not as expected because is not getting all the data inside the <description> tag from at least one item on the list.
For example, this is the item that is giving me problems:

<item>
    <title>Google&#8217;s first DeepMind AI health project is missing something</title>
    <link>http://thenextweb.com/google/2016/02/25/googles-first-deepmind-ai-health-project-is-missing-something/</link>
    <comments>http://thenextweb.com/google/2016/02/25/googles-first-deepmind-ai-health-project-is-missing-something/#respond</comments>
    <pubDate>Thu, 25 Feb 2016 11:36:56 +0000</pubDate>
    <dc:creator><![CDATA[Kirsty Styles]]></dc:creator>
            <category><![CDATA[Google]]></category>
    <category><![CDATA[Insider]]></category>
    <category><![CDATA[Deepmind]]></category>
    <category><![CDATA[doctor]]></category>
    <category><![CDATA[healthcare]]></category>
    <category><![CDATA[NHS]]></category>
    <category><![CDATA[UK]]></category>

    <guid isPermaLink="false">http://thenextweb.com/?p=957096</guid>
    <description><![CDATA[<img width="520" height="245" src="http://cdn1.tnwcdn.com/wp-content/blogs.dir/1/files/2014/04/doctor-crop-520x245.jpg" alt="Doctors Seek Higher Fees From Health Insurers" title="Google&#039;s first DeepMind AI health project is missing something" data-id="750745" /><br />Having been down at Google’s DeepMind office earlier this week its man vs AI machine gaming competition preview, I was tipped off that a potentially-more-serious healthcare announcement would follow soon. That it has, but contrary to what the company’s remit might suggest, this project doesn’t actually contain any artificial intelligence at launch. “To date, no machine learning has been involved in these projects,” the company said. “While there is obvious potential in applying machine learning to these kinds of complex challenges, any decision to do so will led by clinicians.” DeepMind has announced an acquisition in the shape of an Imperial College London&#8230; <br><br><a href="http://thenextweb.com/google/2016/02/25/googles-first-deepmind-ai-health-project-is-missing-something/?utm_source=social&#038;utm_medium=feed&#038;utm_campaign=profeed">This story continues</a> at The Next Web]]></description>
    <wfw:commentRss>http://thenextweb.com/google/2016/02/25/googles-first-deepmind-ai-health-project-is-missing-something/feed/</wfw:commentRss>
    <slash:comments>0</slash:comments>
<enclosure url="http://cdn1.tnwcdn.com/wp-content/blogs.dir/1/files/2014/04/doctor-crop-520x245.jpg" type="image/jpeg" length="0" />
</item>

I'm using this code to parse the data:

from bs4 import BeautifulSoup
import urllib.request

req = urllib.request.urlopen('http://thenextweb.com/feed/')

xml = BeautifulSoup(req, 'xml')

for item in xml.findAll('item'):
    string = item.description.string
    #new_string = string.split('/>', 1)
    #print(new_string[0]+'/><p>')
    print(string)

Everything works when i run the script, but that particular item is failing. The commented lines in the code are for splitting the img and add a <p> tag to order the content.

The result that i get from that item is:

’s DeepMind office earlier this week its man vs AI machine gaming competition preview, I was tipped off that a potentially-more-serious healthcare announcement would follow soon. That it has, but contrary to what the company’s remit might suggest, this project doesn’t actually contain any artificial intelligence at launch. “To date, no machine learning has been involved in these projects,” the company said. “While there is obvious potential in applying machine learning to these kinds of complex challenges, any decision to do so will led by clinicians.” DeepMind has announced an acquisition in the shape of an Imperial College London&#8230; <br><br><a href="http://thenextweb.com/google/2016/02/25/googles-first-deepmind-ai-health-project-is-missing-something/?utm_source=social&#038;utm_medium=feed&#038;utm_campaign=profeed">This story continues</a> at The Next Web

I don't know what is happening. If somebody can help me or guide me through a way to extract the exact <img> tag i would be very thankful.

A quick guess via [this](http://stackoverflow.com/questions/2360598/how-do-i-unescape-html-entities-in-a-string-in-python-3-1): First `import html`. Then try `string = html.unescape(item.description.string)`. Would need to check the the `bs4` API, but you may also want `string = html.unescape(item.description.text)` instead. — scharfmn, Feb 25 '16 at 15:33

mrkzq · Accepted Answer · 2017-05-12T15:49:38.050

Why don't you just search for description tag inside your for loop like this:

for item in xml.findAll('item'):
    s = item.find('description')
    print (s)
    >>> <description>&lt;img width="520" height="245" src="http://cdn1.tnwcdn.com/wp-content/blogs.dir/1/files/2016/02/shutterstock_366588536-520x245.jpg" alt="Fintech" title="5 British companies for FinTech Week" data-id="956789" /&gt;&lt;br /&gt;FinTech, financial technology, is about disrupting the stale financial sector with technology and innovation. Have you accepted the status quo of a bank-led dominance? The people in the flourishing FinTech field have rejected it. Last year, Eileen Burbidge, the UK government’s special envoy for FinTech stated: “London and the UK will lead the FinTech sector.” That’s not hard to believe. With a well-established financial sector, a cultivated tech scene and wide access to capital and talent, London is primed for FinTech. The industry generated over $9 billion in revenue last year. As the UK celebrates #FinTechWeek, we look at five British&amp;#8230; &lt;br&gt;&lt;br&gt;&lt;a href="http://thenextweb.com/insider/2016/02/25/5-british-companies-for-fintech-week/?utm_source=social&amp;#038;utm_medium=feed&amp;#038;utm_campaign=profeed"&gt;This story continues&lt;/a&gt; at The Next Web</description>

Beautiful Soup is getting XML data incomplete

1 Answers1