3

I have been using JSoup to parse lyrics and it has been great until now, but have run into a problem.

I can use Node.html() to return the full HTML of the desired node, which retains line breaks as such:

Glóandi augu, silfurnátt
<br />Bl&oacute;&eth; alv&ouml;ru, starir &aacute;
<br />&Oacute;&eth;ur hundur er &iacute; v&iacute;gam&oacute;&eth;, &iacute; maga... m&eacute;r
<br />
<br />Kolni&eth;ur gref, kvik sem dreg h&eacute;r
<br />Kolni&eth;ur svart, hvergi bjart n&eacute;

But has the unfortunate side-effect, as you can see, of retaining HTML entities and tags.

However, if I use Node.text(), I can get a better looking result, free of tags and entities:

Glóandi augu, silfurnátt Blóð alvöru, starir á Óður hundur er í vígamóð, í maga... mér Kolniður gref, kvik sem dreg hér Kolniður svart,

Which has another unfortunate side-effect of removing the line breaks and compressing into a single line.

Simply replacing <br /> from the node before calling Node.text() yields the same result, and it seems that that method is compressing the text onto a single line in the method itself, ignoring newlines.

Is it possible to have the best of both worlds, and have tags and entities replaced correctly which preserving the line breaks, or is there another method or way of decoding entities and removing tags without having to replace them manually?

joshschreuder
  • 1,413
  • 2
  • 18
  • 32
  • Possible duplicate of [How do I preserve line breaks when using jsoup to convert html to plain text?](https://stackoverflow.com/questions/5640334/how-do-i-preserve-line-breaks-when-using-jsoup-to-convert-html-to-plain-text) – Steve Chambers Apr 16 '18 at 14:51

2 Answers2

2

(disclaimer) I haven't used this API ... but a quick look at the docs suggests that you could visit each descendent node and dump out its text contents. Breaks could be inserted when special tags like <br> are encountered.

The TextNode.getWholeText() call also looks useful.

Asaph
  • 159,146
  • 25
  • 197
  • 199
qwerty
  • 3,801
  • 2
  • 28
  • 43
  • Used TextNode.createFromEncoded(lyrics, "aURI"); to create a TextNode from the HTML, then getWholeText() to get the text itself. It leaves
    intact, but replaces entities fully, and I can just replace the minimal HTML tags manually, so thanks :)     – joshschreuder Mar 18 '11 at 11:32
  • Should mention that I did read the API, but I clearly missed the function that you pointed out. – joshschreuder Mar 18 '11 at 12:58
1

based on another answer from stackoverflow I added a few fixes and came with

    String text = Jsoup.parse(html.replaceAll("(?i)<br[^>]*>", "br2nl").replaceAll("\n", "br2nl")).text();
    text = text.replaceAll("br2nl ", "\n").replaceAll("br2nl", "\n").trim();

Hope this helps

Community
  • 1
  • 1
petrumo
  • 1,116
  • 9
  • 18