6

I want to save text I scrape from various sources without the HTML tags that are on it, but also keeping as much of the structure as I reasonably can.

Markdown seems to be the solution to this (or possibly MultiMarkdown).

There is a question which offers a suggestion on converting from HTML to markdown, but I want to specify some specific things:

  • ALL links (including images) are referenced at the END only (i.e. no inline urls)
  • NO embeded HTML (I'm not even 100% sure yet how I'd like to deal with difficult HTML... but it won't be embeded!)

So my question is as stated in the title: Is there a decent, customisable, HTML to Markdown Java API?

Community
  • 1
  • 1
barryred
  • 1,103
  • 1
  • 17
  • 27

2 Answers2

2

You could try adapting HtmlCleaner which provides a workable interface onto the DOM:

TagNode root = htmlCleaner.clean( stream );
Object[] found = root.evaluateXPath( "//div[id='something']" );
if( found.length > 0 && found instanceof TagNode ) {
    ((TagNode)found[0]).removeFromTree();
}

This would allow you to structure your output stream in any format that you want using a fairly simple API.

Gary
  • 7,167
  • 3
  • 38
  • 57
0

There is a great library for JS called Turndown, you can try it online here. It can be partially customized. For example, links can be referenced at the end. And as far as I know there is no embedded html, everything is transformed.

I needed it for Java (as the linked question), so I ported it. The library for Java is called CopyDown, it has the same test suite as Turndown.

To install with gradle:

dependencies {
        compile 'io.github.furstenheim:copy_down:1.0'
}

Then to use it:

CopyDown converter = new CopyDown();
String myHtml = "<h1>Some title</h1><div>Some html<p>Another paragraph</p></div>";
String markdown = converter.convert(myHtml);
System.out.println(markdown);
> Some title\n==========\n\nSome html\n\nAnother paragraph\n
Gabriel Furstenheim
  • 2,969
  • 30
  • 27