How to Pretty Print HTML to a file, with indentation

Question

I am using lxml.html to generate some HTML. I want to pretty print (with indentation) my final result into an html file. How do I do that?

This is what I have tried and got till now

import lxml.html as lh
from lxml.html import builder as E
sliderRoot=lh.Element("div", E.CLASS("scroll"), style="overflow-x: hidden; overflow-y: hidden;")
scrollContainer=lh.Element("div", E.CLASS("scrollContainer"), style="width: 4340px;")
sliderRoot.append(scrollContainer)
print lh.tostring(sliderRoot, pretty_print = True, method="html")

As you can see I am using the pretty_print=True attribute. I thought that would give indented code, but it doesn't really help. This is the output :

<div style="overflow-x: hidden; overflow-y: hidden;" class="scroll"><div style="width: 4340px;" class="scrollContainer"></div></div>

score 138 · Accepted Answer · edited Mar 17 '22 at 15:48

138

I ended up using BeautifulSoup directly. That is something lxml.html.soupparser uses for parsing HTML.

BeautifulSoup has a prettify method that does exactly what it says it does. It prettifies the HTML with proper indents and everything.

BeautifulSoup will NOT fix the HTML, so broken code, remains broken. But in this case, since the code is being generated by lxml, the HTML code should be at least semantically correct.

In the example given in my question, I will have to do this :

from bs4 import BeautifulSoup as bs
root = lh.tostring(sliderRoot) #convert the generated HTML to a string
soup = bs(root)                #make BeautifulSoup
prettyHTML = soup.prettify()   #prettify the html

edited Mar 17 '22 at 15:48

Tyrannas

4,303
1
11
16

answered May 29 '11 at 11:14

bcosynot

5,653
10
34
47

3

Thank you, but worth to mention that `js` embedded to html will not prettified, if it is important to somebody. – Vitaly Zdanevich Sep 15 '16 at 07:15
16

With version 4 change the first line to `from bs4 import BeautifulSoup as bs` – shao.lo Sep 29 '16 at 16:07
If you just want to prettify html from a string, see AlexG's answer below. – ErikusMaximus Aug 14 '20 at 16:01
9

Be careful with `prettify`, as it changes document semantics: "Since it adds whitespace (in the form of newlines), `prettify()` changes the meaning of an HTML document and should not be used to reformat one. The goal of `prettify()` is to help you visually understand the structure of the documents you work with." – BallpointBen Dec 23 '20 at 15:23
Another warning: With version 4, BeautifulSoup will DECODE html entities, so if you were decoding strings with user-posted content (eg.: forum posts), it will be happy to reverse escaped HTML back, opening you to potential problems. – Ralfp Sep 10 '22 at 21:36

score 50 · Answer 2 · answered May 12 '13 at 09:01

50

Though my answer might not be helpful now, I am dropping it here to act as a reference to anybody else in future.

lxml.html.tostring(), indeed, doesn't pretty print the provided HTML in spite of pretty_print=True.

However, the "sibling" of lxml.html - lxml.etree has it working well.

So one might use it as following:

from lxml import etree, html

document_root = html.fromstring("<html><body><h1>hello world</h1></body></html>")
print(etree.tostring(document_root, encoding='unicode', pretty_print=True))

The output is like this:

<html>
  <body>
    <h1>hello world</h1>
  </body>
</html>

answered May 12 '13 at 09:01

bhootjb

1,501
1
21
33

2

The `pretty_print` flag only works when calling `etree.tostring` with `method='xml'`, which is the default. So, we are dealing with XHTML here. – lenz Jul 08 '15 at 19:46
8

This is an excellent answer, because it doesn't use any external dependencies. However, if the string containing HTML has carriage returns, `etree.tostring` pretties nothing, and returns its input, unchanged, on Python 2.7.10 at least ... once you know, it's a simple matter to replace the carriage returns, but you'll waste a lot of time if you don't know this. – Tom Swirly Mar 03 '16 at 19:48
This is great because it only provides a solution to tabs. This does not alter the HTML in other ways like BeautifulSoup solutions. – earthmeLon May 11 '18 at 20:02
1

NOPE! And here's why. etree.tostring will shorten "" to "" which is not allowed. – shrewmouse Jan 11 '19 at 20:00

score 28 · Answer 3 · answered Nov 13 '17 at 07:26

28

If you store the HTML as an unformatted string, in a variable html_string, it can be done using beautifulsoup4 as follows:

from bs4 import BeautifulSoup
print(BeautifulSoup(html_string, 'html.parser').prettify())

answered Nov 13 '17 at 07:26

Alex

12,078
6
64
74

I've just tried this method to reformat legacy html, and the result is visually different, particularly regarding vertical spacing. Not saying the original html syntax was correct to start with, but be warned that this does not guarantee the same visual output. – Arnaud P Mar 17 '22 at 11:38

score 7 · Answer 4 · answered Mar 19 '18 at 20:49

If adding one more dependency is not a problem, you can use the html5print package. The advantage over the other solutions, is that it also beautifies both CSS and Javascript code embedded in the HTML document.

To install it, execute:

pip install html5print

Then, you can either use it as a command:

html5-print ugly.html -o pretty.html

or as Python code:

from html5print import HTMLBeautifier
html = '<title>Page Title</title><p>Some text here</p>'
print(HTMLBeautifier.beautify(html, 4))

this installs several other dependencies including beautifulsoup4 — byteface, Sep 25 '19 at 23:54

score 5 · Answer 5 · answered May 06 '18 at 08:02

I tried both BeautifulSoup's prettify and html5print's HTMLBeautifier solutions but since I'm using yattag to generate HTML it seems more appropriate to use its indent function, which produces nicely indented output.

from yattag import indent

rawhtml = "String with some HTML code..."

result = indent(
    rawhtml,
    indentation = '    ',
    newline = '\r\n',
    indent_text = True
)

print(result)

score 4 · Answer 6 · answered May 27 '11 at 15:40

Under the hood, lxml uses libxml2 to serialize the tree back into a string. Here is the relevant snippet of code that determines whether to append a newline after closing a tag:

    xmlOutputBufferWriteString(buf, ">");
    if ((format) && (!info->isinline) && (cur->next != NULL)) {
        if ((cur->next->type != HTML_TEXT_NODE) &&
            (cur->next->type != HTML_ENTITY_REF_NODE) &&
            (cur->parent != NULL) &&
            (cur->parent->name != NULL) &&
            (cur->parent->name[0] != 'p')) /* p, pre, param */
            xmlOutputBufferWriteString(buf, "\n");
    }
    return;

So if a node is an element, is not an inline tag and is followed by a sibling node (cur->next != NULL) and isn't one of p, pre, param then it will output a newline.

score 3 · Answer 7 · answered May 27 '11 at 13:14

3

Couldn't you just pipe it into HTML Tidy? Either from the shell or through os.system().

answered May 27 '11 at 13:14

tsm

3,598
2
21
35

I initially thought of using HTML Tidy, but my code is slightly quirky and tidy just ends up playing havoc with it. Decided to use BeautifulSoup instead. Worked like a charm. – bcosynot May 27 '11 at 14:58
HTML Tidy corrects your HTML which can [break things](http://stackoverflow.com/questions/12871339/html-tidy-strip-tags). Such errors are pretty hard to find if you're forgetting that HTML Tidy is processing the results (I know what I'm talking about)... – mzuther Jan 22 '14 at 21:26
1

More recently than the 2011 comments here, see the answer to this 2018 question: https://stackoverflow.com/questions/50380799/cant-figure-out-how-to-invoke-html5tidy-from-python-3. "That library is broken and/or doesn't work with python 3.5." May save someone a bit of time... – RBV May 18 '18 at 20:01

score 2 · Answer 8 · answered May 27 '11 at 12:56

2

If you don't care about quirky HTMLness (e.g. you must support absolutely support those hordes of Netscpae 2.0-using clients, so having <br> instead of <br /> is a must), you can always change your method to "xml", which seems to work. This is probably a bug in lxml or in libxml, but I couldn't find the reason for it.

answered May 27 '11 at 12:56

Boaz Yaniv

6,334
21
30

1

When you set the method to xml, if a tag does not have any sub-elements, then the closing tag is not generated. For instance, in the example in question, the inner div will not have a closing tag. I don't really know why. I ended up using the BeautifulSoup to get a proper output. – bcosynot May 27 '11 at 14:56

score 2 · Answer 9 · answered May 27 '11 at 15:22

not really my code, I picked it somewhere

def indent(elem, level=0):
    i = '\n' + level * '  '
    if len(elem):
        if not elem.text or not elem.text.strip():
            elem.text = i + '  '
        if not elem.tail or not elem.tail.strip():
            elem.tail = i
        for elem in elem:
            indent(elem, level+1)
        if not elem.tail or not elem.tail.strip():
            elem.tail = i
    else:
        if level and (not elem.tail or not elem.tail.strip()):
            elem.tail = i

I use it with:

indent(page)
tostring(page)

If it's not your code, please cite where it's from. What's `tostring()`? — ggorlen, Jan 18 '23 at 06:45

score 0 · Answer 10 · answered Mar 23 '23 at 07:28

This is crude and not very robust but it will balance out the example html string, without using any non-standard libraries

import re
html = """
<A value="X"><B value="X">
<A value="X"><B value="X">
some random text
</B></A><C />
some random text
</B></A><C />
"""
rx_al = r"(<[\/A-Za-z]+[^>]*>)"
rx_op = r"<([A-Za-z]+)[^\/]+>"
rx_cl = r"</([A-Za-z]+)>"
# self-closing not needed
#rx_sc = r"<([A-Za-z]+).* \/>"
matches = re.findall(rx_al, html, flags=re.M)

def lookup_key(match, indent):
  return f"{match[0]}:{indent}"

def balance(nodes):
  builder = []
  indent = 0
  lookup = {}
  for node in nodes:
    is_open = re.match(rx_op, node)
    is_close = re.match(rx_cl, node)
    padl = " " * indent
    if is_open:
      k = lookup_key(is_open, indent)
      lookup[k] = node
      indent += 2
    elif is_close:
      for i in range(0, indent, -2):
        if lookup.pop(lookup_key(is_close, i), None):
          break
      indent -= 2
      padl = padl[:-2]
    builder.append(padl + node)
  return "\n".join(builder)

print(balance(matches))

will produce:

<A value="X">
  <B value="X">
    <A value="X">
      <B value="X">
      </B>
    </A>
    <C />
  </B>
</A>
<C />

How to Pretty Print HTML to a file, with indentation

10 Answers10

Linked

Related