[This is a replacement for my original answer. My original idea was to exploit differences in the behaviour of innerHTML. Although it worked fine in IE9, Firefox and Chrome, it turned out that it failed in Opera, which appears to use an HTML parser for innerHTML even for pages served as application/xhtml+xml]
There's not too many ways to tell XML documents apart from HTML documents. One way however, is to exploit the case handling differences between HTML and XML.
In particular, the behaviour of Element.tagName differs. In an HTML parsed document, the element name will be coerced to upper case for tagName whereas in an XML parsed document it won't be. So we can test document.createElement("div").tagName == "DIV" which will give a different result depending on how the document was parsed.
See this test case:
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Test Case</title>
<script>
window.onload = function() {
document.getElementById("result")
.appendChild(document.createTextNode(
(document.createElement("div").tagName == "DIV")
? "HTML parser" : "XML parser"));
}
</script>
</head>
<body>
<p id="result"></p>
</body>
</html>
See it in action:
