The best answer comes from MDN:
In the old days of the web, pages were typically written in two
versions: One for Netscape Navigator, and one for Microsoft Internet
Explorer. When the web standards were made at W3C, browsers could not
just start using them, as doing so would break most existing sites on
the web. Browsers therefore introduced two modes to treat new
standards compliant sites differently from old legacy sites.
There are now three modes used by the layout engines in web browsers:
quirks mode, almost standards mode, and full standards mode. In quirks
mode, layout emulates nonstandard behavior in Navigator 4 and Internet
Explorer 5 for Windows that is required not to break existing content
on the Web. In full standards mode, the behavior is (hopefully) the
behavior described by the HTML and CSS specifications. In almost
standards mode, there are only a very small number of quirks
implemented.
How do browsers determine which mode to use?
For HTML documents, browsers use a DOCTYPE in the beginning of the
document to decide whether to handle it in quirks mode or standards
mode. To ensure that your page uses full standards mode, make sure
that your page has a DOCTYPE like in this example:
<!DOCTYPE html>
<html>
<head>
<meta charset=UTF-8>
<title>Hello World!</title>
</head>
<body>
</body>
</html>
The DOCTYPE shown in the example, , is the simplest
possible, and the one recommended by HTML5. Earlier versions of the
HTML standard recommended other variants, but all existing browsers
today will use full standards mode for this DOCTYPE, even the dated
Internet Explorer 6. There are no valid reasons to use a more
complicated DOCTYPE. If you do use another DOCTYPE, you may risk
choosing one, which triggers almost standards mode or quirks mode.
Make sure you put the DOCTYPE right at the beginning of your HTML
document. Anything before the DOCTYPE, like a comment or an XML
declaration will trigger quirks mode in Internet Explorer 9 and older.
In HTML5, the only purpose of the DOCTYPE is to activate full
standards mode. Older versions of the HTML standard gave additional
meaning to the DOCTYPE, but no browser has ever used the DOCTYPE for
anything other than switching between quirks mode and standards mode.
To answer your second question, it's recommended that you use the HTML5 doctype:
<!DOCTYPE html>
which triggers standard mode in every browser (including IE6).