The HTML engine and the JavaScript engine are two different things. First, the HTML engine parses the document. It then passes along chunks of JavaScript code to the JavaScript engine. (And CSS code to the CSS engine, and so on.)
Since HTML doesn't know JavaScript syntax, it sees the closing tag as part of the overall content and merrily passes along the contents of the tags to the JavaScript engine:
<script>
var test = "<script>why?
</script>
More specifically, if you follow the parsing rules starting here...
- The first
<script>
put us in "Script data state"
- We remained there until the first
<
, where we entered "Script data less-than sign state"
- We encountered an
s
, where we returned to "Script data state"
- Some characters later, we encounter another
<
, where we enter "Script data less-than sign state"
- We then encounter a
/
, where we enter "Script data end tag open state"
- etc.
You can keep following along in the spec, but essentially it describes the step-by-step details of how the HTML engine is parsing each character as HTML syntax, regardless of how we intuitively interpret those characters.