Frames
Frames used to be the way to go, but as time has gone by they have fallen out of favour with developers for one reason or another - note this article from 2006!
Fortunately, you seem to be in favour of avoiding frames :)
SSI over JavaScript
Secondly, Server-Side Includes (SSI) or some other server-based "include" is favoured over JavaScript, though I accept that this is not necessarily a "pure" HTML/JS/CSS solution.
The format of an SSI statement is as follows:
<!--#include virtual="../quote.txt" -->
http://www.htmlgoodies.com/beyond/webmaster/article.php/3473341/SSI-The-Include-Command.htm
There are many answers that reflect this view on SO - for example, the first three that appeared in searching are
here,
here and
here ...
Note that the accepted answer for that last one is recommending a JS solution, but the final paragraph states a preference for something server-side..
Compilation of your HTML code (my preferred option)
It has been a while since I have needed to create a "pure" HTML/CSS/JS website, but when doing so my preference is to keep the code modular and "compile" the HTML before deployment.
Although it requires a little additional work prior to deployment, it produces the "purest" output to be used within deployed code. You write your code as normal, use a little magic to indicate what you want included and where and then you "compile" this code into bog-standard HTML/CSS/JS files that are deployed onto your site.
This brings the ease and simplicity of using templated header/footer/menu-bar/sidebar files, with the tradeoff of needing to compile the HTML code beforehand.
SASS uses Ruby on Rails to perform this compilation. Unfortunately, a reference for its HTML equivalent is escaping me at this particular moment in time, so I shall update my answer as/when I relocate it.