It is possible to do custom elements in < IE9, but it requires (sans javascript) being careful with doctypes, namespaces and, to be completely proper xhtml, a matching DTD.
Something like this...
<!DOCTYPE html SYSTEM "http://your.domain/xhtml-custom.dtd">
<html xmlns='http://www.w3.org/1999/xhtml'
xmlns:custom="http://your.domain/"
xml:lang='en-US'>
Where the DTD contains things like...
<!ENTITY % attrs "%coreattrs; %i18n; %events;">
<!ENTITY % custom "custom:attribution | custom:quote ">
<!ENTITY % block "p | div | isindex |fieldset | table | %custom; ">
<!ENTITY % Flow "(#PCDATA | %block; | form )*">
<!ENTITY % custom "custom:attribution | custom:quote">
<!ELEMENT custom:attribution %Flow;>
<!ATTLIST custom:attribution %attrs;>
...and so on.
You end up in a situation where a namespace (such as custom:customtag) is required, and targetting it with CSS needs to escape the colon...
custom\:customtag { display:block; }
...which is too much bother - given the whole point of using custom elements is to produce more semantic markup.
I investigated this in detail back in the days of IE6 when xhtml seemed like it may be the future and solved all the problems, but never sought to implement it anywhere for the shear cumbersome nature of the solution.
And the world mostly gave up on xhtml as being too much trouble anyway.
At the end of the day custom elements for better semantics are hardly worth it because no matter what you do your markup will likely be compromised by presentation needs as (and I've been trying to do it for decades now) you just can't separate content from presentation completely online.