I'd try something like this:
window.document.documentElement.innerHTML.length
But to be honest, it is much more complicated than this. It will return the quantity of characters. To get the size of the document, you'd have to actually check the charset/encoding used by the document.
UTF-8 will have 1 bytes for each ASCII char. Anything that isn't ascii might be from 2 to 6 bytes big. If the document is in a different encoding, the size of each char is going to be different. UTF-16 will have 2bytes per char and UTF-32 -> 4 bytes.
One way to "guess" the size of the document would be to scan each char contained in the result of the snippet above. Anything that isn't ascii will have at least 2 bytes for utf-8 (it's unlikely to be more than 2 bytes unless you have chineese or really strange languages in the document). Even that doesn't tell how much space it takes in memory because the text might take more space than the DOM objects. Parsing the whole tree as a String is going to use memory two but it shouldn't be that much heavy if done right and how big is the document.
If you want to know how much space the page is taking in "memory" in the DOM, it will not be possible as each DOM element is dependent specific and there is no API that gives you access to internally.
If for whatever reason, this is a big deal. You could have a look at XPCOM and you could build your own API that returns the amount of memory used in the DOM.
But you should think out of the box
The problem you're having is unlikely because the page is big. But because some JS is creating a lot of dom object. Use a profiler to watch how is the page built and you could possibly consider this: Do we have to keep all the objects attached to the document? If the page is getting big at some place, you could limit the amount of element shown in a block and keep the data in memory. Rebuild the DOM objects when they are needed. Keep data in your js code. If some data is getting unused, you could just forget it or store them inside indexeddb. Reload it from there when needed.
Don't hide dom objects using "display: none". Build DOM objects only when they are required. If you have hidden DOM objects all over the place.