This is not a technical (in depth) answer but may help you as a general explanation. I'm currently looking into this exact problem for a product I'm working on.
The reason fullscreen mode does not persist in the scenario you describe is because switching video amounts to navigating to a different page.
This is by design, out of security considerations: "In addition, navigating to another page [...] while in fullscreen mode exits fullscreen mode as well."
See: Fullscreen API
The trick, then, is to have the element that is in fullscreen be the thing that changes, thus not navigating to another page.
From what I gather, YouTube fullscreens the wrapper element for the videos, and the content of that element changes when you switch to a different video. But the page you are on stays the same.
How YouTube manages to change the window.location without exiting fullscreen... I don't know unfortunately.
But it has to have something to do with fullscreening only part of the page. Otherwise they would have circumvented the security considerations of the API. Doesn't sound like an approach that a devteam at Google could justify.