TL;DR
Since you don't know when new data arrive to the server, having the client request that data is not ideal as you already mentioned. Three alternatives come to my mind, realtime databases, push notifications and WebSockets.
WebSockets seems to be a good choice, because you can easily implement it on the server side and most modern browsers already have it implemented natively. You mention that it is hard to debug, but did you knew about Chrome support for it [1]?
- Launch Chrome Developer tools
- Load your page and initiate the WebSocket connections
- Click the Network Tab.
- Select the WebSocket connection from the list on the left (it will have status of "101 Switching Protocols".
- Click the Frames sub-tab. Binary frames will show up with a length and time-stamp and indicate whether they are masked. Text frames will
show also include the payload content.
Realtime database
Firebase offers multiple services, one of them is a realtime database (old version) and firestore database (new version). Using their SDK you can update the database from the server side and listen for changes on the client side. On the client side you will be using observables
and once a change happen on the database (if you are listening to changes) you will receive that data on the client without you having the perform the request. There are most likely requests being made by the SDK but honestly, I don't know how they deal with this but it lifts the burden from the developer. It is paid for many requests but it is worth to take a look at it.
Push Notifications
Similarly, Firebase (and other companies, such as Amazon) offers you a Push Notification / Cloud Messaging service which allows you to send data to the client without a request is made by the client. This works by using the Notification API and Push API that are built on top of the Service Worker API [2]. Advantage of this technique is that you can notify the user even if he have the app closed.
WebSockets
WebSockets allows you to establish a bidirectional communication which then allow for every party (client and server) to send data to each other. Some tools like Socket.IO gives you more control and features (channels/rooms, auto-reconnection, etc) than traditional WebSockets with the cost of some overhead.