Yes, you'd need to use websockets for a stream of updates. Kafka does not help with this, though, you need to find some solution to combine Kafka with a websocket client.
Such as socket.io ...
// Listen for Kafka
consumer.on('message', ({ value, }) => {
// Parse the JSON value into an object
const { payload, } = JSON.parse(value)
console.log('\n\nemitting from kafka:', payload)
// Emit the message through all connected sockets
io.emit("kafka-event", payload)
Keep in mind, the above code will only work on one client. New sockets do not start new consumers, so will only see updates as of the current offset of the internal Kafka consumer. If you start multiple Kafka consumers (or multiple Node backends), then you may only see a subset of Kafka partitions being consumed in each socket event...
Otherwise, there's nothing unique to Kafka about the question. You would write a loop (e.g. setTimeout() / setInterval()) to query some HTTP API (not the database directly) for all records, and/or new records after the last time you've polled.
Or, depending on your use case, query the whole database table/collection + add some refresh button to accurately capture deletions (unless you have a websocket to send individual delete events as well, and can update the DOM with those events).
currently have a python-kafka consumer, listening to a stream and storing the data in a postgres database
While that may work, Kafka Connect may scale better.
Or, Kafka Streams supports KV queries, so don't need external Postgres database, depending on your query patterns.
thus optionally skipping the database?
If you don't care about retention of historical events, then you don't need any database, no. You'd only then get events in your UI since the consumer-socket gets established, then lose all history on a refresh.
Extra - Frontend to Kafka. Similar answer - you'll need a backend service with a Kafka producer client since there's no native HTTP interface.
