Event Driven Architecture - Response to Mobile/Web Clients

Question

I have the following use case in my microservices

User Experience: A contact details popup asking the user to confirm the contact details, email address & other relevant details on the UI. Once the user submits the data - The user should see contact details saved successfully/failed message on the screen.

Whenever contact details are updated, a unique comment should also be added in the background.

The downstream system exposes a couple of CRUD API endpoints for creating, retrieving, updating & deleting the contact details and the similar CRUD API endpoints for comments.

As each of these API endpoints is independent. I'm thinking of going down the route of implementing SAGA's (Choregraphy/Orchestration) with events to ensure atomicity.

A REST API endpoint(Spring Boot) on my side captures all the data modified by the user from UI as JSON and publishes an event Update_Contact_Details to a Kafka message topic.

Update_Contact_Details Consumer will read the message from the topic, call the downstream update contact details API, and write back either Contact_Details_Updated/Contact_Details_Update_Failed events.

In the happy path scenario : The Contact_Details_Updated Message will be read by Comments Listener and makes a call to downstream API to add a comment.

In the failure Scenario: The Contact_Details_Update_Failed will be read by the Contacts Detailed Failed Listener and makes a call to the Delete Contact Details API.

In either case, as the first capture from UI to API endpoint is REST API - How do I send back the success/failed responses to clients(web/mobile) as the whole process related to orchestrating the above-mentioned flow is asynchronous.

Answer 1

You are basically trying to fit the request-response (synchronous) model to async. We have multiple options here:

1. Block sending the response to the client till the flow completes. This is pretty straight-forward and easy to implement but it comes with a cost and is dependent upon the time taken by the whole system to process the chain of events.

2. Use a callback and notify the consumers over that callback API. In this case, the listener will send a callback to the interested client that invoked the Update_Contact_Details API endpoint. This requires change in your client side code.

3. Use polling. Write an endpoint that queries the status of the update (failed/succeeded) depending on whether the message is received by your failed/success listener and let your client keep polling the endpoint periodically. This require changes in your client side code.

If you have ever used like Kinesis (AWS), you might have noticed that creation/update of resources don't happen immediately. You would see a notification bar on the top saying "Creating kinesis stream" and a gif keeps rotating and after sometime, you see the message that stream is successfully created.

With such long or asynchronous processes, the server typically should give a callback or the client should poll for the result instead of blocking on the response indefinitely.

In the case of AWS, the user understands that creation or alteration of resources doesn't happen in milliseconds like Google search, so [s]he doesn't expect the response immediately.

Anyways, going to the root of the design, you can also consider asking the following questions:

1. Do you really require 2 DBs? You have contact details and comments (to be added to the contacts), why would you require two databases for contact details and comments? If you use a single database (RDBMS), then there is no need for SAGA pattern.

This pattern is applicable only when there are multiple databases (either because the dbs are already existent prior or there are special use-cases).

2. Do you really require Kafka for your use-case? You have tagged apache-kafka in the question, so I suppose you have downstream consumers that require this information as "events", if not there is no use of using Kafka, so the question is "if you really require kafka?"

This being the case, you can update the DB in your UI-facing REST, return the response to the UI once the data is successfully stored in the DB and use CDC (Changed-Data-Capture) to stream DB changes to Kafka.

This ensures that your DB and Kafka are in sync and thereby your Kafka downstream consumers are notified of changes and you update the UI also immediately. If employing CDC is not feasible, then you can schedule a background thread that polls the database for new updates and push them to Kafka.