1

I have a User Microservice, that publishes events when a user is created, modified, or deleted. In addition, I have a Calendar Microservice that needs the user data. The Calendar service, subscribes to the user events and keeps a read only copy of the needed data.

I have just added an Account Microservice that needs the existing user data and I have used the same model as above. What is the best way to get the existing user data into the Account Microservice?

When I did the first Microservice I republished the user-modified event for every user. It was simple, because nothing else was using this event. If I republish now, for the new Account Microservice, the existing Calendar Microservice will also get the events. My logic is idempotent, but this is a lot of wasted work and will only get worse as I get more services.

I've loaded data a lot in the past. I know how to make it work, but I am looking for a best-practice and a way to do it with minimal coordination and dependencies with other services.

Don Chambers
  • 3,798
  • 9
  • 33
  • 74
  • What about querying for events until the remote service catches up? Otherwise the only other way I see this could be done is having a dedicated consumer queue where you re-publish the events only there. This is the same result as if the consumer never went online while messages were buffering in his queue. – plalx Oct 21 '20 at 02:15
  • would timestamp help? – ApprenticeWST Oct 21 '20 at 03:20

1 Answers1

0

When I did the first Microservice I republished the user-modified event for every user. It was simple, because nothing else was using this event. If I republish now, for the new Account Microservice, the existing Calendar Microservice will also get the events. My logic is idempotent, but this is a lot of wasted work and will only get worse as I get more services.

What you probably want here is a design in which you pull, rather than push, copies of the events to new systems: see Greg Young, Polyglot Data.

In broad strokes - the subscriber keeps track of the high water mark, and asks for more events after some mark, and then the event store answers that query with zero or more new events, in order. The twitter timeline api can give you a sense for how that might work -- sadly, the images that explained the ideas seem to have been removed from the docs, but some of them have been captured here at stack overflow, and by the wayback machine

VoiceOfUnreason
  • 52,766
  • 5
  • 49
  • 91