11

Since a couple of days I've been trying to figure it out how to inform to the rest of the microservices that a new entity was created in a microservice A that store that entity in a MongoDB.

I want to:

  • Have low coupling between the microservices

  • Avoid distributed transactions between microservices like Two Phase Commit (2PC)

At first a message broker like RabbitMQ seems to be a good tool for the job but then I see the problem of commit the new document in MongoDB and publish the message in the broker not being atomic.

Why event sourcing? by eventuate.io: enter image description here

One way of solving this issue implies make the schema of the documents a bit dirtier by adding a mark that says if the document have been published in the broker and having a scheduled background process that search unpublished documents in MongoDB and publishes those to the broker using confirmations, when the confirmation arrives the document will be marked as published (using at-least-once and idempotency semantics). This solutions is proposed in this and this answers.

Reading an Introduction to Microservices by Chris Richardson I ended up in this great presentation of Developing functional domain models with event sourcing where one of the slides asked:

How to atomically update the database and publish events and publish events without 2PC? (dual write problem).

The answer is simple (on the next slide)

Update the database and publish events

This is a different approach to this one that is based on CQRS a la Greg Young.

The domain repository is responsible for publishing the events, this would normally be inside a single transaction together with storing the events in the event store.

I think that delegate the responsabilities of storing and publishing the events to the event store is a good thing because avoids the need of 2PC or a background process.

However, in a certain way it's true that:

If you rely on the event store to publish the events you'd have a tight coupling to the storage mechanism.

But we could say the same if we adopt a message broker for intecommunicate the microservices.

The thing that worries me more is that the Event Store seems to become a Single Point of Failure.

If we look this example from eventuate.io enter image description here

we can see that if the event store is down, we can't create accounts or money transfers, losing one of the advantages of microservices. (although the system will continue responding querys).

So, it's correct to affirmate that the Event Store as used in the eventuate example is a Single Point of Failure?

mat-mcloughlin
  • 6,492
  • 12
  • 45
  • 62
gabrielgiussi
  • 9,245
  • 7
  • 41
  • 71
  • What did you end up with? You can use DTC for handling the transactions? And as for single point of failure side of Event Store; I would consider a replication mechanism.. – yakya Aug 29 '17 at 05:35
  • https://www.thoughtworks.com/insights/blog/scaling-microservices-event-stream – Mohsen Jul 02 '18 at 10:08
  • https://groups.google.com/forum/#!topic/event-store/tpz7B2TFEfs – Mohsen Jul 02 '18 at 10:08

5 Answers5

6

What you are facing is an instance of the Two General's Problem. Basically, you want to have two entities on a network agreeing on something but the network is not fail safe. Leslie Lamport proved that this is impossible.

So no matter how much you add new entities to your network, the message queue being one, you will never have 100% certainty that agreement will be reached. In fact, the opposite takes place: the more entities you add to your distributed system, the less you can be certain that an agreement will eventually be reached.

A practical answer to your case is that 2PC is not that bad if you consider adding even more complexity and single points of failures. If you absolutely do not want a single point of failure and wants to assume that the network is reliable (in other words, that the network itself cannot be a single point of failure), you can try a P2P algorithm such as DHT, but for two peers I bet it reduces to simple 2PC.

Akira
  • 4,001
  • 1
  • 16
  • 24
  • Agree also see Byzantine Generals problem. I detest the false guarantees persistent queues give and when it fails ( out of disk/ quota , disk write , poison message , bug ) there is no consideration and the results are often catastrophic. – user1496062 Mar 25 '16 at 05:02
2

We handle this with the Outbox approach in NServiceBus:

http://docs.particular.net/nservicebus/outbox/

This approach requires that the initial trigger for the whole operation came in as a message on the queue but works very well.

Udi Dahan
  • 11,932
  • 1
  • 27
  • 35
  • Thanks Udi!. Two questions: what is the meaning of `Bus.Send(new PreparePurchasedProducts());` and `Bus.Publish(new PurchaseOrderReceived());`? the messages aren't published in the **Dispatch** step? And what is the meaning of the **Queue Tx.** bracket? I understand that the **Db Tx** won't rollback if the **Dispatch** step fails. – gabrielgiussi Mar 22 '16 at 12:17
  • 1
    The messages are dispatched to the broker in the dispatch step, yes, but that could fail (say, because the broker is unavailable). The larger Queue Tx bracket talks about the final ack that is sent to the broker at the end of processing the incoming messages. Any failure would cause that ack not to be dispatched, resulting in the incoming message rolling back and retrying. At that point, all the previously recorded operations are pulled from storage and redispatched by the infrastructure without invoking business code again. – Udi Dahan Mar 23 '16 at 08:00
1

You could also create a flag for each entry inside of the event store which tells if this event was already published. Another process could poll the event store for those unpublished events and put them into a message queue or topic. The disadvantage of this approach is that consumers of this queue or topic must be designed to de-duplicate incoming messages because this pattern does only guarantee at-least-once delivery. Another disadvantage could be latency because of the polling frequency. But since we have already entered the eventually consistent area here this might not be such a big concern.

ra1f
  • 91
  • 2
1

How about if we have two event stores, and whenever a Domain Event is created, it is queued onto both of them. And the event handler on the query side, handles events popped from both the event stores.

Ofcourse every event should be idempotent. But wouldn’t this solve our problem of the event store being a single point of entry?

Naveen Santhanavel
  • 369
  • 3
  • 13
0

Not particularly a mongodb solution but have you considered leveraging the Streams feature introduced in Redis 5 to implement a reliable event store. Take a look this intro here

I find that it has rich set of features like message tailing, message acknowledgement as well as the ability to extract unacknowledged messages easily. This surely helps to implement at least once messaging guarantees. It also support load balancing of messages using "consumer group" concept which can help with scaling the processing part.

Regarding your concern about being the single point of failure, as per the documentation, streams and consumer information can be replicated across nodes and persisted to disk (using regular Redis mechanisms I believe). This helps address the single point of failure issue. I'm currently considering using this for one of my microservices projects.

Anthony Anyanwu
  • 987
  • 11
  • 7