If a large JASON file(10mb) is needed for processing by multiple Microservices what's the best Enterprise Architectural/Design pattern to use? Entire data in the file is needed by each Microservices in order to process it.
Asked
Active
Viewed 1,524 times
3 Answers
1
- Sharing large data set may be an indication for a suboptimal partitioning of the codebase into services. It is preferred all processing of the same domain will be done within a single service.
- When multiple services do have meaningful processing to be done on the same data set - each should have its own copy of it. Sharing databases, is typically - a bad idea!
- When heavyweight data is involved, cloning the data in a "regular" queueing system (such as RabbitMQ / SQS) is quite cumbersome and inefficient.
- A "heavyweight" queuing system such as Kafka / Kinesis - may be most efficient. One copy of the data will be persisted, and each service can read it from a "shared" stream.
Lior Bar-On
- 10,784
- 5
- 34
- 46
-
1Thanks, A dedicated service with Kafka was what I have in mind. This was asked in an interview. Apparently, their loan processing must share this huge file around with multiple services. – user221216 Aug 14 '17 at 16:30
0
Store it somewhere where other microservices can read it from. Using pub/sub or eventing - notify interested microservices where the file is published. The other microservices can read it for themselves.
The g
- 43
- 5
0
- client/system POSTs an event to an API that implements a Saga pattern, passing a link to where the file is located (e.g. uploaded to an AWS S3 bucket).
- Each event/step in the Saga pattern processes the file, per their requirements.
- client/system can poll for an update, or the original POST can be designed to pass a callback/notification service endpoint.
intltechventures
- 181
- 8