Organizing services dataflow / eip

Question

Say I have like 1000 VMs with different services running on them with different technologies used like python, NET, java and different middleware like rabbitmq, redis etc. How can I dynamically handle the interactions between the services and provide scalability?

For Example, say I have Service A which is pushing Data to a rabbitmq then the data is processed by service B while fetching additional data from Service C. You see at the end I have a decentralized system which is pulling data somewhere and pushing it somewhere else... a total mess! Scale it up to 2000 microservices omg XD.

The moment I change one thing a lot of other systems are affected. Do you know something maybe like an ESB where I can couple two services together with a message transform adapter in the middle of it and I can change dependenciesat runtime? Like the stream doesn't end in service F anymore and does end in G for example?

I think microservices are a good idea because they can be stateless, can scale, can easily be deployed as a container. But I don't know a good tool/program for managing the data flow. The rabbitmq doesn't support enough enterprise integration patterns. Do you have any advice?

Answer 1

How can I dynamically handle the interactions -

See if using an existing EIP pattern solves your problem to implement the logistics

Depending on how your design shapes up, you may need to use Distributed Lock Management

Or maybe your application is simple enough to use a Consul K/V store as a semaphore & a simple mosquitto topic based bus.

Provide scalability

What is the solution you are trying to scale? AMQP, Consul, "microservices" in themselves are very scalable & distributed However, to scale your thought process & devops, you need to find a way to see things as patterns that help you split the problem & tackle the complexity

Do you know something maybe like an ESB where I can couple two services together with a message transform adapter in the middle of it and I can change dependenciesat runtime?

Read up on EIP. ESBs are just one of the many ways you can solve your problem. RTFM, & get some perspective.

But I don't know a good tool/program for managing the data flow.

Ask yourself if your problem is related to distributed workflow management, or if a data pipeline is what you are really looking for

Look at Spark, Storm, Luigi, Airflow - they all have a different purpose - but you will know what to do with them if you manage to read up on everything else in this post ;)