1

I am building a micro-services project using docker.

one of my micro-services is a listener that should get data from various number of sources.

What i'm trying to achieve is the ability to start and stop getting data from sources dynamically.

architecture

For example in this drawing, i have 3 sources connected to 3 dockers.

My problem starts because i need to create another docker instance when a new source is available. In this example lets say source #4 is now available and i need to get his data (I know when a new source became available) but i want it to be scaled automatically (with source #4 information for listening)

I came up with two solutions, each has advantages and disadvantages:

1) Create a docker pool of a large number of docker running the listener service and every time a new source is available send a message (using rabbitmq but i think less relevant) to an available docker to start getting data.

in this solution i'm a little bit afraid of the memory consumption of the docker images running for no reason - but it is not a very complex solution.

2) Whenever a new source is becoming available create a new docker (with different environment variables)

With this solution i have a problem creating the docker. At this moment i have achieved this one, but the service that is starting the dockers (lets call it manager) is just a regular nodejs application that is executing commands on the same server - and i need it to be inside a docker container also.

So the problem here is that i couldn't manage create an ssh connection from the main docker to create my new Docker.

I am not quite sure that both of my solutions are on track and would really appreciate any suggestions for my problem.

Yogevnn
  • 1,430
  • 2
  • 18
  • 37

1 Answers1

1

Your question is a bit unclear, but if you just want to scale a service horizontally you should look into a container orchestration technology that will allow you that - For example Kubernetes. I recommend reading the introduction.

All you would need to do for adding additional service containers is to update the number of desired replicas in the Deployment configuration. For more information read this.

Using kubernetes (or short k8s) you will benefit from deployment automation, self healing and service discovery as well as load balancing capabilities in addition to the horizontal scalability.

There are other orchestration alternatives, too ( e.g. Docker Swarm), but I would recommend to look into kubernetes first.

Let me know if that solves your issue or if you have additional requirements that weren't so clear in your original question.

Links for your follow up questions:

1 - Run kubectl commands inside container

2 - Kubernetes autoscaling based on custom metrics

3 - Env variables in Pods

Community
  • 1
  • 1
Oswin Noetzelmann
  • 9,166
  • 1
  • 33
  • 46
  • Thank you for your answer, i will try to make my problem clearer. My sources are updating (adding and deleting ) dynamically, so i need my listeners to scale dynamically, is it possible using `Kubernetes`? if so i would appreciate a reference to that matter (I have an openshift environment on my servers) – Yogevnn May 12 '17 at 07:12
  • 1
    Hi, yes, if you follow my 2nd link in the answer you will see a command like this for scaling: `kubectl scale deployment nginx-deployment --replicas 10` – Oswin Noetzelmann May 12 '17 at 18:10
  • Thanks! I just checked that and it looks like a good idea. Just some questions about it, can i run this commands from a different Docker? Let's say I'll have a dockerManager that will listen to the RabbitMQ, and when a new source is available the dockerManager will use this command to scale the pods. And another question, can i use environment variables when creating the pod? (by the way, I'm using nodejs) – Yogevnn May 12 '17 at 19:08
  • 1
    I added some links to my answer. Yes, you can connect to the API server from inside the Pod (first link). I am not sure that would be a great design though. You should also look into kubernetes autoscaling based on a custom metric (which allows the system to make a rest call to determine if it requires scaling) - See the second link. And yes, you can use Environment Variables when creating a POD. See the third link. – Oswin Noetzelmann May 12 '17 at 20:08
  • It looks like a good solution, but i couldn't figure out how to set my autoscale with different environment variables for each new pod. Because, as mentioned, each pod will be a listener for the new available source. – Yogevnn May 14 '17 at 04:41
  • 1
    Well, you would have to give more details of your use case for me to understand what your requirements are. In general the assumption is that each added pod will provide additional processing capacity for your new source and you address all pods in the deployment through a single kubernetes service. So all you would have to do is ask kubectl what the current number of replicas is and then tell kubectl to increase the number by 1. As an alternative to kubectl you can also use the rest API of kubernetes. – Oswin Noetzelmann May 14 '17 at 09:51
  • I figured out my problem, the answer will do it. Thank you for your time! – Yogevnn May 17 '17 at 07:56