2

We have a web application running on AWS with the following architecture:

  • 1 elasticseach cluster with 2 data nodes
  • 1 auto-scaling load-balanced cluster of web servers

As elasticsearch does some clever internal load balancing we could just point all the web servers at one of the data nodes. But this would create a single point of failure - if that node goes down then I'm not going to get any query results.

My solution thus far has been to have elasticsearch running on each web server as non-data nodes. Each web server queries its local elasticsearch node, which in turn farms the request off to one of the data nodes. This seems to be the suggested approach on the elasticsearch website

This is great in that if one of the data nodes fails in some way we don't lose the ability to serve search queries. However, it does mean elasticsearch is using resources on each web server, and if we migrate to using elastic beanstalk (which I'm keen to do) then we'll need to some how get elasticsearch installed on our web instances. EDIT: I've succeeded with this now, but have yet to figure out how to specify a different config for each environment.

Is there another way to avoid a single point of failure without having elasticsearch running on each web server?

I thought about using a load balancer in front of the data nodes to serve queries from the web servers, but that would also mean opening the cluster up to public access without setting up VPC to restrict access.

Is there a simpler solution I'm missing?

user1207727
  • 1,543
  • 1
  • 15
  • 18
  • If you have 2 data nodes with 1 replica, one node can go down and you can still serve queries without even losing documents. Am I missing anything in your question? – javanna Sep 13 '13 at 21:41
  • You are correct. However, without using a local non-data node I'd lose the built-in ability to handle a node going down. Ie I'd have to detect the failure to connect and switch to a working data node. Maybe that's not such a big deal. It just seems in-optimal – user1207727 Sep 16 '13 at 09:07
  • You mean that you want to use the client node as some kind of load balancer? Client libraries should support more addresses with round-robin and hopefully fallback to the other addresses if the first one doesn't work. Makes sense? – javanna Sep 16 '13 at 10:00
  • Yes that makes sense. We're using elasticsearch with a symfony2 php application, and unfortunately it doesn't support specifying more than one connection. So for the live environment we specify localhost, and leave elasticsearch itself to choose which data node to send the query to. It works great and is quite a neat solution. I was just wondering if I was missing another solution which didn't involve having elasticsearch running on each web server. – user1207727 Sep 18 '13 at 15:20
  • I think you should add this tricky bit to your question as it makes the difference. You could just use a load balancer in front of elasticsearch (Nginx or Apache), your issue doesn't have anything to do with elasticsearch being exposed to single point of failures though! – javanna Sep 18 '13 at 15:51
  • Thanks, but I beg to differ. The reason for mentioning the avoidance of a single point of failure was so that someone wouldn't say 'Just have all your web servers directly connect to one of the data nodes'. Ie this would solve the problem of not having to run elasticsearch on each web server, AND it would provide load balancing (because elasticsearch load balances internally), but if that one data node went down then the whole site would go down with it. I will try to clarify the question though. – user1207727 Sep 18 '13 at 16:17
  • I tip my hat to you by the way - just realised that you work on elasticsearch! – user1207727 Sep 18 '13 at 16:31

1 Answers1

0

I don't think this directly answers your question, but if you are still ok with running ES on your web server nodes, you can customize the software that is installed using the .ebextensions mechanism, which allows you to run scripts and/or install packages when new Elastic Beanstalk instances are started up. If this isn't sufficient you can start your Elastic Beanstalk instances using a custom AMI.

Also, you may not be aware that you can run Elastic Beanstalk in a VPC.

Ken Liu
  • 22,503
  • 19
  • 75
  • 98
  • Thanks Ken. I've managed to get elasticsearch installed using the .ebextensions route. The thing I'm battling with now is how to specify a different elasticsearch config file for each environment, ie dev / staging / production – user1207727 Sep 18 '13 at 15:22
  • this may help you with configuring per-environment settings: http://stackoverflow.com/questions/16585898/referencing-env-variables-from-elastic-beanstalk-ebextensions-config-files – Ken Liu Sep 19 '13 at 03:37