We are trying to working on creating a multi node cluster, the elk package diagrams show the “coordinating node” but the cluster doesn’t like it is getting created. The reason we wanted these was to have Kibana use them so we can load balance the ES cluster for querying.
Please help on how these are created if they are or point us in the right direction to create then using the terraform you got.
Hey there,
Are you looking at our ELK multi-cluster example and seeing that it doesn’t appear to deploy coordinating nodes? If so - you would be correct! For the example, we deploy an Elasticsearch cluster where every node is both a master as well as a data node to keep the example simple. As an aside, coordinating nodes is another name for master nodes. See here
The way that you determine what kind of Elasticsearch node you are deploying is by modifying the Elasticsearch yml configuration file. See the section I’ve highlighted in our example config file.
Our multi-cluster example code brings up one Elasticsearch cluster where each node is both a master and a data node. In production, you would use our Elasticsearch Cluster module two times with two different configuration files. One of the configuration files would create a data only Elasticsearch cluster while the other would create a master only cluster.
The two “instances” of the Elasticsearch-cluster module would automatically auto-discover one another because we’re using the EC2 auto-discovery plugin for Elasticsearch. It’s configured here.
Now, when we deploy our reference architecture, if customers request a self managed ELK stack, then for their production environments we deploy exactly what I described above: two instances of Elasticsearch-cluster with slightly different config files instructing them to be data nodes in one case and master nodes in the other.
Let me know if that helps!
Just wanted to clarify; a dedicated coordinating node is when a node is not a master, data, or ingest node.
Interesting. I think this is something that I misunderstood earlier. Thanks for the clarification!
We based our example elasticsearch.yml from the default elasticsearch.yml that you get when you apt-get install elasticsearch (in our case, version 6.2.4). That is where the comment that I referenced here comes from.
I must have misunderstood what they meant by “coordinator” (in quotes).
As I mentioned in the original post, our intention was to use the coordinating node as load balancer to the ES Cluster. Reading through your answer doesn’t seems the solution, so please clarify if there is a way to do it using your framework.
@kishan4cloudauto at this time I don’t think our package is able to achieve your exact scenario without some modification. You should be able to achieve this by starting off with the mulit-cluster example and creating a new node with node.master , node.data and node.ingest all set to false (you might need to create a new AMI template).
If you want to use it as a load balancer then you need to update the Kibana configuration to point to the new co-ordinating node. You might need to keep the existing load balancer around so Logstash can still forward logs.
Regards
Toni