Consul template is a simple, yet powerful tool. When initiated, it reads one or more template files and queries Consul for all data needed to render them. Typically, you run consul-template as a daemon which will fetch the initial values and then continue to watch for updates, re-rendering the template whenever there are relevant changes in the cluster.

Finally, the template can also run arbitrary commands after the update process completes. For example, it can send the HUP signal to the load balancer service after a configuration change has been made.

https://developer.hashicorp.com/consul/tutorials/developer-configuration/consul-template

The problem in this scenario is that the number of Consul servers, which is currently 3, does not yet equal the value of the parameter bootstrap_expect, which is set to 5. If this setting is not updated, Consul will not bootstrap the cluster until 5 nodes are detected.

The other option is to modify the value of the bootstrap_expect parameter to 3, indicating that you want only 3 nodes in the cluster. Since you have three nodes provisioned, Consul would bootstrap the cluster, and Consul would then be available to use.

Incorrect Answers

Changing the raft_multiplier actually increases the time that Consul will detect and react to node failures, and has no effect on bootstrapping the cluster in this scenario.

Changing the bind_address here won't help since 0.0.0.0 is effectively the same thing as defining the single interface since 0.0.0.0 will use all interfaces and the IPv4 address associate with the single interface.

Enhanced Read Scalability with non-voting members is only available in Enterprise versions of Consul. Consul Enterprise provides the ability to scale clustered Consul servers to include voting and non-voting servers. Non-voting servers still receive data from the cluster replication, however, they do not take part in quorum election operations. Expanding your Consul cluster in this way can scale reads without impacting write latency.

https://developer.hashicorp.com/consul/docs/enterprise/read-scale

https://developer.hashicorp.com/consul/docs/agent/config/config-files#_non_voting_server

This service configuration will register both the front-end service, along with a front-end-sidecar-proxy service.

https://developer.hashicorp.com/consul/docs/connect/registration/sidecar-service

The gossip protocol is powered by Serf, which uses a symmetric key, or shared secret, cryptosystem.

The RPC system supports using end-to-end TLS with optional client authentication.

https://developer.hashicorp.com/consul/docs/security

The key to this question is the phrase It seems some nodes are being provisioned faster than others. Nodes are spinning up faster than others, and since the configuration file only includes a join statement, the nodes being deployed faster cannot communicate with the other nodes, therefore the Consul agent fails. join does not reattempt communication with the listed node(s), therefore the cluster is never bootstrapped.

If the configuration file is updated with retry_join, the nodes will continue to reattempt communication with the other nodes. Assuming all three are successfully provisioned, the cluster will eventually bootstrap once all three nodes are available.

https://developer.hashicorp.com/consul/docs/install/bootstrapping#bootstrapping-the-servers

The key features of Consul are:

Service Discovery: Clients of Consul can register a service, such as api or mysql, and other clients can use Consul to discover providers of a given service. Using either DNS or HTTP, applications can easily find the services they depend upon.

Health Checking: Consul clients can provide any number of health checks, either associated with a given service ("is the webserver returning 200 OK"), or with the local node ("is memory utilization below 90%"). This information can be used by an operator to monitor cluster health, and it is used by the service discovery components to route traffic away from unhealthy hosts.

KV Store: Applications can make use of Consul's hierarchical key/value store for any number of purposes, including dynamic configuration, feature flagging, coordination, leader election, and more. The simple HTTP API makes it easy to use.

Secure Service Communication: Consul can generate and distribute TLS certificates for services to establish mutual TLS connections. Intentions can be used to define which services are allowed to communicate. Service segmentation can be easily managed with intentions that can be changed in real-time instead of using complex network topologies and static firewall rules.

Multi Datacenter: Consul supports multiple datacenters out of the box. This means users of Consul do not have to worry about building additional layers of abstraction to grow to multiple regions.

Consul is designed to be friendly to both the DevOps community and application developers, making it perfect for modern, elastic infrastructures.

Information obtained from the Consul website found here.

The ssh check is the only TCP check listed. The TCP port is defined by the tcp argument, in this case, localhost:22.

web-alias is an alias-check for a local service.

mem-util is a gRPC check for the specific service named my_service

web-app is a TTL health check

https://developer.hashicorp.com/consul/docs/discovery/checks#check-definition

The -once flag will tell the process to run once and then quit.

https://github.com/hashicorp/consul-template#once-mode

The consul reload command triggers a reload of configuration files for the agent. Not all configuration options are reloadable. To see what configuration options are reloaded when a reload command is issued, check out this link here.

https://developer.hashicorp.com/consul/commands/reload