Unable to get Consul Connect to work with Nomad - docker

Question

I'm trying to get Consul Connect side car envoy to work but the health checks for sidecar keeps failing.
I'm using following versions of Consul and Nomad
Consul : 1.7.3
Nomad : 0.11.1
CNI Plugins : 0.8.6
My setup looks like follows.
1 Consul Server running consul in docker container.
docker run -d --net=host --name=server -v /var/consul/:/consul/config consul:1.7 agent -server -ui -node=server-1 -bind=$internal_ip -ui -bootstrap-expect=1 -client=0.0.0.0
internal_ip is the internal IP address of my GCP VM.
1 Nomad Server with Consul Agent in client mode
nohup nomad agent -config=/etc/nomad.d/server.hcl &
docker run -d --name=consul-client --net=host -v ${volume_path}:/consul/config/ consul:1.7 agent -node=$node_name -bind=$internal_ip -join=${server_ip} -client=0.0.0.0
interal_ip is the internal IP address of GCP VM and server_ip is the internal IP address of Server VM.
2 Nomad Client with Consul Agent in client mode
nohup nomad agent -config=/etc/nomad.d/client.hcl &
docker run -d --name=consul-client --net=host -v ${volume_path}:/consul/config/ consul:1.7 agent -node=$node_name -bind=$internal_ip -join=${server_ip} -client=0.0.0.0
On Nomad clients, I also have consul binary available in path.
Now I'm trying to deploy the sample Nomad and Consul Connect job from here
job "countdash" {
datacenters = ["dc1"]
group "api" {
network {
mode = "bridge"
}
service {
name = "count-api"
port = "9001"
connect {
sidecar_service {}
}
}
task "web" {
driver = "docker"
config {
image = "hashicorpnomad/counter-api:v1"
}
}
}
group "dashboard" {
network {
mode = "bridge"
port "http" {
static = 9002
to = 9002
}
}
service {
name = "count-dashboard"
port = "9002"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "count-api"
local_bind_port = 8080
}
}
}
}
}
task "dashboard" {
driver = "docker"
env {
COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
}
config {
image = "hashicorpnomad/counter-dashboard:v1"
}
}
}
}
The docker container for service and sidecar gets started and gets registered in Consul, but I'm unable to access any of the service.
I SSH onto the Nomad Client node and can see the container running.
Odd thing I noticed is that I cannot see port forwarded to the host
I cannot access it via curl from host.
I tried doing curl $internal_ip:9002 but it didn't work.
I checked if Nomad created any new bridge network since that's what I used as mode in the network stanza but there are no new networks.
Is there anything that I'm missing in my setup ?

Answer 1

Have you tried setting COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}" to COUNTING_SERVICE_URL = "http://localhost:8080", since that is the local bind port that the envoy proxy will be listening on to forward traffic to the count-api.
An example of a working connect setup can be found at https://github.com/hashicorp/video-content/tree/master/nomad-connect-integration/nomad_jobs