Running Docker 18.09.7ce with Docker API v1.39 on Ubuntu 18.04 LTS.
I'm trying to set up Traefik 2.2 as a reverse proxy for some swarm services but for some reason Traefik can't connect to the Docker daemon via the TCP port given in the Traefik documentation. These three error messages keep repeating.
level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!: -1" status_code=-1 module=api
level=error msg="Failed to retrieve information of the docker client and server host: Cannot connect to the Docker daemon at tcp://127.0.0.1:2377. Is the docker daemon running?" providerName=docker
level=error msg="Provider connection error Cannot connect to the Docker daemon at tcp://127.0.0.1:2377. Is the docker daemon running?, retrying in 1.461723532s" providerName=docker
It's running on a manager node (I only have one node) and the swarm is working fine, with the API exposed via that TCP port, as shown by the output of the following command.
$ sudo ss --tcp --listening --processes --numeric | grep ":2377"
LISTEN 0 128 *:2377 *:* users:(("dockerd",pid=30747,fd=23))
My architecture is based on this blog post, with a shared overlay network called proxy created with docker network create --driver=overlay proxy.
I tried this but it didn't work, and I can't really find any other related questions. Here are my configuration files:
traefik.toml
[providers.docker]
endpoint = "tcp://127.0.0.1:2377"
swarmMode = true
network = "proxy"
[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.web-secure]
address = ":443"
[certificatesResolvers.le.acme]
email = "my-email#email.com"
storage = "/letsencrypt/acme.json"
caserver = "https://acme-staging-v02.api.letsencrypt.org/directory" # For testing
[certificatesResolvers.le.acme.httpChallenge]
entryPoint = "web"
[log]
level = "DEBUG"
traefik.yml
version: "3.7"
services:
reverse-proxy:
deploy:
placement:
constraints:
- node.role == manager
image: "traefik:v2.2"
ports:
- 80:80
- 443:443
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "/path/to/traefik.toml:/etc/traefik/traefik.toml"
- "letsencrypt:/letsencrypt"
networks:
- "proxy"
networks:
proxy:
external: true
volumes:
letsencrypt:
The only difference I can see is that the blog does not explicitly define an endpoint for the dockers provider. Maybe to removing that?
Related
I am having troubles with microservice health checks in my consul docker setup, which i believe is a symptom of failure in service discovery as i only have one server in my registry.
Below is consul list of members from inside the docker container.
/ # consul members
Node Address Status Type Build Protocol DC Segment
7b1edb14a647 172.19.0.6:8301 alive server 1.7.4 2 dc1 <all>
/ #
Consul container logs repeat the same error below for all the microservices:
consul | 2020-06-16T12:19:11.087Z [WARN] agent: Check socket connection failed: check=service:ffa44b66c4869601c04abdbea6dc5be5 error="dial tcp 172.19.0.6:50044: connect: connection refused"
I am using docker-compose v.3.2 to create a network for containers.
This is a consul service definition
consul:
container_name: consul
ports:
- '8400:8400'
- '8500:8500'
- '8600:53/udp'
image: consul
command: ['agent', '-server', '-bootstrap', '-ui', '-client', '0.0.0.0']
Microservice definition
service-notification:
build:
context: .
dockerfile: apps/service-notification/Dockerfile
args:
NODE_ENV: development
depends_on:
- consul
image: 'service-notification:latest'
restart: always
environment:
- CONSUL_HOST=consul
ports:
- '50044:50044'
I am using CONSUL_HOST env variable to pass in correct host url.
Consul config for the microservice
consul:
host: ${{CONSUL_HOST}}
port: 8500
service:
discoveryHost: ${{CONSUL_HOST}}
healthCheck:
timeout: 1s
interval: 10s
tcp: ${{ service.discoveryHost }}:${{ service.port }}
maxRetry: 5
retryInterval: 5000
tags: ["v1.0.0", "microservice"]
name: io.ultimatebackend.srv.notification
port: 50044
My conclusion so far is that consul server container fails to reach the agents somehow. But i don't know why and i feel like i am missing some obvious peace of consul structure. Please advise.
I was incorrectly configuring my service. The dicoveryHost should be an IP and port of a micro-service inside docker network.
Problem
I'm trying to set up JetBrains Hub, Youtrack, Upsource and Teamcity in a docker container and configure each to be available on their own IP (macvlan) at the default ports 80 redirected to 443 and 443 for HTTPS (so the port numbers do not show up in the browser).
However if I do that I get:
Could not listen on address 0.0.0.0 and port 443
Leaving the teamtools on their default ports 8080 and 8443 works or giving them ports over 2000 seems to work as well.
I checked with fuser 443/tcp and netstat -tulpn but there is nothing running on port 80 or 443. (had to install the packages for those in the container)
I tried setting the listening address to the NICs IP or 172.0.0.1 but this is refused as well:
root#teamtools [ /opt/teamtools ]# docker run --rm -it \
-v /opt/hub/data:/opt/hub/data \
-v /opt/hub/conf:/opt/hub/conf \
-v /opt/hub/logs:/opt/hub/logs \
-v /opt/hub/backups:/opt/hub/backups \
jetbrains/hub:2018.2.9840 \
configure --listen-address=192.168.1.211
* Configuring JetBrains Hub 2018.2
* Setting property 'listen-address' to '192.168.1.211' from arguments
[APP-WRAPPER] Failed to configure Hub: java.util.concurrent.ExecutionException: com.jetbrains.bundle.exceptions.BadConfigurationException: Could not listen on address {192.168.1.211} . Please specify another listen address in property listen-address
Question:
Why can I not set ports 80 and 443?
Why does it work for ports over
2000?
How can I make this work without a reverse proxy?
(reverse-proxy comes with a whole bunch of other issues, that I'm trying to avoid with this setup)
Setup
ESXi 6.7 Host
- vSwitch0 (Allow promiscuous mode: Yes)
- port group: VM Netork (Allow promiscuous mode: No)
- other VMs
- port group: Promiscuous Ports (Allow promiscuous mode: Yes)
- Teamtools VM (Photon OS 2.0, IP: 192.168.1.210)
- firewall based on: https://unrouted.io/2017/08/15/docker-firewall/
- docker/docker-compose
- hub (IP: 192.168.1.211:80/443)
- youtrack (IP: 192.168.1.212:80/443)
- upsource (IP: 192.168.1.213:80/443)
- teamcity-server (IP: 192.168.1.214:80/443)
- teamcity_db (MariaDB 10.3) (IP: 192.168.1.215:3306)
docker-compose.yml
version: '2'
networks:
macnet:
driver: macvlan
driver_opts:
parent: eth0
ipam:
config:
- subnet: 192.168.1.0/24
gateway: 192.168.1.1
services:
hub:
# set a custom container name so no more than one container can be created from this config
container_name: hub
image: "jetbrains/hub:2018.2.9840"
restart: unless-stopped
volumes:
- /opt/hub/data:/opt/hub/data
- /opt/hub/conf:/opt/hub/conf
- /opt/hub/logs:/opt/hub/logs
- /opt/hub/backups:/opt/hub/backups
- /opt/teamtools:/opt/teamtools
expose:
- "80"
- "443"
- "8080"
- "8443"
networks:
macnet:
ipv4_address: 192.168.1.211
domainname: office.mydomain.com
hostname: hub
environment:
- "JAVA_OPTS=-J-Djavax.net.ssl.trustStore=/opt/teamtools/certs/keyStore.p12 -J-Djavax.net.ssl.trustStorePassword=xxxxxxxxxxxxxx"
...
Upsource is running by user jetbrans, which is non-root.
https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html
I'm trying to deploy a private repository on my docker swarm.
I'm following the official docker repository guide to deploy it as a service. I want to be able to use it with https, from outside with a simple url as https://myregistry.mysite.com.
To do so I use following traefik labels in my stack yml file :
traefik.backend: "privateregistry"
traefik.docker.network: "webgateway" # docker overlay external
traefik.enable: "true"
traefik.frontend.entryPoint: "https"
traefik.frontend.redirect.entryPoint: "https"
traefik.frontend.rule: "Host:myregistry.mysite.com"
traefik.port: "5000"
I'm seeing my two frontend/backend in traefik UI but when I access to https://myregistry.mysite.com/v2/ (for example) I've a 500 fatal error. The service log output is
http: TLS handshake error from 10.0.0.68:47796: tls: first record does not look like a TLS handshake
I think I misunderstood something, certs side probably.
Any idea to do that without error ?
Thanks
I suppose you are missing the certificate of the (registry-) server on your client machine. I assume you have two certificate files (used on the server):
myregistry.mysite.com.crt
myregistry.mysite.com.key
Copy myregistry.mysite.com.crt on your client machine to /etc/docker/certs.d/myregistry.mysite.com/ca.crt on Linux or
~/.docker/certs.d/myregistry.mysite.com/ca.crt on Mac. Now you should be able to login from the client:
docker login myregistry.mysite.com
Appendix - Server Setup
Your server setup might look like this:
~/certs/myregistry.mysite.com.crt
~/certs/myregistry.mysite.com.key
~/docker-compose.yml
~/traefik.toml
docker-compose.yml
version: '3'
services:
frontproxy:
image: traefik
command: --api --docker --docker.swarmmode
ports:
- "80:80"
- "443:443"
volumes:
- ./certs:/etc/ssl:ro
- ./traefik.toml:/etc/traefik/traefik.toml:ro
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
docker-registry:
image: registry:2
deploy:
labels:
- traefik.port=5000 # default port exposed by the registry
- traefik.frontend.rule=Host:myregistry.mysite.com
traefik.toml
defaultEntryPoints = ["http", "https"]
# Redirect HTTP to HTTPS and use certificate, see https://docs.traefik.io/configuration/entrypoints/
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ssl/myregistry.mysite.com.crt"
keyFile = "/etc/ssl/myregistry.mysite.com.key"
# Docker Swarm Mode Provider, see https://docs.traefik.io/configuration/backends/docker/#docker-swarm-mode
[docker]
endpoint = "tcp://127.0.0.1:2375"
domain = "docker.localhost"
watch = true
swarmMode = true
To deploy your registry run:
docker stack deploy myregistry -c ~/docker-compose.yml
I am running a single node swarm, I am using traefik to manage all my external connections, and I want to run a registry such that I can connect to it at registry.myhost.com
Now all the examples I can see suggest creating a registry as a normal container rather than a service, however when I do this, I do not have the ability to add it to my traefik network and thus enable it to be found externally.
Do I need to create another internal network and connect both traefik and it to it, and if so, what type. Or do I need to run the registry as a service (I'm only on a single node so volume shouldnt be much of an issue).
And for bonus points, can anyone give me some pointers on how to set it up with s3 as a storage backend?
Overview
You have two machines:
Server: Your (single) Docker Swarm manager node that runs traefik and other Docker containers like the registry.
Client: Another machine that should be able to connect to the registry and push Docker images to it.
I assume you have two certificate files:
registry.myhost.com.crt
registry.myhost.com.key
Server
Your server setup might look like this:
~/certs/registry.myhost.com.crt
~/certs/registry.myhost.com.key
~/docker-compose.yml
~/traefik.toml
docker-compose.yml
version: '3'
services:
frontproxy:
image: traefik
command: --api --docker --docker.swarmmode
ports:
- "80:80"
- "443:443"
volumes:
- ./certs:/etc/ssl:ro
- ./traefik.toml:/etc/traefik/traefik.toml:ro
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
docker-registry:
image: registry:2
deploy:
labels:
- traefik.port=5000 # default port exposed by the registry
- traefik.frontend.rule=Host:registry.myhost.com
- traefik.frontend.auth.basic=user:$$apr1$$9Cv/OMGj$$ZomWQzuQbL.3TRCS81A1g/ # user:password, see https://docs.traefik.io/configuration/backends/docker/#on-containers
traefik.toml
defaultEntryPoints = ["http", "https"]
# Redirect HTTP to HTTPS and use certificate, see https://docs.traefik.io/configuration/entrypoints/
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ssl/registry.myhost.com.crt"
keyFile = "/etc/ssl/registry.myhost.com.key"
# Docker Swarm Mode Provider, see https://docs.traefik.io/configuration/backends/docker/#docker-swarm-mode
[docker]
endpoint = "tcp://127.0.0.1:2375"
domain = "docker.localhost"
watch = true
swarmMode = true
To deploy your registry run:
docker stack deploy myregistry -c ~/docker-compose.yml
Add Another Stack
If your service is not defined in the same docker-compose.yml as traefik you can use the (external) network of the traefik service:
version: '3'
services:
whoami:
image: emilevauge/whoami # A container that exposes an API to show its IP address
networks:
- frontproxy_default # add network of traefik service "frontproxy"
- default
deploy:
labels:
traefik.docker.network: frontproxy_default
traefik.frontend.rule: Host:whoami.myhost.com
traefik.frontend.auth.basic: user:$$apr1$$9Cv/OMGj$$ZomWQzuQbL.3TRCS81A1g/ # user:password, see https://docs.traefik.io/configuration/backends/docker/#on-containers
networks:
frontproxy_default:
external: true # network of traefik service "frontproxy" is defined in another stack
Make sure you add the certificate files of whoami.myhost.com to traefik.toml:
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ssl/registry.myhost.com.crt"
keyFile = "/etc/ssl/registry.myhost.com.key"
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ssl/whoami.myhost.com.crt"
keyFile = "/etc/ssl/whoami.myhost.com.key"
or use a (single) wildcard certificate *.myhost.com
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ssl/myhost.com.crt"
keyFile = "/etc/ssl/myhost.com.key"
See https://docs.traefik.io/configuration/entrypoints/ for further information.
Client
Copy registry.myhost.com.crt on your client machine to /etc/docker/certs.d/registry.myhost.com/ca.crt on Linux or
~/.docker/certs.d/registry.myhost.com/ca.crt on Mac. Now you should be able to login from the client:
docker login -u user -p password registry.myhost.com
Copy an image from Docker Hub to your registry
On your client run:
docker pull hello-world:latest
docker tag hello-world:latest registry.myhost.com/hello-world:latest
docker push registry.myhost.com/hello-world:latest
Now you can pull this image on another machine (for example on the server):
docker pull registry.myhost.com/hello-world:latest
Don't forget to add registry.myhost.com.crt on that client machine, too.
I try to setup traefik with a Docker Swarm. I have to VMs - one manger-node and one worker-node.
In addition I have created a external network with:
docker network create --driver=overlay proxy-net
I start traefik as a service within my manager-node with the following docker-compose.yml file:
version: '3'
services:
traefik:
image: traefik:v1.4.4
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- $PWD/management/traefik/traefik.toml:/etc/traefik/traefik.toml
ports:
- 80:80
- 8100:8080
deploy:
placement:
constraints:
- node.role == manager
networks:
default:
external:
name: proxy-net
My traefik.toml file looks like this:
Debug : "DEBUG"
defaultEntryPoints = ["http"]
[entryPoints]
[entryPoints.http]
address = ":80"
[web]
address = ":8080"
[docker]
watch = true
swarmmode = true
domain = "mydomain.com"
exposedbydefault = true
When I now start a new service (e.g. emilevauge/whoami) with:
docker service create \
--name whoami1 \
--publish mode=host,target=80,published=8002 \
--network proxy-net \
--label traefik.docker.network=proxy-net \
--label traefik.frontend.rule=Host:whoami.mydomain.com \
--label traefik.port=8002 \
emilevauge/whoami
The service is seen by the traefik web frontend. So at first every thing looks fine. I can access the service directly on my worker node on port 8002.
But traefik does not seem to be able to proxy this service. When I browse my endpoint URL (whomai.mydomain.com) I get the answer:
Bad Gateway
The traefik logfile (logLevel=DEBUG) shown messages like this:
proxy_traefik.1.zl50yv6got5f#tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Filtering container without port and no traefik.port label swarmpit_app.1 : strconv.Atoi: parsing "": invalid syntax"
proxy_traefik.1.zl50yv6got5f#tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Filtering container without port and no traefik.port label proxy_traefik.1 : strconv.Atoi: parsing "": invalid syntax"
proxy_traefik.1.zl50yv6got5f#tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Filtering container without port and no traefik.port label swarmpit_db.1 : strconv.Atoi: parsing "": invalid syntax"
proxy_traefik.1.zl50yv6got5f#tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Validation of load balancer method for backend backend-whoami1-whoami1-whoami1 failed: invalid load-balancing method ''. Using default method wrr."
proxy_traefik.1.zl50yv6got5f#tocidoc001 time="2017-12-03T20:09:28Z" level=debug msg="Configuration received from provider docker: {"backends":{"backend-whoami1-whoami1-whoami1":{"servers":{"service-0":{"url":"http://10.0.1.5:8002","weight":0}},"loadBalancer":{"method":"wrr"}}},"frontends":{"frontend-whoami1-whoami1-whoami1":{"entryPoints":["http"],"backend":"backend-whoami1-whoami1-whoami1","routes":{"service-whoami1":{"rule":"Host:whoami.mydomain.com"}},"passHostHeader":true,"priority":0,"basicAuth":[],"headers":{}}}}"
I played around several hours with different configurations. I also read the very concise documentation about traefik and docker-swarm. But I don't get any idea what I'm doing wrong.
Can any body help me with some tips how to better understand the problem?
I think it is not working because you Træfik service is not on the same docker network as your whoami1 service.
You should try to add proxy-net network to your Træfik service in your compose file.
There is a warning in Træfik documentation at the end of this page https://docs.traefik.io/configuration/backends/docker/
when running inside a container, Træfik will need network access through:
docker network connect <network> <traefik-container>
As already mentioned, they need to be in the same overlay network which is not ingress. The ingress network is only for manager nodes.
Further more, your traefik service is not assigned to the proxy-net network. You're creating proxy-net in your traefik config part, but don't assigned it to it
version: '3'
services:
traefik:
image: traefik:v1.4.4
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- $PWD/management/traefik/traefik.toml:/etc/traefik/traefik.toml
ports:
- 80:80
- 8100:8080
networks:
- proxy-net
deploy:
placement:
constraints:
- node.role == manager
networks:
proxy-net:
driver: overlay
Further more, you should create a config with docker config create. Otherwise with $PWD/management/traefik/traefik.toml you need to copy the traefik.toml file to every manager node.
Append your compose file with
configs:
traefik_conf_v1:
file: ./traefik.toml
and your traefik part with
configs:
- source: traefik_conf_v1
target: /etc/traefik/traefik.toml
Now back to your problem.
What's your service is missing is the label to the backend. Otherwise traefik doesn't know where the service is running (network assignment isn't enough!).
docker service create \
--name whoami1 \
--publish mode=host,target=80,published=8002 \
--network proxy-net \
--label traefik.backen=whoami1 \
--label traefik.docker.network=proxy-net \
--label traefik.frontend.rule=Host:whoami.mydomain.com \
--label traefik.port=8002 \
emilevauge/whoami
This should work. And when it does, stop publishing ports of your services. That makes everything complicated when you're in a hurry and need to scale. Remember, work balancing is handle by the swarm itself.
And yeah, dynamic flexible reverse proxys is still a problem nowadays :)
Remember, you got your entry points on manager nodes with traefik, but not on the worker nodes.
I finally I solved this issue. It was actually not a Traefik problem.
The problem was, that both VMs from my provider have the same private IPv4 address.
To register and join the docker-swarm it is important to provide the public IPv4 addresses with the option --advertise-addr
To register the swarm I have to run:
docker swarm init --advertise-addr [manager-ip-address]
to join the swarm by a worker-node also the public IPv4 address need to be set explicitly:
docker swarm join \
--token SWMTKN-1-xxxxxxxxxxxxxxxxxxxx-xxxxxxxx \
--advertise-addr [worker-ip-address]\
[manager-ip-address]:2377
I would say that your setup of service labels was wrong. Traefik redirects requests to swarm service port so it should go to port 80, not to published port 8002. I think that correct service create command should be:
docker service create \
--name whoami1 \
--publish mode=host,target=80,published=8002 \
--network proxy-net \
--label traefik.docker.network=proxy-net \
--label traefik.frontend.rule=Host:whoami.mydomain.com \
--label traefik.port=80 \
emilevauge/whoami
And publishing the 80 port for whoami service is not needed.