I have a problem with traefik. The situation:
I have a docker gitlab with gitlab-registry activated, no problem with that.
My gitlab answer on port 80 and registry on port 80 as well, and this is not a problem nether (tried with port mapping and worked well.
Now I try to configure these guys in traefik and that's when problems occur. When ther is just one router and one service, all good but when I try to add the registry I have an error:
The involved part in the docker-compose file:
labels:
- traefik.http.routers.gitlab.rule=Host(`git.example.com`)
- traefik.http.routers.gitlab.tls=true
- traefik.http.routers.gitlab.tls.certresolver=lets-encrypt
- traefik.http.routers.gitlab.service=gitlabservice
- traefik.http.services.giltabservice.loadbalancer.server.port=80
- traefik.http.routers.gitlab.entrypoints=websecure
- traefik.http.routers.gitlabregistry.rule=Host(`registry.example.com`)
- traefik.http.routers.gitlabregistry.tls=true
- traefik.http.routers.gitlabregistry.tls.certresolver=lets-encrypt
- traefik.http.routers.gitlabregistry.service=gitlabregistryservice
- traefik.http.services.giltabregistryservice.loadbalancer.server.port=80
- traefik.http.routers.gitlabregistry.entrypoints=websecure
And the errors I have in traefik:
time="2021-03-27T08:48:20Z" level=error msg="the service "gitlabservice#docker" does not exist" entryPointName=websecure routerName=gitlab#docker
time="2021-03-27T08:48:20Z" level=error msg="the service "gitlabregistryservice#docker" does not exist" entryPointName=websecure routerName=gitlabregistry#docker
Thanks in advance if you have any idea.
Related
I am trying to run Gitlab Docker image on mac Big Sur locally following the steps from documentation: https://docs.gitlab.com/ee/install/docker.html . And always cannot see locally https://gitlab.example.com. I tried both Gitlab EE and CE, and different versions of images, including latest. I also tried to use usual Docker basic run and docker-compose. I also updated Docker Desktop to latest version 4.10.1. I refined the logs and for all cases I have same error in Gitally log below:
~/gitlab/logs/gitaly/current:
{"level":"warning","msg":"[core] [Channel #2 SubChannel #4] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0\",\n \"ServerName\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0: connect: no such file or directory\"","pid":344,"system":"system","time":"2022-07-26T09:57:38.226Z"}
{"level":"warning","msg":"[core] [Channel #3 SubChannel #5] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1\",\n \"ServerName\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1: connect: no such file or directory\"","pid":344,"system":"system","time":"2022-07-26T09:57:38.228Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","344","/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","supervisor.pid":400,"time":"2022-07-26T09:57:38.228Z"}
{"address":"/var/opt/gitlab/gitaly/gitaly.socket","level":"info","msg":"listening at unix address","time":"2022-07-26T09:57:38.234Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","344","/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1"],"supervisor.name":"gitaly-ruby.1","supervisor.pid":401,"time":"2022-07-26T09:57:38.234Z"}
But I am not sure that this is the case - also checked other logs and seems to me no errors in other logs - but of course I will be happy to provide more logs if you want.
This is my 'docker ps -a' status, docker version: 4.10.1, is run with docker-compose:
web:
image: 'gitlab/gitlab-ce:latest'
container_name: 'gitlab'
restart: unless-stopped
hostname: 'gitlab.example.com'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://gitlab.example.com'
ports:
- '80:80'
- '443:443'
- '22:22'
volumes:
- '$GITLAB_HOME/config:/etc/gitlab'
- '$GITLAB_HOME/logs:/var/log/gitlab'
- '$GITLAB_HOME/data:/var/opt/gitlab'
list of docker active containers says container is healthy - but actually I see nothing in browser via https://gitlab.example.com.
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b0fb6252c7f3 gitlab/gitlab-ce:latest "/assets/wrapper" 3 days ago Up 3 minutes (healthy) 0.0.0.0:22->22/tcp, 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp gitlab
I also tried to add docker host IP to my hosts file - but no success:
192.168.31.182 host.docker.internal
192.168.31.182 gateway.docker.internal
192.168.31.182 gitlab.example.com
I still see nothing in browser by https://gitlab.example.com
Adding more info to #sytech "That's just a placeholder URL... Did you replace gitlab.example.com with your actual GitLab instance URL?"
You either need to add an /etc/hosts, setting to route gitlab.example.com to localhost or modify that to be localhost
hostname: 'localhost'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'localhost'
These settings are telling the nginx proxy what url should be forwarded to gitlab. That's useful when you're running multiple sites from one server. If you are just running it locally, you should just use localhost.
I also tried to add docker host IP to my hosts file - but no success:
You are forwarding the ports from the docker ip. Just use localhost. It probably didn't work because the docker IP you're looking at is for a different docker network.
I am struggling with Go requests between containers.
The issue that I have that the rest of my containers can send request to the node Container that give response, but when I send request from my GoLang application to node I get that refuse error "dial tcp 172.18.0.6:3050: connect: connection refused".
So my whole docker set up is:
version: "3.3"
services:
##########################
### SETUP SERVER CONTAINER
##########################
node:
# Tell docker what file to build the server from
image: myUserName/mernjs:node-dev
build:
context: ./nodeMyApp
dockerfile: Dockerfile.dev
# The ports to expose
expose:
- 3050
# Port mapping
ports:
- 3050:3050
# Volumes to mount
volumes:
- ./nodeMyApp/src:/app/server/src
# Run command
# Nodemon for hot reloading (-L flag required for polling in Docker)
command: nodemon -L src/app.js
# Connect to other containers
links:
- mongo
# Restart action
restart: always
react:
ports:
- 8000:8000
build:
context: ../reactMyApp
dockerfile: Dockerfile.dev
volumes:
- ../reactMyApp:/usr/src/app
- /usr/src/app/node_modules
- /usr/src/app/.next
restart: always
environment:
- NODE_ENV=development
golang:
build:
context: ../goMyApp
environment:
- MONGO_URI=mongodb://mongo:27017
# Volumes to mount
volumes:
- ../goMyApp:/app/server
links:
- mongo
- node
restart: always
So my React app can send the request to "http://node:3050/api/greeting/name" and it get the response even that react app is not linked to the node app but when Golang app sends request to node docker container it gets connection refuse message GetJson err: Get "http://node:3050/api/greeting/name": dial tcp 172.18.0.6:3050: connect: connection refused
func GetJson(url string, target interface{}) error {
r, err := myClient.Get(url)
if err != nil {
fmt.Println("GetJson err: ", err)
return err
}
defer r.Body.Close()
return json.NewDecoder(r.Body).Decode(target)
}
type ResultsDetails struct {
Greeting string `bson:"greatingMessage" json:"greatingMessage"`
Message string `bson:"message" json:"message"`
}
func GetGreetingDetails(name string) ResultsDetails {
var resp ResultsDetails
GetJson("http://node:3050/api/greeting/"+name, &resp)
return resp
}
So how do I solve the Golang request to another Docker Node Container when docker doesnt see the host as the name of my container 'node'?
Update:
By accident i put Golang port, which it doenst run on any port since it is application that checks on database records. So it hasnt got any api, therefore it is not running on any port.
Is that could be the problem why my golang application cannot communication to other containers?
Since i have also another golang application which is api application and it is running on 5000 port and it is well communicating to my node application?
Network info:
After checking the network if node and golang share the same network and the answer is yes. All containers share the same network
(Unrelated to my issue) To anyone who has "dial tcp connection refused" issue I suggest to go though that guide https://maximorlov.com/4-reasons-why-your-docker-containers-cant-talk-to-each-other/. Really helpful. To those who this guide wont help prob read bellow this, maybe you trying to request the container api after just containers were built :D
For those who was interested what was wrong:
Technically reason why I was getting this error is because of the request that I was trying to run, was just when all containers were built.
I believe there is some delay to the network after containers are built. Thats why there host was throwing "dial tcp 172.18.0.6:3050: connect: connection refused" I've run that test on other containers that could possibly send request to that node container and they were all failing after the build time. But when re-requesting after few seconds all worked out.
Sorry to bother you guys. I really spent 3 days into this issue. And I was looking into completely wrong direction. Never thought that the issue is that silly :D
Thanks for you time.
I've met the same error in my harbor registry service.
After I docker exec -it into the container, and check if the service is available, and finally I found that http_proxy has been set.
Remove the http_proxy settings for docker service, then it works like a charm.
Failed on load rest config err:Get "http://core:8080/api/internal/configurations": dial tcp 172.22.0.8:8080: connect: connection refused
$docker exec -it harbor-jobservice /bin/bash
$echo $http_proxy $https_proxy
I am having this error, after running skaffold dev.
Step 1/6 : FROM node:current-alpine3.11
exiting dev mode because first build failed: unable to stream build output: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: read udp 192.168.49.2:35889->192.168.49.1:53: i/o timeout. Please fix the Dockerfile and try again..
Here is skaffold.yml
apiVersion: skaffold/v2beta11
kind: Config
metadata:
name: *****
build:
artifacts:
- image: 127.0.0.1:32000/auth
context: auth
docker:
dockerfile: Dockerfile
deploy:
kubectl:
manifests:
- infra/k8s/auth-depl.yaml
local:
push: false
artifacts:
- image: 127.0.0.1:32000/auth
context: auth
docker:
dockerfile: Dockerfile
sync:
manual:
- src: "src/**/*.ts"
dest: .
I have tried all possible solutions I saw online, including adding 8.8.8.8 as the DNS, but the error still persists. I am using Linux and running ubuntu, I am also using Minikube locally. Please assist.
This is a Community Wiki answer, posted for better visibility, so feel free to edit it and add any additional details you consider important.
In this case:
minikube delete && minikube start
solved the problem but you can start from restarting docker daemon. Since this is Minikube cluster and Skaffold uses for its builds Minikube's Docker daemon, as suggested by Brian de Alwis in his comment, you may start from:
minikube stop && minikube start
or
minikube ssh
su
systemctl restart docker
I searched for similar errors and in many cases e.g. here or in this thread, setting up your DNS to something reliable like 8.8.8.8 may also help:
sudo echo "nameserver 8.8.8.8" >> /etc/resolv.conf
in case you use Minikube you should first:
minikube ssh
su ### to become root
and then run:
echo "nameserver 8.8.8.8" >> /etc/resolv.conf
The following error message:
Please fix the Dockerfile and try again
may be somewhat misleading in similar cases as Dockerfile is probably totally fine, but as we can read in other part:
lookup registry-1.docker.io on 192.168.49.1:53: read udp 192.168.49.2:35889->192.168.49.1:53: i/o timeout.
it's definitely related with failing DNS lookup. This is well described here as well known issue.
Get i/o timeout
Get https://index.docker.io/v1/repositories//images: dial tcp: lookup on :53: read udp :53: i/o timeout
Description
The DNS resolver configured on the host cannot resolve the registry’s
hostname.
GitHub link
N/A
Workaround
Retry the operation, or if the error persists, use another DNS
resolver. You can do this by updating your /etc/resolv.conf file
with these or other DNS servers:
nameserver 8.8.8.8 nameserver 8.8.4.4
I'm trying to start the change of backends to be compatible with traefik v2.0.
The old configuration was:
labels:
- traefik.port=8500
- traefik.docker.network=proxy
- traefik.frontend.rule=Host:consul.{DOMAIN}
I assumed, the network is not necessary anymore, it would change the new traefik for:
- traefik.http.routers.consul-server-bootstrap.rule=Host('consul.scoob.thrust.com.br')
But how I set, that this should forward to my backend at port 8500? and not 80 where the entrypoint was reached at Traefik?
My goal would try to accomplish something like this:
https://docs.traefik.io/user-guide/cluster-docker-consul/#migrate-configuration-to-consul
Is it still possible?
I saw, there was no --consul or storeconfig command in v2.0
You need traefik.http.services.{SERVICE}.loadbalancer.server.port
labels:
- "traefik.http.services.{SERVICE}.loadbalancer.server.port=8500"
- "traefik.docker.network=proxy"
- "traefik.http.routers.{SERVICE}.rule=Host(`{DOMAIN}`)"
Replace {SERVICE} with the name of your service.
Replace {DOMAIN} with your domain name.
If you want to remove the proxy network you'll need to look at https://docs.traefik.io/v2.0/providers/docker/#usebindportip
We have a treafik installation on docker swarm with several services balanced through traefik. Each service has at least two backends balanced with wrr and a healthcheck.
Is there a way (api, rest endpoint, logfile whatever) to find out which frontends have dead backends? By dead I mean on which backends treafik has detected via healthcheck that they are not eligible for balancing?
What is the best practice for this?
I see two ways of getting that info:
Traefik log
Look at Traefik log which provides traces for healthchecks:
time="2019-03-05T22:19:35Z" level=debug msg="Refreshing health check for backend: backend-web-so-55004614",
time="2019-03-05T22:19:35Z" level=warning msg="Health check still failing. Backend: \"backend-web-so-55004614\" URL: \"http://192.168.80.2:80\" Reason: received error status code: 404",
time="2019-03-05T22:19:36Z" level=debug msg="Refreshing health check for backend: backend-web-so-55004614",
time="2019-03-05T22:19:36Z" level=warning msg="Health check still failing. Backend: \"backend-web-so-55004614\" URL: \"http://192.168.80.2:80\" Reason: received error status code: 404",
Traefik /metrics
If it is not convenient to parse Traefik logs, you could active Traefik Prometheus metrics (which is enabled by default):
docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p "80:80" -p "8080:8080" traefik --api --docker
Then you can make an HTTP query on http://localhost:8080/metrics and look for lines containing _backend_server_up. Each of these lines tells you that your backend is up and healthy. If a backend is missing, that means it is unhealthy or stopped:
traefik_backend_server_up{backend="backend-robots",url="http://172.23.0.3:80"} 1
traefik_backend_server_up{backend="backend-smtp-ui",url="http://172.25.0.3:8025"} 1
traefik_backend_server_up{backend="backend-varnish-admin",url="http://172.23.0.8:6085"} 1
traefik_backend_server_up{backend="backend-varnish-http",url="http://172.23.0.8:6081"} 1
traefik_backend_server_up{backend="backend-web-apps",url="http://172.21.0.2:80"} 1
traefik_backend_server_up{backend="backend-web-report",url="http://172.19.0.6:80"} 1
You could have a script querying this URL or you could install Prometheus which has alerting rules