I'm setting up a VPS via ansible to run a web app that's already in a docker container. That part is up and running. Moving on to the proxy part, I added jwilder/nginx-proxy and jrcs/letsencrypt-nginx-proxy-companion to my playbook. In general that is working. The SSL cert were created, and I can access the web app through the proxy. Unfortunately the images don't seem to be getting through. I think I need to tweak the "Connection" to be "keep-alive". Though I've yet to figure out how to get that set. Any suggestions? Thanks in advance.
nginx-proxy env vars:
VIRTUAL_HOST: "{{ webserver_hostname }}"
VIRTUAL_PORT: 9000
LETSENCRYPT_HOST: "{{ webserver_hostname }}"
LETSENCRYPT_EMAIL: "{{ letsencrypt_email }}"
SSL_POLICY: Mozilla-Modern
docker images in use:
jwilder/nginx-proxy
jrcs/letsencrypt-nginx-proxy-companion
custom docker with app.
Note:
I have removed the letsencrypt-nginx-proxy-companion and the image
problem remains.
I don't have problems with the images if I directly access the port on web app's docker.
Browser responses:
Safari:
"Failed to load resource: The network connection was lost."
Error: "The network connection was lost."
Chrome:
Status is "(failed)".
Checking the response header the status is 200.
Under the timing, Connection Start is "Stalled".
Firefox:
Status is 200, but Response payload is empty.
Thanks again for taking a look.
Related
I am trying to run Gitlab Docker image on mac Big Sur locally following the steps from documentation: https://docs.gitlab.com/ee/install/docker.html . And always cannot see locally https://gitlab.example.com. I tried both Gitlab EE and CE, and different versions of images, including latest. I also tried to use usual Docker basic run and docker-compose. I also updated Docker Desktop to latest version 4.10.1. I refined the logs and for all cases I have same error in Gitally log below:
~/gitlab/logs/gitaly/current:
{"level":"warning","msg":"[core] [Channel #2 SubChannel #4] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0\",\n \"ServerName\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0: connect: no such file or directory\"","pid":344,"system":"system","time":"2022-07-26T09:57:38.226Z"}
{"level":"warning","msg":"[core] [Channel #3 SubChannel #5] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1\",\n \"ServerName\": \"/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1: connect: no such file or directory\"","pid":344,"system":"system","time":"2022-07-26T09:57:38.228Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","344","/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.0"],"supervisor.name":"gitaly-ruby.0","supervisor.pid":400,"time":"2022-07-26T09:57:38.228Z"}
{"address":"/var/opt/gitlab/gitaly/gitaly.socket","level":"info","msg":"listening at unix address","time":"2022-07-26T09:57:38.234Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","344","/var/opt/gitlab/gitaly/run/gitaly-344/sock.d/ruby.1"],"supervisor.name":"gitaly-ruby.1","supervisor.pid":401,"time":"2022-07-26T09:57:38.234Z"}
But I am not sure that this is the case - also checked other logs and seems to me no errors in other logs - but of course I will be happy to provide more logs if you want.
This is my 'docker ps -a' status, docker version: 4.10.1, is run with docker-compose:
web:
image: 'gitlab/gitlab-ce:latest'
container_name: 'gitlab'
restart: unless-stopped
hostname: 'gitlab.example.com'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://gitlab.example.com'
ports:
- '80:80'
- '443:443'
- '22:22'
volumes:
- '$GITLAB_HOME/config:/etc/gitlab'
- '$GITLAB_HOME/logs:/var/log/gitlab'
- '$GITLAB_HOME/data:/var/opt/gitlab'
list of docker active containers says container is healthy - but actually I see nothing in browser via https://gitlab.example.com.
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b0fb6252c7f3 gitlab/gitlab-ce:latest "/assets/wrapper" 3 days ago Up 3 minutes (healthy) 0.0.0.0:22->22/tcp, 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp gitlab
I also tried to add docker host IP to my hosts file - but no success:
192.168.31.182 host.docker.internal
192.168.31.182 gateway.docker.internal
192.168.31.182 gitlab.example.com
I still see nothing in browser by https://gitlab.example.com
Adding more info to #sytech "That's just a placeholder URL... Did you replace gitlab.example.com with your actual GitLab instance URL?"
You either need to add an /etc/hosts, setting to route gitlab.example.com to localhost or modify that to be localhost
hostname: 'localhost'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'localhost'
These settings are telling the nginx proxy what url should be forwarded to gitlab. That's useful when you're running multiple sites from one server. If you are just running it locally, you should just use localhost.
I also tried to add docker host IP to my hosts file - but no success:
You are forwarding the ports from the docker ip. Just use localhost. It probably didn't work because the docker IP you're looking at is for a different docker network.
I am struggling with Go requests between containers.
The issue that I have that the rest of my containers can send request to the node Container that give response, but when I send request from my GoLang application to node I get that refuse error "dial tcp 172.18.0.6:3050: connect: connection refused".
So my whole docker set up is:
version: "3.3"
services:
##########################
### SETUP SERVER CONTAINER
##########################
node:
# Tell docker what file to build the server from
image: myUserName/mernjs:node-dev
build:
context: ./nodeMyApp
dockerfile: Dockerfile.dev
# The ports to expose
expose:
- 3050
# Port mapping
ports:
- 3050:3050
# Volumes to mount
volumes:
- ./nodeMyApp/src:/app/server/src
# Run command
# Nodemon for hot reloading (-L flag required for polling in Docker)
command: nodemon -L src/app.js
# Connect to other containers
links:
- mongo
# Restart action
restart: always
react:
ports:
- 8000:8000
build:
context: ../reactMyApp
dockerfile: Dockerfile.dev
volumes:
- ../reactMyApp:/usr/src/app
- /usr/src/app/node_modules
- /usr/src/app/.next
restart: always
environment:
- NODE_ENV=development
golang:
build:
context: ../goMyApp
environment:
- MONGO_URI=mongodb://mongo:27017
# Volumes to mount
volumes:
- ../goMyApp:/app/server
links:
- mongo
- node
restart: always
So my React app can send the request to "http://node:3050/api/greeting/name" and it get the response even that react app is not linked to the node app but when Golang app sends request to node docker container it gets connection refuse message GetJson err: Get "http://node:3050/api/greeting/name": dial tcp 172.18.0.6:3050: connect: connection refused
func GetJson(url string, target interface{}) error {
r, err := myClient.Get(url)
if err != nil {
fmt.Println("GetJson err: ", err)
return err
}
defer r.Body.Close()
return json.NewDecoder(r.Body).Decode(target)
}
type ResultsDetails struct {
Greeting string `bson:"greatingMessage" json:"greatingMessage"`
Message string `bson:"message" json:"message"`
}
func GetGreetingDetails(name string) ResultsDetails {
var resp ResultsDetails
GetJson("http://node:3050/api/greeting/"+name, &resp)
return resp
}
So how do I solve the Golang request to another Docker Node Container when docker doesnt see the host as the name of my container 'node'?
Update:
By accident i put Golang port, which it doenst run on any port since it is application that checks on database records. So it hasnt got any api, therefore it is not running on any port.
Is that could be the problem why my golang application cannot communication to other containers?
Since i have also another golang application which is api application and it is running on 5000 port and it is well communicating to my node application?
Network info:
After checking the network if node and golang share the same network and the answer is yes. All containers share the same network
(Unrelated to my issue) To anyone who has "dial tcp connection refused" issue I suggest to go though that guide https://maximorlov.com/4-reasons-why-your-docker-containers-cant-talk-to-each-other/. Really helpful. To those who this guide wont help prob read bellow this, maybe you trying to request the container api after just containers were built :D
For those who was interested what was wrong:
Technically reason why I was getting this error is because of the request that I was trying to run, was just when all containers were built.
I believe there is some delay to the network after containers are built. Thats why there host was throwing "dial tcp 172.18.0.6:3050: connect: connection refused" I've run that test on other containers that could possibly send request to that node container and they were all failing after the build time. But when re-requesting after few seconds all worked out.
Sorry to bother you guys. I really spent 3 days into this issue. And I was looking into completely wrong direction. Never thought that the issue is that silly :D
Thanks for you time.
I've met the same error in my harbor registry service.
After I docker exec -it into the container, and check if the service is available, and finally I found that http_proxy has been set.
Remove the http_proxy settings for docker service, then it works like a charm.
Failed on load rest config err:Get "http://core:8080/api/internal/configurations": dial tcp 172.22.0.8:8080: connect: connection refused
$docker exec -it harbor-jobservice /bin/bash
$echo $http_proxy $https_proxy
I have tried a lot, but I can't find a solution to this problem.
I am running a nexus sonatype (3.21.1-01) docker image on a centos7 server behind a vthunder a10 proxy.
The docker login and pull works great but docker push fail with EOF after some retrying.
Here the interested routes:
docker image port 8081 > my.server:8081
docker image port 8443 > my.server:8443
proxy.domain.local:443 > my.server:8081
proxy.domain.local:8443 > my.server:8443
I have created a docker repository in nexus which have the http connector exposed on 8443
The proxy is exposed under ssl with self signed certificate
The client's /etc/docker/daemon.json file contains the insecure registry options:
"insecure-registries": ["proxy.domain.local:8443","proxy.domain.local"]
Here the situation:
If I try to push from the client an image of which all layers already exist on the remote server (but missing on nexus repository), it works.
If I try the same but adding some difference to the same image (such as a new LABEL), it fail in this way:
(9c27e219663c: Layer already exists
Patch https://proxy.domain.local:8443/v2/test4/blobs/uploads/6862fe60-d63b-4942-bbb6-f403307e677a: EOF)
If I push directly from my.server machine, pointing to localhost:8443 it works.
If i push from the client machine an image with new layers it fail in this way after some retrying (the same behavior with smaller images):
docker push proxy.domain.local:8443/ara
The push refers to repository [proxy.domain.local:8443/ara]
edb7a4f74e22: Retrying in 8 seconds
de421654540d: Retrying in 8 seconds
-------------
The push refers to repository [proxy.domain.local:8443/ara]
edb7a4f74e22: Pushing [==================================================>] 172.6MB/172.6MB
de421654540d: Pushing [==================================================>] 200.8MB/200.8MB
EOF
this is a summary of what happen in wireshark
the.client my.server HTTP 316 GET /v2/ HTTP/1.1
...
my.server the.client HTTP 654 HTTP/1.1 401 Unauthorized (application/json)
...
the.client my.server HTTP 442 HEAD /v2/alpine-test/blobs/sha256:95f5ecd24e438e09033c8e69ec136079f8774ab8284f1431f5433a829054b5e7 HTTP/
(asking to nexus if the image is already uploaded)
my.server the.client HTTP 493 HTTP/1.1 404 Not Found
(it isn't)
the.client my.server HTTP 437 POST /v2/alpine-test/blobs/uploads/ HTTP/1.1
(so it start to post the image)
my.server the.client HTTP 584 HTTP/1.1 202 Accepted
...
the.client my.server HTTP 437 POST /v2/alpine-test/blobs/uploads/ HTTP/1.1
...
my.server the.client HTTP 584 HTTP/1.1 202 Accepted
..
and so on with some FIN/ACK in the middle until the client stops to send it...
** on nexus server log there is absolutely no trace about this **
this is the nexus docker compose:
services:
nexus:
build:
context: .
args:
DOCKER_GID: ${DOCKER_GID}
NEXUS_UID: ${NEXUS_UID}
NEXUS_GID: ${NEXUS_GID}
restart: always
environment:
- NEXUS_UID_GID=${NEXUS_UID_GID}
- HOSTNAME_DOCKER_NEXUS=${HOSTNAME_DOCKER_NEXUS}
ports:
- "8081:8081"
- "8443:8443"
user: ${NEXUS_UID_GID}
hostname: ${HOSTNAME_DOCKER_NEXUS}
volumes:
- /var/nexus-data:/nexus-data
- /etc/hosts:/etc/hosts
- /var/run/docker.sock:/var/run/docker.sock
Can you help me?
I was thinking about a possibile nexus-docker-user permission issue on the local machine/docker binary permissions (if i try from localhost it works, yes, but the image is already stored on the system of course) - but I think it is not so probable.
I was thinking also about proxy configuration issue (more probable), but I don't know much about proxy.
[Workaround]
Because I can not figure out the problem, I ended up with make proxy transparent and configuring nexus to serve directly in https throught it's jetty.xml, jetty.https and nexus.properties.
Serving https directly from jetty instead of let the proxy upgrade the connection solved the above problem.
I have an issue where a self-signed certificate has been added to a testing environment.
So this means my selenium grid that is hosted in Docker containers is unable to get to this environment due to the certificate.
I get this error when executing tests
Message: OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://xxx.xx.x.x:4444/wd/hub/session/0ee03d72bff0d5527cff926121b496bb/url timed out after 60 seconds.
----> System.Net.WebException : The request was aborted: The operation has timed out.
TearDown : OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://xxx.xx.x.x:4444/wd/hub/session/0ee03d72bff0d5527cff926121b496bb/screenshot timed out after 60 seconds.
----> System.Net.WebException : The operation has timed out
The docker environment is set up with docker-compose and using chrome and hub images.
Compose file is this
version: "3"
services:
selenium-hub:
image: selenium/hub:latest
container_name: selenium-hub
ports:
- "4444:4444"
chrome:
image: selenium/node-chrome:latest
volumes:
- /dev/shm:/dev/shm
depends_on:
- selenium-hub
environment:
- HUB_HOST=selenium-hub
- HUB_PORT=4444
I added the certificates to the host hoping this would be enough but obviously not as each container is separated.
My question is how do I insert the certificates into each chrome node that spins up?
More information
When running a curl from within the container I get the following error
#b94ed81b0110:/etc# curl https://xxxx.xxxx.co.uk
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
But I have installed the required certificates to the container
root#b94ed81b0110:/etc# update-ca-certificates
Updating certificates in /etc/ssl/certs...
2 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Adding debian:admin.pem
Adding debian:assessor.pem
done.
done.
I'm running a kubernetes cluster on docker-desktop (mac).
It has a local docker registry inside it.
I'm able to query the registry no problem via the API calls to get the list of tags.
I was able to push an image before, but it took multiple attempts to push.
I can't push new changes now. It looks like it pushes successfully for layers, but then doesn't acknowledge the layer has been pushed and then retries.
Repo is called localhost:5000 and I am correctly port forwarding as per instructions on https://blog.hasura.io/sharing-a-local-registry-for-minikube-37c7240d0615/
I'm ot using ssl certs as this is for development on local machine.
(The port forwarding is proven to work otherwise API call would fail)
e086a4af6e6b: Retrying in 1 second
35c20f26d188: Layer already exists
c3fe59dd9556: Pushing [========================> ] 169.3MB/351.5MB
6ed1a81ba5b6: Layer already exists
a3483ce177ce: Retrying in 16 seconds
ce6c8756685b: Layer already exists
30339f20ced0: Retrying in 1 second
0eb22bfb707d: Pushing [==================================================>] 45.18MB
a2ae92ffcd29: Waiting
received unexpected HTTP status: 502 Bad Gateway
workaround (this will suffice but not ideal, as have to build each container
apiVersion: v1
kind: Pod
metadata:
name: producer
namespace: aetasa
spec:
containers:
- name: kafkaproducer
image: localhost:5000/aetasa/cta-user-create-app
imagePullPolicy: Never // this line uses the built container in docker
ports:
- containerPort: 5005
Kubectl logs for registry
10.1.0.1 - - [20/Feb/2019:19:18:03 +0000] "POST /v2/aetasa/cta-user-create-app/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/18.09.2 go/go1.10.6 git-commit/6247962 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \x5C(darwin\x5C))" "-"
2019/02/20 19:18:03 [warn] 12#12: *293 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000011, client: 10.1.0.1, server: localhost, request: "PATCH /v2/aetasa/cta-user-create-app/blobs/uploads/16ad0e41-9af3-48c8-bdbe-e19e2b478278?_state=qjngrtaLCTal-7-hLwL9mvkmhOTHu4xvOv12gxYfgPx7Ik5hbWUiOiJhZXRhc2EvY3RhLXVzZXItY3JlYXRlLWFwcCIsIlVVSUQiOiIxNmFkMGU0MS05YWYzLTQ4YzgtYmRiZS1lMTllMmI0NzgyNzgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDItMjBUMTk6MTg6MDMuMTU2ODYxNloifQ%3D%3D HTTP/1.1", host: "localhost:5000"
2019/02/20 19:18:03 [error] 12#12: *293 connect() failed (111: Connection refused) while connecting to upstream, client: 10.1.0.1, server: localhost, request: "PATCH /v2/aetasa/cta-user-create-app/blobs/uploads/16ad0e41-9af3-48c8-bdbe-e19e2b478278?_state=qjngrtaLCTal-7-hLwL9mvkmhOTHu4xvOv12gxYfgPx7Ik5hbWUiOiJhZXRhc2EvY3RhLXVzZXItY3JlYXRlLWFwcCIsIlVVSUQiOiIxNmFkMGU0MS05YWYzLTQ4YzgtYmRiZS1lMTllMmI0NzgyNzgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDItMjBUMTk6MTg6MDMuMTU2ODYxNloifQ%3D%3D HTTP/1.1", upstream: "http://10.104.68.90:5000/v2/aetasa/cta-user-create-app/blobs/uploads/16ad0e41-9af3-48c8-bdbe-e19e2b478278?_state=qjngrtaLCTal-7-hLwL9mvkmhOTHu4xvOv12gxYfgPx7Ik5hbWUiOiJhZXRhc2EvY3RhLXVzZXItY3JlYXRlLWFwcCIsIlVVSUQiOiIxNmFkMGU0MS05YWYzLTQ4YzgtYmRiZS1lMTllMmI0NzgyNzgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDItMjBUMTk6MTg6MDMuMTU2ODYxNloifQ%3D%3D", host: "localhost:5000"
Try configure --max-concurrent-uploads=1 for your docker client. You are pushing quite large layers (350MB), so probably you are hitting some limits (request sizes, timeouts) somewhere. Single concurrent upload may help you, but it is only a work around. Real solution will be configuration (buffer sizes, timeouts, ...) of registry + reverse proxy in front of registry eventually.
It may be a disk space issue. If you store docker images inside the Docker VM you can fill up the disk space quite fast.
By default, docker-desktop VM disk space is limited to 64 gigabytes. You can increase it up to 112Gb on the "Disk" tab in Docker Preferences.
I have encountered this issues quite few times and unfortunately couldn't get to the permanent fix.
Most likely the image should have been corrupted in the registry. As a work around, i suggest you delete the image from registry and do a fresh push. it would work and subsequent pushes would work too.
This issue must be related to the missing layers of the image. sometimes we delete the image using --force option, in that case it is possible that some of the common layers might get deleted and would affect other images that share the deleted layers.