docker can't connect to another docker service - docker

I'm trying to run sentry onpremise version (https://github.com/getsentry/onpremise)
logs shows
relay_1 | 2020-09-09T10:45:13Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1 | caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web.router703710.com. type: AAAA class: IN
relay_1 | caused by: Failed resolving hostname: no record found for name: web.router703710.com. type: AAAA class: IN
relay_1 | caused by: Failed resolving hostname: no record found for name: web.router703710.com. type: AAAA class: IN
router703710.com is the dns assigned by our cisco router..
although we don't use it in any way.
I just see cisco has that dns setup in the router's setting page.
From the hostmachine where the docker is running, I can't connect to the router703710.com or www.router703710.com .
So how do I tell the docker not to use the dns and some other dns which is actually working?
Only thing I can think of are the following settings and not sure it'll make difference
network_mode: host
dns:
- 192.168.1.1
The docker image seems to be https://hub.docker.com/r/getsentry/relay/ but I don't know how to see the actuall dockerfile

It has been discussed in https://github.com/getsentry/onpremise/issues/771 and it seems this issue is from Docker. agrevtcev has provided an workaround in that thread, to downgrade Docker to 19.03.
I made some changes on the commands. The latest Docker of 19.03 is 19.03.15 (not 19.03.14 in the original workaround), and use apt-mark to prevent from upgrading docker automatically (note this is for Ubuntu 20.04, focal):
apt install -y --allow-downgrades docker-ce=5:19.03.15~3-0~ubuntu-focal docker-ce-cli=5:19.03.15~3-0~ubuntu-focal
apt-mark hold docker-ce docker-ce-cli

Related

port 80 refused - digital ocean droplet web console w/ caprover instance

I have a cap rover instance in my digital ocean instance that I created. I want to use teh caprover instance to run cap rover sample apps.
I opened the digital ocean droplet web console in order to run a caprover isntance.
I ran the following lines of code:
ufw allow 80,443,3000,996,7946,4789,2377/tcp; ufw allow 7946,4789,2377/udp;
and got this:
Skipping adding existing rule
Skipping adding existing rule (v6)
Skipping adding existing rule
Skipping adding existing rule (v6)
I then ran this:
docker run -p 80:80 -p 443:443 -p 3000:3000 -v /var/run/docker.sock:/var/run/docker.sock -v /captain:/captain caprover/caprover
I got this:
Unable to find image 'caprover/caprover:latest' locally
latest: Pulling from caprover/caprover
Digest: sha256:39c3f188a8f425775cfbcdc4125706cdf614cd38415244ccf967cd1a4e692b4f
Status: Downloaded newer image for caprover/caprover:latest
docker: Error response from daemon: driver failed programming external connectivity on endpoint priceless_sammet (9da9028cfc4873818f113458237ebd00f9c64fa648b853730a60b10bea39c720): Bind for 0.0.0.0:3000 failed: port is already allocated.
I tried changing the ports to:
docker run -p 81:81 -p 444:444 -p 3321:3321 -v /var/run/docker.sock:/var/run/docker.sock -v /captain:/captain caprover/caprover
and got this:
Captain Starting ...
Installing Captain Service ...
Installation of CapRover is starting...
For troubleshooting, please see: https://caprover.com/docs/troubleshooting.html
>>> Checking System Compatibility <<<
Docker Version passed.
Ubuntu detected.
X86 CPU detected.
Total RAM 1033 MB
Are your trying to run CapRover on a local machine or a machine without public IP?
In that case, you need to add this to your installation command:
-e MAIN_NODE_IP_ADDRESS='127.0.0.1'
Otherwise, if you are running CapRover on a VPS with public IP:
Your firewall may have been blocking an in-use port: 80
A simple solution on Ubuntu systems is to run "ufw disable" (security risk)
Or [recommended] just allowing necessary ports:
ufw allow 80,443,3000,996,7946,4789,2377/tcp; ufw allow 7946,4789,2377/udp;
See docs for more details on how to fix firewall issues
Finally, if you are an advanced user, and you want to bypass this check (NOT RECOMMENDED),
you can append the docker command with an addition flag: -e BY_PASS_PROXY_CHECK='TRUE'
Installation failed.
Error: Port seems to be closed: 80
at Request._callback (/usr/src/app/built/utils/CaptainInstaller.js:149:24)
at Request.self.callback (/usr/src/app/node_modules/request/request.js:185:22)
at Request.emit (events.js:400:28)
at Request.<anonymous> (/usr/src/app/node_modules/request/request.js:1154:10)
at Request.emit (events.js:400:28)
at IncomingMessage.<anonymous> (/usr/src/app/node_modules/request/request.js:1076:12)
at Object.onceWrapper (events.js:519:28)
at IncomingMessage.emit (events.js:412:35)
at endReadableNT (internal/streams/readable.js:1334:12)
at processTicksAndRejections (internal/process/task_queues.js:82:21)
How can I open port 80, 443, and 3000 so that I can run the cap rover instance

Skaffold dev fails

I am having this error, after running skaffold dev.
Step 1/6 : FROM node:current-alpine3.11
exiting dev mode because first build failed: unable to stream build output: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.49.1:53: read udp 192.168.49.2:35889->192.168.49.1:53: i/o timeout. Please fix the Dockerfile and try again..
Here is skaffold.yml
apiVersion: skaffold/v2beta11
kind: Config
metadata:
name: *****
build:
artifacts:
- image: 127.0.0.1:32000/auth
context: auth
docker:
dockerfile: Dockerfile
deploy:
kubectl:
manifests:
- infra/k8s/auth-depl.yaml
local:
push: false
artifacts:
- image: 127.0.0.1:32000/auth
context: auth
docker:
dockerfile: Dockerfile
sync:
manual:
- src: "src/**/*.ts"
dest: .
I have tried all possible solutions I saw online, including adding 8.8.8.8 as the DNS, but the error still persists. I am using Linux and running ubuntu, I am also using Minikube locally. Please assist.
This is a Community Wiki answer, posted for better visibility, so feel free to edit it and add any additional details you consider important.
In this case:
minikube delete && minikube start
solved the problem but you can start from restarting docker daemon. Since this is Minikube cluster and Skaffold uses for its builds Minikube's Docker daemon, as suggested by Brian de Alwis in his comment, you may start from:
minikube stop && minikube start
or
minikube ssh
su
systemctl restart docker
I searched for similar errors and in many cases e.g. here or in this thread, setting up your DNS to something reliable like 8.8.8.8 may also help:
sudo echo "nameserver 8.8.8.8" >> /etc/resolv.conf
in case you use Minikube you should first:
minikube ssh
su ### to become root
and then run:
echo "nameserver 8.8.8.8" >> /etc/resolv.conf
The following error message:
Please fix the Dockerfile and try again
may be somewhat misleading in similar cases as Dockerfile is probably totally fine, but as we can read in other part:
lookup registry-1.docker.io on 192.168.49.1:53: read udp 192.168.49.2:35889->192.168.49.1:53: i/o timeout.
it's definitely related with failing DNS lookup. This is well described here as well known issue.
Get i/o timeout
Get https://index.docker.io/v1/repositories//images: dial tcp: lookup on :53: read udp :53: i/o timeout
Description
The DNS resolver configured on the host cannot resolve the registry’s
hostname.
GitHub link
N/A
Workaround
Retry the operation, or if the error persists, use another DNS
resolver. You can do this by updating your /etc/resolv.conf file
with these or other DNS servers:
nameserver 8.8.8.8 nameserver 8.8.4.4

Windows Container unable to access internet

Running Windows-based containers I am unable to access the internet from within. Example:
From my host machine I can run the following command:
PS C:\Developer> nslookup aka.ms
Server: cache100.ns.tdc.net
Address: 193.162.153.164
Non-authoritative answer:
Name: aka.ms
Address: 88.221.62.148
When I try to do this from inside a container:
PS C:\Developer> docker run mcr.microsoft.com/dotnet/framework/runtime:4.8-windowsservercore-ltsc2019 powershell nslookup aka.ms
*** UnKnown can't find aka.ms: Server failed
Server: UnKnown
Address: 172.28.112.1
While I am not specifically interested in aka.ms, this error happens for all services I try to connect to, so I am not able to install external libraries, etc.
I am running Docker Desktop v19.03.12. The behaviour occurs regardless of whether I have WSL 2 enabled or not, and my Docker setup is all defaults.
Note: I have some time ago experienced this behaviour. Back then I added the following snippet to my Dockerfile:
RUN powershell -command certutil -generateSSTFromWU roots.sst && certutil -addstore -f root roots.sst && del roots.sst
To my understanding this would install an SSH certificate, which solved the issue. This command, however, now fails:
PS C:\> certutil -generateSSTFromWU roots.sst
The server name or address could not be resolved 0x80072ee7 (WinHttp: 12007 ERROR_WINHTTP_NAME_NOT_RESOLVED) -- http://ctldl.windowsupdate.com/msdownload/update/v3/static/truste
dr/en/authrootstl.cab
CertUtil: -generateSSTFromWU command FAILED: 0x80072ee7 (WinHttp: 12007 ERROR_WINHTTP_NAME_NOT_RESOLVED)
CertUtil: The server name or address could not be resolved
I tested this out on a basic server core image and I got it worked with adding DNS settings.
I connected to the container interactively to test this, but you can probably add the command to a DockerFile too.
docker run -it container powershell
Type netsh to start network configuration
First we look up the network we want to change
( in my case "Ethernet 2")
Then we add an static DNS server to this interface
interface ip show config
interface ipv4 set dns name="Ehternet 2" static 8.8.8.8
exit
nslookup aka.ms
S C:\> nslookup aka.ms
Server: dns.google
Address: 8.8.8.8
Non-authoritative answer:
Name: aka.ms
Address: 23.38.17.26
Reference Docker Networking
Reference howto

Clustering vernemq docker containers running on different machines

I was hoping this would be an easy one by just using the below snippet on the second instance's docker-compose.yml file
- DOCKER_VERNEMQ_DISCOVERY_NODE=<ip address of the first instance>
but that doesn't seem to work.
Log of the second instance confirms it's attempting to cluster:
13:56:09.795 [info] Sent join request to: 'VerneMQ#<ip address of the first instance>'
13:56:16.800 [info] Unable to connect to 'VerneMQ#<ip address of the first instance>'
While the log of the first instance does not show anything at all.
From within the second instance I can confirm that the endpoint is accessible:
$ docker exec -it vernemq /bin/sh
$ curl <ip address of the first instance>:44053
curl: (56) Recv failure: Connection reset by peer
then in the log of the first instance I see an error which is totally expected and confirms I've reached the first instance
13:58:33.572 [error] CRASH REPORT Process <0.3050.0> with 0 neighbours crashed with reason: bad argument in vmq_cluster_com:process_bytes/3 line 142
13:58:33.572 [error] Ranch listener {{172,19,0,2},44053} terminated with reason: bad argument in vmq_cluster_com:process_bytes/3 line 142
It might have to do with the fact that ip address as seen from within the docker container is 172.19.0.2 while the external one is 10. ....
Also tried adding hostname of the first instance to known_hosts to no avail.
Please advise.
I'm using erlio/docker-vernemq:1.10.0
$ docker --version
Docker version 19.03.13, build 4484c46d9d
$ docker-compose --version
docker-compose version 1.27.2, build 18f557f9
I managed to get this sorted by creating a docker overlay network
on machine1: docker swarm init
on machine2: docker swarm join --token ...
on machine1: docker network create --driver=overlay --attachable vernemq-overlay-net
The relevant bits of my dockerfile are:
version: '3.6'
services:
vernemq:
container_name: ${NODE_NAME:?Node name not specified}
image: vernemq/vernemq:1.10.4.1
environment:
- DOCKER_VERNEMQ_NODENAME=${NODE_NAME:?Node name not specified}
- DOCKER_VERNEMQ_DISCOVERY_NODE=${DISCOVERY_NODE:-}
networks:
default:
external:
name: vernemq-overlay-net
with the following env vars:
machine1:
NODE_NAME=vernemq1.example.com
DISCOVERY_NODE=
machine2:
NODE_NAME=vernemq2.example.com
DISCOVERY_NODE=vernemq1.example.com
Note:
Chances are machine2 won't find vernemq-overlay-net due to a bug in docker-compose as far as I remember.
In that case you start a container with docker: docker run -dit --name alpine --net=vernemq-overlay-net alpine which will make it available for docker-compose.

Docker error: HTTP 408 response body: invalid character '<' looking for beginning of value

When I go docker pull hello-world I get the below error message:
Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
Installed Docker version:
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:47:50 2016
OS/Arch: linux/amd64
Installed using: curl -fsSL https://get.docker.com/ | sh
I have ensured that network exists and I can reach other sites. Please help
Update 1: The issue cannot be of setting MTU because I could pull the images from Docker hub a few days back on the same machine.
The issue cannot be of HTTP_PROXY either because I am in my home network
I have run across this issue a couple times with Raspberry Pi boards running various flavors of Debian/Raspbian (RPi model info was obtained by cat /proc/cpuinfo | grep Model):
Raspberry Pi Model B Rev 1 with Raspbian based on Debian 11 (bullseye)
Raspberry Pi 4 MOdel B Rev 1.4 with Debian 10 (buster)
In both cases, running docker run --rm hello-world resulted in the 408 HTTP status code reported in the original question in this thread:
$ docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: error parsing
HTTP 408 response body: invalid character '<'
looking for beginning of value: "<html><body>
<h1>408 Request Time-out</h1>\nYour browser
didn't send a complete request in time.\n</body>
</html>\n".
See 'docker run --help'.
The solution (noted as an aside by #Romaan) was to adjust the MTU. I did this as follows:
sudo ip link set dev eth0 mtu 1400
docker run --rm hello-world
and the hello-world container was successfully pulled and executed.
Examples of how to permanently adjust the MTU for a network interface on Debian may be found here.
That error message looks like it's coming from a proxy server. From the docker pull documentation
Proxy configuration
If you are behind an HTTP proxy server, for example in corporate
settings, before open a connect to registry, you may need to configure
the Docker daemon’s proxy settings, using the HTTP_PROXY, HTTPS_PROXY,
and NO_PROXY environment variables. To set these environment variables
on a host using systemd, refer to the control and configure Docker
with systemd for variables configuration.
The link to the instructions for configuring systemd with a proxy is straightforward.
The error message is little misleading. The problem was not that there was invalid character, but the network was misconfigured. I had one LAN interface and one WLAN interface.
LAN interface connects to a router A which forward requests to router B which was connected to internet. While the WLAN interface was directly connected to router B. I forgot to remove the WLAN configuration.
Once I ensured the WLAN interface is removed, things worked smoothly.
In short: Ensured DNS resolution works and that MTU is set right.
Another possible reason for error
If you are using Mac, please ensure to allow Unrestriction Access to Web Content like below:
Another possible step in troubleshooting
Ensure there is no proxy or web filter in your network, that is, if possible connect to your 3G network and try again to see if the results are different
I ran into this problem on Ubuntu. I managed to solve it by disconnecting from NordVPN:
$ nordvpn disconnect
You are disconnected from NordVPN.
It seems the VPN somehow slowed down the dockerhub traffic and broke my docker pulls.
There is a high chance that this is caused due to internet connectivity issue, try to rerun when the internet connection is stable.

Resources