We're currently having an issue where curl bigquery.googleapis.com sometimes hangs indefinitely inside a ubuntu:20.04 local docker container, but always works (returns a 404) on the host macOS machine and in production. The container is running a Python Flask application.
Some observations:
It's flakey, curl bigquery.googleapis.com works when the container initializes, but trying again ~10 mins later hangs. Eventually, the command will work again and curl -v shows it's trying to connect to a different IP.
We don't have this issue on production, or on any host machine.
Different people on our team experience the issue at different times, on different IPs.
During the issue, connections to other google services (e.g. curl servicemanagement.googleapis.com) works fine.
During the issue, the command works in another docker container with a different image.
Issue has occurred across many macOS Docker versions.
Curling directly to the IPv4 address also hangs.
Interestingly, doing netstat on the host machine shows all connections as ESTABLISHED, but doing netstat inside the container shows them as SYN_SENT.
We hypothesize that the connection is being kept alive on the host, but is killed in the container. Or, certain ACKs aren't being forwarded correctly to the container for some reason.
netstat on macOS
netstat inside container
TCP Dump
curl -v output
root#cf8bd850e9ab:/code# curl -v bigquery.googleapis.com
* Trying 142.251.35.170:80...
* TCP_NODELAY set
* Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address
* Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address
* Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address
* Trying 2607:f8b0:4006:81e::200a:80...
* TCP_NODELAY set
* Immediate connect fail for 2607:f8b0:4006:81e::200a: Cannot assign requested address
Additional output:
netstat -p: https://pastebin.com/sKQ23yah
curl --ipv4: https://pastebin.com/3N7Q6aB4
You can specify the DNS while running the container with --dns flag or you can configure it by adding an entry in /etc/docker/daemon.json which could probably help with the resolution.
Related
i have a super weird issue which i can reproduce.
Most of my clients in my network at home are connected via wireuard vpn with another system in another country and use this system as default gateway. No issues so far.
My QNAP NAS also uses this gateway for communication to the internet. Now i have setup a docker container in my container station and the issue appears. For now, one specific website is not reachable anymore from inside of any docker container on the QNAP.
curl https://rapidgator.net/ --verbose
* Trying 195.211.222.2:443...
* Connected to rapidgator.net (195.211.222.2) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to rapidgator.net:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to rapidgator.net:443
Same issue appeears with wget/openssl s_client, etc. and yes i can still reach every other website as usual.
If I turn off the VPN Gateway in my QNAP and the communication uses my usual public IP the issue for this one specific website disapears. I was able to reproduce this issue with a simple ubuntu:latest docker on my manjaro which is also using wireguard. If i turn off wireguard on Manjaro, issue with the website inside the docker disappears.
I should also note that on every system with wireguard VPN and VPN default gateway without docker I can reach this specific website without any issues. Its only with the combination of both and i don't even know what i could do to fix this issue.
Any suggestion on this issue? I can provide you any information if needed.
Thanks!
I am trying to connect to a mosquitto broker, across linux clients. I can get everything working from the local machine, but when trying to connect from another machine I get the error ConnectionRefusedError: [Error 111] Connection refused.
Here is the process:
On the local machine, I install mostquitto, stop the service and start a live instance:
#Terminal 1
sudo service mosquitto stop
mosquitto
I then try and pub and sub, from distinct terminals on that machine:
#Terminal 2
mosquitto_sub -t 'test'
#Terminal 1 shows new connection
#Terminal 3
mosquitto_pub -t 'test' -m 'Hello, world!'
#Terminal 1 shows new connection, and then disconnect.
#Terminal 2 shows 'Hello, world!'
I now try to connect from a remote machine. First I edit the mosquitto config file to allow unauthorized connections:
sudo nano /etc/mosquitto/mosquitto.conf
#Add the following:
listener 1883
allow_anonymous true
protocol mqtt
I note that the mosquitto logs previously showed only local connections allowed, after editing the config file and restarting, the logs no longer show that message.
Then I install paho-mqtt on another machine. I run the following python script:
import paho.mqtt.client as mqtt
client = mqtt.Client('131')
client.connect('192.168.0.146') #The IP of machine 1, running the broker where the code above ran correctly across the terminals
I get the error mentioned above:ConnectionRefusedError: [Error 111] Connection refused. The mosquitto instance on machine 1 shows nothing. Logs show nothing.
I can't work out what is going on. I have read every question on SO that I can find. Nothing goes beyond changing the config file. I have tried running the broker on two machines (laptop and pi). I have tried connecting from multiple different sources: esp32 board, different laptop and pi. Nothing works. I can only assume there is some network-wide problem, but my network isn't isolating devices as I ssh into my pi all the time and have wifi lights and switches running on the LAN.
If anyone can help me troubleshoot I would be very grateful.
Mosquitto will not pick up a default configuration file, you must always pass the configuration file with the -c command line argument or it will fall back to the baked in config (that will only listen on localhost)
The service includes -c /etc/mosquitto/mosquitto.conf to force it to use the config file.
I'm running a setup of 3 Ubuntu virtual machines. Two running the Python production code base and the other has a Memcached Docker container. On the Memcached machine I ran docker run -dit --name production-memcached --publish 11211:11211 memcached:latest.
The code base gets the following error message when trying to interact with it:
"exception": "TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None)"
I have ran docker exec -it production-memcached memcached stats and get the error message below.
failed to listen on TCP port 11211: Address already in use
However I've ran netstat -plnt and get tcp6 0 0 :::11211 :::* LISTEN 35030/docker-proxy, which looks fine to me.
I was able to get this to work by opening port 80 and using iptables to forward incoming port 80 to port 11211 but would prefer to use the Memcached port number.
The Memcached client is created by the following line:
client = base.Client(("domain.co.uk", 11211))
Any help would be appreciated, thanks.
DigitalOcean doesn't allow in-traffic on port 11211 to its virtual machines. If you want a Memcached machine you'll also need a virtual machine to act a proxy between you and it. I hope this saves someone the headache it caused me!
Here is my configuration:
On the server I have a dhcp server that gives IP addresses to clients (connected on eth1) in order for the clients to be connected to the internet (on eth0).
For a special operational use, I would like to stop the IP connection between a client and the server for 30 seconds and rewarm up the link after that period. Currently I have tried to use iptable black list to put the client IP inside the black ip list with this command:
command 1: sudo iptables -I INPUT -s '.$ip.' -j DROP
then I use another iptable command to resume the ip link with this command:
command 2: sudo iptables -D INPUT -s '.$ip.' -j DROP
the both commands are encapsulated in a PHP program stored in an Ubuntu server and launched from a Windows workstation. Both command works perfectly but, unfortunately, I never get the internet connection back. From a Windows command screen I can monitor the behaviour of the line with ping commands. Here are the result of command 1:stop the network
Here are the result of command 2:start the network
Here is my Question:
Can someone thell me how to rewarm up the local IP socket in order to relaunch the tcp connection after command 2?
Another way of doing this may could have be to stop using iptable command and dynamically modify the lease time of the client IP address given by dhcp service.
Stopping the connection: by anticipating the end of lease period of the IP address. Rewarming up the connection: by attempting a http command to invoke the establishment of a tcp connection with a new lease time.
Can someone tell me how to overcome this?
Thanks very much.
I am using a Dockerfile to hit our corporate Nexus (npm) server for 'npm install' commands. I am seeing:
* Hostname was NOT found in DNS cache
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 216.xxx.xxx.xxx...
* connect to 216.xxx.xxx.xxx port 443 failed: Connection refused
* Failed to connect to nexus.<something>.com port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to nexus.<something>.com port 443: Connection refused
I can resolve www.google.com. I can hit and and use our corporate NPM registery from my local box. It appears that only our internal dns names are the problem and only when attempting to access them from inside a docker container. I've googled and not been able to determine the changes I need to make to fix this problem.
Dockerfile (I've trimmed the irrelevant commands):
FROM node:6.3
RUN curl -k -v https://www.google.com
RUN curl -k -vv https://nexus.<something>.com/repository/npm-all/
The curl to google.com succeeds. The curl to our internal repo fails.
I am starting it with the command:
docker build .
Contents of /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
I am running Ubuntu 15.10.
Solution
#BMitch is correct. Modify the contents of /etc/resolv.conf by adding dns server addresses associated with your corporate network. In the case of Ubuntu 15+ (i am running gnome3) your config file will be overwritten by the Network Manager so it cannot be hand edited. Make the changes via the Network Manager gui. Open Network Settings, select the DNS tab and add servers.
The local dns address is the issue you're facing. The container can't connect to your localhost ip from inside the container. The solution is to pass an ip address of the DNS server in your docker run or update your /etc/resolv.conf to point to a non-loopback ip address.
From Docker's DNS documentation:
Filtering is necessary because all localhost addresses on the host are
unreachable from the container’s network. After this filtering, if
there are no more nameserver entries left in the container’s
/etc/resolv.conf file, the daemon adds public Google DNS nameservers
(8.8.8.8 and 8.8.4.4) to the container’s DNS configuration. If IPv6 is
enabled on the daemon, the public IPv6 Google DNS nameservers will
also be added (2001:4860:4860::8888 and 2001:4860:4860::8844).
Note: If you need access to a host’s localhost resolver, you must modify your DNS service on the host to listen on a non-localhost
address that is reachable from within the container.