This confuses me as it's just started happening...
My workstation (Ubuntu 18.04 running docker 20.10.17) can resolve a specific internet host without issue. The container that my workstation hosts, however, is unable to resolve the same host.
Any other internet host appears to resolve without any issue within the container : actually, from what I can see, it's unable to resolve any host assigned to that specific domain name (eg flora.com) but only in the container - on the host it's just fine.
I've read that docker takes a copy of /etc/resolv.conf from the host and places it into the container, but I don't think that's correct as my container's /etc/resolv.conf reads as follows:
search lan
nameserver 127.0.0.11
options edns0 ndots:0
whereas my workstation's /etc/resolv.conf read:
# This file is managed by man:systemd-resolved(8). Do not edit.
...
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0
search lan
Within my docker-compose.yml I have my containers using a defined bridged network (which has been working just fine for the last three years):
console-vpacs:
...
depends_on: [console-mysql]
...
networks:
- dev-vlt-console
...
networks:
dev-vlt-console:
name: dev-vlt-console
driver: bridge
I'm at a bit of a loss where to go from here as I've :
restarted the docker service a number of time
deleted the docker0 interface, restarted the service
searched SO and google and nothing appears to be working (the only thing I've not done is to remove docker and reinstall)
Can anyone give any insight?
A complete reinstall of docker resolved this issue. Very odd that it happened, but I'm still running 20.10.17 so there was no update there ... but I'm back up and running again
Related
I am running docker-compose for my containers and I have noticed that once I am logged in one of those containers, I am able to resolve (from hostname to IP address) some addresses like bigquery.googleapis.com but not pubsub.googleapis.com (I used the command host to test it). Instead, if I run the same commands on a local shell they get resolved.
Any idea why with Docker I am not able to resolve all the addresses?
UPDATE
Adding this to the yaml conf makes it working but I do not think is great:
dns:
- x.x.x.x -- from my /etc/resolv.conf
- y.y.y.1 -- based on my local ip
I'm trying to run a Thingsboard PE docker-compose cluster (basic configuration) on an Azure Linux VM (ubuntu 20.04). The main "monolith" container shuts down after about a minute and the logs report it can't access the license server. I'm assuming it's shutting down because of license server access, and assuming the problem is that the container can't access the internet (but any advice on further troubleshooting would be appreciated).
Within the container cat /etc/resolv.conf returns:
search 1lt4eb1hmraebffqmvlsi2dp5g.px.internal.cloudapp.net
nameserver 127.0.0.11
options ndots:0
On the host it's:
nameserver 168.63.129.16
search 1lt4eb1hmraebffqmvlsi2dp5g.px.internal.cloudapp.net
There's no problem with internet access from the host and I can ping Google's dns servers.
I've read a lot of posts/advice on setting DNS server settings for docker containers and (separately) tried the following but the service still fails:
Added Google dns entries to docker-compose.yml
Added Google dns to /etc/docker/daemon.json
Added Google dns to /etc/default/docker
Updated /etc/resolv.conf symlink with: sudo ln -sf /run/systemd/resolve/resolv.conf
It looks like I can't edit iptables on the Azure VM (but if it's possible please let me know).
If anyone has experienced the same or similar issue I'd be interested to know how you resolved it.
For anyone else having this problem, after researching further it seems the issue was actually the "options ndots:0" configuration which isn't compatible with the default host DNS config on Ubuntu 20.04. I added the below setting to /etc/docker/daemon.json and after that the containers were able to access the internet. It might be possible to limit this to specific containers by adding the setting to the service in docker-compose.yml, but I haven't tried that yet.
{
"dns-opts":["ndots:1"]
}
The question is basic: how to I connect to the localhost of the host machine from inside of a Docker container?
I tried answers from this post, using add-host host.docker.internal:host-gateway or writing --network=host when running my container but none of these methods seem to work.
I have a simple hello world webserver up on my machine, and I can see it's contents with curl localhost:8000 from my host, but I can't curl it from inside the container. I tried curl host.docker.internal:8000, curl localhost:8000, and curl 127.0.0.1:8000 from inside the container (based on the solution I used to make localhost available there) but none of them seem to work and I get a Connection refused error every time.
I asked somebody else to try this out for me on their own machine and it worked for them, so I don't think I'm doing anything wrong.
Does anybody have any idea what is wrong with my containers?
Host machine: Ubuntu 20.04.01
Docker version: 20.10.7
Used image: Ubuntu 20.04 (and i386/ubuntu18.04)
Temporary solution
This does not completely solve the problem for production purposes, but at least in order to get the localhost working, by adding these lines into docker-compose.yml it solved my issue for now (source):
services:
my-service:
network_mode: host
I am using apache nifi to use Java REST endpoints with the same ubuntu and docker versions, so in my case, it looks like this:
services:
nifi:
network_mode: host
After changing docker-compose.yml, I recommend stopping docker container, removing containers(docker-compose rm - do not use if you need some containers, otherwise use docker container rm container_id) and build with docker-compose up --build again.
In this case, I needed to use another localhost IP for my service to access with a browser (nifi started on other ip - 127.0.1.1 but works fine as well).
Searching for the problem / deeper into ubuntu-docker networking
Firstly, I will write down some useful commands that may be useful to find out a solution for the docker-ubuntu networking issue:
ip a - show all routing, network devices, interfaces and tunnels (mainly I can observe state DOWN with docker0)
ifconfig - list all interfaces
brctl show - ethernet bridge administration (docker0 has no attached interface / veth pair)
docker network ls - manages docker networks - names, drivers, scope...
docker network inspect bridge - I can see for docker0 bridge has no attached docker containers - empty and not used bridge
(useful link for ubuntu-docker networking explanation)
I guess that problem lays within veth pair (see link above), because when docker-compose occurs, there is a new bridge created (not docker0) that is connected to veth pair in my case, and docker0 is not used. My guess is that if docker0 is used, then host.docker.internal:host-gateway would work. Somehow in ubuntu networking, there is docker0 not used as the default bridge and this maybe should be changed.
I don't have much time left actually, well, I suppose someone can use this information and resolve the core of the problem later on.
I am working behind a company proxy.
Because of the many limitations enforced, I have to switch to public network when I come to build my docker images (mainly ubuntu-based images). The build is performed on the same computer (thus, the same dns conf).
Apart from the build, I always run my containers behind this proxy.
The company also (and indeed) has its own dns.
Unfortunately, I don't understand how to pass the host dns to the containers the "proper way" and don't understand how docker manages to build the containers resolv.conf.
When I look at my host, I see such a conf :
$ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.8.4
nameserver 10.xxx.yyy.z
search rennes.mycompany.fr
And from within my containers, I see:
search rennes.mycompany.fr
nameserver 127.0.0.11
options ndots:0
What looks odd to me is that only a part of the host configuration can be found in the container...
And when I try to reach any company-hosted name from the container, I have a name resolution failure.
But if I add the host's resolv.conf nameserver to the container's one, it works:
$ echo nameserver 192.168.1.1 >> /etc/resolv.conf
So, this is indeed not the way to do it... but I am not sure how I should perform it.
I tried to add the 'dns' to my services in the docker-compose, but I didn't work (or not a way I understand), and the documentation there is quite spartan...
Is there a way to tell docker to use/share host's dns configuration?
May my problem come from my building the image outside of the company network?
EDIT: the docker-compose.yml
version: '3.3'
services:
my-apache:
image: apache:latest
ports:
- 8088:80
networks:
- my-project-net
dns:
- 192.168.1.1
depends_on:
- my-project
my-project:
image: my-project:latest
networks:
- my-project-net
dns:
- 192.168.1.1
networks:
my-project-net:
driver: bridge
only 'my-project' requires access to company dns, but I added dns to both in case I missed some clue...
EDIT 2 : few more details and attempts
docker version : 18.03.0-ce, build 0520e24
docker-compose version : 1.18.0, build 8dd22a9
The name I try to resolve looks like this :
some-pf-db.network.mycompany.fr
The (modified) docker daemon conf looks like this :
$ cat /etc/docker/daemon.json :
{
"insecure-registries": [
"some.test.registry.mycompany.fr:5000"
],
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
},
"dns-opts": ["ndots:15"]
}
Following #BMitch link, I tried to update docker daemon (+restart) with the ndots options up to 15 (don't think I'd need that many, but it's the lazy way!)...
and unfortunately it didn't solve the problem. The new value replaces the former one within the container, but it keeps failing reaching the dns
could not translate host name "some-pf-db.network.mycompany.fr" to
address: Temporary failure in name resolution
EDIT3 : I was looking at the wrong container... so, changing the threshold for dots (ndots) up to (eg.) 15 within the docker daemon conf (/etc/docker/daemon.json) AND removing the "dns" param from the service makes the magic operate! I have now access to my company dns for names (with a lot of dots in them!!)
The 127.0.0.11 entry inside the container is expected even when you override DNS on the container. This points back to the loopback interface inside the container which has a mapping for port 53 to go back to the docker engine for DNS resolution. You need docker to do the DNS resolution to give you container to container networking with DNS for discovery.
You should still see the docker engine call out to your DNS server even with the 127.0.0.11 entry inside the container, so it's not a bug, or lack of configurability, you just don't see this configuration from inside the container.
We'd need more details on the actual issue you are encountering, but one possible problem I've seen from this before is DNS not resolving external hosts without a fully qualified name in some specific scenarios. You can read about that in this issue/thread here:
https://github.com/moby/moby/issues/32093
if you are using docker-compose you need to add dns section onto your service definition in yaml file.
If you running docker directly for command line you may use --dns=IP_ADDRESS... argument for defining your company nameserver.
more details in documentation here: https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/
I'm trying to figure out a problem with identical docker containers being run on different hosts, where one container can find/ping/nslookup a domain on a private network, and another can't. One host is OSX 10.11, the other is Ubuntu 16.04. Both are running docker 1.12. I'm using docker-compose to bring up my application, and I'm hoping to figure out what is going on and how to fix it, or some configuration changes I could make without resorting to hardcoding domains or ip addresses that would make the container behave the same on both hosts.
On my OSX box, I have the following dns nameservers set automatically by my domain:
osx:$ cat /etc/resolv.conf
domain redacted.lan
nameserver 172.16.20.19
nameserver 10.43.0.11
I'm aware that resolv.conf isn't used by most OSX tools, but System Preferences > Network shows the same settings.
I have similar settings on my Ubuntu 16 box as well (command from https://askubuntu.com/questions/152593/command-line-to-list-dns-servers-used-by-my-system):
ubu:$ cat /etc/resolv.conf
nameserver 127.0.1.1
search redacted.lan
ubu:$ nmcli device show eno1 | grep IP4.DNS
IP4.DNS[1]: 172.16.20.19
IP4.DNS[2]: 10.43.0.11
Then, on both OSX and Ubuntu, I start my container with this:
$ docker run -it redacted_web bash
And then I run these commands to diagnose my problem:
$ apt-get update
$ apt-get install -y dnsutils
$ cat /etc/resolv.conf
$ nslookup redacted.lan
On OSX, the output from the last 2 commands is:
root#d19f49322fda:/app# cat /etc/resolv.conf
search local
nameserver 192.168.65.1
root#d19f49322fda:/app# nslookup redacted.lan
Server: 192.168.65.1
Address: 192.168.65.1#53
Name: redacted.lan
Address: 172.18.0.23
On Ubuntu, the output is:
root#91e82d652e07:/app# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
search redacted.lan
nameserver 8.8.8.8
nameserver 8.8.4.4
root#91e82d652e07:/app# nslookup redacted.lan
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find redacted.lan: NXDOMAIN
Possible differences I can think of:
On OSX there is a vm running docker, where as on ubuntu it's native
On Ubuntu, docker is run with sudo, possibly picking up different configuration settings
Updated Answer (2017-06-20)
Newer versions of Ubuntu (17.04+) don't use dnsmasq (it now uses systemd-resolved). You'll run into a similar problem with host resolution, but the original solution here no longer works. In fact, Docker containers can't even communicate with systemd-resolvd because it's running its own DNS in a location that's unreachable from within a container.
A good solution on newer versions of Ubuntu is to put the following configuration in /etc/docker/daemon.json (create this file if it doesn't exist):
{
"dns": ["172.16.20.19", "10.43.0.11"],
"dns-search": ["redacted.lan"]
}
This allows you to configure the DNS servers and search domains. The DNS IPs above are from the original question, but you can use your own custom ones too. You probably want to match the DNS config on your host machine. Search domain is optional, and you could entirely omit that line (careful with your commas!). Again, you probably want to match your host machine.
Essentially, what these daemon.json options do, is automatically inject the DNS config and search domain into the config files inside the container for any container that is started on that daemon. This is necessary because you cannot use systemd-resolved from the host to resolve DNS within the container due to limitations of the way systemd-resolved works. The docs are here and here.
Original Answer
The problem is that the host is using dnsmasq to resolve the private IP and Docker is not using dnsmasq on the host.
The simple fix is to turn off dnsmasq on the host machine.
Run sudo vi /etc/NetworkManager/NetworkManager.conf
Comment out this line: #dns=dnsmasq
Run sudo service network-manager restart
Now, you should be able to use the docker container and it will resolve your private DNS correctly.
Check your startup scripts for the docker daemon, it includes the following option to adjust the DNS used when creating containers:
$ dockerd --help
# ...
--dns=[] DNS server to use
--dns-opt=[] DNS options to use
--dns-search=[] DNS search domains to use
On Ubuntu, I believe these settings are in /etc/default/docker which is read by /etc/init.d/docker. Leaving these unset should default to the /etc/resolv.conf values.
Update: Docker's DNS networking documentation has a lot of detail that should point you in the right direction.
if there are no more nameserver entries left in the container’s /etc/resolv.conf file, the daemon adds public Google DNS nameservers (8.8.8.8 and 8.8.4.4) to the container’s DNS configuration.
This looks like it's probably happening in your situation. The filters are looking for local addresses, which I wouldn't think is happening in your situation with a 172.16.0.0/12 private addr, but it's possible.
You might wonder what happens when the host machine’s /etc/resolv.conf
file changes. The docker daemon has a file change notifier active
which will watch for changes to the host DNS configuration.
Note: The file change notifier relies on the Linux kernel’s inotify
feature. Because this feature is currently incompatible with the
overlay filesystem driver, a Docker daemon using “overlay” will not be
able to take advantage of the /etc/resolv.conf auto-update feature.
If this is happening, then giving the daemon a restart would likely resolve it and would indicate you want Docker to start after the NetworkManager.
Update #2: looking over this github issue it's possible that they also included 172.16.0.0/12 to avoid conflicts with docker's bridged networks. If you can be sure to avoid using the same network inside docker, then passing the dns server in /etc/defaults/docker would likely force the correct behavior. There's also a mention at the end of the post about lxc causing conflicts, so if you have that installed and can remove it, give that a try first.