When running containers on startup I noticed some were using resolv.conf before systemd-resolved had updated it from the default using DHCP. This meant that containers that started too early after boot could not resolve anything and needed to be restarted to use the proper DNS settings. This is happening for different reasons for both rkt and Docker; Docker's method for updating resolv.conf inside containers is not compatible with the overlay filesystem driver and since systemd-resolved does not update the file in-place (rather creates a temporary one and renames) rkt's bind mounting does not update what the container sees.
Currently I am using a hacky systemd.unit to delay the network-online.target which docker.service and my rkt pods depend on.
[Unit]
Description=Wait for DNS
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/sh -c 'while ! getent ahosts google.com >dev/null; do sleep 1; done'
[Install]
WantedBy=network-online.target
But this significantly delays my start-up time
# systemd-analyze blame
18.068s wait-for-dns.service
...
and if resolv.conf changes again it won't help. So I was wondering if there's a more elegant solution to my problem. Idealy I'd like to be able to trigger a resolv.conf update in both rkt and Docker containers every time it changes.
Run containers on a user defined network so they will use the embedded DNS server that will forward lookups to the systems DNS.
The default docker0 bridge has some special rules that were left in place for legacy support. Using a mounted /etc/resolv.conf is one of those legacy things.
If rkt doesn't support the same type of DNS then the general solution could be to setup a DNS server like Unbound to be a local forwarding resolver. Then containers have a static DNS server to reference.
Related
How do you access remote Docker container by its hostname?
I need to access remote Docker containers by its hostnames (or some constant IP's) for development and testing purposes. I have tried:
looking for any DNS approach (have not found any clues),
importing /ets/hosts (probably impossible),
creating tunnes (only this works but it is very time consuming).
It's the same as running any other process on a host, Docker or not Docker: you access it via the host name or IP address of the host and the port the service is listening on (the first port of the docker run -p argument). Docker containers don't have externally visible individual IP addresses any more than non-Docker HTTP or ssh daemons do.
If you do have DNS infrastructure available to you, you could set up CNAME records to resolve particular service names to the specific hosts that are running them.
One solution that may help you is some sort of service registry; in the past I've used Consul with some success. You can configure Consul with some health checks or other probes ("look for an HTTP service on port 12345 that answers GET / calls"), and it will provide its own DNS service ("okay, http://whatevername.service.consul:12345/ will reach your service on whichever hosts it happens to be running on").
Nothing in the Docker infrastructure specifically helps this. Using /etc/hosts is distinctly not a best practice: the name-to-IP mapping needs to be kept in sync across all machines and you'll start wishing you had a network service to publish it for you, which is exactly what DNS is for.
I've currently got two fairly vanilla CentOS 7 boxes that are running under a Docker Swarm, one a master and the other joined to it. In that swarm, I want to have a stack running that will essentially be my Plex / multimedia system. I've got the docker_compose.yml file for that linked. I can deploy the file to the swarm using the following command:
docker stack deploy --compose-file docker-compose.yml plexsystem
That works fine, it deploys the containers like you would expect. The issue I'm having is that the containers do not have external internet access, so if the container needs to download any files or interact with APIs, they fail. I interact with containers with docker exec -it container /bin/bash and try to ping out, and it always fails.
What do I need to add or change to my docker-compose.yml file so that my networking can work and I can finally get my stack working as it should. I've been banging my head with this one and I cannot figure out how to get swarm networking. Thank you very much!
Checkout your docker daemon file and look for the iptables parameter configuration. Most probably it is set to false and hence it is not able to access the internet.
--iptables=false prevents the Docker daemon from adding iptables rules. If multiple daemons manage iptables rules, they may overwrite rules set by another daemon. Be aware that disabling this option requires you to manually add iptables rules to expose container ports. If you prevent Docker from adding iptables rules, Docker will also not add IP masquerading rules, even if you set --ip-masq to true. Without IP masquerading rules, Docker containers will not be able to connect to external hosts or the internet when using network other than default bridge.
I'm trying to figure out a problem with identical docker containers being run on different hosts, where one container can find/ping/nslookup a domain on a private network, and another can't. One host is OSX 10.11, the other is Ubuntu 16.04. Both are running docker 1.12. I'm using docker-compose to bring up my application, and I'm hoping to figure out what is going on and how to fix it, or some configuration changes I could make without resorting to hardcoding domains or ip addresses that would make the container behave the same on both hosts.
On my OSX box, I have the following dns nameservers set automatically by my domain:
osx:$ cat /etc/resolv.conf
domain redacted.lan
nameserver 172.16.20.19
nameserver 10.43.0.11
I'm aware that resolv.conf isn't used by most OSX tools, but System Preferences > Network shows the same settings.
I have similar settings on my Ubuntu 16 box as well (command from https://askubuntu.com/questions/152593/command-line-to-list-dns-servers-used-by-my-system):
ubu:$ cat /etc/resolv.conf
nameserver 127.0.1.1
search redacted.lan
ubu:$ nmcli device show eno1 | grep IP4.DNS
IP4.DNS[1]: 172.16.20.19
IP4.DNS[2]: 10.43.0.11
Then, on both OSX and Ubuntu, I start my container with this:
$ docker run -it redacted_web bash
And then I run these commands to diagnose my problem:
$ apt-get update
$ apt-get install -y dnsutils
$ cat /etc/resolv.conf
$ nslookup redacted.lan
On OSX, the output from the last 2 commands is:
root#d19f49322fda:/app# cat /etc/resolv.conf
search local
nameserver 192.168.65.1
root#d19f49322fda:/app# nslookup redacted.lan
Server: 192.168.65.1
Address: 192.168.65.1#53
Name: redacted.lan
Address: 172.18.0.23
On Ubuntu, the output is:
root#91e82d652e07:/app# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
search redacted.lan
nameserver 8.8.8.8
nameserver 8.8.4.4
root#91e82d652e07:/app# nslookup redacted.lan
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find redacted.lan: NXDOMAIN
Possible differences I can think of:
On OSX there is a vm running docker, where as on ubuntu it's native
On Ubuntu, docker is run with sudo, possibly picking up different configuration settings
Updated Answer (2017-06-20)
Newer versions of Ubuntu (17.04+) don't use dnsmasq (it now uses systemd-resolved). You'll run into a similar problem with host resolution, but the original solution here no longer works. In fact, Docker containers can't even communicate with systemd-resolvd because it's running its own DNS in a location that's unreachable from within a container.
A good solution on newer versions of Ubuntu is to put the following configuration in /etc/docker/daemon.json (create this file if it doesn't exist):
{
"dns": ["172.16.20.19", "10.43.0.11"],
"dns-search": ["redacted.lan"]
}
This allows you to configure the DNS servers and search domains. The DNS IPs above are from the original question, but you can use your own custom ones too. You probably want to match the DNS config on your host machine. Search domain is optional, and you could entirely omit that line (careful with your commas!). Again, you probably want to match your host machine.
Essentially, what these daemon.json options do, is automatically inject the DNS config and search domain into the config files inside the container for any container that is started on that daemon. This is necessary because you cannot use systemd-resolved from the host to resolve DNS within the container due to limitations of the way systemd-resolved works. The docs are here and here.
Original Answer
The problem is that the host is using dnsmasq to resolve the private IP and Docker is not using dnsmasq on the host.
The simple fix is to turn off dnsmasq on the host machine.
Run sudo vi /etc/NetworkManager/NetworkManager.conf
Comment out this line: #dns=dnsmasq
Run sudo service network-manager restart
Now, you should be able to use the docker container and it will resolve your private DNS correctly.
Check your startup scripts for the docker daemon, it includes the following option to adjust the DNS used when creating containers:
$ dockerd --help
# ...
--dns=[] DNS server to use
--dns-opt=[] DNS options to use
--dns-search=[] DNS search domains to use
On Ubuntu, I believe these settings are in /etc/default/docker which is read by /etc/init.d/docker. Leaving these unset should default to the /etc/resolv.conf values.
Update: Docker's DNS networking documentation has a lot of detail that should point you in the right direction.
if there are no more nameserver entries left in the container’s /etc/resolv.conf file, the daemon adds public Google DNS nameservers (8.8.8.8 and 8.8.4.4) to the container’s DNS configuration.
This looks like it's probably happening in your situation. The filters are looking for local addresses, which I wouldn't think is happening in your situation with a 172.16.0.0/12 private addr, but it's possible.
You might wonder what happens when the host machine’s /etc/resolv.conf
file changes. The docker daemon has a file change notifier active
which will watch for changes to the host DNS configuration.
Note: The file change notifier relies on the Linux kernel’s inotify
feature. Because this feature is currently incompatible with the
overlay filesystem driver, a Docker daemon using “overlay” will not be
able to take advantage of the /etc/resolv.conf auto-update feature.
If this is happening, then giving the daemon a restart would likely resolve it and would indicate you want Docker to start after the NetworkManager.
Update #2: looking over this github issue it's possible that they also included 172.16.0.0/12 to avoid conflicts with docker's bridged networks. If you can be sure to avoid using the same network inside docker, then passing the dns server in /etc/defaults/docker would likely force the correct behavior. There's also a mention at the end of the post about lxc causing conflicts, so if you have that installed and can remove it, give that a try first.
An application server is running as one Docker container and database running in another container. IP address of the database server is obtained as:
sudo docker inspect -f '{{ .NetworkSettings.IPAddress }}' db
Setting up JDBC resource in the application server to point to the database gives "java.net.ConnectException".
Linking containers is not an option since that only works on the same host.
How do I ensure that IP address of the database container is visible to the application server container?
If you want private networking between docker containers on remote hosts you can use weave to setup an overlay network between docker containers. If you don't need a private network just expose the ports using the -p switch and configure the addresses of the host machine as the destination IP in the required docker container.
One simple way to solve this would be using Weave. It allows you to create many application-specific networks that can span multiple hosts as well as datacenters. It also has a very neat DNS-based service discovery mechanism.
I should disclaim, I am one of Weave engineering team.
Linking containers is not an option since that only works on the same host.
So are you saying your application is a container running on docker server 1 and your db is a container on docker server 2? If so, you treat it like ordinary remote hosts. Your DB port needs to be exposed on docker server 2 and that IP:port needs to be configured into your application server, typically via environment variables.
The per host docker subnetwork is a Private Network. It's perhaps possible to have this address be routable, but it would be much pain. And it's further complicated because container IP's are not static.
What you need to do is publish the ports/services up to the host (via PORT in dockerfile and -p in your docker run) Then you just do host->host. You can resolve hosts by IP, Environment Variables, or good old DNS.
Few things were missing that were not allowing the cross-container communication:
WildFly was not bound to 0.0.0.0 and thus was only accepting requests on eht0. This was fixed using "-b 0.0.0.0".
Firewall was not allowing the containers to communication. This was removed using "systemctl stop firewall; systemctl disable firewall"
Virtual Box image required a Host-only adapter
After this, the containers are able to communicate. Complete details are available at:
http://blog.arungupta.me/2014/12/wildfly-javaee7-mysql-link-two-docker-container-techtip65/
the default dns 8.8.8.8 and 8.8.4.4 are blocked by my nation's firewall, and I do not want to stop the container for I install a lot of software in it.
If you want to change the DNS, you can just do it in the same way you would on a normal host and this should just work. For example editing the /etc/resolv.conf file in the container.
Although if it is entirely blocked, I'd recommend a more permanent solution.
In /etc/default/docker (may be /etc/default/docker.io on Debian based hosts) you can add a line to set the default DNS on new containers. This line would look something like
DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4"
You can just add a new -dns x.x.x.x for every DNS server you wish to use to resolve.
Once you have done this you will need to restart the Docker process. This is done in a few ways depending on the host again. For Debian based hosts:
service docker.io restart
When you docker run a new container, it will have those DNS servers set.