I am a newbie to docker. When I go through docker tutorial, I saw that "Docker client can communicate with more than one daemon". What does that mean exactly?
By default, the Docker daemon listens on a Unix socket, /var/run/docker.sock. However, Docker can also be configured to listen on a TCP socket. In fact, it is often configured this way on Mac and Windows systems because Docker is actually running inside a virtual machine and the default Docker socket is not available on the host filesystem.
Because there are different ways of connecting to Docker, you must be able to configure the Docker client to connect to a Docker daemon at a specific location. You can do this using the DOCKER_HOST environment variable. You can point this at a network location:
export DOCKER_HOST=tcp://192.168.99.101:2376
Or at an alternate socket location:
export DOCKER_HOST=unix:///tmp/docker.sock
If you have Docker configured to listen for tcp connections, you can use the Docker client on a single machine to communicate with Docker on multiple hosts (but if you decide to do something like this, read through "Protect the Docker daemon socket").
Per the Docker Documentation,
The Docker client can communicate with more than one daemon.
This means that the command-line utility docker can connect to different services that run in the background,
Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers.
So for example, you could configure the daemon to run on a separate machine and connect to it from your workstation.
Related
I use docker-compose to spin up a few containers as part of an application I'm developing. One of the containers needs to start a docker swarm service on the host machine. On Docker for Windows and Docker for Mac, I can connect to the host docker daemon using the REST Api by using the "host.docker.internal" DNS name and this works great. However, if I run the same compose file on linux, "host.docker.internal" does not work (yet, seems it may be coming in the next version of docker). To make matters worse, on Linux I can use network mode of "host" to work around the issue but that isn't supported on Windows or Mac.
How can I either:
Create a docker-compose file or structure a containerized application to be slightly different based on the host platform (windows|mac|linux) without having to create multiple docker-compose.yml files or different application code?
Access the host docker daemon in a consistent way regardless of the host OS?
If it matters, the container that is accessing the docker daemon of the host is using the docker python sdk and making api calls to docker over tcp without TLS (this is used for development only).
Update w/ Solution Detail
For a little more background, there's a web application (aspnet core/C#) that allows users to upload a zip file. The zip file contains, among other things, an exported docker image file. There's also an nginx container in front of all of this to allow for ssl termination and load balancing. The web application pulls out the docker image, then using the docker daemon's http api, loads the image, re-tags the image, then pushes it to a private docker repository (which is running somewhere on the developer's network, external to docker). After that, it posts a message to a message queue where a separate python application uses the python docker library to deploy the docker image to a docker swarm.
For development purposes, the applications all run as containers and thus need to interact with docker running on the host machine as a stand alone swarm node. SoftwareEngineer's answer lead me down the right path. I mapped the docker socket from the host into the web application container at first but ran into a limitation of .net core that won't be resolved until .net 5 which is that there's no clean way of doing http over a unix socket.
I worked around that issue by eventually realizing that nginx can reverse proxy http traffic to a unix socket. I setup all containers (including the dynamically loaded swarm service from the zips) to be part of an overlay network to give them all access to each other and allowing me to hit an http endpoint to control the host machine's docker/swarm daemon over http.
The last hurdle I ran into was that nginx couldn't write to the mapped in /var/run/docker.sock file so I modified nginx.conf to allow it to run as root within the container.
As far as I can tell, the docker socket is available at the path /var/run/docker.sock on all systems. I have personally verified this with a recent Linux distro (Ubuntu), Windows 10 Pro running Docker for Windows (2.2.0) with both WSL2 (Ubuntu and Alpine) and the windows cmd (cli) and powershell. From memory, it works with OSX too, and I used to do the same thing in WSL1.
Mapping this into a container is achieved on any terminal with the -v, --volume, or --mount flags. So,
docker container run -v /var/run/docker.sock:/var/run/docker.sock
Mounts the socket into an identical path within the container. This means that you can access the socket using the standard docker client (docker) from within the container with no extra configuration. Using this path inside a Linux container is recommended because the standard location and is likely to be less confusing to anyone maintaining your code in the future (including yourself).
I'm seeing problems mounting local volumes when running docker on Windows 10. The problems only appear when I have my company VPN enabled.
C:\Users\matt> docker run --rm -v d:/tmp:/data alpine ls /data
my_local_test_file.txt
When connected to VPN, I get this:
C:\Users\matt> docker run --rm -v d:/tmp:/data alpine ls /data
C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: error while creating mount source path '/host_mnt/d/tmp': mkdir /host_mnt/d: file exists.
Docker version is 17.12.0-ce-win47
I believe the problem is that docker uses the network when mounting local volumes, and the VPN routes ALL network traffic via the VPN gateway, so docker can't see the local drive.
Is there a workaround for this?
I'm aware I could run docker within a linux VM, or use docker toolbox. Neither of those are particularly good.
Is there another possible workaround?
the VPN routes ALL network traffic via the VPN gateway
You're probably right, in which case all traffic routed from Docker client to Docker daemon will also be through the VPN. When you use Docker CLI on Windows, it will connect to the Docker daemon which is accessible through the network. Using a VPN may disrupt this mechanism.
I think what's happening is:
When VPN is disabled, you use the Docker daemon on your machine and everything works
When VPN is enabled, another Docker daemon is used either because your VPN redirect traffic addressed to your Docker host (127.0.0.1 by default or set via -H flag or DOCKER_HOST env variable). This means that somehow this IP or host exists on your VPN network and there is a Docker daemon listening on it (which is kind of odd admittedly, it may be risky to use that daemon)
If that's really happening, you'll certainly see different output from docker ps -a, docker images, etc. because you are connecting to different daemons. (the daemon accessible through your VPN is actually being owned by someone else, you'd better not use it!)
What you can do:
Do not route 127.0.0.1 (or whatever is configured as Docker host) through your VPN
Action to take will depend on the VPN software you are using, or you can add route directly on your windows machine (here is a good article on the subject)
Find out your IP when VPN is enabled and configure Daemon to listen to this IP
When your VPN is enabled, run ipconfig /all and find the interface used by your VPN and it's IP address, for example 10.142.0.12 (you can compare output before/after enabling VPN to identify which one it is)
Configure your Docker daemon to listen this IP address and restart it. Either use the UI, or on Windows config file is located at %programdata%\docker\config\daemon.json by default, you need to specify "hosts": ["10.142.0.12", "127.0.0.1"] for example (see docs for details)
Configure Docker host to 10.142.0.12 when VPN is enabled, either by setting environment variable DOCKER_HOST=10.142.0.12 or with client docker -H 10.142.0.12 <cmd>
/!\ Security note: this may present a security issue as anyone knowing your IP on the VPN network will be able to use the Daemon on your machine
Hope this helps. I am not a Windows expert so I was not able to give details on Windows-related issues, but feel free to ask details if needed.
Now I have two laptops (not necessary in one local network) and docker installed on both of them. My goal is to run docker daemon on the first laptop and be able to execute commands using docker client on the second laptop. What should I do to achieve the goal?
Follow the public API? Docker Engine API
Setup Docker to listen for TCP connections on a specified port and protect that port with TLS. You must setup some environment variables so the Docker client communicates with the Docker daemon.
Here's the relevant documentation:
https://docs.docker.com/engine/admin/
https://docs.docker.com/engine/security/https/
Enjoy, and have fun.
new to docker and docker swarm. Trying docker and docker swarm both.
initially i had started a docker daemon and was able to connect it on http port i.e. 2375. I had installed docker colud plugin in jenkins and added http://daemon-IP:2375 and was able to create containers. well it creates a container, does my build inside it and destroys the container.
My Query is, will i be able to connect to docker swarm on http port, the same way i a am connecting to a standalone docker daemon ? is there any documentation on it. or the my understanding about the swarm is wrong.
please suggest.
Thanks
Yeah you can connect to a remote host the same way you are doing via the Unix Socket. People very often forget that docker is a client-server architecture and your "docker run..." commands are basically just commands issued by the docker client.
If you set certain environment variables:
DOCKER_HOST=tcp:ip.address.of.host:port
DOCKER_TLS_VERIFY=1
DOCKER_CERTS=/directory/where/certs/are
(The last two are optional for TLS connections, which I would highly recommend. You'd have to setup https://docs.docker.com/engine/security/https/ which is recommended for a production environment)
Once you've set your DOCKER_HOST environment variable, if you issue a docker command and get a response, it will be from the remote host if everything is setup correctly.
An application server is running as one Docker container and database running in another container. IP address of the database server is obtained as:
sudo docker inspect -f '{{ .NetworkSettings.IPAddress }}' db
Setting up JDBC resource in the application server to point to the database gives "java.net.ConnectException".
Linking containers is not an option since that only works on the same host.
How do I ensure that IP address of the database container is visible to the application server container?
If you want private networking between docker containers on remote hosts you can use weave to setup an overlay network between docker containers. If you don't need a private network just expose the ports using the -p switch and configure the addresses of the host machine as the destination IP in the required docker container.
One simple way to solve this would be using Weave. It allows you to create many application-specific networks that can span multiple hosts as well as datacenters. It also has a very neat DNS-based service discovery mechanism.
I should disclaim, I am one of Weave engineering team.
Linking containers is not an option since that only works on the same host.
So are you saying your application is a container running on docker server 1 and your db is a container on docker server 2? If so, you treat it like ordinary remote hosts. Your DB port needs to be exposed on docker server 2 and that IP:port needs to be configured into your application server, typically via environment variables.
The per host docker subnetwork is a Private Network. It's perhaps possible to have this address be routable, but it would be much pain. And it's further complicated because container IP's are not static.
What you need to do is publish the ports/services up to the host (via PORT in dockerfile and -p in your docker run) Then you just do host->host. You can resolve hosts by IP, Environment Variables, or good old DNS.
Few things were missing that were not allowing the cross-container communication:
WildFly was not bound to 0.0.0.0 and thus was only accepting requests on eht0. This was fixed using "-b 0.0.0.0".
Firewall was not allowing the containers to communication. This was removed using "systemctl stop firewall; systemctl disable firewall"
Virtual Box image required a Host-only adapter
After this, the containers are able to communicate. Complete details are available at:
http://blog.arungupta.me/2014/12/wildfly-javaee7-mysql-link-two-docker-container-techtip65/