Running Docker Compose on Docker Swarm - docker

I've started a docker master with:
docker swarm init --advertise-addr <MANAGER-IP>
so, im trying to have my shell point to swarm master via:
eval $(docker-machine env --swarm <MANAGER-IP>)
but it's giving me an error: Host does not exists
docker info:
-bash-4.2$ docker info
Containers: 18
Running: 1
Paused: 0
Stopped: 17
Images: 20
Server Version: 1.12.0
Storage Driver: devicemapper
Pool Name: docker-253:1-25646-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.124 GB
Data Space Total: 107.4 GB
Data Space Available: 13.4 GB
Metadata Space Used: 5.071 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.142 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null overlay
Swarm: active
NodeID: 05szzy2z96ypgl5k21swggoil
Is Manager: true
ClusterID: a2wrfuga2tu4cm4k0lxxorqtm
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot interval: 10000
Heartbeat tick: 1
Election tick: 3
Dispatcher:
Heartbeat period: 5 seconds
CA configuration:
Expiry duration: 3 months
Node Address: 10.193.46.89
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51 GiB
Name: scsor0004331002.rtp.openenglab.netapp.com
ID: T52U:6MWQ:XEDM:2TGH:ITLQ:YD6B:R3MR:MWF5:CFBM:G6PX:W4LG:6SR7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: eugenepark3
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Anyone know what i need to put for eval $(docker-machine env --swarm <MANAGER-IP>) so my compose can run on swarm cluster?
Im supposed to put master name but i dont know how to find it
-bash-4.2$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
05szzy2z96ypgl5k21swggoil * scsor0004331002.rtp.openenglab.netapp.com Ready Active Leader
59t110b0wjhitj1fr8erys600 scsor0004331003.rtp.openenglab.netapp.com Ready Active
dhm6utu2w3dw1to0zh3n71moq scsor0004331004.rtp.openenglab.netapp.com Ready Active

You're mixing up the container based swarm commands with the newer swarmkit based Swarm that's been embedded directly into the Docker CLI. With the new version of Swarm, docker-compose isn't directly supported, yet. Consider this a beta product that works well for a limited scope. You can try the experimental release of the docker engine which adds support for DAB files that are managed with the docker stack CLI. The DAB files are exported from docker-compose bundle and then imported into Docker. This feature is still very experimental and expected to change.
Without that, anything with docker-compose will only operate on a single docker engine since the swarm access is all done under a different docker service CLI interface.

Related

Cannot deploy Docker Swarm stack in rootless mode, mkdir /var/lib/docker: permission denied

I've set up Docker in rootless mode under Ubuntu 20.04 and Debian 11 (in my case, using Ansible and this role). I want to deploy a simple Docker stack to the node via Docker Swarm. No other hosts are involved, just one Swarm node from the same machine, acting as a manager.
I can run this project with Docker and Docker Compose just fine, also in rootless mode. All that changes for the rootless setup is that DOCKER_HOST is overwritten in .bashrc:
export XDG_RUNTIME_DIR="/run/user/1000"
export DOCKER_HOST="unix:///run/user/1000/docker.sock"
When I deploy the stack though, none of the services can start (here is an excerpt of the status):
$ docker stack deploy -c docker-stack.yml demo-stack
$ docker stack ps demo-stack --no-trunc
jig6zyewkem2g225509x91nt5 demo-stack_db.1 registry.example.com/db:v1.20.2 bullseye Shutdown Rejected 15 seconds ago "mkdir /var/lib/docker: permission denied"
ox6x5w7du9o5ew2v70g5mfg9e demo-stack_redis.1 registry.example.com/redis:v1.20.2 bullseye Shutdown Rejected 15 seconds ago "mkdir /var/lib/docker: permission denied"
ipme447wrrsjc8jw6cpfak4hq demo-stack_web.1 registry.example.com/web:v1.20.2 bullseye Shutdown Rejected 14 seconds ago "mkdir /var/lib/docker: permission denied"
The services all error with mkdir /var/lib/docker: permission denied. I suppose that it tries to start them as if the system was using rootful Docker, but it's a rootless installation.
I guess the question is: how do I get the Swarm node (which is the very same machine) to use the correct Docker rootless configuration for launching the services? That would include using the correct DOCKER_HOST configuration.
I am unsure if this is even supposed to work. I hear that overlay networks are not supported, but I am only on one machine, so I don't really need this. I do need Swarm for its usable implementation of secrets (compared to the mock implementation from Docker Compose).
Note that I have the same setup with Docker running in (normal) rootful mode, and there, all services can be started. It's therefore not an issue with the Docker stack file itself.
More details with docker info:
Client:
Context: default
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 12
Server Version: 20.10.13
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: hpzsmez48acse9yo1frnx37fo
Is Manager: true
ClusterID: zkv7wsoun193kyvbxe1k3hdph
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 127.0.0.1
Manager Addresses:
127.0.0.1:2377
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
runc version: v1.0.3-0-gf46b6ba2
init version: de40ad0
Security Options:
seccomp
Profile: default
rootless
cgroupns
Kernel Version: 5.10.0-13-amd64
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.936GiB
Name: bullseye
ID: 3R5P:2UV6:FIP4:UIJV:TDNQ:35DT:DEDI:SMGN:FDUY:JSWO:FRU6:O2HF
Docker Root Dir: /home/vagrant/.local/share/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
The solution is simple: Docker Rootless does not work with Docker Swarm. You can have either, but not both.

How to list only images located in a specific, private registry

I'm having problems getting a listing of images from a specific registry that I've set up on a local server, or, maybe, I'm having issues publishing them to that registry in the first place, as this is my first adventure into docker registries, I may just be confused with the terms used.
There's an old question, here, that kind of looks like what I want to achieve, but it appears that docker has gained built-in support for this, in the meanwhile, so the methods mentioned here are no longer relevant.
I have 2 servers (for the purpose of this question):
rancher-server: This server has a rancher:v2.6.0 container running and a registry:2 container.
k8s-server: This is just a freshly installed server, with the docker and kubernetes packages installed, that I want the rancher server to administer.
On k8s-server, I'm trying to spin up a docker image rancher/rancher-agent:v2.6.0 with a few arguments, that should let it relinquish control to the rancher server.
The trick here is, that this is all required to work without internet access (currently there IS internet access, but it's a PoC for a task that requires to be air-gapped). For the purposes of this question, I really just want to be able to spin up docker containers on k8s-server, using the registry on rancher-server.
Currently, this is the state of rancher-server:
# docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b9a15ea00d5e registry:2 "/entrypoint.sh /e..." About an hour ago Up About an hour 0.0.0.0:5000->5000/tcp local-registry
1b6bc6b88a8e 08c9693b4357 "entrypoint.sh 08c..." 26 hours ago Up 2 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp goofy_minsky
# docker image ls --all (the list is big, this is just a sample):
REPOSITORY TAG IMAGE ID CREATED
rancher/rancher-agent v2.6.0 9c35a790aa16 2 weeks ago
rancher-server.example.com:5000/rancher/rancher-agent v2.6.0 9c35a790aa16 2 weeks ago
# docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 225
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 66aedde759f33c190954815fb765eedc1d782dd9 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-1160.41.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.701 GiB
Name: rancher-server
ID: SA2T:G2IA:CGER:6BC5:HIV2:4T6T:LF3Q:2YVS:SYU7:SQ5V:ACUS:BMEX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
rancher-server.example.com:5000
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
On the k8s-server, I try to list the contents of that registry:
# docker image ls --all rancher-server.example.com:5000
REPOSITORY TAG IMAGE ID CREATED SIZE
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 66aedde759f33c190954815fb765eedc1d782dd9 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-1160.41.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.701 GiB
Name: k8s-server
ID: QETJ:QSPQ:VS36:OOOA:ZPYL:CDHK:AJ5G:N4BD:ZQUH:UL6O:PHAB:5UOE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
rancher-server.example.com:5000
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
I had to jump through a few hoops to get there, in the first place, marking the registry as unsafe in /etc/docker/daemon.json on the k8s-server and disabling selinux on the rancher-server, for example.
I've tried to docker login rancher-server.example.com:5000 first, but that made no difference. It does look like, to me, that the k8s-server is configured correctly, but that the images on rancher-server haven't been tagged/pushed properly, but when I look back at the registry, I don't know how to do it differently, and, as far as I understand the registry, it looks fine to me?
I've changed the server names for anonymity and the output has been lightly edited for presentation.
EDIT:
I think I found a clue to what's happening here, it turns out that I can actually run the images from this registry remotely, just fine, it just so happens that I have no way to discover the names of the images, however, if I do a docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher-server.example.com:5000/rancher/rancher-agent:v2.6.0 --server https://rancher-server.example.com:5000 --token <token> --ca-checksum <ca-checksum> --etcd --controlplane it actually pulls and runs the container, so it looks like the registry itself is fine, but maybe the index isn't?

Can't recover docker swarm from pending state

There was a crash and I have this issue now where it says docker swarm status is pending and the node status is UNKNOWN. This is my docker info result
swarm#swarm-manager-1:~$ docker info
Containers: 270
Running: 0
Paused: 0
Stopped: 270
Images: 160
Server Version: 1.12.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1211
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null overlay
Swarm: pending
NodeID: d9hq8wzz6skh9pzrxzhbckm97
Is Manager: true
ClusterID: 5zgab5w50qgvvep35eqcbote2
Managers: 1
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: HIDDEN
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-91-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Name: swarm-manager-1
ID: AXPO:VFSV:TDT3:6X7Y:QNAO:OZJN:U23R:V5S2:FU33:WUNI:CRPK:2E2C
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
This is my docker node ls result:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
9tlo3rg7tuc23xzc3am28lak1 swarm-worker-1 Unknown Active
d9hq8wzz6skh9pzrxzhbckm97 * swarm-manager-1 Unknown Active Leader
I've tried restarting docker engine and the VM but doesn't help in any way. The system is actually running as when I say docker ps in the worker it shows all the containers but on the manager there is nothing on docker ps.
Any idea?
In my experience with Swarm the only solution to similar trouble was to destroy the swarm. And when you do this you should probably also do a docker system prune (only if theres nothing valuable that could be deleted) and service docker restart. And then set up a new swarm.
It sucks. I know
Instead of just rebuilding the whole swarm all at once, you can attempt to remove and re-add each node one at a time - the advantage of this is that the swarm state is not destroyed and, on larger swarms, services can continue while you fix it. This process is considerably more complicated when you don't have a quorum of managers, though.
First, note the node IDs (I'll refer to here as $WORKER_ID and $MANAGER_ID).
On manager node:
docker node update --availability drain $WORKER_ID
^ This is optional, but it's a good habit when working with live services on a swarm.
docker swarm join-token manager
^ This command will give you the join command to run on each node after it's removed. I'll refer to it as $JOIN_COMMAND below. We will demote the worker once the manager re-joins.
On worker:
docker swarm leave
$JOIN_COMMAND
This node is now re-joined as a manager, but I'll continue calling it the 'worker' to avoid confusion.
On manager:
docker node rm $WORKER_ID
docker node update --availability drain $MANAGER_ID
docker swarm leave -f
$JOIN_COMMAND
docker node rm $MANAGER_ID
docker node ls
Find the worker's new id (pay attention to the hostname, not the role) -> $NEW_WORKER_ID
docker node demote $NEW_WORKER_ID
Your swarm should be refreshed - if there were more nodes, the services running on each would have migrated across the swarm when you drained each node.
If it still doesn't work (and regardless), you really should consider upgrading to docker v17.06 or newer. Swarm networking was very unstable before that, causing a lot of issues stemming from race conditions.

Unable to start container

I'm new to docker and trying to implement docker using chef on centos 7.1
Below is the basic code I wrote for installing, pulling centos image and creating container.
All the 3 tasks are executing successfully. Since the containers are in stop mode I tried to start manually by typing docker start containerid. When I checked docker ps I found the container is not started . I tried for several times but couldn't start the container.
Docker code using chef
docker_service 'default' do
action [:create, :start]
end
docker_image 'centos' do
action :pull
end
docker_container 'check2' do
repo 'centos'
action :create
end
Docker info:
Containers: 6
Running: 1
Paused: 0
Stopped: 5
Images: 3
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-8:1-523814-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 441.3 MB
Data Space Total: 107.4 GB
Data Space Available: 28.08 GB
Metadata Space Used: 1.159 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.4.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 6.807 GiB
ID: R24R:ORHY:XJQW:2HNI:U5TV:UGF7:B7VX:P6Z6:UHSR:YIMR:VGJT:4URU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Would you please help me
By default, the centos image will run /bin/bash which will exit immediately without a tty (-t) and standard input (-i) available.
Try running something in the container
docker_container 'check2' do
repo 'centos'
command 'top -b -d 5'
end

Docker daemon restart and reattaching to containers

If I kill my docker daemon process and then restart it, any containers that were running are now listed with Exited status and cannot be restarted using docker-compose as it will complain about the container name already being in use.
Docker containers can be started again using docker start but this could be hard when you've many containers.
Is there any way to restart the docker daemon, leave containers running (so as to not disrupt traffic) and have the daemon reattach to the containers?
uname -a:
Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
docker info:
Containers: 23
Running: 2
Paused: 0
Stopped: 21
Images: 16
Server Version: 1.11.1
Storage Driver: devicemapper
Pool Name: docker-253:0-1567975-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.738 GB
Data Space Total: 107.4 GB
Data Space Available: 28 GB
Metadata Space Used: 7.688 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.14 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 7.64 GiB
Name:
ID:
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username:
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Looks like this is something that will be addressed in 1.12:
https://github.com/docker/docker/issues/2658

Resources