Docker not picking up flannel subnet. Any help will be greatful. I am
using coreos as my container Linux and docker version is 1.12.6
my docker startup file look like below.
Flannel is working as expected
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=containerd.service docker.socket network.target
Requires=containerd.service docker.socket
[Service]
Type=notify
EnvironmentFile=-/run/flannel/flannel_docker_opts.env
# the default is not to use systemd for cgroups because the delegate issues
still
# exists and systemd currently does not support the cgroup feature set
required
# for containers run by docker
ExecStart=/usr/lib/coreos/dockerd --host=fd:// --
containerd=/var/run/docker/libcontainerd/docker-containerd.sock
$DOCKER_OPTS $DOCKER_CGROUPS $
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting
overhead
# in the kernel. We recommend using cgroups to do container-local
accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker
containers
Delegate=yes
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/docker.service.d/40-flannel.conf
[Unit]
Requires=flanneld.service
After=flanneld.service
[Service]
EnvironmentFile=/etc/kubernetes/cni/docker_opts_cni.env
# /etc/systemd/system/docker.service.d/40-storage.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --host=fd:// --graph="/abc/docker" $DOCKER_OPTS
$DOCKER_CGROUPS $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ
# /etc/systemd/system/docker.service.d/50-insecure-registry.conf
[Service]
Environment=DOCKER_OPTS='--insecure-registry="10.x.x.x:5000"'
# /etc/systemd/system/docker.service.d/50-require-flannel.conf
[Unit]
Requires=flanneld.service
After=flanneld.service
Check if /run/flannel/flannel_docker_opts.env exists and its content.
My /run/flannel/subnet.env looks like:
FLANNEL_NETWORK=10.252.0.0/16
FLANNEL_SUBNET=10.252.127.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
Add --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} to dockerd args.
If this cannot work, rm -rf /var/lib/docker/overlay2/* and restart docker.service
Related
My ultimate goal is to have docker wait for my mergerfs mount before it starts docker and for mergerfs to wait for 3 rclone mounts before it starts itself. I've tried 'requires' and bash script for 'execprestart' and 'requiresmountsfor', 'wants=', 'after=', and a combination of all of these things. I've even tried mounting mergerfs via /etc/fstab. None of it works. The systemd services just 'fail' and don't try again because of requirements etc. Nothing waits or keeps trying.
What's the proper way to mount rclone, have mergerfs wait until those mounts are successful, then have docker override wait for mergerfs to successfully mount to start?
docker.service.d/override.conf
[Unit]
After=merge.service
{tv,movies,music}.service
[Unit]
Description=tv
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
Environment=RCLONE_CONFIG=/data/rclone/config/rclone.conf
RestartSec=5
ExecStart=/usr/bin/rclone mount tv: /mnt/media/tv \
--<options>
ExecStop=/bin/fusermount -uz /mnt/media/tv
ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --rc-addr 127.0.0.1:5572 _async=true
Restart=on-failure
User=user
Group=user
[Install]
WantedBy=multi-user.target
merge.service
[Unit]
Description=merge
After=tv.service movies.service music.service
[Service]
Type=forking
ExecStart=/usr/bin/mergerfs /mnt/local:/mnt/media=NC /mnt/merge -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full,nonempty
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
A quick way to accomplish the goal is to test for existence of the mount via pre start script and have systemd retry on failure.
[Service]
ExecStartPre=/usr/bin/test -f /mnt/media/tv
RestartSec=5s
Restart=on-failure
My VM crashed because it was out of memory. After rebooting the machine docker was not running:
systemctl status docker
● docker.service
Loaded: error (Reason: Invalid argument)
Active: inactive (dead)
Dec 19 08:18:21 my-vm-single-instance systemd[1]: [/lib/systemd/system/docker.service:1] Assignment outside of section. Ignoring.
Dec 19 08:18:21 my-vm-single-instance systemd[1]: docker.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
I installed docker using the offical documentation: https://docs.docker.com/engine/install/debian/
The VM is running:
Debian GNU/Linux 9 (stretch)
Docker version 19.03.14, build 5eb3275d40
docker-compose version 1.25.4, build 8d51620a
I got docker up and running again with
dockerd
However I would like to get it running again through systemctl.
The contents of /lib/systemd/system/docker.service are:
Environment="GOOGLE_APPLICATION_CREDENTIALS=/etc/docker/key.json"
Any ideas how to fix this problem?
If the docker.service contains only one line as it mentioned it's bogus.
As it says
docker.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
the executions scripts are missing at least.
Here is a sample service file:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
This is my default service file. I've never modified it after installation.
When I am trying to start docker using the command:
sudo systemctl start docker
I am getting below error
Failed to start docker.service: Unit not found.
I tried finding some suggestions over the web to resolve this issue and followed that but it didn't solve the issue.
Cannot start docker daemon in CentOS7
This is my docker.socket file [which is just a copy-paste of one of the answer]
[Unit]
Description=Docker Socket for the API
PartOf=docker.service
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
This is the error I am getting error while starting docker.socket
sudo systemctl start docker.socket
See "systemctl status docker.socket" and "journalctl -xe" for details.
output of "systemctl status docker.socket"
systemctl status docker.socket
systemd[1]: Socket service docker.service not loaded, refusing.
systemd[1]: Failed to listen on Docker Socket for the API.
docker version details
Client: Docker Engine - Community
Version: 19.03.2
API version: 1.40
Go version: go1.12.8
Git commit: 6a30dfca03
Built: Thu Aug 29 05:26:30 2019
OS/Arch: linux/amd64
Experimental: false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
To me, it looks like that docker.service is an issue. Could you please suggest how I can resolve it.
There should be a docker.service unit file at either /lib/systemd/system or /etc/systemd/system. Mine looks like what's shown below.
If you have one there, you can try to make sure it's enabled via:
sudo systemctl enable docker.service
Here's an exmaple of the docker.service unit file:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I followed the steps provided in the documentation here to add tls security for docker api. Certificates are located in ~/.docker/ as well as /etc/docker/ssl/ folders. I added override.conf to /etc/systemd/system/docker.service.d/ with content
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem
Then, I used daemon-reload and docker start
$ systemctl daemon-reload
$ service docker start
The errors in journalctl -xe is:
-- Unit docker.socket has finished starting up.
--
-- The start-up result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Start request repeated too quickly.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 15 21:43:24 cynicalplyaground systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has failed.
--
-- The result is RESULT.
Jan 15 21:43:24 cynicalplyaground systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 15 21:45:01 cynicalplyaground CRON[12769]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jan 15 21:45:01 cynicalplyaground CRON[12768]: pam_unix(cron:session): session closed for user root
How can I sort this issue?
In the present case the same error occured after the latest manjaro update (2020-01-20).
Tried to change the systemd docker service, as adviced in other cases, but I reverted those changes and finally this was solved with:
a reboot of the system
(like advised here: https://www.reddit.com/r/archlinux/comments/7ya4ug/installing_docker_on_arch_linux/)
Getting to the root of the problem;
systemctl status docker.service
has this:
/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Trying to run that command, it complains about
unable to configure the Docker daemon with file /etc/docker/daemon.json: EOF
ls -l /etc/docker/daemon.json
-rw-r--r-- 1 root root 0 Jul 30 10:32 /etc/docker/daemon.json
NOTE that the JSON file is empty. Delete it.
For me it was because the docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables.
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
dockerd, should start fine after switching to iptables-legacy.
I have the same issue and just modify the "/usr/bin/dockerd" to "/usr/sbin/dockerd", then it works.
You can check the dockerd path first.
in my case... the host was part of a docker swarm...but the IPv6 was no longer reachable or automatically assigned to the host...
I manually add the old_IPv6
ip -6 address add 28xx:xxxx:x:x:xx:ebff:fe14:xxx dev ens3x
the journalctl -u docker.service mention:
level=fatal msg="Error starting cluster component: could not find local IP address: dial udp [2xxx:xxx:xxxx:xxx]:2377: connect: network is unreachable"
after add manually the IPv6 I was able to start docker so with docker running I leave the "swarm" and reboot
docker swarm leave --force
after reboot the docker services run as usual
For me it was missing disk space. Reboot also helped, but I was stillnot able to build any container.
After pruning some outdated stuff from the docker volumes I was able to continue.
I faced a similar issue on Ubuntu because I added the hosts option to /etc/docker/daemon.json file. That's ok, but for systems that use systemd it may cause conflict with the arguments passed to dockerd on start.
The solution was to delete the /etc/docker/daemon.json's hosts entry and set this config on file /etc/systemd/system/docker.service.d/options.conf.
$ cat /etc/systemd/system/docker.service.d/options.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://
After that, restart the service.
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
You may check that your changes has been applied by running docker info. Also, you may note on the docker service status that Drop-In field is using the options.conf created, and dockerd was executed with the specified host list.
$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset>
Drop-In: /etc/systemd/system/docker.service.d
└─options.conf
Active: active (running) since Fri 2022-11-18 01:02:18 EST; 1h 50min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 1111 (dockerd)
Tasks: 18
Memory: 58.5M
CPU: 1.294s
CGroup: /system.slice/docker.service
└─1111 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://
References:
Daemon configuration file
Control Docker with systemd
I had a similar issue on nixOS installed in a btrfs filesystem.
For me the solution was to add virtualisation.docker.storageDriver = "btrfs"; to my /etc/nixos/configuration.nix
Which according to the docker docs should equate to adding the following to /etc/docker/daemon.json in most other distros:
{
"storage-driver": "btrfs"
}
I was able to solve the problem by disabling the firewalld
systemctl disable firewalld
systemctl stop firewalld
I have installed VirtualBox and installed Ubuntu server version in VirtualBox VM. My host machine is Windows 10.
I have also installed Docker in my host Windows box. My intention is to use the docker CLI in Windows to connect to docker daemon (server) inside the VM.
I have made the changes in the Ubuntu VM and it is listening at port 2375.
tcp 0 0 127.0.0.1:2375 0.0.0.0:* LISTEN 2305/dockerd
Also I have set the environment variable DOCKER_HOST in my host(Windows) to the VM machine IP and port.
set DOCKER_HOST=tcp://192.168.56.107:2375
My Windows machine IP is 192.168.56.1 and the ping is working fine.
Pinging 192.168.56.107 with 32 bytes of data:
Reply from 192.168.56.107: bytes=32 time<1ms TTL=64
Reply from 192.168.56.107: bytes=32 time<1ms TTL=64
But when I try to connect from my Windows machine, it gives the following error:
error during connect: Get http://192.168.56.107:2375/v1.27/info: dial tcp 192.168.56.107:2375: connectex: No connection could be made because the target machine actively refused it.
Please find docker info output:
controller#ubuntuserver:~$ docker info
Containers: 4
Running: 0
Paused: 0
Stopped: 4
Images: 2
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-50-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.79GiB
Name: ubuntuserver
ID: AWDW:34ET:4J2J:2NWB:UPK7:EQHB:W64E:22AT:W6J4:BMRD:NDO6:CNR2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: API is accessible on http://127.0.0.1:2375 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
WARNING: No swap limit support
cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
Can you please help me to resolve this?
You need to configure the Docker daemon in your ubuntu server in order for it to accept tcp connection.
By default Docker listen on the unix socket /var/run/docker.sock.
To configure your daemon, you can have a look at the documentation here
Step-by-step configuration (in this example, everything is done on the Ubuntu VM) :
Configure the daemon
On Ubuntu, by default you are using systemd. You need to edit the configuration file (usually located in /lib/systemd/system/docker.service) :
[Service]
ExecStart=/usr/bin/dockerd --containerd=/run/containerd/containerd.sock -H tcp://0.0.0.0:2375
With this example, the Docker daemon no longer listen on the unix socket. It only listen on tcp call from localhost.
Restart the daemon :
$> sudo systemctl daemon-reload
$> sudo systemctl restart docker.service
Configure the client (still on the VM)
After restarting the daemon, your docker client does not work anymore (as you've just told the client to only listen to tcp connection). Thus, if you do docker image ls it should not respond. In order for your client to work, you need to tell it which server to connect to :
$> export DOCKER_HOST="tcp://0.0.0.0:2375"
Now, your client should be able to connect to the daemon (i.e : docker image ls should print all the images)
This should work fine on your Ubuntu server. You just need to apply the same client configuration on Windows. If it does not work on Windows, then it means something else is blocking the trafic (probably a firewall).
Hope this helps.
Maybe your server ICMP protocol has been prohibited,check it by this cmd:
iptables -L INPUT --line-numbers
and if terminal shows:
and delete this record by cmd
iptables -D INPUT 7
Hope this helps.