Docker restartmanger prevents restart despite restart policy - docker

Question

I have a docker container that likes to shutdown without restarting, despite having a restart=unless-stopped policy set.
Other containers are running on the same host (with similar startup configuration parameters) which I don't have any problems with. The host is a node in a swarm on a somewhat unstable network, and the container is frequent user of the node network (talking to the master node) so I'm not surprised that it would fail regularly, but I expect it to restart itself.
This is due to the restartmanger. The docker inspect State.error shows a message which clearly came from docker and not my container. The logs show:
... time="2019-09-21T02:06:31.969473802Z" level=error msg="restartmanger wait error: Could not attach to network cqr3v2jode1boqh2yofqrh7bx: context deadline exceeded"
So it appears that -- occasionally -- when the container gets restarted the network is down and the manger decides stop restarting. The question becomes how to override this behavior.
docker info:
Client:
Debug Mode: false
Server:
Containers: 4
Running: 2
Paused: 0
Stopped: 2
Images: 43
Server Version: 19.03.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: wgn64s7lx9jvgw36gtlu0dsou
Is Manager: false
Node Address: 10.0.0.2
Manager Addresses:
10.0.0.1:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.66-v7+
Operating System: Raspbian GNU/Linux 9 (stretch)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 874.5MiB
Name: sensors-2
ID: NTRC:WPLS:GH2P:ZTLM:EDAN:H7HB:HGP6:6G6A:3YVW:T2I7:TVJU:XV3N
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Here are the relevant bits from docker inspect on the non-restarting container. Note that it has restarted a few times, it exited due to a network error, and the MaximumRetryCount is 0 (which I assume is unlimited). Most recently it wasn't up for long... but my understanding of unless-stopped is that docker would continue restarting the container, though it would increase the delay between restarts.
[
{
"Id": "fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4",
"Created": "2019-06-24T05:25:10.792698029Z",
"Path": "/srv/bin/weather_collector_server",
"Args": [
"/etc/config.ini"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "Could not attach to network cqr3v2jode1boqh2yofqrh7bx: context deadline exceeded",
"StartedAt": "2019-09-21T03:56:40.911764904Z",
"FinishedAt": "2019-09-21T03:58:07.234852939Z"
},
"Image": "sha256:ee0e5023f37917f074dd0bf03dca328833eafd117fe69041203533768a196789",
"ResolvConfPath": "/var/lib/docker/containers/fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4/hostname",
"HostsPath": "/var/lib/docker/containers/fa7c59dfa38f25c70d4c1293db27965c2e76af950fa19a2097b4ce63e1af2be4/hosts",
"LogPath": "",
"Name": "/weather_collector_server",
"RestartCount": 3,
"Driver": "overlay2",
"Platform": "linux",
...
"HostConfig": {
...
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
...
],
"NetworkSettings": {
"Bridge": "",
"SandboxID": "0e901219511bb618d66943a12af1e09d8bbcb78ca4caa0bad88880f21d843c55",
...
"Networks": {
"hostnet": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"fa7c59dfa38f"
],
"NetworkID": "cqr3v2jode1boqh2yofqrh7bx",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": {}
}
}
}
}
]