I tried to setup the tool chain of mosquitto, telegraf, and influxdb. All three are installed
on a raspberry pi using apt. To debug, I use a file output from telegraf.
This connection does not work when the pi boots. Mosquito is working if subscribed from outside.
telegraf collects system and disk information. However telegraf does not collect mqtt information.
When I restart mosquitto like
sudo service mosquitto stop
mosquitto -v
the connection is working.
When I restart mosquitto like
sudo service mosquitto stop
sudo service mosquitto start
it is again not working.
What could be the difference?
I just upgraded to the latest versions, but that did not change anything.
mosquitto 1.5.7
telegraf 1.15.3
influxdb 1.8.2
The boot messages of mosquitto are fine:
Sep 14 21:34:30 raspberrypi systemd[1]: Starting Mosquitto MQTT v3.1/v3.1.1 Broker...
Sep 14 21:34:31 raspberrypi systemd[1]: Started Mosquitto MQTT v3.1/v3.1.1 Broker.
The boot messages from telegraf report connection to mosquitto, though there is some trouble with influxdb
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! Starting Telegraf 1.15.3
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.300652Z lvl=info msg="Opened shard" log_id=0PFXdCuW000 service=store trace_id=0PFXdEbG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/base/autogen/16 duration=598.507ms
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.300796Z lvl=info msg="Opened shard" log_id=0PFXdCuW000 service=store trace_id=0PFXdEbG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/base/autogen/152 duration=675.711ms
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.366628Z lvl=info msg="Opened file" log_id=0PFXdCuW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/base/autogen/2/000000001-000000001.tsm id=0 duration=11.324ms
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.374469Z lvl=info msg="Opened file" log_id=0PFXdCuW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/base/autogen/24/000000319-000000002.tsm id=0 duration=22.091ms
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! Loaded inputs: system mqtt_consumer disk
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! Loaded aggregators:
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! Loaded processors:
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! Loaded outputs: influxdb file
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! Tags enabled: host=raspberrypi
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"raspberrypi", Flush Interval:10s
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.489708Z lvl=info msg="Opened shard" log_id=0PFXdCuW000 service=store trace_id=0PFXdEbG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/base/autogen/2 duration=188.821ms
Sep 14 21:34:54 raspberrypi telegraf[401]: 2020-09-14T19:34:54Z I! [inputs.mqtt_consumer] Connected [tcp://localhost:1883]
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.548591Z lvl=info msg="Opened shard" log_id=0PFXdCuW000 service=store trace_id=0PFXdEbG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/base/autogen/24 duration=239.663ms
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.552787Z lvl=info msg="Opened file" log_id=0PFXdCuW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/base/autogen/32/000000271-000000002.tsm id=0 duration=22.821ms
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.788229Z lvl=info msg="Opened file" log_id=0PFXdCuW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/base/autogen/62/000000006-000000002.tsm id=0 duration=203.005ms
Sep 14 21:34:54 raspberrypi influxd[407]: ts=2020-09-14T19:34:54.842928Z lvl=info msg="Opened shard" log_id=0PFXdCuW000 service=store trace_id=0PFXdEbG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/base/autogen/32 duration=352.965ms
Sep 14 21:34:56 raspberrypi influxd[407]: ts=2020-09-14T19:34:56.503706Z lvl=info msg="Opened file" log_id=0PFXdCuW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/base/autogen/40/000000004-000000002.tsm id=0 duration=71.762ms
Sep 14 21:34:58 raspberrypi systemd[1]: systemd-fsckd.service: Succeeded.
Sep 14 21:34:59 raspberrypi influxd[407]: ts=2020-09-14T19:34:59.734290Z lvl=info msg="Opened shard" log_id=0PFXdCuW000 service=store trace_id=0PFXdEbG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/base/autogen/62 duration=5185.491ms
Sep 14 21:34:59 raspberrypi influxd[407]: ts=2020-09-14T19:34:59.762419Z lvl=info msg="Opened file" log_id=0PFXdCuW000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/base/autogen/41/000000001-000000001.tsm id=0 duration=8.874ms
Sep 14 21:34:59 raspberrypi influxd[407]: ts=2020-09-14T19:34:59.785965Z lvl=info msg="Opened shard" log_id=0PFXdCuW000 service=store trace_id=0PFXdEbG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/base/autogen/40 duration=4942.818ms
The relevant parts oc telegraf.conf are
[[outputs.influxdb]]
urls = ["http://127.0.0.1:8086"]
database = "base"
skip_database_creation = true
username = "telegraf"
password = "****"
content_encoding = "identity"
[[outputs.file]]
files = ["stdout", "/tmp/metrics.out"]
[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
topics = ["home/garden/+"]
topic_tag = "mqtt_topic"
qos = 1
max_undelivered_messages = 1
persistent_session = true
client_id = "lord_of_the_pis"
data_format = "json"
The client_id was the problem.
client_id = "lord_of_the_pis"
With a shorter client_id, it works fine.
Related
Goal
I'm running docker in a live CD and I want to cache the results of docker build, mostly, for when I reboot into this live CD. My idea was to set up an overlayfs in /var/lib/docker. So, I have the below in /etc/fstab:
overlay /var/lib/docker overlay lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work,x-systemd.requires=/mnt/root,x-systemd.requires=/var/lib/docker,nofail 0 2
However, I also want to use the overlay2 storage driver. I'm having a hard time getting both to cooperate. There are 2 ways that I've tried to get this to work.
Approach 1 (docker service before mount)
1a. Start docker.{socket,service} systemd units.
sudo systemctl restart docker.service # restarts docker.socket automatically
1b. Start overlayfs
sudo mount -t overlay overlay -o lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work
Both of these steps execute successfully. However, if I try to build/run a new image then I get the following error.
Failure Kind 1
$ make up
mkdir -p .dynamodb-data/
docker-compose up -d --build
Pulling dynamodb (amazon/dynamodb-local:)...
latest: Pulling from amazon/dynamodb-local
2cbe74538cb5: Pull complete
2b0e6045b331: Extracting [==================================================>] 150MB/150MB
667cfda26bc2: Download complete
ERROR: failed to register layer: Error processing tar file(exit status 1): failed to mknod("/etc/sysconfig/clock", S_IFCHR, 0): operation not permitted
make: *** [Makefile:15: up] Error 1
Failure Kind 2
$ docker run -it --rm alpine
docker: Error response from daemon: open /var/lib/docker/overlay2/011d016883746bd72b20f298ecd73545a982019acd9eb3f600ff3b2bd4768888/committed: no such file or directory.
See 'docker run --help'.
Approach 2 (mount before docker service)
2a. Start overlayfs
sudo mount -t overlay overlay -o lowerdir=/var/lib/docker,upperdir=/mnt/root/var/lib/docker,workdir=/mnt/root/.docker_work
2b. Start docker.{socket,service} systemd units.
sudo systemctl restart docker.service # restarts docker.socket automatically
2a. succeeds (of course), but 2b. fails. I've tried to capture some of the systemd logs, below, to try to help diagnose this problem.
Failure Kind 1
Running Command
$ sudo systemctl restart docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
journalctl
$ journalctl -u docker
-- Journal begins at Sat 2022-01-08 12:36:55 EET, ends at Sat 2022-01-08 13:30:15 EET. --
Jan 08 12:37:11 johnos systemd[1]: Starting Docker Application Container Engine...
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.548295176+02:00" level=info msg="Starting up"
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575509563+02:00" level=info msg="libcontainerd: started new containerd process" pid=2346
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575558875+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575566468+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575590744+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:12 johnos dockerd[2267]: time="2022-01-08T12:37:12.575605990+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.212959027+02:00" level=info msg="starting containerd" revision=v1.4.11 version=v1.4.11
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.229766943+02:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.229835283+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233082343+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /r>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233119579+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233257178+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (tmpfs) must be a btrfs filesystem to be>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233270945+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233289127+02:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233297875+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233320748+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233391289+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233485569+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the>
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233500101+02:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233516933+02:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233524181+02:00" level=info msg="metadata content store policy set" policy=shared
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233604310+02:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233619120+02:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233663958+02:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233690208+02:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233702617+02:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233713672+02:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233728996+02:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233740040+02:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233751585+02:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233763065+02:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233773387+02:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233828995+02:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.233872779+02:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234064302+02:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234082499+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234118718+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234131212+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234141269+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234150671+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234160162+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234172118+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234181945+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234191381+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234200863+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234238659+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234251883+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234262989+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234272388+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234387869+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234413901+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234435704+02:00" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
Jan 08 12:37:13 johnos dockerd[2346]: time="2022-01-08T12:37:13.234447184+02:00" level=info msg="containerd successfully booted in 0.021879s"
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237690927+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237706112+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237717947+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.237724235+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238078323+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238095004+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238105606+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.238112459+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.293257377+02:00" level=info msg="Loading containers: start."
Jan 08 12:37:13 johnos dockerd[2267]: time="2022-01-08T12:37:13.360740919+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 08 12:37:14 johnos dockerd[2267]: time="2022-01-08T12:37:14.902691695+02:00" level=info msg="Loading containers: done."
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.113537187+02:00" level=info msg="Docker daemon" commit=v20.10.12 graphdriver(s)=overlay2 version=20.10.12
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.113579097+02:00" level=info msg="Daemon has completed initialization"
Jan 08 12:37:15 johnos systemd[1]: Started Docker Application Container Engine.
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.121263835+02:00" level=info msg="API listen on /run/docker.sock"
Jan 08 12:37:15 johnos dockerd[2267]: time="2022-01-08T12:37:15.124035385+02:00" level=info msg="API listen on /run/docker.sock"
Jan 08 12:42:57 johnos systemd[1]: Stopping Docker Application Container Engine...
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.199583930+02:00" level=info msg="Processing signal 'terminated'"
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.199951358+02:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jan 08 12:42:57 johnos dockerd[2267]: time="2022-01-08T12:42:57.200178572+02:00" level=info msg="Daemon shutdown complete"
Overlay on top of overlay is not supported by the kernel. There is no docker configuration that will change that. You can find a list of supported backing filesystems in Docker's documentation. The least bad option if your backing filesystem is overlay is going to be fuse-overlayfs, which runs in user space. And the easiest but also worst option will be vfs, aka the native storage driver, which is a full copy of the filesystem for each layer and container (no overlay at all).
ok I installed (in ubuntu 20.04) as it said the official page of influxdb https://portal.influxdata.com/downloads/, specifically these commands:
wget https://dl.influxdata.com/influxdb/releases/influxdb_2.0.2_amd64.deb
sudo dpkg -i influxdb_2.0.2_amd64.deb
then add commands to start and create persistence with the daemon.
systemctl enable --now influxdb
systemctl status influxdb
and it comes out as if it was activated and running normally
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-11-20 17:43:54 -03; 55min ago
Docs: https://docs.influxdata.com/influxdb/
Main PID: 750 (influxd)
Tasks: 7 (limit: 1067)
Memory: 33.8M
CGroup: /system.slice/influxdb.service
└─750 /usr/bin/influxd
Nov 20 17:44:03 hypercc influxd[750]: ts=2020-11-20T20:44:03.754479Z lvl=info msg="Open store (start)" log_id=0QarEkHl000 service=storage-engine op_name=tsdb_open op_event=start
Nov 20 17:44:03 hypercc influxd[750]: ts=2020-11-20T20:44:03.754575Z lvl=info msg="Open store (end)" log_id=0QarEkHl000 service=storage-engine op_name=tsdb_open op_event=end op_elapsed=0.098ms
Nov 20 17:44:03 hypercc influxd[750]: ts=2020-11-20T20:44:03.754661Z lvl=info msg="Starting retention policy enforcement service" log_id=0QarEkHl000 service=retention check_interval=30m
Nov 20 17:44:03 hypercc influxd[750]: ts=2020-11-20T20:44:03.754888Z lvl=info msg="Starting precreation service" log_id=0QarEkHl000 service=shard-precreation check_interval=10m advance_period=30m
Nov 20 17:44:03 hypercc influxd[750]: ts=2020-11-20T20:44:03.755164Z lvl=info msg="Starting query controller" log_id=0QarEkHl000 service=storage-reads concurrency_quota=10 initial_memory_bytes_quota_per_query=9223372036854775807 memory_bytes_quota_per_query=9223372036854775807 max_memory_bytes=0 queue_size=10
Nov 20 17:44:03 hypercc influxd[750]: ts=2020-11-20T20:44:03.755725Z lvl=info msg="Configuring InfluxQL statement executor (zeros indicate unlimited)." log_id=0QarEkHl000 max_select_point=0 max_select_series=0 max_select_buckets=0
Nov 20 17:44:04 hypercc influxd[750]: ts=2020-11-20T20:44:04.071001Z lvl=info msg=Starting log_id=0QarEkHl000 service=telemetry interval=8h
Nov 20 17:44:04 hypercc influxd[750]: ts=2020-11-20T20:44:04.071525Z lvl=info msg=Listening log_id=0QarEkHl000 transport=http addr=:8086 port=8086
Nov 20 18:14:03 hypercc influxd[750]: ts=2020-11-20T21:14:03.757182Z lvl=info msg="Retention policy deletion check (start)" log_id=0QarEkHl000 service=retention op_name=retention_delete_check op_event=start
Nov 20 18:14:03 hypercc influxd[750]: ts=2020-11-20T21:14:03.757233Z lvl=info msg="Retention policy deletion check (end)" log_id=0QarEkHl000 service=retention op_name=retention_delete_check op_event=end op_elapsed=0.074ms
What should I add to be able to write "influx" and go directly to the DB to make queries? is it something with the ip address?
When I enter influx, I only get help options but it doesn't say anything about connecting or something like that.
by the way here https://docs.influxdata.com/influxdb/v2.0/get-started/ it is installed in a different way but it is supposed that both ways work well.
thanks.
Usually tools like Telegraf are used to collect data and write it to InfluxDB. You can install Telegraf on each server you want to collect data from.
https://docs.influxdata.com/telegraf/v1.17/
You can browse to http://your_server_ip:8086 and login to chronograf (included to InfluxDB 2.0). Here you can create dashboards and query data from InfluxDB.
Its also possible to do manual queries via the InfluxDB CLI. You can simply use it with the influx query command in your terminal.
https://docs.influxdata.com/influxdb/v2.0/query-data/
Note that some commands need authentication before you are allowed to execute them (e.g. the user command). You can authenticate by adding the -t parameter followed by a valid user token (can be found in the web interface).
Example: influx -t token_here user list
Hope this helps you out.
I am facing a strange issue with the docker image pull command where the command fails with this error;
[desai#brilp0017 ~]$ docker image pull nginx:latest
latest: Pulling from library/nginx
d121f8d1c412: Extracting [==================================================>]
27.09MB/27.09MB
ebd81fc8c071: Download complete
655316c160af: Download complete
d15953c0e0f8: Download complete
2ee525c5c3cc: Download complete
failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root534731447: device or resource busy
After this error the docker daemon is no longer accessible and all docker commands return following error;
[desai#brilp0017 ~]$ docker info
Client:
Debug Mode: false
Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info
The command systemctl status docker however shows it as running;
[desai#brilp0017 ~]$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-09-11 14:25:53 BST; 14min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 2380 (dockerd)
Tasks: 14
Memory: 249.5M
CGroup: /system.slice/docker.service
└─2380 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
The journalctl log has the same error line as encountered in the pull command;
Sep 11 14:25:52 brilp0017 systemd[1]: Starting Docker Application Container Engine...
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.345006155+01:00" level=info msg="Starting up"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348597478+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348667479+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348733420+01:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.348765306+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353865701+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353908904+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353944835+01:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.353988191+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.497701794+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816295801+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816318357+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Sep 11 14:25:52 brilp0017 dockerd[2380]: time="2020-09-11T14:25:52.816442165+01:00" level=info msg="Loading containers: start."
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.101411528+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.125378601+01:00" level=info msg="Loading containers: done."
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.291896277+01:00" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.292711063+01:00" level=info msg="Docker daemon" commit=48a66213fe graphdriver(s)=overlay2 version=19.03.12-ce
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.293190069+01:00" level=info msg="Daemon has completed initialization"
Sep 11 14:25:53 brilp0017 dockerd[2380]: time="2020-09-11T14:25:53.340381428+01:00" level=info msg="API listen on /run/docker.sock"
Sep 11 14:25:53 brilp0017 systemd[1]: Started Docker Application Container Engine.
Sep 11 14:32:38 brilp0017 dockerd[2380]: time="2020-09-11T14:32:38.011501405+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 11 14:33:11 brilp0017 dockerd[2380]: time="2020-09-11T14:33:11.592234770+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Sep 11 14:34:52 brilp0017 dockerd[2380]: time="2020-09-11T14:34:52.864254519+01:00" level=info msg="Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root534731447: device or resource busy"
After this the error remains the same even after stopping and starting docker service multiple times with systemctl. After restarting the laptop fully and starting the docker service it starts work as expected until the next time docker pull command is used.
I have searched for solution on the internet but majority of them point to the user not being in docker group but that is not the case for me;
[desai#brilp0017 ~]$ groups
sys network power vboxusers wireshark sambashare docker lp wheel desai
Here is the output of docker version before it crashes for version details;
[desai#brilp0017 ~]$ docker version
Client:
Version: 19.03.12-ce
API version: 1.40
Go version: go1.14.5
Git commit: 48a66213fe
Built: Sat Jul 18 01:33:21 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.12-ce
API version: 1.40 (minimum version 1.12)
Go version: go1.14.5
Git commit: 48a66213fe
Built: Sat Jul 18 01:32:59 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.0.m
GitCommit: 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a.m
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.18.0
GitCommit: fec3683
I am using Manjaro linux OS;
Operating System: Manjaro Linux
KDE Plasma Version: 5.19.4
KDE Frameworks Version: 5.73.0
Qt Version: 5.15.0
Kernel Version: 4.19.141-2-MANJARO
OS Type: 64-bit
Processors: 8 × Intel® Core™ i7-8550U CPU # 1.80GHz
Memory: 31.2 GiB of RAM
Graphics Processor: Mesa Intel® UHD Graphics 620
Any help on this would be appreciated.
This issue was resolved by updating the kernel to 5.4 version.
I have a Influxdb database that is losing data due the activation of the retention policy.
I upgraded the influxdb code from the v1.6.3 to v1.7.7, but the behavior is the same.
> SHOW RETENTION POLICIES ON "telegraf"
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 false
forever 0s 168h0m0s 1 true
Aug 16 06:02:25 influxdb influxd[805]: ts=2019-08-16T09:02:25.623073Z lvl=info msg="Retention policy deletion check (start)" log_id=0HEpQh70000 service=retention trace_id=0HIQTFLW000 op_name=retention_delete_check op_event=start
Aug 16 06:02:25 influxdb influxd[805]: ts=2019-08-16T09:02:25.623477Z lvl=info msg="Retention policy deletion check (end)" log_id=0HEpQh70000 service=retention trace_id=0HIQTFLW000 op_name=retention_delete_check op_event=end op_elapsed=0.487ms
Aug 16 06:32:25 influxdb influxd[805]: ts=2019-08-16T09:32:25.623033Z lvl=info msg="Retention policy deletion check (start)" log_id=0HEpQh70000 service=retention trace_id=0HISB6aW000 op_name=retention_delete_check op_event=start
Aug 16 06:32:25 influxdb influxd[805]: ts=2019-08-16T09:32:25.623339Z lvl=info msg="Retention policy deletion check (end)" log_id=0HEpQh70000 service=retention trace_id=0HISB6aW000 op_name=retention_delete_check op_event=end op_elapsed=0.352ms
Aug 16 07:02:25 influxdb influxd[805]: ts=2019-08-16T10:02:25.622970Z lvl=info msg="Retention policy deletion check (start)" log_id=0HEpQh70000 service=retention trace_id=0HITtyqW000 op_name=retention_delete_check op_event=start
Aug 16 07:02:25 influxdb influxd[805]: ts=2019-08-16T10:02:25.623272Z lvl=info msg="Retention policy deletion check (end)" log_id=0HEpQh70000 service=retention trace_id=0HITtyqW000 op_name=retention_delete_check op_event=end op_elapsed=0.362ms
Aug 16 07:32:25 influxdb influxd[805]: ts=2019-08-16T10:32:25.622899Z lvl=info msg="Retention policy deletion check (start)" log_id=0HEpQh70000 service=retention trace_id=0HIVbq5W000 op_name=retention_delete_check op_event=start
Aug 16 07:32:25 influxdb influxd[805]: ts=2019-08-16T10:32:25.623780Z lvl=info msg="Retention policy deletion check (end)" log_id=0HEpQh70000 service=retention trace_id=0HIVbq5W000 op_name=retention_delete_check op_event=end op_elapsed=0.917ms
Aug 16 08:02:25 influxdb influxd[805]: ts=2019-08-16T11:02:25.622839Z lvl=info msg="Retention policy deletion check (start)" log_id=0HEpQh70000 service=retention trace_id=0HIXKhLW000 op_name=retention_delete_check op_event=start
Aug 16 08:02:25 influxdb influxd[805]: ts=2019-08-16T11:02:25.622987Z lvl=info msg="Retention policy deletion check (end)" log_id=0HEpQh70000 service=retention trace_id=0HIXKhLW000 op_name=retention_delete_check op_event=end op_elapsed=0.171ms
I should not see the retention policy being activated ever, as the duration is set to '0s'. Any help is much appreciated.
If you dont want forever retention policy to stay just write following query to influx.
> DROP RETENTION POLICY "forever" ON "telegraf"
And make autogen retention policy as default for telegraf database.
> ALTER RETENTION POLICY "autogen" ON "telegraf" DEFAULT
I'm running several docker containers with restart=always on Ubuntu 18.04.1 LTS. The physical server reboots every morning at 2am via cronjob executing reboot now.
So far, I haven't had any problems with that in the past 5 or 6 months running that particular setup.
But today, containers didn't start after the daily reboot. The output of docker ps was empty, all containers were in state "Exited".
Why does this happen all of a sudden? Was my setup mis-configured from the beginning, or does the recent docker-ce package upgrade play a role?
Here are logs before and after the reboot as well as the docker.service unit and version info:
root#skprov2:~# journalctl -b -1 -x -u docker
Nov 15 02:00:02 skprov2 systemd[1]: Stopping Docker Application Container Engine...
-- Subject: Unit docker.service has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has begun shutting down.
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.189764841+01:00" level=info msg="Processing signal 'terminated'"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.595098434+01:00" level=info msg="shim reaped" id=c929d444a6eb59a69a0da738ca782a9feb92ac1f80e5c4576bf85376c3d4c17a
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.601217756+01:00" level=info msg="shim reaped" id=98a8c1b99cf986e6a889474f0fc28fe3635e466b21f8a37ef3c10a1050495c78
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.604880385+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.670918937+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.732991633+01:00" level=info msg="shim reaped" id=9b3badc752786df08d00138c0222042a6bd80bb2c971f5a96b71e57105cea95c
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.748732351+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.843982385+01:00" level=info msg="shim reaped" id=ae7531405113db8b4754491a12c2ababf09fa0c8f501bfe6f1b33e3ff18b6462
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.869023019+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:03 skprov2 dockerd[1504]: time="2018-11-15T02:00:03.863568729+01:00" level=info msg="shim reaped" id=b335536f5f07b1db3f32ba4452fc4aadacc02c6184cef7fc9df619ab81bbf002
Nov 15 02:00:04 skprov2 dockerd[1504]: time="2018-11-15T02:00:04.279347144+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.233635995+01:00" level=info msg="Container 77e00eabebea97357f05f564597d167acad8f2596b25d295b4366baf08ef3127 failed to exit within 10 seconds of signal 15 - using the force"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.253563540+01:00" level=info msg="Container 7f7d2a92bcdbb240a9400942c9301f5cd77bf9d3fbde1d38f41a2bd1226f9b09 failed to exit within 10 seconds of signal 15 - using the force"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.253563179+01:00" level=info msg="Container f6b49cc85eb7f9226ac192498b1e319d68e0de2faff6b4e3e67adabba43a093a failed to exit within 10 seconds of signal 15 - using the force"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.654403249+01:00" level=info msg="shim reaped" id=7f7d2a92bcdbb240a9400942c9301f5cd77bf9d3fbde1d38f41a2bd1226f9b09
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.679675304+01:00" level=info msg="shim reaped" id=f6b49cc85eb7f9226ac192498b1e319d68e0de2faff6b4e3e67adabba43a093a
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.680699340+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.689801078+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:13 skprov2 dockerd[1504]: time="2018-11-15T02:00:13.088891655+01:00" level=info msg="shim reaped" id=77e00eabebea97357f05f564597d167acad8f2596b25d295b4366baf08ef3127
Nov 15 02:00:13 skprov2 dockerd[1504]: time="2018-11-15T02:00:13.111193244+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:15 skprov2 dockerd[1504]: time="2018-11-15T02:00:15.233286510+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Nov 15 02:00:15 skprov2 dockerd[1504]: time="2018-11-15T02:00:15.233684167+01:00" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
Nov 15 02:00:15 skprov2 dockerd[1504]: time="2018-11-15T02:00:15.233695697+01:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Nov 15 02:00:15 skprov2 dockerd[1504]: time="2018-11-15T02:00:15.234287398+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4209ec610, TRANSIENT_FAILURE" module=grpc
Nov 15 02:00:15 skprov2 dockerd[1504]: time="2018-11-15T02:00:15.234328545+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4209ec610, CONNECTING" module=grpc
Nov 15 02:00:16 skprov2 systemd[1]: Stopped Docker Application Container Engine.
-- Subject: Unit docker.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has finished shutting down.
==================================================================================
==================================================================================
==================================================================================
root#skprov2:~# journalctl -b 0 -x -u docker
-- Logs begin at Thu 2018-07-05 13:16:23 CEST, end at Thu 2018-11-15 08:16:31 CET. --
Nov 15 02:04:00 skprov2 systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has begun starting up.
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.152961544+01:00" level=info msg="systemd-resolved is running, so using resolvconf: /run/systemd/resolve/resolv.conf"
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.432271212+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.432315437+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.439772198+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.439800208+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.471564855+01:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.471618580+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.471653422+01:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>}]" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.475146270+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.475678777+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420abc010, CONNECTING" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.475795536+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42080b0b0, CONNECTING" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.475769194+01:00" level=info msg="blockingPicker: the picked transport is not ready, loop back to repick" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.476273893+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42080b0b0, READY" module=grpc
Nov 15 02:04:12 skprov2 dockerd[1690]: time="2018-11-15T02:04:12.476346309+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420abc010, READY" module=grpc
Nov 15 02:04:14 skprov2 dockerd[1690]: time="2018-11-15T02:04:14.769703354+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Nov 15 02:04:23 skprov2 dockerd[1690]: time="2018-11-15T02:04:23.247573731+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 15 02:04:23 skprov2 dockerd[1690]: time="2018-11-15T02:04:23.247926863+01:00" level=warning msg="Your kernel does not support swap memory limit"
Nov 15 02:04:23 skprov2 dockerd[1690]: time="2018-11-15T02:04:23.247998928+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Nov 15 02:04:23 skprov2 dockerd[1690]: time="2018-11-15T02:04:23.248016977+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Nov 15 02:04:23 skprov2 dockerd[1690]: time="2018-11-15T02:04:23.254944197+01:00" level=info msg="Loading containers: start."
Nov 15 02:04:25 skprov2 dockerd[1690]: time="2018-11-15T02:04:25.856323528+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Nov 15 02:04:35 skprov2 dockerd[1690]: time="2018-11-15T02:04:35.182112549+01:00" level=error msg="Failed to start container c929d444a6eb59a69a0da738ca782a9feb92ac1f80e5c4576bf85376c3d4c17a: id already in use"
Nov 15 02:04:35 skprov2 dockerd[1690]: time="2018-11-15T02:04:35.206030890+01:00" level=error msg="Failed to start container b335536f5f07b1db3f32ba4452fc4aadacc02c6184cef7fc9df619ab81bbf002: id already in use"
Nov 15 02:04:35 skprov2 dockerd[1690]: time="2018-11-15T02:04:35.235647072+01:00" level=error msg="Failed to start container ae7531405113db8b4754491a12c2ababf09fa0c8f501bfe6f1b33e3ff18b6462: id already in use"
Nov 15 02:04:35 skprov2 dockerd[1690]: time="2018-11-15T02:04:35.374241415+01:00" level=error msg="Failed to start container 9b3badc752786df08d00138c0222042a6bd80bb2c971f5a96b71e57105cea95c: id already in use"
Nov 15 02:04:35 skprov2 dockerd[1690]: time="2018-11-15T02:04:35.410173049+01:00" level=error msg="Failed to start container 7f7d2a92bcdbb240a9400942c9301f5cd77bf9d3fbde1d38f41a2bd1226f9b09: id already in use"
Nov 15 02:04:36 skprov2 dockerd[1690]: time="2018-11-15T02:04:36.171600568+01:00" level=error msg="Failed to start container 98a8c1b99cf986e6a889474f0fc28fe3635e466b21f8a37ef3c10a1050495c78: id already in use"
Nov 15 02:04:36 skprov2 dockerd[1690]: time="2018-11-15T02:04:36.970077586+01:00" level=error msg="Failed to start container f6b49cc85eb7f9226ac192498b1e319d68e0de2faff6b4e3e67adabba43a093a: id already in use"
Nov 15 02:04:36 skprov2 dockerd[1690]: time="2018-11-15T02:04:36.993993749+01:00" level=error msg="Failed to start container 77e00eabebea97357f05f564597d167acad8f2596b25d295b4366baf08ef3127: id already in use"
Nov 15 02:04:36 skprov2 dockerd[1690]: time="2018-11-15T02:04:36.994202774+01:00" level=info msg="Loading containers: done."
Nov 15 02:04:37 skprov2 dockerd[1690]: time="2018-11-15T02:04:37.492457742+01:00" level=info msg="Docker daemon" commit=4d60db4 graphdriver(s)=overlay2 version=18.09.0
Nov 15 02:04:37 skprov2 dockerd[1690]: time="2018-11-15T02:04:37.494916840+01:00" level=info msg="Daemon has completed initialization"
Nov 15 02:04:37 skprov2 dockerd[1690]: time="2018-11-15T02:04:37.669139526+01:00" level=info msg="API listen on /var/run/docker.sock"
Nov 15 02:04:37 skprov2 systemd[1]: Started Docker Application Container Engine.
-- Subject: Unit docker.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has finished starting up.
--
-- The start-up result is RESULT.
==================================================================================
==================================================================================
==================================================================================
root#skprov2:~# cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H unix://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
==================================================================================
==================================================================================
==================================================================================
root#skprov2:~# docker info && docker version
Containers: 14
Running: 5
Paused: 0
Stopped: 9
Images: 61
Server Version: 18.09.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-39-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.39GiB
Name: skprov2
ID: EDC2:AGFH:BHKP:P4HS:M5DA:ZPXM:AU6B:TV6E:6KIU:YC4S:F3NN:35A4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
localhost:5000
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:49:01 2018
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:16:44 2018
OS/Arch: linux/amd64
Experimental: false
I stumbled on the same issue.
In my case the issue was due to Docker not cleaning up after himself properly.
As seen in Docker's log:
time="2018-12-31T17:38:54.330555181+02:00" level=error msg="2089c8095e62011b0dc05e66c51ae59d648d909ca7a8e806af0fdf39b2e3006c cleanup: failed to delete container from containerd: transport is closing: unknown"
These ids are used on startup because of restart=always.
So Docker says
time="2018-12-31T17:40:04.648261275+02:00" level=error msg="Failed to start container 2089c8095e62011b0dc05e66c51ae59d648d909ca7a8e806af0fdf39b2e3006c: id already in use"
Seems like the Docker daemon shuts down faster than the containers are cleaned (probably because of the container's ignoring signals or something)
So seems the solution for me was to change the daemon's shutdown-timeout in the Docker daemon configuration file. The default is 10s or something, I changed it to 60s and no I don't experience these issues anymore.
I still think this is a legitimate scenario that should work out-of-the-box though.
The containerd "id already in use" errors appear to have been fixed in the latest release of Docker CE. Try upgrading to 18.09.3 to see if that corrects your issue.
https://github.com/docker/docker-ce/releases/tag/v18.09.3