I have an AWS EC2 instance with EBS mounted as /docker:
root#ip-10-0-0-235:~# lsblk /dev/xvdc1
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdc1 202:33 0 100G 0 part /docker
Then I have /etc/docker/daemon.json with:
{
"graph": "/docker/"
}
After Docker version was updated - I had to change graph to data-root as mentioned here:
{
"data-root": "/docker/"
}
But after that change - I faced with another error:
Nov 22 09:15:31 ip-10-0-0-235 dockerd[5671]: time="2017-11-22T09:15:31.168591095Z" level=info msg="libcontainerd: new containerd process, pid: 5679"
Nov 22 09:15:32 ip-10-0-0-235 dockerd[5671]: time="2017-11-22T09:15:32.170033835Z" level=warning msg="failed to rename /docker/tmp for background deletion: %!s(<nil>). Deleting synchronously"
Nov 22 09:15:32 ip-10-0-0-235 dockerd[5671]: time="2017-11-22T09:15:32.172631277Z" level=error msg="[graphdriver] prior storage driver aufs failed: invalid argument"
Nov 22 09:15:32 ip-10-0-0-235 dockerd[5671]: Error starting daemon: error initializing graphdriver: invalid argument
What's the correct way now to mount it?
Docker 17.05.0-ce, running on Ubuntu 16.04.3.
P.S. During writing this Q - found weird mounted partition:
root#ip-10-0-0-235:~# mount | grep xvd
/dev/xvda1 on / type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda1 on /docker/aufs type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb1 on /jenkins type ext4 (rw,relatime,data=ordered)
/dev/xvdc1 on /docker type ext4 (rw,relatime,data=ordered)
/dev/xvda1 on /docker/aufs and /dev/xvdc1 on /docker. That doesn't looks correct...
Related
I am trying to fetch mass of urls with Selenium WebDriver (selenium/standalone-chrome:96.0 image), running in container on EC2 instance with 30GB storage. I put many efforts to avoid disc space leaking during this proccess, but finally gave up. So after a while container run out of space and i get error from WebDriver like selenium.common.exceptions.WebDriverException: Message: unknown error: cannot create temp dir for user data dir
As workaround I can force container exit after a while, so docker will restart container (with restart:always policy), but disc space is not reclaimed, and sooner or later docker restart manager throws error like
restartmanger wait error: mount /dev/mapper/docker-259:3-394503-72f7b76024003665f890079f6f681414587483fa2f30e0f080c027cd516ba7d2:/var/lib/docker/devicemapper/mnt/72f7b76024003665f890079f6f681414587483fa2f30e0f080c027cd516ba7d2: input/output error\nFailed to mount; and leaves container stopped.
Are there any technique to reclaim disk space on container restart?
UPDATE
creating/closing webdriver, performed after each driver.get()
def create_webdriver():
global driver
try:
logger.info("WebDriver: creating...")
options = Options()
options.add_argument("start-maximized")
options.add_argument("enable-automation")
options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("--disable-infobars")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-browser-side-navigation")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(options=options)
except Exception:
logger.exception("WebDriver: exception while creating, can not manage, exiting.")
exit(1)
def close_webdriver():
global driver
if driver is not None:
driver.quit()
driver = None
UPDATE2
It seems that there are no disk space leakage, but some issues with docker devicemapper fs on EC2 instance. I carefully investigate disk and docker space usage during the proccess, and find no issues
devtmpfs 16323728 120 16323608 1% /dev
tmpfs 16333664 0 16333664 0% /dev/shm
/dev/nvme0n1p1 8189348 1919080 6170020 24% /
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 9 1 9.033GB 8.571GB (94%)
Containers 8 6 144.8MB -2B (0%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
but anyway container feels bad
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot create temp dir for user data dir
and exit, docker can't restart it and there are errors in /var/log/docker
time="2021-12-26T01:36:06.030765815Z" level=error msg="Driver devicemapper couldn't return diff size of container 258399ca6d95cb3510e5e02fec9253b2f22852e8a3553cfad8774b9f913ed279: Failed to mount; dmesg: <3>[ 3761.830462] Buffer I/O error on dev dm-8, logical block 2185471, lost async page write\n<4>[ 3761.839429] JBD2: recovery failed\n<3>[ 3761.843623] EXT4-fs (dm-8): error loading journal\n: mount /dev/mapper/docker-259:3-394503-26a311e2927d080ef4895f43d7dcd6ddaa26e5c0d8e71b6eb46bcdc8d1601194:/var/lib/docker/devicemapper/mnt/26a311e2927d080ef4895f43d7dcd6ddaa26e5c0d8e71b6eb46bcdc8d1601194: input/output error"
time="2021-12-26T01:36:25.009915383Z" level=info msg="ignoring event" container=f47ab38bdab172205bd30c3cdbc6723162e4422ef4dcda23f6fec0ac99a20035 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2021-12-26T01:36:25.010710566Z" level=info msg="shim disconnected" id=f47ab38bdab172205bd30c3cdbc6723162e4422ef4dcda23f6fec0ac99a20035
time="2021-12-26T01:36:25.010797187Z" level=error msg="copy shim log" error="read /proc/self/fd/36: file already closed"
time="2021-12-26T01:36:28.788036177Z" level=warning msg="error locating sandbox id c1e0abc725ee3e88f388042a34b8e46db09a8fd8024774862899d0f7d9af721b: sandbox c1e0abc725ee3e88f388042a34b8e46db09a8fd8024774862899d0f7d9af721b not found"
time="2021-12-26T01:36:28.788396052Z" level=error msg="Error unmounting device 8de02009e67a0fea87313b35b117eaed6cf654837532e04ce16a6fc0846d1954: invalid argument" storage-driver=devicemapper
time="2021-12-26T01:36:28.788426923Z" level=error msg="error unmounting container" container=f47ab38bdab172205bd30c3cdbc6723162e4422ef4dcda23f6fec0ac99a20035 error="invalid argument"
time="2021-12-26T01:36:28.789562261Z" level=error msg="f47ab38bdab172205bd30c3cdbc6723162e4422ef4dcda23f6fec0ac99a20035 cleanup: failed to delete container from containerd: no such container"
time="2021-12-26T01:36:28.794739546Z" level=error msg="restartmanger wait error: mount /dev/mapper/docker-259:3-394503-8de02009e67a0fea87313b35b117eaed6cf654837532e04ce16a6fc0846d1954:/var/lib/docker/devicemapper/mnt/8de02009e67a0fea87313b35b117eaed6cf654837532e04ce16a6fc0846d1954: input/output error\nFailed to mount; dmesg: <3>[ 3784.574178] Buffer I/O error on dev dm-10, logical block 1048578, lost async page write\n<4>[ 3784.583183] JBD2: recovery failed\n<3>[ 3784.587446] EXT4-fs (dm-10): error loading journal\n\ngithub.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).MountDevice\n\t/builddir/build/BUILD/docker-20.10.7-3.71.amzn1/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:2392\ngithub.com/docker/docker/daemon/graphdriver/devmapper.(*Driver).Get\n\t/builddir/build/BUILD/docker-20.10.7-3.71.amzn1/src/github.com/docker/docker/daemon/graphdriver/devmapper/driver.go:208\ngithub.com/docker/docker/layer.(*referencedRWLayer).Mount\n\t/builddir/build/BUILD/docker-20.10.7-3.71.amzn1/src/github.com/docker/docker/layer/mounted_layer.go:104\ngithub.com/docker/docker/daemon.(*Daemon).Mount\n\t/builddir/build/BUILD/docker-20.10.7-3.71.amzn1/src/github.com/docker/docker/daemon/daemon.go:1320\ngithub.com/docker/docker/daemon.(*Daemon).conditionalMountOnStart\n\t/builddir/build/BUILD/docker-20.10.7-3.71.amzn1/src/github.com/docker/docker/daemon/daemon_unix.go:1360\ngithub.com/docker/docker/daemon.(*Daemon).containerStart\n\t/builddir/build/BUILD/docker-20.10.7-3.71.amzn1/src/github.com/docker/docker/daemon/start.go:145\ngithub.com/docker/docker/daemon.(*Daemon).handleContainerExit.func1\n\t/builddir/build/BUILD/docker-20.10.7-3.71.amzn1/src/github.com/docker/docker/daemon/monitor.go:84\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1374"
SOLVED
It was really issue with default AMI Linux docker configuration with devicemapper storage driver on EC2 instance. Clean install docker on Ubuntu 18.04 with overlay2 storage driver solves the issue completely.
docker.log
time="2020-02-18T10:05:51.740757000+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded." storage-driver=overlay2
time="2020-02-18T10:05:51.747446000+08:00" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
time="2020-02-18T10:05:51.758181000+08:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded." storage-driver=overlay
time="2020-02-18T10:05:51.990661000+08:00" level=error msg="Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/dockerd/#storage-driver-options" storage-driver=devicemapper
time="2020-02-18T10:05:52.146274000+08:00" level=warning msg="Your kernel does not support cgroup memory limit"
time="2020-02-18T10:05:52.146389000+08:00" level=warning msg="Unable to find cpu cgroup in mounts"
time="2020-02-18T10:05:52.146445000+08:00" level=warning msg="Unable to find blkio cgroup in mounts"
time="2020-02-18T10:05:52.146504000+08:00" level=warning msg="Unable to find cpuset cgroup in mounts"
time="2020-02-18T10:05:52.146553000+08:00" level=warning msg="mountpoint for pids not found"
time="2020-02-18T10:05:52.149506000+08:00" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2020-02-18T10:05:52.160535000+08:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Devices cgroup isn't mounted
docker version
Client: Docker Engine - Community
Version: 19.03.6
API version: 1.40
Go version: go1.12.16
Git commit: 369ce74
Built: Thu Feb 13 01:40:25 2020
OS/Arch: linux/arm
Experimental: false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I have this literally bookmarked, and I go here every time I need to start docker on my netbooting Alpine. Single command to mount the cgroup pseudofs, w/o touching fstab:
sudo mount -t cgroup cgroup /sys/fs/cgroup
FYI: this may be also caused by your kernel not having cgroup support compiled in, but it's highly unlikely.
I've installed Docker on Centos 7 using the yum process. Below is the output for systemctl status docker.service -l:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-01-02 14:11:09 GMT; 1h 15min ago
Docs: https://docs.docker.com
Main PID: 136537 (dockerd)
Memory: 71.3M
CGroup: /system.slice/docker.service
├─136537 /usr/bin/dockerd
└─136556 docker-containerd --config /var/run/docker/containerd/containerd.toml
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157175048Z" level=error msg="c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5 cleanup: failed to delete container from containerd: no such container"
Jan 02 15:00:33 colossus dockerd[136537]: time="2018-01-02T15:00:33.157230927Z" level=error msg="Handler for POST /v1.35/containers/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c60a6261b7381f5739fa11e6df8b4219d8db94e766c6e4c8c4c2d45a44ee03a5/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.837926048Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/shim.sock" debug=false module="containerd/tasks" pid=145298
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44Z" level=info msg="shim reaped" id=3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f module="containerd/tasks"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899560066Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.899585062Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 02 15:09:44 colossus dockerd[136537]: time="2018-01-02T15:09:44.921068216Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309201262Z" level=error msg="3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f cleanup: failed to delete container from containerd: no such container"
Jan 02 15:09:45 colossus dockerd[136537]: time="2018-01-02T15:09:45.309254679Z" level=error msg="Handler for POST /v1.35/containers/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/start returned error: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown"
The error i get (sudo docker run hello-world) is:
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/3e04ce212f3603d2b93339494d74e305ea6a2e8d08a3ba9c30a0c741c9232b1f/log.json: no such file or directory): docker-runc did not terminate sucessfully: unknown.
ERRO[0001] error waiting for container: context canceled
docker --version
Docker version 17.12.0-ce, build c97c6d6
and here is the docker info
Containers: 27
Running: 0
Paused: 0
Stopped: 27
Images: 47
Server Version: 17.12.0-ce
Storage Driver: devicemapper
Pool Name: docker-253:0-135160107-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 14.4GB
Data Space Total: 107.4GB
Data Space Available: 25.82GB
Metadata Space Used: 12.6MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.135GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: N/A (expected: b2567b37d7b75eb4cf325b77297b140ea686ce8f)
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 88
Total Memory: 125.6GiB
Name: colossus
ID: HV7I:KP2J:N6SQ:SAEP:MWSO:O5T3:Y2PE:L6TL:2D6L:IHVD:SQ3E:TM5P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
You most likely have an old package installed that is breaking runc. Make sure to upgrade your operating system to the current release (for CentOS, this is 1708, or the RHEL equivalent of 7.4). Others have reported similar issues related to packages being out of date.
I had 2 systems with identical CentOS 7.2 versions and kernel. Had this problem with one but not the other. Comparing package versions showed a difference in the version of containerd.io -- I downgraded that on the non-working version and it worked:
$ sudo yum downgrade containerd.io-1.2.0-3.el7
Removed:
containerd.io.x86_64 0:1.2.4-3.1.el7
Installed:
containerd.io.x86_64 0:1.2.0-3.el7
I am trying to install docker on an Ubuntu 16.04 on a remote webserver. However, I get an error that is really frustrating me. I have been installing docker a lot of times already, also on this OS, but this never happened.
I am stuck at sudo apt-get install -y docker-engine, when docker-engine cannot be installed:
~# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since So 2017-03-05 17:47:20 CET; 32s ago
Docs: https://docs.docker.com
Main PID: 18194 (code=exited, status=1/FAILURE)
dockerd[18194]: time="2017-03-05T17:47:20.567753592+01:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please e
dockerd[18194]: time="2017-03-05T17:47:20.569299675+01:00" level=error msg="'overlay' not found as a supported filesystem on this host. Please e
dockerd[18194]: time="2017-03-05T17:47:20.591796895+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
dockerd[18194]: time="2017-03-05T17:47:20.592394882+01:00" level=warning msg="Your kernel does not support oom control"
dockerd[18194]: time="2017-03-05T17:47:20.592410368+01:00" level=warning msg="Your kernel does not support memory swappiness"
dockerd[18194]: time="2017-03-05T17:47:20.592421460+01:00" level=warning msg="Your kernel does not support kernel memory limit"
dockerd[18194]: time="2017-03-05T17:47:20.592427398+01:00" level=warning msg="Unable to find cpu cgroup in mounts"
dockerd[18194]: time="2017-03-05T17:47:20.592458649+01:00" level=warning msg="Unable to find cpuset cgroup in mounts"
dockerd[18194]: time="2017-03-05T17:47:20.592490516+01:00" level=warning msg="mountpoint for pids not found"
dockerd[18194]: Error starting daemon: Devices cgroup isn't mounted
I added root to the group, also I found the advice to add GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1" to the file /etc/default/grub, but that file does not exist!
I also tried sudo apt-get install cgroupfs-mount but without success :-(
Thank you for your help!
So, folks. There is no solution, unfortunately. The reason is the architecture of the remote webserver, a hosted VM.
My provider told me all VMs use the same kernels in one environment and therefore dokker can't access it.
The only option now is to switch to another server.
I been tackling trying to get docker-in-docker working for a CentOS 7 image, with ubuntu as the host image.
As of now i have not started building this as a docker image. And is currently experimenting with bash on how to "get docker in docker to work"
Currently systemctl start docker ran inside the inner docker image CentOS gives the following error
Error: No space left on device
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
Further investigation on the error systemctl status docker gives the following
Oct 13 04:32:08 codenvy docker[6520]: time="2015-10-13T04:32:08Z" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Oct 13 04:32:08 codenvy docker[6520]: time="2015-10-13T04:32:08Z" level=warning msg="Running modprobe bridge nf_nat failed with message: , error: exit status 1"
Oct 13 04:32:08 codenvy docker[6520]: time="2015-10-13T04:32:08Z" level=info msg="-job init_networkdriver() = OK (0)"
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=warning msg="Your kernel does not support cgroup swap limit."
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=info msg="Loading containers: start."
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=info msg="Loading containers: done."
Oct 13 04:32:09 codenvy docker[6520]: time="2015-10-13T04:32:09Z" level=fatal msg="Shutting down daemon due to errors: inotify_add_watch: no space left on device"
Oct 13 04:32:09 codenvy systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 13 04:32:09 codenvy systemd[1]: Failed to start Docker Application Container Engine.
Oct 13 04:32:09 codenvy systemd[1]: Unit docker.service entered failed state.
Additional Information
Host OS: Ubuntu 14.04.2 LTS
Docker Image: codenvy/onprem-multi (which is based on centos:centos7)
Mounted Volumes
/sys/fs/cgroup
/sys/fs/cgroup:/sys/fs/cgroup:ro
/mnt/docker-files-lvm/docker/codenvy/docker:/var/lib/docker
/mnt/docker-files-lvm/docker/codenvy/ldap:/var/lib/ldap
/mnt/docker-files-lvm/docker/codenvy/mongo:/var/lib/mongo
/mnt/docker-files-lvm/docker/codenvy/home:/home
Privileged mode
Note
This is not regarding how to install codenvy, its regarding getting docker itself installed and working. Before installing codenvy
Added: df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/hc--dawn--vg-root 27G 3.5G 23G 14% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 16G 12K 16G 1% /dev
tmpfs 3.2G 1.1M 3.2G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 16G 37M 16G 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 236M 95M 129M 43% /boot
/dev/mapper/base--storage-docker--files 886G 52G 790G 7% /mnt/docker-files-lvm
Note: /mnt/docker-files-lvm/docker maps to the /mnt/docker-files-lvm (which is 790GB)
If it is a no space left problem you can configure docker to have it store images and containers elsewhere.
Since you are using systemctl the config file is located here:
/lib/systemd/system/docker.service
You can add a -g option to change where docker store things.
For example:
ExecStart=/usr/bin/docker daemon -g /there_is_space_here -H fd://