docker 20: change docker image location - docker

i want to change the pull location for the docker image, i already follow the instruction from this link https://quick-adviser.com/how-do-i-change-docker-location/.
i already try adding file daemon.json to C:\ProgramData\Docker\config and fill it with following code:
{
"data-root": "D:\\docker-image"
}
then restarting docker desktop. After that, i trying pull a small size docker image like adminer docker pull adminer and check to D:\docker-image. After the execution is completed, the folder still empty
here's my docker info output:
$ docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.7.1)
compose: Docker Compose (Docker Inc., v2.2.3)
scan: Docker Scan (Docker Inc., v0.16.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.16.3-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 6.04GiB
Name: docker-desktop
ID: V6FY:3JES:DVIP:5ZLG:6J26:IXE7:RKCB:T3MK:RR4B:X2XC:JR7B:LEIH
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Basically you're running Docker Desktop with WSL2. So files are no longer located in Windows directly. Docker creates an Linux-Virtual-Machine (WSL2) where all files are located inside in a Linux folder structure.
# Image-Layers are somewhere here:
\\wsl$\docker-desktop-data\version-pack-data\community\docker\image\overlay2
\\wsl$\docker-desktop-data\version-pack-data\community\docker\overlay2
If you want to change the folder on Windows side (e.g. to use disk space on another drive) then you have to move the image file. This is described in the question I posted. How can I change the location of docker images when using Docker Desktop on WSL2 with Windows 10 Home?
Two different possibilities given: Moving the WSL2 in total or Move the file and generate a Symlink (mklink) to keep WSL2 like it is and just redirect to the file in another location.
By default docker locates the WSL-Image-File for the Data here: %homepath%\AppData\Local\Docker\wsl\data as ext4.vhdx
With this Docker Desktop and WSL2 setup you can't just change the path where the images are located by just using a windows path because under the hood docker works in Linux. So images are stored in Linux.
If you just want to get the image data to move them to another system check docker save and docker load https://docs.docker.com/engine/reference/commandline/save/
Does this fit your use case?

Related

docker compose prompted error: no configuration file provided: not found

I had setup docker desktop with windows WSL integration version 2 and I run into issue when execute certain docker compose command with following errors
docker compose logs
no configuration file provided: not found
However, there were no problem found when executing the following
docker compose up
and image built and fired up successfully.
Is there anyone can help on this?
docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.7.1)
compose: Docker Compose (Docker Inc., v2.2.1)
scan: Docker Scan (Docker Inc., 0.9.0)
Server:
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 4
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.60.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 24.95GiB
Name: docker-desktop
ID: FUMA:ZOXR:BA4L:YSOZ:4NQT:HHIZ:ASAD:EJGA:NJRG:SO4S:GXN3:JG5H
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
UPDATED
It was my mistake that I should execute the docker compose command under the correct directory where docker-compose.yml file located at.
The issue resolved
execute docker-compose command where docker-compose.yml file located at should resolved it.
or specify the docker-compose.yml file as bellow
docker-compose -f <docker-compose.yml> logs
as suggested
For anyone who's getting
no configuration file provided: not found
I was getting the same message on Mac trying to execute
docker-compose up
The problem was with docker-compose.yml file. I saved it from Visual Studio Code as a text file though OS determined it like a proper YAML file. Look at the screenshot: left file is wrong but seems the same.
Make sure you select YAML format when you saving it.
I was facing the same issue on Windows PowerShell . I was using the docker-compose up command from within the newly created directory located in C:\WINDOWS\system32\ .
In my case, I rectified the problem by moving it to C:\Users\
Faced the same issue on a Windows PC. The problem was with the extension of the file being assumed by the OS as a known file type.
Solution
Go to Folder options and disable the Hide extensions for known file types.
You should be able to see the actual file extension which may looked like proper .txt files now (instead of .yml). Rename the file(s) accordingly, run the command again and viola!
Create the following docker-compose.yaml file in an empty directory of your choice
I was also receiving this error and I had mis-spelt the file name(oops), so also double check spelling of files if anyone else finds their way to this post!

docker import with Docker for Windows using Linux containers no space left on device error with Storage Driver: overlay2

I'm importing a large database image at 15 GB (I can import an image of 9.5 GB without a problem.) with Docker for Windows using Linux containers. I'm using Windows 10 Pro 1803, build 17134.1006.
Error:
PS C:\WINDOWS\system32> docker import "C:\Users\oscar\Desktop\MSSQL.tar" mssql
Error response from daemon: Error processing tar file(exit status 1): write /var/opt/mssql/data/TestDatabase.mdf: no space left on device
I have removed all dangling volumes and unused images as suggested here:
https://stackoverflow.com/a/37287054/3850405
When reading about storage drivers I came across some limitations for devicemapper and other drivers.
https://docs.docker.com/engine/reference/commandline/dockerd/#options-per-storage-driver
For devicemapper:
Specifies the size to use when creating the base device, which limits
the size of images and containers. The default value is 10G. Note,
thin devices are inherently “sparse”, so a 10G device which is mostly
empty doesn’t use 10 GB of space on the pool. However, the filesystem
will use more space for the empty case the larger the device is.
https://docs.docker.com/engine/reference/commandline/dockerd/#dmbasesize
I can't find any image limitation for overlay2, only 20 GB limit for containers as default.
Specifies the size to use when creating the sandbox which is used for
containers. Defaults to 20G.
I tried to run the example command but I got an error. dockerd is not mapped to command.
C:\>dockerd --storage-opt size=40G
'dockerd' is not recognized as an internal or external command,
operable program or batch file.
https://docs.docker.com/engine/reference/commandline/dockerd/#size
Since it should work anyway I'm not sure I'm on the right path there.
System information:
PS C:\WINDOWS\system32> docker --version
Docker version 19.03.2, build 6a30dfc
PS C:\WINDOWS\system32> docker info
Client:
Debug Mode: false
Server:
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 8
Server Version: 19.03.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.184-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.837GiB
Name: docker-desktop
ID: XD32:TQJ4:EKWP:BPE4:ETXW:XFXE:LB3L:J4WB:PCFR:DAXK:MJ62:47RI
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 28
Goroutines: 42
System Time: 2019-09-17T10:00:50.1259999Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Storage space should not be an issue:
I should learn to read... read it as 55.29 GB unused. After increasing Disk image max size in Docker -> Settings -> Advanced everything started working.

How to increase docker disk image size in Ubuntu

I am trying to increase the docker image size on ubuntu. When I do docker info I get following info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-87-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.67GiB
Name: no1010042033112.corp.adobe.com
ID: PYZE:KYTG:DXED:QI37:43ZM:56BB:TLM6:X2OJ:WDPA:35UP:Z4CU:DSNC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
As you can see that total memory is Total Memory: 15.67GiB. I couldn't find a way to do it on Ubuntu. I tried following ways
1) sudo dockerd --storage-opt dm.basesize=100G
2) Changing DOCKER_OPTS ="--storage-opt dm.basesize=50G" in /etc/default/docker.
But none of these helped. This option is easily available in Docker config in Windows. But how to do it from a ubuntu terminal
Docker, on Linux, with the overlay2 storage driver, uses all of the host system's disk (and memory). There's no way to make it use less disk (without repartitioning your main system disk) and no way to give it more (without adding new hardware).
Docker for Mac, the Linux-flavored Docker for Windows, and Docker Machine all work by launching virtual machines that run a minimal Linux OS. That VM has a specific disk and memory allocation and there's UI controls for it, but it's because the containers are running on a different OS and need an actual virtualization layer.
On Linux, Docker also supports several storage drivers. These require varying amounts of Linux kernel support. Early versions of Docker used something called devicemapper which worked by allocating space in (most often) a fixed-size file, and then the dm.basesize option you give matters ("dm" is for "devicemapper"). Current versions of Docker on current versions of Linux use a different driver called overlay2 which just stores image and container content in ordinary directories. You still need kernel support for it but there's no reserved or limited disk space. That's also why the only size number in the docker info output is memory, which is a different resource.

How to find whether a image is present in the docker repository

Trying to pull a docker image in my local repo.
Command used:
docker pull buzybox:1.24
Error response from daemon: repository buzybox not found: does not exist or no pull access
My questions:
1:How to find the repository name which am connected with?
2.Whats the access issue in the error its talking about? Do we also have any kind of access issues in the default repo?? How to find that out?
3.How to change the repo if in case i dont find a image in my currect repo.
4.Do we have any web portal from where we can search if the image is present in repository.
Output id docker info :
docker info
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 2
Server Version: 17.06.2-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk
syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.41-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.941GiB
Name: moby
ID: RJWZ:HXFZ:5FQX:V2AB:YAXP:BKSU:RLQH:H24C:LNUD:USDA:VIF7:W2M3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 15
Goroutines: 26
System Time: 2018-01-28T06:34:45.7011515Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Thanks in advance Guys!!
Nilotpal
You do not connect to a repository. You connect to a registry. You are probably connected to https://index.docker.io/v1/ but to be sure, you can run sudo docker info and look for the registry line. The repository in your case is buzybox and you are trying to download the image with tag 1.24 from that repository.
The access issue is probably that you mispelled busybox (notice the 's' instead of a 'z').
To log into a different registry, then run
docker login https://<YOUR-DOMAIN>:8080
and then make sure to specify the new registry when you download an image like so:
docker pull <YOUR-DOMAIN>:8080/busybox:1.24

Docker volume mapping file corruption when filesystem is xfs and storage driver is overlay

when I volume mapping in Docker 1.13.0, some file is corrupted.
In docker container, when "ls -l" the folder, it displayed like this:
"?????????? ? ? ? ? ? file_corrpted_and_cant_access.conf"
and can't edit it or delete it.
it just show "No such file or directory".
I think it can't link file inode and path.
how to fix it?
Additional information
after volume mapping, I soft link volume mapping folder inside docker container.
docker run --privileged -d -v /opt/volume_mapping_folder/:/inside_container/inside_folder --restart=always testcontainer
and insde docker container I soft link folder
docker -it testcontainer /bin/bash
ln -s /inside_container/inside_folder /opt/appFolder
touch /opt/appFolder/file_corrpted_and_cant_access.conf
Output of docker info:
Server Version: 1.13.0
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.6.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.671 GiB
ID: 2D2E:73MA:BJQ3:WQAJ:BR3W:TYF5:F3MQ:E7S3:KZGV:A64K:ASZK:UEXE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
This is is related xfs filesystem with overlay mode.
https://github.com/docker/docker/issues/27358
the solution is one of these.
xfs partitions need to be formatted with ftype set to 1
use ext4 filesystem instead of xfs
setting docker Storage Driver to 'devicemapper'
I hit a similar problem today using Docker 1.13.0 running a CentOS image on a RHEL 7.3 system. Files in my containers root filesystem were acting odd. For example, I would try to delete a file but it would not be removed from the filesystem and display the same '?' you posted when running 'ls'. I also had problems trying to change ownership of files. Anyway, my guess is Docker 1.13.0 changed the default Storage Driver to overlay when in the past (Docker 1.12.5) it was devicemapper. I changed the default back to devicemapper and the problems went away.
I am not a Docker or Linux Filesystem expert and I am not sure if the change to overlay as a default was intentional.

Resources