All my docker containers suddenly exist. I try to restart the container and get following error:
Error response from daemon: Cannot start container 1559: Error getting container 1559fdbc6c2ab8f12d9efe1a066880ddedb2c424d3a3ed8a1f8a2eb181e1c3ba from driver devicemapper: Error mounting '/dev/mapper/docker-253:2-33554560-1559fdbc6c2ab8f12d9efe1a066880ddedb2c424d3a3ed8a1f8a2eb181e1c3ba' on '/data/docker/devicemapper/mnt/1559fdbc6c2ab8f12d9efe1a066880ddedb2c424d3a3ed8a1f8a2eb181e1c3ba': invalid argument
Here is docker info
[root#localhost ~]# docker info
Containers: 9
Images: 189
Storage Driver: devicemapper
Pool Name: docker-253:2-33554560-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 7.367 GB
Data Space Total: 107.4 GB
Data Space Available: 54.15 GB
Metadata Space Used: 11.58 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.136 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /data/docker/devicemapper/devicemapper/data
Metadata loop file: /data/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 1
Total Memory: 3.703 GiB
Name: localhost.localdomain
ID: UBGK:AERA:AYMM:XB6P:XCOG:MUGB:NKZM:GSIY:AH25:UGN7:FUF3:ID44
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
I tried to restart the docker and it still didn't work.
I cannot remove all the containers or images.
The disk used to run out of space.
My docker version is 1.8
What can I do to recover my container?
Really thank you for your help!!!!!!!
Can you post your docker info command output?
If your Docker Root dir -
/var/lib/docker (default)
is under the same disk running out of space as mentioned by you then you wouldn't be able to pull image or run docker containers. Try to clean up some space and then you may be able to recover your container. BTW when your say recover - Are you writing any data into the container?
Any specific reason to be on older version of Docker?
The recommended storage driver for Docker is overlay2 & FYI the devicemapper storagedriver is deprecated in 18.09 release and would be removed in future release.
Read more about storage drivers here
Related
Hi i have 3 node Swarm docker setup (1 manager, 2 nodes) on Centos 7. I'm using devicemapper data storage for all three VM's. The problem is that after some time folder:
/var/lib/docker/devicemapper/metadata and ../mnt
have so many files / folders they cannot be even "ls'ed". docker system prune, and docker volume prune, are not clearing antyhing.
Any ideas why it does happen and how to fix it?
Server Version: 17.06.0-ce
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 3.055GB
Data Space Total: 20.4GB
Data Space Available: 17.34GB
Metadata Space Used: 120.5MB
Metadata Space Total: 213.9MB
Metadata Space Available: 93.4MB
Thin Pool Minimum Free Space: 2.039GB
I'm new to docker and trying to implement docker using chef on centos 7.1
Below is the basic code I wrote for installing, pulling centos image and creating container.
All the 3 tasks are executing successfully. Since the containers are in stop mode I tried to start manually by typing docker start containerid. When I checked docker ps I found the container is not started . I tried for several times but couldn't start the container.
Docker code using chef
docker_service 'default' do
action [:create, :start]
end
docker_image 'centos' do
action :pull
end
docker_container 'check2' do
repo 'centos'
action :create
end
Docker info:
Containers: 6
Running: 1
Paused: 0
Stopped: 5
Images: 3
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-8:1-523814-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 441.3 MB
Data Space Total: 107.4 GB
Data Space Available: 28.08 GB
Metadata Space Used: 1.159 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.4.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 6.807 GiB
ID: R24R:ORHY:XJQW:2HNI:U5TV:UGF7:B7VX:P6Z6:UHSR:YIMR:VGJT:4URU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Would you please help me
By default, the centos image will run /bin/bash which will exit immediately without a tty (-t) and standard input (-i) available.
Try running something in the container
docker_container 'check2' do
repo 'centos'
command 'top -b -d 5'
end
I've started a docker master with:
docker swarm init --advertise-addr <MANAGER-IP>
so, im trying to have my shell point to swarm master via:
eval $(docker-machine env --swarm <MANAGER-IP>)
but it's giving me an error: Host does not exists
docker info:
-bash-4.2$ docker info
Containers: 18
Running: 1
Paused: 0
Stopped: 17
Images: 20
Server Version: 1.12.0
Storage Driver: devicemapper
Pool Name: docker-253:1-25646-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 3.124 GB
Data Space Total: 107.4 GB
Data Space Available: 13.4 GB
Metadata Space Used: 5.071 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.142 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null overlay
Swarm: active
NodeID: 05szzy2z96ypgl5k21swggoil
Is Manager: true
ClusterID: a2wrfuga2tu4cm4k0lxxorqtm
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot interval: 10000
Heartbeat tick: 1
Election tick: 3
Dispatcher:
Heartbeat period: 5 seconds
CA configuration:
Expiry duration: 3 months
Node Address: 10.193.46.89
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51 GiB
Name: scsor0004331002.rtp.openenglab.netapp.com
ID: T52U:6MWQ:XEDM:2TGH:ITLQ:YD6B:R3MR:MWF5:CFBM:G6PX:W4LG:6SR7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: eugenepark3
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8
Anyone know what i need to put for eval $(docker-machine env --swarm <MANAGER-IP>) so my compose can run on swarm cluster?
Im supposed to put master name but i dont know how to find it
-bash-4.2$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
05szzy2z96ypgl5k21swggoil * scsor0004331002.rtp.openenglab.netapp.com Ready Active Leader
59t110b0wjhitj1fr8erys600 scsor0004331003.rtp.openenglab.netapp.com Ready Active
dhm6utu2w3dw1to0zh3n71moq scsor0004331004.rtp.openenglab.netapp.com Ready Active
You're mixing up the container based swarm commands with the newer swarmkit based Swarm that's been embedded directly into the Docker CLI. With the new version of Swarm, docker-compose isn't directly supported, yet. Consider this a beta product that works well for a limited scope. You can try the experimental release of the docker engine which adds support for DAB files that are managed with the docker stack CLI. The DAB files are exported from docker-compose bundle and then imported into Docker. This feature is still very experimental and expected to change.
Without that, anything with docker-compose will only operate on a single docker engine since the swarm access is all done under a different docker service CLI interface.
I accidentally removed my container with "docker rm [CONTAINER_ID]".
Is there anyway that I can undo this, or restore the data in the container?
It was a CentOS image.
$docker info
Containers: 1
Images: 30
Server Version: 1.9.1
Storage Driver: devicemapper
Pool Name: docker-202:16-1179651-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem:
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 8.998 GB
Data Space Total: 107.4 GB
Data Space Available: 43.66 GB
Metadata Space Used: 11.45 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.136 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /srv/docker/devicemapper/devicemapper/data
Metadata loop file: /srv/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.17-22.30.amzn1.x86_64
Operating System: Amazon Linux AMI 2016.03
CPUs: 2
Total Memory: 3.862 GiB
Name: ip-172-31-14-126
ID: QXWS:5VBE:CIZV:NF57:KNTZ:ZOIV:HIZZ:PXKW:44LT:KVFZ:ECQI:FPIX
Username: hogehoge
Registry: https://index.docker.io/v1/
Similar question:
how to retrieve volume from a removed Docker container?
The above question is asking how to retrieve data from a "data volume
container", but my one was not. I stored everything inside the CentOS
image.
The answer is probably no.
According to the comment I received from docker forum:
docker rm is just like rm on the host, there is no going back
Personal story:
I lost 3 weeks of my business data, and having big trouble with it.
Good lesson to teach myself that never "rm" unless you are 100% sure what you are doing, and backup your data!
I've built an image from a Dockerfile, committed it and am now trying to push it to the Hub. The command I run:
sudo docker push lisahelm/mongo:v2
What prints out:
The push refers to a repository [lisahelm/mongo] (len: 1)
7b6d0719b415: Image already exists
975e0be2d43f: Image already exists
ee08822aa3f9: Image already exists
96f2191238d5: Image already exists
07f8e8c5e660: Image already exists
37bea4ee0c81: Image already exists
a82efea989f9: Image already exists
e9e06b06e14c: Image already exists
FATA[0015] Error pushing to registry: Server error: 400 trying to push lisahelm/mongo:v2 manifest
Info I've seen people ask for in other questions:
Docker version 1.6.0, build 4749651/1.6.0
Containers: 4
Images: 22
Storage Driver: devicemapper
Pool Name: docker-202:1-263695-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.222 GB
Data Space Total: 107.4 GB
Data Space Available: 4.88 GB
Metadata Space Used: 2.58 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.145 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.14.35-28.38.amzn1.x86_64
Operating System: Amazon Linux AMI 2015.03
CPUs: 1
Total Memory: 1.957 GiB
Name: ip-172-31-11-134
ID: WVRW:3RM3:L4KL:YABF:JGSK:S6ML:U2CH:5Z5G:67CY:24BF:3DIE:E6TA
Username: lisahelm
Registry: [https://index.docker.io/v1/]
You are using an Amazon AMI running RHEL.
The docker shipped by Amazon on these machines is defective.
The solution is to either switch to Ubuntu, or update docker manually on the instance.
You can read here for context and an answer from an AMZ dev: https://github.com/docker/docker/issues/13143#issuecomment-102522728
And here for another user coming up with an update solution: https://github.com/docker/distribution/issues/538#issuecomment-104241554
Finally here for the Amazon support forum about this bug: https://forums.aws.amazon.com/thread.jspa?messageID=622774