Docker (in Azure App Service) Running out of space - docker

A docker container running jsReport is running out of sapce and the container is failing.
Error in logfile
2021-07-13T07: 37: 25.469Z ERROR - Container start failed for xxxx
with System.AggregateException, One or more errors occurred. (Docker
API responded with status code=InternalServerError, response= {
"message": "OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container
init caused: rootfs_linux.go:59: mounting
"/var/LWASFiles/Sites/xxxx/appsvctmp" to rootfs at
"/mnt/data/docker/images/231072.231072/aufs/mnt/7c0065c198b3881806bb293ccee0ceef29b8b619bb47a7512f9b5485557811d5/appsvctmp"
caused: mkdir
/mnt/data/docker/images/231072.231072/aufs/mnt/7c0065c198b3881806bb293ccee0ceef29b8b619bb47a7512f9b5485557811d5/appsvctmp:
no space left on device: unknown" }
Problem started when I was updating to service where each dowload was about 200MB+.
I could probably fix this by removing the service and create a new one OR up the service plan which adds more space.
However I would think there must be a way to use docker commands to clean up some of these upgrade files. Can anyone point me in the right direction?

you can use docker system df command to see the reclaimable space as shown in below image
you can use the docker prune command to remove the unused objects such as images, containers, volumes, and networks
https://docs.docker.com/config/pruning/

Related

Docker error failed to create shim: OCI runtime create failed: container_linux.go:380:

ERROR: for app_web
Cannot start service app_web: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting "/host_mnt/Users/akanwar/Documents/c/cbax-config" to rootfs at "/cbax-apply-platform/node_modules/#c/cbax-config" caused: mkdir /var/lib/docker/overlay2/a3bccebb167966c795860d95a5a758f244ae5da780f962333f0d51d2d8b2def7/merged/cbax-apply-platform/node_modules/#c/cbax-config: operation not permitted: unknown
STEPS TAKEN TO RESOLVE
deleted the docker data ran everything.
From Docker widget Clean / Purge data
From Docker widget reset to factory settings
docker system prune --all
https://github.com/docker/for-mac/issues/1396 I followed this link as well and tried the solutions mentioned in it , but still none of it worked for me.
None of the solutions worked for me
You need to remove this from all services:
- ${CBAX_PATH}/cbax-apply-platform:/cbax-apply-platform:rw
You cannot mount a folder and then also sub-folders. This is wrong:
- ${CBAX_PATH}/cbax-apply-platform:/cbax-apply-platform:rw
- ${CBAX_PATH}/cbax-application-pages:/cbax-apply-platform/app/assets/components/cbax-application-pages:rw
Also you mount a lot of host folders in multiple service with read-write access. I think that makes a mess of your files on the host if all services start modifying files. It's better to mount them read-only.
This happens when the docker-compose file was looking for the folder cbax-config path on my machine on mac OS. MacOS doesn't have an cbax-config therefore it was not able to mount.

cannot start any docker containers sometime anymore

Sometimes after:
a container has created a lot of threads
I restarted a container several time because it crashed
it got when starting (docker-composer up) an arbitrary container:
ERROR: for container_name Cannot start service container_name: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/memory/docker/99dec670f659c8411b960d99bbafc13f0429e178b1ea8417e6cfc8d56182291b: permission denied: unknown
There is neither a high load (<100) in number of thread not in memory usage of the server reported by docker stats``or htop`.
The problem can not be fixed by restarting the docker process (docker.service and docker.socket) - but by restarting the (virtual) machine.
Also after sometime (2 days / weekend) of not doing anything on the server (but keep other containers running) the problem disappears.

OCI runtime exec failed

since today i have got an error message with various docker commands. Unfortunately I don't really know what to do with it. Does anyone have any idea what the problem could be and how I fix it?
Error:
OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: process_linux.go:95: starting setns process caused: fork/exec /proc/self/exe: resource temporarily unavailable: unknown
Another Error:
ERROR: for hosting_mail_1 Cannot start service mail: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/5fabf9edf67fbd6455bdc955c56c063683aa78e8e31514660661799aaa867391/log.json: no such file or directory): runc did not terminate successfully: unknown
ERROR: for mail Cannot start service mail: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/5fabf9edf67fbd6455bdc955c56c063683aa78e8e31514660661799aaa867391/log.json: no such file or directory): runc did not terminate successfully: unknown
ERROR: Encountered errors while bringing up the project.
I don't know if you solved your problem finally, but this really looks like bad file system authorization that may have been corrupted from an update on the file systems.
Regarding the error : container_linux.go:370:,/run/containerd/io.containerd.runtime.v2.task/moby/5fabf9edf67fbd6455bdc955c56c063683aa78e8e31514660661799aaa867391/log.json
I can see that :
docker manage to initiate a volume ID
did not manage to mount that volume on the disk
0/ Check docker basic command
docker ps
docker images
docker pull ubuntu:latest
If one of these commands is failing, you are up to review docker installation, seems that maybe docker is not installed properly.
1/
To check if you need to completely re-install docker, you may try the following basic command
docker run --name checkDocker -it ubuntu:latest bash
If this is not displaying any docker shell, then you have a problem on running a container, not necessarly docker installation.
2/
Check your docker volumes and rights, I don't have your installation setup, but It seems you are using docker-compose and maybe there is some conflicts when mounting the volume of your containers with specific rights and the host's rights and user id
3/
If you are ending up here you should follow that work around of re-installation, which would be the fatest solution to restore your application if you have backup (hope you have )

Docker Image running correctly on one machine but failing on another

When Trying to install Istio 1.2.3 on my cluster using Helm, I encountered an issue with the istio/kubectl image being used in the istio-init jobs with the following error:
container_linux.go:295: starting container process caused "exec: \"kubectl\": executable file not found in $PATH"
docker: Error response from daemon: oci runtime error: container_linux.go:295: starting container process caused "exec: \"kubectl\": executable file not found in $PATH".
Running the kubectl command in my local docker also gives the same error, however on another machine it works correctly
docker run <istio/kubectl-imageid> kubectl
What could cause this issue? And what would I need to change to overcome it?
It is definitely the same docker image and from my understanding a docker image should work identically in different environments assuming the same cpu architecture.
Turns out when I copied the image across machines, I did a
docker import istio-kubectl.1.2.3.tar
instead of a
docker load istio-kubectl.1.2.3.tar
The difference according to the documentation is:
docker load: Load an image from a tar archive or STDIN
docker import: Import the contents from a tarball to create a filesystem image
Loading the image instead of importing corrected the observed issue.

Docker rootfs_linux.go permission denied when mounting /proc

I'm using meteor-up to deploy to docker instances inside an LXD container. When it attempts to bring up a MongoDB docker instance, it fails with the following error:
docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\\"proc\\\" to rootfs \\\"/var/lib/docker/vfs/dir/20a621a73755db3f4030bc67bbf7ff9540bd1d63bb348d8be440d8ca63d3c922\\\" at \\\"/proc\\\" caused \\\"permission denied\\\"\"".
I've tried deploying as the ubuntu user to my LXD container, and as root, and that hasn't helped. There's obviously some permissions issue here, but I don't know where to start debugging this problem.
According to https://discuss.linuxcontainers.org/t/having-trouble-using-docker/383 it sounds like you might need to set security.nesting=true. Stop the container (lxc stop deploy-container), configure nesting on (lxc config set deploy-container security.nesting true as per https://insights.ubuntu.com/2015/10/30/nested-containers-in-lxd) and lxc start deploy-container.
For Proxmox users: https://www.youtube.com/watch?v=79KiCBNbsbg
Enable Nesting feature on the container to fix the following error
ERROR: for mysql-8.0 Cannot start service mysql-8.0:
failed to create shim: OCI runtime create failed:
container_linux.go:380: starting container process caused: process_linux.go:545:
container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at
"/proc" caused: mount through procfd: permission denied: unknown
But other users Proxmox users say to enable keyctl too.
If you still have errors you may need to remove domain_name and hostname from the docker-compose config to fix the following errors as said here.
ERROR: for mariadb10-4 Cannot start service mariadb10-4: failed to create shim: OCI runtime create failed: container_linux.go:380:
starting container process caused: process_linux.go:545:
container init caused: write sysctl key kernel.domainname:
open /proc/sys/kernel/domainname: permission denied: unknown

Resources