Sometimes after:
a container has created a lot of threads
I restarted a container several time because it crashed
it got when starting (docker-composer up) an arbitrary container:
ERROR: for container_name Cannot start service container_name: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385: applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/memory/docker/99dec670f659c8411b960d99bbafc13f0429e178b1ea8417e6cfc8d56182291b: permission denied: unknown
There is neither a high load (<100) in number of thread not in memory usage of the server reported by docker stats``or htop`.
The problem can not be fixed by restarting the docker process (docker.service and docker.socket) - but by restarting the (virtual) machine.
Also after sometime (2 days / weekend) of not doing anything on the server (but keep other containers running) the problem disappears.
Related
A docker container running jsReport is running out of sapce and the container is failing.
Error in logfile
2021-07-13T07: 37: 25.469Z ERROR - Container start failed for xxxx
with System.AggregateException, One or more errors occurred. (Docker
API responded with status code=InternalServerError, response= {
"message": "OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container
init caused: rootfs_linux.go:59: mounting
"/var/LWASFiles/Sites/xxxx/appsvctmp" to rootfs at
"/mnt/data/docker/images/231072.231072/aufs/mnt/7c0065c198b3881806bb293ccee0ceef29b8b619bb47a7512f9b5485557811d5/appsvctmp"
caused: mkdir
/mnt/data/docker/images/231072.231072/aufs/mnt/7c0065c198b3881806bb293ccee0ceef29b8b619bb47a7512f9b5485557811d5/appsvctmp:
no space left on device: unknown" }
Problem started when I was updating to service where each dowload was about 200MB+.
I could probably fix this by removing the service and create a new one OR up the service plan which adds more space.
However I would think there must be a way to use docker commands to clean up some of these upgrade files. Can anyone point me in the right direction?
you can use docker system df command to see the reclaimable space as shown in below image
you can use the docker prune command to remove the unused objects such as images, containers, volumes, and networks
https://docs.docker.com/config/pruning/
since today i have got an error message with various docker commands. Unfortunately I don't really know what to do with it. Does anyone have any idea what the problem could be and how I fix it?
Error:
OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: process_linux.go:95: starting setns process caused: fork/exec /proc/self/exe: resource temporarily unavailable: unknown
Another Error:
ERROR: for hosting_mail_1 Cannot start service mail: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/5fabf9edf67fbd6455bdc955c56c063683aa78e8e31514660661799aaa867391/log.json: no such file or directory): runc did not terminate successfully: unknown
ERROR: for mail Cannot start service mail: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/5fabf9edf67fbd6455bdc955c56c063683aa78e8e31514660661799aaa867391/log.json: no such file or directory): runc did not terminate successfully: unknown
ERROR: Encountered errors while bringing up the project.
I don't know if you solved your problem finally, but this really looks like bad file system authorization that may have been corrupted from an update on the file systems.
Regarding the error : container_linux.go:370:,/run/containerd/io.containerd.runtime.v2.task/moby/5fabf9edf67fbd6455bdc955c56c063683aa78e8e31514660661799aaa867391/log.json
I can see that :
docker manage to initiate a volume ID
did not manage to mount that volume on the disk
0/ Check docker basic command
docker ps
docker images
docker pull ubuntu:latest
If one of these commands is failing, you are up to review docker installation, seems that maybe docker is not installed properly.
1/
To check if you need to completely re-install docker, you may try the following basic command
docker run --name checkDocker -it ubuntu:latest bash
If this is not displaying any docker shell, then you have a problem on running a container, not necessarly docker installation.
2/
Check your docker volumes and rights, I don't have your installation setup, but It seems you are using docker-compose and maybe there is some conflicts when mounting the volume of your containers with specific rights and the host's rights and user id
3/
If you are ending up here you should follow that work around of re-installation, which would be the fatest solution to restore your application if you have backup (hope you have )
The ssh disconnected when i using a container, and then i find all containers were dead.
I want to restart my container by
docker restart mysql2
but fail with error
Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused "process_linux.go:396: setting cgroup config for procHooks process caused \"failed to write max to pids.max: write /sys/fs/cgroup/pids/docker/3e8d10946219b7ba3a2259f8f21222c978dc51fade62fd882f02d191e24c8cb5/pids.max: invalid argument\""": unknown Error: failed to start containers: mysql2
Then i tried to restart docker and restart linux, both not work.
Is there any way to troubleshoot the problem?
i resolved this problem, the step:
docker update --pids-limit 40000 mysql2
docker restart mysql2
by reset the pids limit of docker, i can restart my container now.
I'm trying to set net.core.rmem_max=26214400 on a container
I have this docker-compose
develop:
sysctls:
- net.core.rmem_max=26214400
image: centos6
but when I do
docker-compose up -d develop
I get this error
ERROR: for develop_1 Cannot start service develop: OCI runtime create
failed: container_linux.go:348: starting container process caused
"process_linux.go:402: container init caused \"open
/proc/sys/net/core/rmem_max: no such file or directory\"": unknown
ERROR: for develop Cannot start service develop: OCI runtime create
failed: container_linux.go:348: starting container process caused
"process_linux.go:402: container init caused \"open
/proc/sys/net/core/rmem_max: no such file or directory\"": unknown
What docker image are you using? Not all namespaced kernel parameters on the host are available in the container. Most likely the one you mention is not.
You can quickly inspect which are indeed available by running:
docker run --rm develop ls -l /proc/sys/net/core
I'd also recommend checking the corresponding compose docs.
I updated docker and compose to
Docker version 17.06.0-ce, build 02c1d87
docker-compose version 1.14.0, build c7bdf9e
Now when I am running container with docker run it works fine. But I have 10+ services defined in compose file. For last 1 year it was working file and after update and restarting docker service it suddenly giving me -
ERROR: for extremis_maxwell_1 Cannot start service maxwell: oci runtime error: container_linux.go:262: starting container process caused "process_linux.go:339: container init caused \"read init-p: connection reset by peer\""
ERROR: for maxwell Cannot start service maxwell: oci runtime error: container_linux.go:262: starting container process caused "process_linux.go:339: container init caused \"read init-p: connection reset by peer\""
I tried re-installing everything but error remains the same.
I was facing the same issue in case I use memory limit on CentOS 7.2. All you need to do is update your RedHat or CentOS kernel to -514 patch or wait for docker 17.06.1.
yum update kernel
See https://github.com/moby/moby/issues/34046