I am trying to kill containers I launched through docker-compose. Either by gracefully stopping (Ctrl+C) or by docker-compose down I encounter the following error:
ERROR: for <container-name> cannot stop container: b60c1c4d886899504b...2a022e4d39429dc6ca6e4784afdd: Cannot kill container b60c1c4d886899504b...2a022e4d39429dc6ca6e4784afdd: unknown error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused "permission denied"
: unknown
I am just looking for the answer WHY. I am trying to understand AppArmor better but understanding WHY I can't stop the containers before everything would help to understand what's going on.
I see that this is an error many people come across. 1 2 3 4
However, most of the answers suggest workarounds and no solutions. Even the explanatory answers like 1 dive directly into AppArmor and profiles. From docker documentation I see docker has a default AppArmor policy docker-default. I partly understand the concept but still don't get WHY I can't stop the containers through the user and the environment I've started them in.
If I try to wrap my questions:
I started some containers as a user, WHY can't I stop them. sudo does not work either. Who can stop them then?
Do I need an AppArmor profile for each container?
I don't feel it's a good idea to restart AppArmor or disable it. Should I do that? What is the ideal solution?
Any feedback or explanations welcome. Thanks.
I couldn't find the exact reason but came close. It seems there were conflicting docker installations on my Ubuntu 19.10 and this was causing this access control issue.
As stated here, I've removed the snap installation. As I already had another installation I didn't have to install it via another way.
sudo snap remove docker
Related
I am trying to stop a docker container and get the following error:
This happens randomly on occasion and it is very frustrating to have restart the docker service and relaunch all my containers.
Would anyone know what could be happening to cause this? As far I have seen or know, there has not been any changes made to the container since they have been launched, may some changes in the content of the data in the containers. Also if people need more information, I would be happy to provide.
FYI everything that I am doing I am doing as a root user.
ALSO -- ABSOLUTLEY CANNOT STOP THE DOCKER DAMON OR RESTART IT, THIS MUST BE RESOLVED WHILE KEEPING THE CURRENT CONTAINERS OPEN AND RUNNIN.
I have a multi-services environment that is hosted with docker swarm. There are multiple stacks that are created. All the docker containers which are running have an inbuild Spring Boot application. The issue is coming that all my stacks get restarted on their own. Now I know that in compose file I have mentioned that restart_policy as on failure. Hence it auto restarted. The issue comes that when services are restarted, I get errors from a particular service and this breaks everything.
I am not able to figure out what actually happens.
I did quite a lot of research and found out about these things.
Docker daemon is not restarted. I double-checked this with the uptime of the docker daemon.
I checked the docker service ps <Service_ID> and there I can see service showing shutdown and starting. No other information.
I checked the docker service logs <Service_ID> but no error in there too.
I checked for resource crunch. I can assure you that there was quite a good resource available at the host as well as each container level.
Can someone help where exactly to find logs for this even? Any other thoughts on this?
My host is actually a VM hosted on VMWare Vcenter.
After a lot of research and going through all docker logs, I could not find the solution. Later on, I discovered that there was a memory snapshot taken for backup every 24 hours.
Here is what I observe:
Whenever we take a snapshot, all docker services running on the host restart automatically. There will be no errors in that but they will just restart gracefully.
I found some questions already having this problem with VMware snapshots.
As far as I know, when we take a snapshot, it points to a different memory location and saves the previous one. I am not able to find why it's happening but yes Root cause of the problem was this. If anyone is a VMWare snapshots expert, please let us know.
I've got a docker container running a service, and I need that service to send logs to rsyslog. It's an ubuntu image running a set of services in the container. However, the rsyslog service cannot start inside this container. I cannot determine why.
Running service rsyslog start (this image uses upstart, not systemd) returns only the output start: Job failed to start. There is no further information provided, even when I use --verbose.
Furthermore, there are no error logs from this failed startup process. Because rsyslog is the service that can't start, it's obviously not running, so nothing is getting logged. I'm not finding anything relevant in Upstart's logs either: /var/log/upstart/ only contains the logs of a few things that successfully started, as well as dmesg.log which simply contains dmesg: klogctl failed: Operation not permitted. which from what I can tell is because of a docker limitation that cannot really be fixed. And it's unknown if this is even related to the issue.
Here's the interesting bit: I have the exact same container running on a different host, and it's not suffering from this issue. Rsyslog is able to start and run in the container just fine on that host. So obviously the cause is some difference between the hosts. But I don't know where to begin with that: There are LOTS of differences between the hosts (the working one is my local windows system, the failing one is a virtual machine running in a cloud environment), so I wouldn't know where to even begin about which differences could cause this issue and which ones couldn't.
I've exhausted everything that I know to check. My only option left is to come to stackoverflow and ask for any ideas.
Two questions here, really:
Is there any way to get more information out of the failure to start? start itself is a binary file, not a script, so I can't open it up and edit it. I'm reliant solely on the output of that command, and it's not logging anything anywhere useful.
What could possibly be different between these two hosts that could cause this issue? Are there any smoking guns or obvious candidates to check?
Regarding the container itself, unfortunately it's a container provided by a third party that I'm simply modifying. I can't really change anything fundamental about the container, such as the fact that it's entrypoint is /sbin/init (which is a very bad practice for docker containers, and is the root cause of all of my troubles). This is also causing some issues with the docker logging driver, which is why I'm stuck using syslog as the logging solution instead.
I have an issue. I typed the minikube start command and it stuck. What should I do? Is deleting minikube the only solution?
Restarting existing docker container for "minikube"
You have provided too little information to conclusively solve your problem. But one way is to actually delete the minikube and restart. You can see this similar question. Make sure that you have proper privileges to run docker containers.
Generally, this problem occurs quite often on Ubuntu. You can find very extensive thread on github.
In addition to the Restarting existing docker container for "minikube", you should also get some other information (like specific error). If they are insufficient, you can always open an issue on github.
In the thread above you can find a couple of potential solutions. Here is one of them:
When I run minikube --start --driver=docker --alsologtostderr, I get the same error message with "no such file or directory".
Edit: I was able to fix this by changing to .deb docker instead of snap docker.
Per https://kubernetes.io/docs/tasks/tools/install-minikube/:
"If you're using the none driver in Debian or a derivative, use the .deb packages for Docker rather than the snap package, which does not work with Minikube. You can download .deb packages from Docker."
I did $ snap remove docker, then followed these instructions:
https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
Maybe the error messages could be amended to tell this to the user?
You can try sudo minikube delete to delete the container first,
then minikube start and see if the issue is fixed or not.
Can anyone help me make sense of the below error and others like it? I've Googled around, but nothing makes sense for my context. I download my Docker Image, but the container refuses to start. The namespace referenced is not always 26, but could be anything from 20-29. I am launching my Docker container onto an EC2 instance and pulling the image from AWS ECR. The error is persistent no matter if I re-launch the instance completely or restart docker.
docker: Error response from daemon: oci runtime error:
container_linux.go:247: starting container process caused
"process_linux.go:334: running prestart hook 0 caused \"error running
hook: exit status 1, stdout: , stderr: time=\\\"2017-05-
11T21:00:18Z\\\" level=fatal msg=\\\"failed to create a netlink handle:
failed to set into network namespace 26 while creating netlink socket:
invalid argument\\\" \\n\"".
Update from my Github issue: https://github.com/moby/moby/issues/33656
It seems like the DeepSecurity agent (ds_agent) running on a container with Docker can cause this issue invariably. A number of other users reported this problem, causing me to investigate. I previously installed ds_agent on these boxes, before replacing it with other software as a business decision, which is when the problem went away. If you are having this problem, might be worthwhile to check if you are running the ds_agent process, or other similar services that could be causing a conflict using 'htop' as the user in the issue above did.
Did you try running it with the --privileged option?
If it still doesn't run, try adding --security-opts seccomp=unconfined and either --security-opts apparmor=unconfined or --security-opts selinux=unconfined depending whether you're running Ubuntu or a distribution with SELinux enabled, respectively.
If it works, try substituting the --privileged option with --cap-add=NET_ADMIN` instead, as running containers in privileged mode is discouraged for security reasons.