Cannot create container for service xxx: open /var/lib/docker/overlay2/969rf5...fdf-init/merged/etc/resolv.conf: Operation not permitted - docker

while deploying docker images on the production server using docker-compose, I got the following error.
Cannot create container for service xxx: open /var/lib/docker/overlay2/969rf5...fdf-init/merged/etc/resolv.conf: Operation not permitted
What I tried:
Changed permissions ( So much so that gave 777 for all the directories involved)
Upgraded kernel (Saw somewhere that it could be kernel issue)
Removed ACL using setfacl -Rb /var/lib/docker
Added the "graph: /var/lib/docker" line in daemon.json
Restarted docker service couple of times.
Tried running the images individually and not using docker-compose.
Tried running with sudo
Set the --storage-opt overlay2.override_kernel_check=1 (Since it is rhel and kernel version supported for docker overlay2 is >4.0, so was suggested to override it.)
I failed everytime!
Have no idea what the issue is or what it could be. Every time I run the docker-compose file, it creates a new directory in /var/lib/docker/overlay2 with old permissions even though I change the permissions for all the other directories in it. At this point I'm not even sure if it is permissions issue or not.
Any help would be appreciated! Thank you!

Finally after trying everything on the internet and nothing working out, the issue was resolved. It was due to the Antivirus that was installed on the server. Had it removed by the client and docker/docker-compose started working absolutely fine.
To give a little more information on the antivirus, it was some FIM antivirus that was installed on a rhel 3.10-xxx which was blocking docker. It was an older version and thank goodness newer versions are compatible with docker. So that's a wrap, thank you all!

Related

Docker failed to initialize Docker Desktop is shutting down

This question was already asked a lot, but I didn't found a solution which is working for me.
I downloaded Docker Desktop with his guide for my windows 10 machine. After a successful installation the application does not start.
Following this solution didn't work. I deleted settings.json and restarted everything without success. Even reinstalling did not work. Any one has an other idea?
Edit:Reinstalling an older version (3.6.0 in my case) fixed the issue, but there has to be some other way...
Stop all processes with Docker and remove all content from the directory
C:\Users\<User>\AppData\Roaming\Docker
After that start Docker Desktop.

Docker not starting on Windows 11 with WSL 2

I'm trying to setup docker with WSL 2 to run a Dockerfile. I downloaded Docker Desktop, and when I tried to follow the quick start guide, I got the following error:
docker: error during connect: This error may indicate that the docker daemon is not running.: Post "http://%2F%2F.%2Fpipe%d2Fdocker_engine/v1.24/containers/create?name=repo": open //./pipe/docker_enginer: The system cannot find the file specified.
I set com.docker.service to run in the Task Manager, and have run:
“c:\Program Files\Docker\Docker\DockerCli.exe” -SwitchDaemon
I have also quit Docker Desktop and reopened it in admin mode, and I still get the message that Docker Desktop has stopped.
Please let me know if there's any other options, thanks!
I managed to solve the problem in this way:
delete config file AppData/Roaming/Docker/settings.json
start Docker Desktop (it will create new default settings.json)
You need to re-configure settings, so it is also good idea to save the old settings.json and compare settings with the new one.
If you are running Docker Desktop for Windows 4.5.0 then you should be aware of an existing issue where the default backend selected after installing is not the correct one.
You can switch to the correct backend manually by editing the file located at: %AppData%\Docker\settings.json (full path: C:\Users\%UserName%\AppData\Roaming\Docker\settings.json) and at the bottom of the file change the value for the wslEngineEnabled field to true. After that Docker Desktop should start correctly.
Similar issue and solution is mentioned here by the user mccaa25.
latest version 4.5.0 It won't work for me too, for me downgrade to 4.4.4 it worked. It may be problem with the latest version
for released note and older version Here
Note: You can install an older version and update to latest version 4.5.0
Wow! I spent multiple days trying to get Docker for windows 11 to work. What finally did it was downgrading to version 4.4.4 as #GinglrBell recommended. Here's a link Docker version 4.4.4.
Make sure WSL is installed and virtualization is enabled. If you try to open WSL and you get an error telling you to enable virtualization in BIOS, run this command as Admin in powershell: bcdedit /set hypervisorlaunchtype auto (Docker src). Then restart your PC.
I hit this issue today, and the steps below are what worked for me - the gist is that I needed to install WSL 2 beforehand. Most of my steps were derived from here:
open powershell as admin
wsl -l -v # Windows Subsystem for Linux has no installed distributions
install linux kernel update from here # in my case, x64-based
wsl --set-default-version 2
wsl --list --online # shows several linux distributions, I picked a recent one
wsl --install -d Ubuntu-20.04 # then create default linux user name and password
install windows docker desktop from here # now starts successfully
Manged to switch to Windows Containers. (Docker v4.9.0)
What I did was to change the following in the settings.json:
"useWindowsContainers": true,
"wslEngineEnabled": true,
Also set security on folder:
C:\ProgramData\Docker (I set "Everyone" full control.)
After that I used the "Switch to Windows Containers"-feature from the task-bar icon content menu
I tried Janne Rantala's method.
But the same problem occurs again after several days.
My final solution is to uninstall Docker Desktop, then download and run the installation file as an administrator.
Then the problem no longer appears again.
I was able to resolve this issue on my machine by installing the Linux kernel update package as described in the Microsoft guide
I have been having some Docker setting up troubles for a fair amount of time. I think I had to just make sure I removed all references of Docker. After buying Windows 10 pro I still had issues, I did have to look into the following folders:
C:\Users\%UserName%\AppData\Local\Temp\Docker
C:\Users\%UserName%\AppData\Roaming\Docker
I also deleted the Docker file in my programfiles folder
Navigated to said file and deleted Docker files.
Used the docker installer, checkboxes (to use wsl2).
I was faicing the same issue and resolved. I have just open Microsoft Store application and after that opened the docker and its working fine. This is wired but sharing this so it might helps someone.
I solved this issue by enabling the windows feature "Windows Hypervisor Platform" and restarting my computer.
I installed Docker Desktop v4.16.2

Docker Failed to Initialize on Windows 10

I have an issue with running the Docker Desktop on my Windows 10 PC. After trying to run it I got an error:
Docker failed to initialize - Docker Desktop is shutting down
I already tried reinstalling (as admin), restarting PC, restarting docker services, removing content from .../AppData/Local/Docker and .../AppData/Roaming/Docker folders, but still receiving the same error message.
Does anyone have an idea how can I fix it?
Thank you a lot!
Simply local Appdata on your windows box, go to Roaming->Docker and delete the settings.json file. Restart docker and you will be fine.
Appdata can be located at this path: C:\Users\LifeIsGood\AppData\Roaming\Docker
NB. LifeisGood is my username
Downgrade to the older version solved my issue
You can try uninstalling docker and deleting
/Users/AppData/Local/Docker
/Users/AppData/Roaming/Docker
C:/Program Files/Docker
C:/ProgramData/Docker
Reinstalling Docker
However, what worked for me ultimately was deleting .docker in /Users folder and restarting docker.
I get that a lot on my laptop. I restart the "Hyper-V Host Compute Service" in Services. When that has restarted, I can start Docker Desktop.
I haven't found a permanent fix.
I had the same issue while starting the "Linux container" but windows container was working fine.
After trying All the possible methods to make docker desktop work, In case that is not worked then try this one. This one worked for me.
Please make sure to use older version of docker desktop ( mine is 4.4)
In docker desktop,
Go to settings --> General --> Use the WSL 2 based engine
Where If that check box turned On, Just Turn that off.

java.lang.IllegalStateException: Could not connect to Ryuk at localhost:49167

I am using test containers library to launch a container. It worked for a while but currently running into this
java.lang.IllegalStateException: Could not connect to Ryuk at localhost:49167
at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:201)
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
at org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
I looked at the resource reaper code and it seems like it is failing here
public synchronized void performCleanup() {
this.registeredContainers.forEach(this::stopContainer);
this.registeredNetworks.forEach(this::removeNetwork); //FAILS HERE
this.registeredImages.forEach(this::removeImage);
}
I can confirm that this is resolved by updating testcontainers to 1.16.0 version
Was facing the same exception with testcontainers 1.15.3
Windows 10, Docker desktop 4.0.0
I tried everything mentioned here but nothing worked.
if anyone is still facing the issue they can give this one a try...this worked for me
this works for windows
open you power shell in administrator mode.
run "net stop winnat"
run "net start winnat"
doing this resolved this issue on my setup. hope this helps someone
I ran into the same issue. Same Docker-for-Mac version.
Solution for me was to disable "Use gRPC FUSE for file sharing" in Preferences:
For me, restarting the local docker engine solved the problem
I'm on a Windows 11 machine with the same issue:
It turned out that a vast number of ports were allocated by the service "IP helper" (relates to ipv6 support), i.e. ports prior allocated by testcontainers, or other services. They were allocated for weeks by Ip helper but never, or rarely, released. By permanently disabling service Ip helper all issues were resolved and the set-up works great since then without any observed side effects that I am in need of.
Setting: Windows11 + Rancher Desktop
Restarting Rancher Desktop did not help. However, restarting my computer did the job.

Erlang machine stopped instantly (distribution name conflict?). The service is not restarted as OnFail is set to ignore

I am using RabbitMQ. For some reason the rabbitMQ service stops as soon as you start it. I saw following error in the event log:
RabbitMQ: Erlang machine stopped instantly (distribution name conflict?). The service is not restarted as OnFail is set to ignore.
Someone told me to run this command: erl -sname rabbit
This command generates following output:
{(no error logger present")i neirtr otre: r"mEirnraotri nign ipnr odcoe_sbso o<
t0".,2{.b0a>d awrigt,h[ {eexrilt_p rviaml_uleo:a d{ebra,dcahregc,k[_{feirlle__pr
reismu_llto,a3d,e[r{,fcihleec,k"_efrill_e_prreismu_llto,a3d,e[r{.feirlle",}\,"{e
lriln_ep,r29i3m}_]l}o,a{dienri.te,rgle\t"_}b,o{olti,n1e,,[2{9f3i}l]e},,"{iinniit
t.,egrelt"_}b,o{olti,n1e,,[78{9f}i]l}e,,{\i"niinti,tg.eetr_lb\o"o}t,,{2l,i[n{ef,
i7l8e9,}"]i}n,i{ti.neirtl,"g}e,t{_lbionoet,,7762},][}{,f{iilnei,t\,"dion_ibto.oe
tr,l3\,"[}{,f{illien,e",i77n6i}t].}e,r{li"n}i,t{,ldion_eb,o74o3t},]3},][}{}f
ile,\"init.erl\"},{line,743}]}]}\n"
I am not sure how to interpret this output. I wonder the error is specific to RabbitMQ or erlang.
I have no idea how to procceed. Please suggest.
I have just run into this problem setting up RabbitMq as a service up on a new Windows server. The only thing I can think of that broke it for me is renaming the new windows box after installing the RabbitMq service, but before testing it for the first time.
First off I noticed it ran as an application fine. I solved it by installing the service again using the command from the manual install instructions:
rabbitmq-service install
Assuming that you have your path variables included for the RabbitMq sbin directory.
The only thing that worked for me was to clear the directory C:\Users\xxxxx\AppData\Roaming\RabbitMQ.
(cf. https://groups.google.com/forum/#!topic/rabbitmq-users/138RHzzsORU)
In my scenario, Two directories of Erlang under C:\Program Files with different versions were there, I uninstalled one of the version, also uninstalled RabbitMQ service from Windows services list - Restarted the system.
Again ran RabbitMQ setup - RabbitMQ service was setup successfully.
I ran into the same issue when installing RabbitMQ 3.7.17 via Chocolatey on a Windows Server 2016.
After trying most of the suggested solutions, the one that worked for me was:
rabbitmq-service remove
rabbitmq-service install
rabbitmq-service start
PS: if your PATH is not configured for RabbitMQ, this is the folder you need to run the commands from: C:\Program Files\RabbitMQ Server\rabbitmq_server-3.7.17\sbin (if your version is also 3.7.17).
For anyone else looking up this error: double check your config files and SSL files. I ran into this issue when I had specified the ssl_options.cacertfile with ca.pem but the file was mistyped as ca-pem in the directory. Unfortunately RabbitMQ wasn't smart enough to catch the missing file and was dumping with no logs.
I found a name conflict with an env-variable, I use since years - means, this was not a problem with the previous version.
I have "Logs" and apps will write into that directory, usually with their own subdirectories. RMQ uses the same variable name and means a plain filename.
So using: "C:\Users\rabbit\AppData\Roaming\RabbitMQ\log\log" made it working for me - this is in the rabbit's users private environment. So the global settings are now not seen by rabbit. Uff. And it looks like, this is really meant as a filename and after I changed it again to "rabbit#c4711-node.log", it writes like the earlier version. The service starts now for me - but this was really messy and I don't trust it at the moment ;-)
From my perspective, one should run such a service under its own account. If the service is already there, create a local user account - I've used "rabbit" and give it a password. The account I created, got admin right from me - but I currently just dont know, it this is needed. At least it should not - will see this later. If you have account/credentials, go to the service manager and click properties for the service. On the second tab ("log on"), check "this account" and enter username an password. If you have an account for the service you should be able to login with user.
Then you can specify environment variables with user scope.
To do this, logon with the user you created. Go to ControlPanel/System and click "advanced":
In the Environment UI, enter user specific variables
in the top panel:
Note: This was not my rabbit user, because I currently cannot login there. The variables, I entered - not guaranteed, it is correct - are the following:
RABBITMQ_BASE=C:\Users\rabbit\AppData\Roaming\RabbitMQ
RABBITMQ_CONFIG_FILE=C:\Users\rabbit\AppData\Roaming\RabbitMQ\rabbitmq
RABBITMQ_LOGS=C:\Users\rabbit\AppData\Roaming\RabbitMQ\log
RABBITMQ_LOG_BASE=C:\Users\rabbit\AppData\Roaming\RabbitMQ\log
RABBITMQ_NODE_IP_ADDRESS=192.168.26.3
This works for me.
The last time I installed it - some years ago - it was better to understand - this time, sorry, I dont .....
But made it workig.
According to RabbitMQ Install on Windows guide here
Troubleshooting When Running as a Service
In the event that the Erlang VM crashes whilst RabbitMQ is running as
a service, rather than writing the crash dump to the current directory
(which doesn't make sense for a service) it is written to an
erl_crash.dump file in the base directory of the RabbitMQ server (set
by the RABBITMQ_BASE environment variable, defaulting to
%APPDATA%\%RABBITMQ_SERVICENAME% - typically %APPDATA%\RabbitMQ
otherwise).
Basically it means to add a Environment Variable named RABBITMQ_BASE with value %APPDATA%\RabbitMQ
This fixed my problem.
I ran into this issue and the only way I could solve it was by unintalling RabbitMQ, unsintalling Erlang, rebooting the server and installing a clean Erlang and a clean RabbitMQ.
After all this, I could finally install and start the RabbitMQ instance as a windows service.
Tried all the solutions in this post and nothing worked.
Lucky for me it was in our development server, so the loss was acceptable.
The downside to this approach is that you loose all configs (all users, virtual hosts, etc).
It's all gone and you have to reconfigure the RabbitMQ instance from scratch.
Checking in from 2021:
None of this worked for me, the problem was actually that I had another instance of RabbitMQ running inside my WSL Ubuntu distro.
I had the same issue and I just downloaded the latested version of erlang and RabbitmQ and this resolved the issue for me.
While I got the same error, and the root cause for me seems related to Erlang cookie, I fixed it by doing:
Create a folder to store cookie, for example I am using C:\erl-23.2\home .
Add new system environment variable HOMEDRIVE, set the value to C:\
Add new system environment variable HOMEPATH, set the value to erl-23.2\home
This is making use of the rule:
%HOMEDRIVE%%HOMEPATH%.erlang.cookie (usually C:\Users%USERNAME%.erlang.cookie for user %USERNAME%) if both the HOMEDRIVE and HOMEPATH environment variables are set
Since I was doing a migration when the error popped up, I still had my original .erlang.cookie in C:\Users\Me, but the new installation generated a new .erlang.cookie during installation in C:\Windows\System32\config\systemprofile. After making them equal again and performing these steps from the sbin dir, it worked again.
rabbitmq-service remove
rabbitmq-service install
rabbitmq-service start
I had this today trying to install rabbitmq 3.8.0 with erlang 22.0 (64Bit).
Even completely re-installing both erlang and rabbit, deleteing all directories and registry did not help at all. Also i tried to set the needed PATH variables for erlang manually and re-installing the service each time.
The only solution working for me was installing another version of erlang. In my sepcific case i used erlang 21.3 in the 32bit version.
Doing that, no manually action was necessary and rabbit was up and running (after re-installing the service).

Resources