while deploying docker images on the production server using docker-compose, I got the following error.
Cannot create container for service xxx: open /var/lib/docker/overlay2/969rf5...fdf-init/merged/etc/resolv.conf: Operation not permitted
What I tried:
Changed permissions ( So much so that gave 777 for all the directories involved)
Upgraded kernel (Saw somewhere that it could be kernel issue)
Removed ACL using setfacl -Rb /var/lib/docker
Added the "graph: /var/lib/docker" line in daemon.json
Restarted docker service couple of times.
Tried running the images individually and not using docker-compose.
Tried running with sudo
Set the --storage-opt overlay2.override_kernel_check=1 (Since it is rhel and kernel version supported for docker overlay2 is >4.0, so was suggested to override it.)
I failed everytime!
Have no idea what the issue is or what it could be. Every time I run the docker-compose file, it creates a new directory in /var/lib/docker/overlay2 with old permissions even though I change the permissions for all the other directories in it. At this point I'm not even sure if it is permissions issue or not.
Any help would be appreciated! Thank you!
Finally after trying everything on the internet and nothing working out, the issue was resolved. It was due to the Antivirus that was installed on the server. Had it removed by the client and docker/docker-compose started working absolutely fine.
To give a little more information on the antivirus, it was some FIM antivirus that was installed on a rhel 3.10-xxx which was blocking docker. It was an older version and thank goodness newer versions are compatible with docker. So that's a wrap, thank you all!
I have installed and pulled from docker.io successfully many times on Redhat Enterprise Linux. But now I am having a very difficult time with a slightly different environment behind a proxy server.
I am starting to think there is a modification Redhat has done to the docker packages preventing my proxy setting from working correctly. When I try to do a simple "docker pull centos", I get the frustrating error: "Tag latest not found in repository docker.io/library/centos". I have tried other tags as well with no luck. Before I put the proxy setting in, I received "authorizationrequired" messages. I got past that by adding the proxy info to /etc/sysconfig/docker. Then I received x509 errors. I got past that by adding the appropriate certificate file to docker. Now, with everything seemingly correct, I am unable to do a simple pull. From what I can tell, this is not an issue with stock docker. Is there something else that needs to be done to make the redhat docker package work with a proxy? Any help is greatly appreciated. I tried everything in another stackoverflow question to no avail.
Everything looked like it was working from the cli. However, I ran /var/log/messages | grep docker to look at the docker daemon logs. I was still getting certificate errors. I loaded hub.docker.com into my browser and extracted the certificate chain. I put those in /etc/docker/certs.d/docker.io and it worked part-way.
I needed to add those proxy certs to the global location and now it works all the way.
I am using RabbitMQ. For some reason the rabbitMQ service stops as soon as you start it. I saw following error in the event log:
RabbitMQ: Erlang machine stopped instantly (distribution name conflict?). The service is not restarted as OnFail is set to ignore.
Someone told me to run this command: erl -sname rabbit
This command generates following output:
{(no error logger present")i neirtr otre: r"mEirnraotri nign ipnr odcoe_sbso o<
t0".,2{.b0a>d awrigt,h[ {eexrilt_p rviaml_uleo:a d{ebra,dcahregc,k[_{feirlle__pr
reismu_llto,a3d,e[r{,fcihleec,k"_efrill_e_prreismu_llto,a3d,e[r{.feirlle",}\,"{e
lriln_ep,r29i3m}_]l}o,a{dienri.te,rgle\t"_}b,o{olti,n1e,,[2{9f3i}l]e},,"{iinniit
t.,egrelt"_}b,o{olti,n1e,,[78{9f}i]l}e,,{\i"niinti,tg.eetr_lb\o"o}t,,{2l,i[n{ef,
i7l8e9,}"]i}n,i{ti.neirtl,"g}e,t{_lbionoet,,7762},][}{,f{iilnei,t\,"dion_ibto.oe
tr,l3\,"[}{,f{illien,e",i77n6i}t].}e,r{li"n}i,t{,ldion_eb,o74o3t},]3},][}{}f
ile,\"init.erl\"},{line,743}]}]}\n"
I am not sure how to interpret this output. I wonder the error is specific to RabbitMQ or erlang.
I have no idea how to procceed. Please suggest.
I have just run into this problem setting up RabbitMq as a service up on a new Windows server. The only thing I can think of that broke it for me is renaming the new windows box after installing the RabbitMq service, but before testing it for the first time.
First off I noticed it ran as an application fine. I solved it by installing the service again using the command from the manual install instructions:
rabbitmq-service install
Assuming that you have your path variables included for the RabbitMq sbin directory.
The only thing that worked for me was to clear the directory C:\Users\xxxxx\AppData\Roaming\RabbitMQ.
(cf. https://groups.google.com/forum/#!topic/rabbitmq-users/138RHzzsORU)
In my scenario, Two directories of Erlang under C:\Program Files with different versions were there, I uninstalled one of the version, also uninstalled RabbitMQ service from Windows services list - Restarted the system.
Again ran RabbitMQ setup - RabbitMQ service was setup successfully.
I ran into the same issue when installing RabbitMQ 3.7.17 via Chocolatey on a Windows Server 2016.
After trying most of the suggested solutions, the one that worked for me was:
rabbitmq-service remove
rabbitmq-service install
rabbitmq-service start
PS: if your PATH is not configured for RabbitMQ, this is the folder you need to run the commands from: C:\Program Files\RabbitMQ Server\rabbitmq_server-3.7.17\sbin (if your version is also 3.7.17).
For anyone else looking up this error: double check your config files and SSL files. I ran into this issue when I had specified the ssl_options.cacertfile with ca.pem but the file was mistyped as ca-pem in the directory. Unfortunately RabbitMQ wasn't smart enough to catch the missing file and was dumping with no logs.
I found a name conflict with an env-variable, I use since years - means, this was not a problem with the previous version.
I have "Logs" and apps will write into that directory, usually with their own subdirectories. RMQ uses the same variable name and means a plain filename.
So using: "C:\Users\rabbit\AppData\Roaming\RabbitMQ\log\log" made it working for me - this is in the rabbit's users private environment. So the global settings are now not seen by rabbit. Uff. And it looks like, this is really meant as a filename and after I changed it again to "rabbit#c4711-node.log", it writes like the earlier version. The service starts now for me - but this was really messy and I don't trust it at the moment ;-)
From my perspective, one should run such a service under its own account. If the service is already there, create a local user account - I've used "rabbit" and give it a password. The account I created, got admin right from me - but I currently just dont know, it this is needed. At least it should not - will see this later. If you have account/credentials, go to the service manager and click properties for the service. On the second tab ("log on"), check "this account" and enter username an password. If you have an account for the service you should be able to login with user.
Then you can specify environment variables with user scope.
To do this, logon with the user you created. Go to ControlPanel/System and click "advanced":
In the Environment UI, enter user specific variables
in the top panel:
Note: This was not my rabbit user, because I currently cannot login there. The variables, I entered - not guaranteed, it is correct - are the following:
RABBITMQ_BASE=C:\Users\rabbit\AppData\Roaming\RabbitMQ
RABBITMQ_CONFIG_FILE=C:\Users\rabbit\AppData\Roaming\RabbitMQ\rabbitmq
RABBITMQ_LOGS=C:\Users\rabbit\AppData\Roaming\RabbitMQ\log
RABBITMQ_LOG_BASE=C:\Users\rabbit\AppData\Roaming\RabbitMQ\log
RABBITMQ_NODE_IP_ADDRESS=192.168.26.3
This works for me.
The last time I installed it - some years ago - it was better to understand - this time, sorry, I dont .....
But made it workig.
According to RabbitMQ Install on Windows guide here
Troubleshooting When Running as a Service
In the event that the Erlang VM crashes whilst RabbitMQ is running as
a service, rather than writing the crash dump to the current directory
(which doesn't make sense for a service) it is written to an
erl_crash.dump file in the base directory of the RabbitMQ server (set
by the RABBITMQ_BASE environment variable, defaulting to
%APPDATA%\%RABBITMQ_SERVICENAME% - typically %APPDATA%\RabbitMQ
otherwise).
Basically it means to add a Environment Variable named RABBITMQ_BASE with value %APPDATA%\RabbitMQ
This fixed my problem.
I ran into this issue and the only way I could solve it was by unintalling RabbitMQ, unsintalling Erlang, rebooting the server and installing a clean Erlang and a clean RabbitMQ.
After all this, I could finally install and start the RabbitMQ instance as a windows service.
Tried all the solutions in this post and nothing worked.
Lucky for me it was in our development server, so the loss was acceptable.
The downside to this approach is that you loose all configs (all users, virtual hosts, etc).
It's all gone and you have to reconfigure the RabbitMQ instance from scratch.
Checking in from 2021:
None of this worked for me, the problem was actually that I had another instance of RabbitMQ running inside my WSL Ubuntu distro.
I had the same issue and I just downloaded the latested version of erlang and RabbitmQ and this resolved the issue for me.
While I got the same error, and the root cause for me seems related to Erlang cookie, I fixed it by doing:
Create a folder to store cookie, for example I am using C:\erl-23.2\home .
Add new system environment variable HOMEDRIVE, set the value to C:\
Add new system environment variable HOMEPATH, set the value to erl-23.2\home
This is making use of the rule:
%HOMEDRIVE%%HOMEPATH%.erlang.cookie (usually C:\Users%USERNAME%.erlang.cookie for user %USERNAME%) if both the HOMEDRIVE and HOMEPATH environment variables are set
Since I was doing a migration when the error popped up, I still had my original .erlang.cookie in C:\Users\Me, but the new installation generated a new .erlang.cookie during installation in C:\Windows\System32\config\systemprofile. After making them equal again and performing these steps from the sbin dir, it worked again.
rabbitmq-service remove
rabbitmq-service install
rabbitmq-service start
I had this today trying to install rabbitmq 3.8.0 with erlang 22.0 (64Bit).
Even completely re-installing both erlang and rabbit, deleteing all directories and registry did not help at all. Also i tried to set the needed PATH variables for erlang manually and re-installing the service each time.
The only solution working for me was installing another version of erlang. In my sepcific case i used erlang 21.3 in the 32bit version.
Doing that, no manually action was necessary and rabbit was up and running (after re-installing the service).
I'm having this issue since yesterday and I am not sure what could be the problem. I have a Rails application that I want to run against a remote SDK installed on my VirtualBox (Ubuntu); yesterday I was able to set it up on another VM I have and when I went to Settings in Rubymine and added the SSH credentials, it took some time but finally a lot of gems were downloaded and I was able to run the project directly from Rubymine environment, trying to prepare everything for debug.
I had to change my VirtualBox and now when I connect to it via SSH it starts "Collecting files..." for a while and then "Downloading..." the gems... it looks like it restarts for a couple of times and then at some point, after a while and without any further notice, it stops.
If it helps, my host is Windows 7, my guest is Ubuntu 14.04; my RVM is 1.9.3-p392.
Any ideas? I'm surprised because this worked like a charm yesterday and today is not. Thank you in advance for any help.
What version is your Rubymine? I had issue back with version 6.x with remote rubies freezing the system. Also are you using a Vagrant box? I have seen some of the Vagrant boxes have issues as well. I would log in to the Ubuntu Guest via command line/SSH and see if you still have issues downloading gems.