Docker rabbitmq image fails with [error] Too short cookie string - docker
it seems like I do have a problem with the rabbitmq and rabbitmq-management docker image on my Windows machine running docker-desktop.
When trying to run it, the following log comes up before it shuts down:
21:01:21.726 [error] Failed to write to cookie file '/var/lib/rabbitmq/.erlang.cookie': enospc
21:01:22.355 [error] Too short cookie string
21:01:22.356 [error] Too short cookie string
21:01:23.161 [error] Too short cookie string
21:01:23.162 [error] Too short cookie string
21:01:23.783 [error] Too short cookie string
21:01:23.784 [error] Too short cookie string
21:01:24.405 [error] Too short cookie string
21:01:24.406 [error] Too short cookie string
21:01:25.027 [error] Too short cookie string
21:01:25.028 [error] Too short cookie string
21:01:25.661 [error] Too short cookie string
21:01:25.662 [error] Too short cookie string
21:01:26.281 [error] Too short cookie string
21:01:26.282 [error] Too short cookie string
21:01:26.910 [error] Too short cookie string
21:01:26.911 [error] Too short cookie string
21:01:27.533 [error] Too short cookie string
21:01:27.534 [error] Too short cookie string
21:01:28.161 [error] Too short cookie string
Distribution failed: {{:shutdown, {:failed_to_start_child, :auth, {'Too short cookie string', [{:auth, :init_cookie, 0, [file: 'auth.erl', line: 290]}, {:auth, :init, 1, [file: 'auth.erl', line: 144]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 417]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 385]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"rabbitmqcli-47-rabbit#90cc77cefcb8", :shortnames, 15000], false, :net_sup_dynamic]}, :permanent, 1000, :supervisor, [:erl_distribution]}}
Configuring logger redirection
21:01:29.717 [error]
21:01:29.715 [error] Too short cookie string
21:01:29.715 [error] Supervisor net_sup had child auth started with auth:start_link() at undefined exit with reason "Too short cookie string" in auth:init_cookie/0 line 290 in context start_error
21:01:29.715 [error] CRASH REPORT Process <0.201.0> with 0 neighbours crashed with reason: "Too short cookie string" in auth:init_cookie/0 line 290
21:01:29.719 [error] BOOT FAILED
BOOT FAILED
21:01:29.719 [error] ===========
===========
21:01:29.719 [error] Exception during startup:
Exception during startup:
21:01:29.720 [error]
21:01:29.720 [error] supervisor:children_map/4 line 1171
supervisor:children_map/4 line 1171
supervisor:'-start_children/2-fun-0-'/3 line 355
21:01:29.721 [error] supervisor:'-start_children/2-fun-0-'/3 line 355
21:01:29.721 [error] supervisor:do_start_child/2 line 371
supervisor:do_start_child/2 line 371
21:01:29.721 [error] supervisor:do_start_child_i/3 line 385
supervisor:do_start_child_i/3 line 385
21:01:29.721 [error] rabbit_prelaunch:run_prelaunch_first_phase/0 line 27
rabbit_prelaunch:run_prelaunch_first_phase/0 line 27
21:01:29.721 [error] rabbit_prelaunch:do_run/0 line 111
rabbit_prelaunch:do_run/0 line 111
21:01:29.722 [error] rabbit_prelaunch_dist:setup/1 line 15
rabbit_prelaunch_dist:setup/1 line 15
rabbit_prelaunch_dist:duplicate_node_check/1 line 51
21:01:29.722 [error] rabbit_prelaunch_dist:duplicate_node_check/1 line 51
21:01:29.722 [error] error:{badmatch,
error:{badmatch,
{error,
21:01:29.722 [error] {error,
21:01:29.722 [error] {{shutdown,
{{shutdown,
21:01:29.722 [error] {failed_to_start_child,auth,
{failed_to_start_child,auth,
21:01:29.723 [error] {"Too short cookie string",
{"Too short cookie string",
21:01:29.723 [error] [{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},
[{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},
21:01:29.723 [error] {auth,init,1,[{file,"auth.erl"},{line,144}]},
{auth,init,1,[{file,"auth.erl"},{line,144}]},
21:01:29.723 [error] {gen_server,init_it,2,
{gen_server,init_it,2,
[{file,"gen_server.erl"},{line,417}]},
21:01:29.723 [error] [{file,"gen_server.erl"},{line,417}]},
21:01:29.724 [error] {gen_server,init_it,6,
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,385}]},
21:01:29.724 [error] [{file,"gen_server.erl"},{line,385}]},
21:01:29.724 [error] {proc_lib,init_p_do_apply,3,
{proc_lib,init_p_do_apply,3,
21:01:29.724 [error] [{file,"proc_lib.erl"},{line,226}]}]}}},
[{file,"proc_lib.erl"},{line,226}]}]}}},
21:01:29.724 [error] {child,undefined,net_sup_dynamic,
{child,undefined,net_sup_dynamic,
21:01:29.725 [error] {erl_distribution,start_link,
{erl_distribution,start_link,
21:01:29.725 [error] [[rabbit_prelaunch_510#localhost,shortnames],
[[rabbit_prelaunch_510#localhost,shortnames],
21:01:29.725 [error] false,net_sup_dynamic]},
false,net_sup_dynamic]},
21:01:29.725 [error] permanent,1000,supervisor,
permanent,1000,supervisor,
21:01:29.725 [error] [erl_distribution]}}}}
21:01:29.726 [error]
[erl_distribution]}}}}
21:01:30.726 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Too short cookie string",[{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},{auth,init,1,[{file,"auth.erl"},{line,144}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_510#localhost,shortnames],false,net_sup_dynamic]},...}}}} in context start_error
21:01:30.726 [error] CRASH REPORT Process <0.153.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Too short cookie string",[{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},{auth,init,1,[{file,"auth.erl"},{line,144}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_510#localhost,...],...]},...}}}}}},...} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{\"Too short cookie string\",[{auth,init_cookie,0,[{file,\"auth.erl\"},{line,290}]},{auth,init,1,[{file,\"auth.erl\"},{line,144}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,417}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_510#localhost,shortnames],false,net_sup_dynamic]},permanent,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Too
Crash dump is being written to: erl_crash.dump...
I've been using this image for months now without any problems, but all of sudden it doesn't work anymore.
I also tried running this on my raspberry pi. Turns out it works there, so it has to be more of a local thing for me, which is kind of weird as docker is basically meant to avoid these problems.
I also tried setting the RABBITMQ_ERLANG_COOKIE environment variable to a long name, but with no success. Any ideas?
I had exactly the same error and followed the advice of Owen Brown. Unfortunately, deleting the rabbitmq images could not solve my problems but deleting all other redundant images as well (especially the ones of big size) did it for me.
In case you are wondering how much all images are in size, you can check these statistics with
docker system df
Edit:
As i encountered the same issue again just a couple of hours later i researched again and found this answer which resolved my troubles.
More important than deleting images is the deletion of volumes using:
docker volume rm $(docker volume ls -f dangling=true -q) // removes all volumes
It seems that RabbitMQ has no space left to write the cookie:
21:01:21.726 [error] Failed to write to cookie file '/var/lib/rabbitmq/.erlang.cookie': enospc
Maybe you have some space limitations in the docker, or you don't actually have space left in the host?
Launch rabbitmq in isolation using docker run rabbitmq:management.
If you still get the "Failed to write to cookie file '/var/lib/rabbitmq/.erlang.cookie': enospc" error, then the problem is caused because a file used by RabbitMQ has been corrupted.
Since the container exits as soon as the error occurs, and the container is launching from a cached docker image, you won't be able to enter the container to fix the problem. Instead, you'll need remove the cached images.
Remove the cause of the original issues, which may have been a shortage of memory assigned to Docker. You can delete images or assign more memory through Docker's utility.
Delete both the rabbitmq docker image
docker images | grep rabbit
then
run docker rmi <image_id(s)> for each image id.
Delete all cached images.
List all images using $ docker images --filter "dangling=true will
Then, remove them with $ docker rmi $(docker images -f "dangling=true" -q)
Finally, run rabbitmq again with docker run rabbitmq:management. If it runs without erring, you're back in business.
I tried all this https://stackoverflow.com/a/63461987
And then what did the trick was docker-compose down and then docker-compose up again.
Seems like it can also be an issue with docker desktop as pointed out here:
https://github.com/docker-library/rabbitmq/issues/423
My fellow had the same problem. He says that he always do docker-compose down -v before building a solution, but it appears that volumes didn't clean up.
After docker system prune volumes were still there.
docker volume prune command helped.
Tried #Jose M solution without success. docker prune command helped me.
Related
Rancher pod is crashing
I'm trying to install Rancher in AKS, but unfortunately I'm not able to reach the rancher pod. When I check the pod logs, I see the following error messages: 2023/01/28 20:22:50 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax ... /01/28 20:22:50 [ERROR] failed to cleanup orphan catalog bindings .. failed to sync cache for rke-machine-config.cattle.io/v1, Kind=VmwarevsphereConfig E0128 20:22:51.712910 34 reflector.go:139] pkg/mod/github.com/rancher/client-go#v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *unstructured.Unstructured: Get "https://10.0.0.1:443/apis/rke-machine-config.cattle.io/v1/vmwarevsphereconfigs?allowWatchBookmarks=true&resourceVersion=112947&timeout=9m58s&timeoutSeconds=598&watch=true": context canceled 2023/01/28 20:22:51 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Cluster 2023/01/28 20:22:51 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineSet 2023/01/28 20:22:51 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck .... 2023/01/28 20:23:05 [ERROR] error syncing 'cattle-fleet-system/helm-operation-l496h': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-l496h), requeuing I've followed these instructions: https://ranchermanager.docs.rancher.com/v2.5/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks https://ranchermanager.docs.rancher.com/v2.5/pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster#install-the-rancher-helm-chart Any idea what could be the issue? Many thanks in advance for your help
OOM when building quarkus native application
Not sure what am I missing? After reading multiple blogs and StackOverflow questions, I tried increasing the memory to Xmx12G but with no success. I keep getting msg to increase memory. Has anyone faced the issue in the recent past and is it been solved? This is just the plain application downloaded directly from the quarkus website. docker run --env LANG=C --rm -v qaurkus-test/target/qaurkus-test-1.0.0-SNAPSHOT-native-image-source-jar:/project:z --name build-native-bkbGA quay.io/quarkus/ubi-quarkus-native-image:21.2-java11 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -H:+AllowFoldMethods -jar qaurkus-test-1.0.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -J-Xmx12G -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-UseServiceLoaderFeature -H:+StackTrace -H:-ParseOnce qaurkus-test-1.0.0-SNAPSHOT-runner [qaurkus-test-1.0.0-SNAPSHOT-runner:25] classlist: 6,968.00 ms, 0.94 GB [qaurkus-test-1.0.0-SNAPSHOT-runner:25] (cap): 534.58 ms, 0.94 GB [qaurkus-test-1.0.0-SNAPSHOT-runner:25] setup: 2,329.28 ms, 0.94 GB The bundle named: messages, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath. 19:29:29,666 INFO [org.jbo.threads] JBoss Threads version 3.4.2.Final Error: Image build request failed with exit status 137 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 43.324 s [INFO] Finished at: 2021-09-01T00:59:39+05:30 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal io.quarkus.platform:quarkus-maven-plugin:2.2.1.Final:build (default) on project qaurkus-test: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors [ERROR] [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image [ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:235) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:566) [ERROR] at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:820) [ERROR] at io.quarkus.builder.BuildContext.run(BuildContext.java:277) [ERROR] at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18) [ERROR] at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449) [ERROR] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478) [ERROR] at java.base/java.lang.Thread.run(Thread.java:829) [ERROR] at org.jboss.threads.JBossThread.run(JBossThread.java:501) [ERROR] Caused by: java.lang.RuntimeException: Image generation failed. Exit code was 137 which indicates an out of memory error. Consider increasing the Xmx value for native image generation by setting the "quarkus.native.native-image-xmx" property [ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:372) [ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:205) [ERROR] ... 11 more [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Finally, got it working. check how much memory docker is allocated using docker stats command Follow the process mentioned in the post and increase it to atleast 8G, in my case max memory consumed was around 7G
How to fix springboot error Failed to retrieve RMIServer stub: javax.naming.CommunicationException error during JRMP connection establishment
I am having issue with springboot maven project and cassandra in docker. I want to execute mvn verify and in pre-integration-test to start cassandra docker and api server, execute test and then in post-integration-test to stop server but I am facing error Could not contact Spring Boot application: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; I have created github repo so you can easily reproduce error. Steps to reproduce: git clone https://github.com/gandra/docker-cassandra-with-initial-seed.git cd docker-cassandra-with-initial-seed mvn verify this will produce error like this: [INFO] [INFO] --- spring-boot-maven-plugin:2.2.2.RELEASE:start (api-server-start) # docker-cassandra-with-initial-seed --- [INFO] Attaching agents: [] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 18.501 s [INFO] Finished at: 2021-01-21T18:39:05+01:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.2.2.RELEASE:start (api-server-start) on project docker-cassandra-with-initial-seed: Could not contact Spring Boot application: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: [ERROR] java.io.EOFException] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Any idea how to fix. p.s. I am working on mac and this solution would work on mac and linux as well.
Code https://github.com/gandra/docker-cassandra-with-initial-seed is fixed and now working. Basically 2 thing are changed: Removed port mapping 9001 from docker-cassandra.sh. Added sleep 20 seconds in cassandra_start function in docker-cassandra.sh in order to give time cassandra to properly boot up. Not nice but at least working. Now followiung code should work with 1 integration test executing successfully: git clone https://github.com/gandra/docker-cassandra-with-initial-seed.git cd docker-cassandra-with-initial-seed mvn verify
If you are using Windows, there was an issue solved in 2020 that is a little bit tricky to find (here the complete thread on Github if you want to read the full story). Just to recap, the solution is to fix the jmxPort. For example: <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <jmxPort>9011</jmxPort> </configuration> [...]
The problem here is that he tries to use a port that is already in use. I had a similar problem with spring boot (not with cassandra). Two ways i see to solving this: First way: Find the program that is using that port (in my case it was "Intel(R) Graphics Command Center" on port 9001). (you can use netstat /a /b in command prompt) or use tcpview (external free program) disable the program Profit Second way configuring the jmx port in your pom on the spring-boot-maven-plugin (Answer from sixro) BUT, i don't like this way because you basically force the others in your project to use your workaround, while the issue is on your machien.
LINKERD: Unable to build docker image from linkerd
https://github.com/linkerd/linkerd#docker From the instruction on Readme, I have executed the following commands, ; linkerd/docker ;namerd/docker I get the following exception, [info] Done packaging. [trace] Stack trace suppressed: run last linkerd/bundle:docker for the full output. [error] (linkerd/bundle:docker) java.io.IOException: Cannot run program "docker" (in directory "/home/shaikk/linkerd/linkerd/target/docker"): error=2, No such file or directory [error] Total time: 284 s, completed Mar 6, 2017 9:13:49 AM
I think the No such file or directory error message is referring to the docker binary itself. Can you try running which docker to see if it's in your path? If it's not there, you can install it using the instructions here: https://docs.docker.com/engine/installation/#platform-support-matrix
what does the error mean:"unexpected_delivery_and_no_default_consumer"
I use rabbitmq-erlang-client rabbitmq_2.7.0 in my production env. Recently, I have found some error as "unexpected_delivery_and_no_default_consumer" in my server app log. 2016-08-26 15:25:00.465 [error] CRASH REPORT Process with 0 neighbours exited with reason: {unexpected_delivery_and_no_default_consumer,{gen_server2,call,[,{consumer_call,{'basic.deliver',>,15804,false,>,>},{amqp_msg,{'P_basic',undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},>}},infinity]}} in gen_server:terminate/7 line 826 rabbitmq consumer res: rabbitmqctl list_consumers Why "amq.ctag-2I715Fu1Q2m9AHgrhlN1Og" is not in the consumers tags? PS: 1. I do not use the method 'basic.cancel'. 2. I config the msg as "no_ack = true"