what does the error mean:"unexpected_delivery_and_no_default_consumer" - erlang

I use rabbitmq-erlang-client rabbitmq_2.7.0 in my production env. Recently, I have found some error as "unexpected_delivery_and_no_default_consumer" in my server app log.
2016-08-26 15:25:00.465 [error] CRASH REPORT Process with 0 neighbours exited with reason: {unexpected_delivery_and_no_default_consumer,{gen_server2,call,[,{consumer_call,{'basic.deliver',>,15804,false,>,>},{amqp_msg,{'P_basic',undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined,undefined},>}},infinity]}} in gen_server:terminate/7 line 826
rabbitmq consumer res:
rabbitmqctl list_consumers
Why "amq.ctag-2I715Fu1Q2m9AHgrhlN1Og" is not in the consumers tags?
PS:
1. I do not use the method 'basic.cancel'.
2. I config the msg as "no_ack = true"

Related

Kazoo pusher apns not start with reason {'function not exported',{string,lowercase,1}}

I'm new in Erlang and telefony stuff. I'm trying to wake up an iOS app by a push notification sent by Pusher, a Kazoo module that use apns4erl to do the job.
When Pusher try to send a notification, It should start the apns but it get this error:
10:44:22.756 [error] |0000000000|Undefined:Undefined(<0.29186.843>) gen_statem 'apns_com.test.viavoip' in state connected terminated with reason: {'function not exported',{string,lowercase,1}} in gun:normalize_headers/1 line 661
10:44:22.756 [error] |0000000000|string:Undefined(<0.29186.843>) CRASH REPORT Process 'apns_com.test.viavoip' with 0 neighbours crashed with reason: call to undefined function string:lowercase(<<"apns-topic">>)
10:44:22.756 [error] |0000000000|Undefined:Undefined(<0.26408.843>) Supervisor apns_sup had child apns_connection started with {apns_connection,start_link,undefined} at <0.29186.843> exit with reason undef in context child_terminated
10:44:22.760 [error] |0000000000|Undefined:Undefined(<0.31997.843>) gen_server pm_apple terminated with reason: {undef,{gen_statem,call,[<0.29186.843>,{push_notification,<<"ac79eb0d7f28ee04d3539b80a3b6f519cc3db78e7b3840f18b497fadcedc5264">>,<<"{\"aps\":{\"alert\":{\"loc-args\":[\"1089 - Dario Test3\"],\"loc-key\":\"IC_SIL\"},\"call-id\":\"2d25fee2-74d9-43c7-9bfe-90a167f5941a\",\"sound\":\"ring.caf\"},\"call-id\":\"2d25fee2-74d9-43c7-9bfe-90a167f5941a\",\"caller-id-name\":\"Dario Test3\",\"caller-id-number\":\"1089\",\"proxy\":\"sip:94.138.35.201:5060\",\"registration-token\":\"4ad4ad69-6939-4f04-ada6-e827983a4b23\",\"utc_unix_timestamp_ms\":\"16...">>,...},...]}} in gen:do_call/4 line 177
10:44:22.760 [error] |0000000000|Undefined:Undefined(<0.31997.843>) CRASH REPORT Process pm_apple with 0 neighbours exited with reason: {undef,{gen_statem,call,[<0.29186.843>,{push_notification,<<"ac79eb0d7f28ee04d3539b80a3b6f519cc3db78e7b3840f18b497fadcedc5264">>,<<"{\"aps\":{\"alert\":{\"loc-args\":[\"1089 - Dario Test3\"],\"loc-key\":\"IC_SIL\"},\"call-id\":\"2d25fee2-74d9-43c7-9bfe-90a167f5941a\",\"sound\":\"ring.caf\"},\"call-id\":\"2d25fee2-74d9-43c7-9bfe-90a167f5941a\",\"caller-id-name\":\"Dario Test3\",\"caller-id-number\":\"1089\",\"proxy\":\"sip:94.138.35.201:5060\",\"registration-token\":\"4ad4ad69-6939-4f04-ada6-e827983a4b23\",\"utc_unix_timestamp_ms\":\"16...">>,...},...]}} in gen_server:terminate/7 line 812
10:44:22.761 [error] |0000000000|Undefined:Undefined(<0.30380.843>) Supervisor pusher_module_sup had child pm_apple started with pm_apple:start_link() at <0.31997.843> exit with reason {undef,{gen_statem,call,[<0.29186.843>,{push_notification,<<"ac79eb0d7f28ee04d3539b80a3b6f519cc3db78e7b3840f18b497fadcedc5264">>,<<"{\"aps\":{\"alert\":{\"loc-args\":[\"1089 - Dario Test3\"],\"loc-key\":\"IC_SIL\"},\"call-id\":\"2d25fee2-74d9-43c7-9bfe-90a167f5941a\",\"sound\":\"ring.caf\"},\"call-id\":\"2d25fee2-74d9-43c7-9bfe-90a167f5941a\",\"caller-id-name\":\"Dario Test3\",\"caller-id-number\":\"1089\",\"proxy\":\"sip:94.138.35.201:5060\",\"registration-token\":\"4ad4ad69-6939-4f04-ada6-e827983a4b23\",\"utc_unix_timestamp_ms\":\"16...">>,...},...]}} in context child_terminated
It seems that the apns cannot start because of this: string:lowercase(<<"apns-topic">>).
In erl console this command works fine, maybe there is few dependences with a wrong version or some wrong comfiguration.
Any advise?

Docker rabbitmq image fails with [error] Too short cookie string

it seems like I do have a problem with the rabbitmq and rabbitmq-management docker image on my Windows machine running docker-desktop.
When trying to run it, the following log comes up before it shuts down:
21:01:21.726 [error] Failed to write to cookie file '/var/lib/rabbitmq/.erlang.cookie': enospc
21:01:22.355 [error] Too short cookie string
21:01:22.356 [error] Too short cookie string
21:01:23.161 [error] Too short cookie string
21:01:23.162 [error] Too short cookie string
21:01:23.783 [error] Too short cookie string
21:01:23.784 [error] Too short cookie string
21:01:24.405 [error] Too short cookie string
21:01:24.406 [error] Too short cookie string
21:01:25.027 [error] Too short cookie string
21:01:25.028 [error] Too short cookie string
21:01:25.661 [error] Too short cookie string
21:01:25.662 [error] Too short cookie string
21:01:26.281 [error] Too short cookie string
21:01:26.282 [error] Too short cookie string
21:01:26.910 [error] Too short cookie string
21:01:26.911 [error] Too short cookie string
21:01:27.533 [error] Too short cookie string
21:01:27.534 [error] Too short cookie string
21:01:28.161 [error] Too short cookie string
Distribution failed: {{:shutdown, {:failed_to_start_child, :auth, {'Too short cookie string', [{:auth, :init_cookie, 0, [file: 'auth.erl', line: 290]}, {:auth, :init, 1, [file: 'auth.erl', line: 144]}, {:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 417]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 385]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"rabbitmqcli-47-rabbit#90cc77cefcb8", :shortnames, 15000], false, :net_sup_dynamic]}, :permanent, 1000, :supervisor, [:erl_distribution]}}
Configuring logger redirection
21:01:29.717 [error]
21:01:29.715 [error] Too short cookie string
21:01:29.715 [error] Supervisor net_sup had child auth started with auth:start_link() at undefined exit with reason "Too short cookie string" in auth:init_cookie/0 line 290 in context start_error
21:01:29.715 [error] CRASH REPORT Process <0.201.0> with 0 neighbours crashed with reason: "Too short cookie string" in auth:init_cookie/0 line 290
21:01:29.719 [error] BOOT FAILED
BOOT FAILED
21:01:29.719 [error] ===========
===========
21:01:29.719 [error] Exception during startup:
Exception during startup:
21:01:29.720 [error]
21:01:29.720 [error] supervisor:children_map/4 line 1171
supervisor:children_map/4 line 1171
supervisor:'-start_children/2-fun-0-'/3 line 355
21:01:29.721 [error] supervisor:'-start_children/2-fun-0-'/3 line 355
21:01:29.721 [error] supervisor:do_start_child/2 line 371
supervisor:do_start_child/2 line 371
21:01:29.721 [error] supervisor:do_start_child_i/3 line 385
supervisor:do_start_child_i/3 line 385
21:01:29.721 [error] rabbit_prelaunch:run_prelaunch_first_phase/0 line 27
rabbit_prelaunch:run_prelaunch_first_phase/0 line 27
21:01:29.721 [error] rabbit_prelaunch:do_run/0 line 111
rabbit_prelaunch:do_run/0 line 111
21:01:29.722 [error] rabbit_prelaunch_dist:setup/1 line 15
rabbit_prelaunch_dist:setup/1 line 15
rabbit_prelaunch_dist:duplicate_node_check/1 line 51
21:01:29.722 [error] rabbit_prelaunch_dist:duplicate_node_check/1 line 51
21:01:29.722 [error] error:{badmatch,
error:{badmatch,
{error,
21:01:29.722 [error] {error,
21:01:29.722 [error] {{shutdown,
{{shutdown,
21:01:29.722 [error] {failed_to_start_child,auth,
{failed_to_start_child,auth,
21:01:29.723 [error] {"Too short cookie string",
{"Too short cookie string",
21:01:29.723 [error] [{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},
[{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},
21:01:29.723 [error] {auth,init,1,[{file,"auth.erl"},{line,144}]},
{auth,init,1,[{file,"auth.erl"},{line,144}]},
21:01:29.723 [error] {gen_server,init_it,2,
{gen_server,init_it,2,
[{file,"gen_server.erl"},{line,417}]},
21:01:29.723 [error] [{file,"gen_server.erl"},{line,417}]},
21:01:29.724 [error] {gen_server,init_it,6,
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,385}]},
21:01:29.724 [error] [{file,"gen_server.erl"},{line,385}]},
21:01:29.724 [error] {proc_lib,init_p_do_apply,3,
{proc_lib,init_p_do_apply,3,
21:01:29.724 [error] [{file,"proc_lib.erl"},{line,226}]}]}}},
[{file,"proc_lib.erl"},{line,226}]}]}}},
21:01:29.724 [error] {child,undefined,net_sup_dynamic,
{child,undefined,net_sup_dynamic,
21:01:29.725 [error] {erl_distribution,start_link,
{erl_distribution,start_link,
21:01:29.725 [error] [[rabbit_prelaunch_510#localhost,shortnames],
[[rabbit_prelaunch_510#localhost,shortnames],
21:01:29.725 [error] false,net_sup_dynamic]},
false,net_sup_dynamic]},
21:01:29.725 [error] permanent,1000,supervisor,
permanent,1000,supervisor,
21:01:29.725 [error] [erl_distribution]}}}}
21:01:29.726 [error]
[erl_distribution]}}}}
21:01:30.726 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Too short cookie string",[{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},{auth,init,1,[{file,"auth.erl"},{line,144}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_510#localhost,shortnames],false,net_sup_dynamic]},...}}}} in context start_error
21:01:30.726 [error] CRASH REPORT Process <0.153.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Too short cookie string",[{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},{auth,init,1,[{file,"auth.erl"},{line,144}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_510#localhost,...],...]},...}}}}}},...} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{\"Too short cookie string\",[{auth,init_cookie,0,[{file,\"auth.erl\"},{line,290}]},{auth,init,1,[{file,\"auth.erl\"},{line,144}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,417}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[rabbit_prelaunch_510#localhost,shortnames],false,net_sup_dynamic]},permanent,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,auth,{"Too
Crash dump is being written to: erl_crash.dump...
I've been using this image for months now without any problems, but all of sudden it doesn't work anymore.
I also tried running this on my raspberry pi. Turns out it works there, so it has to be more of a local thing for me, which is kind of weird as docker is basically meant to avoid these problems.
I also tried setting the RABBITMQ_ERLANG_COOKIE environment variable to a long name, but with no success. Any ideas?
I had exactly the same error and followed the advice of Owen Brown. Unfortunately, deleting the rabbitmq images could not solve my problems but deleting all other redundant images as well (especially the ones of big size) did it for me.
In case you are wondering how much all images are in size, you can check these statistics with
docker system df
Edit:
As i encountered the same issue again just a couple of hours later i researched again and found this answer which resolved my troubles.
More important than deleting images is the deletion of volumes using:
docker volume rm $(docker volume ls -f dangling=true -q) // removes all volumes
It seems that RabbitMQ has no space left to write the cookie:
21:01:21.726 [error] Failed to write to cookie file '/var/lib/rabbitmq/.erlang.cookie': enospc
Maybe you have some space limitations in the docker, or you don't actually have space left in the host?
Launch rabbitmq in isolation using docker run rabbitmq:management.
If you still get the "Failed to write to cookie file '/var/lib/rabbitmq/.erlang.cookie': enospc" error, then the problem is caused because a file used by RabbitMQ has been corrupted.
Since the container exits as soon as the error occurs, and the container is launching from a cached docker image, you won't be able to enter the container to fix the problem. Instead, you'll need remove the cached images.
Remove the cause of the original issues, which may have been a shortage of memory assigned to Docker. You can delete images or assign more memory through Docker's utility.
Delete both the rabbitmq docker image
docker images | grep rabbit
then
run docker rmi <image_id(s)> for each image id.
Delete all cached images.
List all images using $ docker images --filter "dangling=true will
Then, remove them with $ docker rmi $(docker images -f "dangling=true" -q)
Finally, run rabbitmq again with docker run rabbitmq:management. If it runs without erring, you're back in business.
I tried all this https://stackoverflow.com/a/63461987
And then what did the trick was docker-compose down and then docker-compose up again.
Seems like it can also be an issue with docker desktop as pointed out here:
https://github.com/docker-library/rabbitmq/issues/423
My fellow had the same problem. He says that he always do docker-compose down -v before building a solution, but it appears that volumes didn't clean up.
After docker system prune volumes were still there.
docker volume prune command helped.
Tried #Jose M solution without success. docker prune command helped me.

Elasticbeanstalk Deployment Fails

I am trying to deploy a docker based environment that is using 1-1 application load balancer in AWS elasticbeanstalk. I am getting the below error message. It seemed to deploy very well before but for unknown reasons the new deploys fail.
2019-01-31 18:52:23 INFO Environment update is starting.
2019-01-31 18:53:03 INFO Deploying new version to instance(s).
2019-01-31 18:53:12 INFO Successfully pulled node:10.15.0
2019-01-31 18:57:34 INFO Successfully built aws_beanstalk/staging-app
2019-01-31 18:57:42 ERROR Failed to start nginx, abort deployment
2019-01-31 18:57:44 ERROR [Instance: i-0d4b3811324b90c27] Command failed on instance. Return code: 1 Output: nginx: [emerg] no host in upstream ":8080" in /etc/nginx/conf.d/elasticbeanstalk-nginx-docker-upstream.conf:2
nginx: configuration file /etc/nginx/nginx.conf test failed
Failed to start nginx, abort deployment.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
2019-01-31 18:57:45 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
2019-01-31 18:57:45 ERROR Unsuccessful command execution on instance id(s) 'i-0d4b3811324b90c27'. Aborting the operation.
2019-01-31 18:57:45 ERROR Failed to deploy application.
This issue was caused of an Uncaught error in my Node js application. Issue is not with Beanstalk.

Docker container crashes after execution of python script

After deploying a Python Docker container and successfully executing a script the container crashes and restarts in a loop after showing the following error message:
2017-06-19 13:22:49 [APP/PROC/WEB/0] OUT Exit status 0
2017-06-19 13:22:49 [CELL/0] OUT Exit status 0
2017-06-19 13:22:49 [CELL/0] OUT Destroying container
2017-06-19 13:22:49 [API/0] OUT Process has crashed with type: "web"
2017-06-19 13:22:49 [API/0] OUT App instance exited with guid 85e7922e-5a0c-4430-994a-324e5abc0c14 payload: {"instance"=>"", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"2 error(s) occurred:\n\n* Codependent step exited\n* cancelled", "crash_count"=>1, "crash_timestamp"=>1497871369566402154, "version"=>"b9800e3a-b057-4cc5-b7e4-c01f9b3c6594"}
Executing the same docker image locally it does not throw any errors. The Python script I execute is doing a simple print command and I even implemented a handler for the SIGTERM signal that is sent into the container after execution.
In CF, applications are not supposed to finish. But if your script only just prints something, it'll perform an exit 0 afterwards. Thus the app container is stopped and CF registers a "crash", and will then restart the application in accordance with the app lifecycle:
https://docs.cloudfoundry.org/devguide/deploy-apps/app-lifecycle.html

Connecting to RabbitMQ server from MongooseIM

I am trying to communicate with a rabbitmq server from my mongooseim server. For that first I copied rabbitmq-client library (for erlang) to my apps directory, updated my makefile and then recompiled mongooseim code. (as per the instruction from here). This way I can compile the code with no error and can see amqp_client has been started from mongooseim console. Now when I try to run amqp_connection:start() from the console I get the below error. How to fix this?
(mongooseim#localhost)8> {ok, Connection} = amqp_connection:start(#amqp_params_network{}).
2015-04-23 15:07:27.320 [debug] <0.790.0> Supervisor {<0.790.0>,amqp_connection_sup} started amqp_connection_type_sup:start_link() at pid <0.791.0>
2015-04-23 15:07:27.324 [debug] <0.790.0> Supervisor {<0.790.0>,amqp_connection_sup} started amqp_gen_connection:start_link(<0.791.0>, {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5672,0,0,0,infinity,none,[#Fun<amq..>,...],...}) at pid <0.792.0>
** exception exit: {{function_clause,[{amqp_gen_connection,terminate,
[{undef,[{rabbit_net,connection_string,
[#Port<0.6063>,outbound],
[]},
{amqp_network_connection,try_handshake,3,[]},
{amqp_gen_connection,handle_call,3,[]},
{gen_server,try_handle_call,4,
[{file,"gen_server.erl"},{line,607}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,639}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,237}]}]},
{<0.791.0>,
#amqp_params_network{username = <<"guest">>,
password = <<"guest">>,virtual_host = <<"/">>,
host = "localhost",port = 5672,channel_max = 0,
frame_max = 0,heartbeat = 0,connection_timeout = infinity,
ssl_options = none,
auth_mechanisms = [#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
client_properties = [],socket_options = []}}],
[]},
{gen_server,try_terminate,3,
[{file,"gen_server.erl"},{line,621}]},
{gen_server,terminate,7,
[{file,"gen_server.erl"},{line,787}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,237}]}]},
{gen_server,call,[<0.792.0>,connect,infinity]}}
in function gen_server:call/3 (gen_server.erl, line 190)
(mongooseim#localhost)9> 2015-04-23 15:07:27.411 [error] <0.792.0> gen_server <0.792.0> terminated with reason: no function clause matching amqp_gen_connection:terminate({undef,[{rabbit_net,connection_string,[#Port<0.6063>,outbound],[]},{amqp_network_connection,try_handshake,...},...]}, {<0.791.0>,{amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5672,0,0,0,infinity,...}})
2015-04-23 15:07:27.412 [error] <0.792.0> CRASH REPORT Process <0.792.0> with 0 neighbours exited with reason: no function clause matching amqp_gen_connection:terminate({undef,[{rabbit_net,connection_string,[#Port<0.6063>,outbound],[]},{amqp_network_connection,try_handshake,...},...]}, {<0.791.0>,{amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5672,0,0,0,infinity,...}}) in gen_server:terminate/7 line 792
2015-04-23 15:07:27.414 [error] <0.790.0> Supervisor {<0.790.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.791.0>, {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5672,0,0,0,infinity,none,[#Fun<amq..>,...],...}) at <0.792.0> exit with reason no function clause matching amqp_gen_connection:terminate({undef,[{rabbit_net,connection_string,[#Port<0.6063>,outbound],[]},{amqp_network_connection,try_handshake,...},...]}, {<0.791.0>,{amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5672,0,0,0,infinity,...}}) in context child_terminated
2015-04-23 15:07:27.416 [error] <0.790.0> Supervisor {<0.790.0>,amqp_connection_sup} had child connection started with amqp_gen_connection:start_link(<0.791.0>, {amqp_params_network,<<"guest">>,<<"guest">>,<<"/">>,"localhost",5672,0,0,0,infinity,none,[#Fun<amq..>,...],...}) at <0.792.0> exit with reason reached_max_restart_intensity in context shutdown
The undef error message tells us that the rabbit_net module or the connection_string function is missing and rabbit_net is a part of rabbit_common package. I guess you didn't add rabbit_common. You can find the package on the website you pasted in the previous question - here. Repeat the steps with rabbit_common.ez and it should work.

Resources