My use case is that I have a queue from which my application reads messages. I need to delete the message if it is processed correctly, else I want it to stay in the queue. If my application processes the same message again, does that count as a "retry" based on which maximum receives works, or does maximum receives work as a different metric? if i cannot use it as a retry, can you help me with how I could go around implementing it?
Really sorry about the crude ideas I have compiled, but I am a newbie in terms of AWS, and wanted this done a bit quicker than expected.
Thanks for any help.
Maybe can you take a look on dead letter queues. When a message generate an unexpected error it can be retried according to maxReceiveCount, if there is still an error after all allowed retries then message will be moved in a dead letter queue according to redrive policy :
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
Hope this helps.
Related
We are working on a billing invoice system. As a part of processing our request, we need to make an asynchronous call by placing a message in a queue. We work at 20TPS and have SLA for entire transaction of 12 sec. Occasionally, we have observed that when MQ server becomes very slow but still operational it's taking a lot of time just to write the message in the queue. We want to handle this scenario and have a system that throws an exception when it exceeds a predefined limit for writing the message in the queue.
In simple words, we want to implement a write timeout when there is a delay in writing a message in the queue. Any help is appreciated.
We are aware of mentioning timeout for receiving the response but we are unable to find any fix for mentioning timeout while writing the message in the queue.
We have found some suggestions on revalidating the destination. But in our case, we already know the destination is operational and our system becomes slow only during the response.
Let's say my Erlang application receives an important message from the outside (through an exposed API endpoint, for example). Due to a bug in the application or an incorrectly formatted message the process handling the message crashes.
What happens to the message? How can I influence what happens to the message? And what happens to the other messages waiting in the process mailbox? Do I have to introduce a hierarchy of processes just to make sure that no messages are lost?
Is there something like Akka's dead letter queue in Erlang? Let's say I want to handle the message later - either by fixing the message or fixing the bug in the application itself, and then rerunning the message processing.
I am surprised how little information about this topic is available.
There is no information because there is no dead letter queue, if you application crashed while processing your message the message would be already received, why would it go on a dead letter queue (if one existed).
Such a queue would be a major scalability issue with not much use (you would get arbitrary messages which couldn't be sent and would be totally out of context)
If you need to make sure a message is processed you usually use a way to get a reply back when the message is processed like a gen_server call.
And if your messages are such important that it would be a catastrophe if lost you should probably persist it in a external DB, because otherwise if your computer crashes what would happen to all the messages in transit?
Is there any functionality built in to spring-amqp that would make it easy to requeue dead letter messages? I can write code to do it but it seems like such a common use case that would fit well into the framework.
This is outside of Spring AMQP, but you can configure a TTL on the dead letter queue and configure that queue to dead-letter back to the original queue when expired.
You can check the x-death header if you want to give up completely after some number of retry cycles.
See this answer and its question for more information.
I am running into out of memory issues after a certain amount of time when I run my quickfixj app. After a little investigation I found out that this was being caused by messages that quickfixj caches for re sending when a resend request is received.
So for testing I set this flag to N on a particular session. After that my memory problems completely disappeared. But I do not understand why quickfixj is keeping these message in memory when I have properly set this property : FileStorePath. These messages should be stored into a file but they are not. I do see some files present in the directory I set in FileStorePath but none of them seems to be storing messages, I can only see sequence number in them. Do I need to set other flags besides this in order to make this work?
I do not plan to use PersisMessages flag outside testing. I would prefer FileStoreMaxCachedMsgs flag with a reasonable figure. I also need to know what will happen if my app receives a resend request when I have set PersisMessages to N? Will quickfixj send gapfills instead or will it crash with some exception message?
Thanks
i ve found that quickfixj sends gap fills when it could not find messages. also the config flag FileStoreMaxCachedMsgs is used to tell quickfixj about how much messages it should keep in cache before pushing them down to files. so this flag, in my opinion, should be altered in order to get your app to work without running out of memory due to message caching.
hope it ll be helpful for somebody. Thanks
We are getting close to releasing our new rails app and so far interest seems very strong we are a little worried about where bottle necks will be. One seems to be system e-mails on signup and in other situations. Is this correct?
Should individual e-mails to users be sent asynchronously in the background? If so, what would be the best solution?
I have looked at a few solutions and can't seem to find anything definitive.
In the backgroud using http://github.com/tobi/delayed_job
I would say it depends on your requirements. If you need to be able to inform the user that sending mail failed, do it in the same thread.
If not, sending mail should support things like retries etc, so I would put the message into a queue/the filesystem/database table/etc and have another thread/process deal with the details of sending.
Same thread, if you ask me.... by generating a file in a drop folder, which an email server picks up. Then there is not too much overhead, so a seaprate threads makes little sense.
At least this is how I always handle this.