We have more than 100 SQS queues and its dynamic as well. Hence, creating alert for each metric name would be challenging. Is there any different solution to monitor SQS queues?
I configured cloudwatch event rule, to direct all SQS events to cloudwatch log group. But no logs are recorded in log group. Can someone help me out with a solution to monitor these queues.
One possible solution is to read the SQS information and then use it as a variable to create cloudwatch alarms via terraform.
see:
AWS CLI SQS List Queues
Terraform alarm resource
Terraform apply variables
Related
I wanted to check if there is scenario where there are 30-40 jobs running concurrently in cloud dataflow. Is there a setting by which the workers used on 1 job can be shared across other or use managed instance group as compute option.
The reason for asking is if the risk of running out of compute instances or exceeding the quota can be managed.
Cloud Dataflow manages the GCE instances internally. This means that it is unable to share the instances with other jobs. Please see here for more information.
I am using a SNS to send out notifications to SQS in another AWS account. There is a DLQ configured to this SQS listener for processing failure of messages in SQS. Is it feasible to use same DLQ for handling SNS subscription failures as well?
While you certainly can use the same SQS as a DLQ for both the SNS topic and the SQS subscription, doing so might not be advisable. You'd be writing two different kinds of messages to the same queue, which might need different types of processing. The DLQ queue would also need a wider set of permissions to accommodate multiple writers. I would recommend using separate DLQs for the two different use cases.
I have a use case where I publish to a topic and listen via Queues in solace. Due to the increased number of Queues , we have decided to create temporary queues. When I tried with a temp Queue, I was able to publish and subscribe directly. But I'm unable to attach a topic endpoint to the queue. Is it possible t attach a topic endpoint to a temp Queue in Solace, if so how to do it?
Yes, it is possible. Which API are you using? Assuming it is the Solace Java API, then to add a topic subscription to a queue you can use the JCSMPSession.addSubscription(...) method. For other Solace APIs see the documentation here: https://docs.solace.com/Solace-PubSub-Messaging-APIs/API-Developer-Guide/Adding-Topic-Subscriptio.htm
The process is same whether the queue is temporary or durable.
Queue queue = JCSMPFactory.onlyInstance().createQueue("Q/tutorial/topicToQueueMapping");
Topic tutorialTopic = JCSMPFactory.onlyInstance().createTopic("T/mapped/topic/sample");
session.addSubscription(queue, tutorialTopic, JCSMPSession.WAIT_FOR_CONFIRM);
In the add subscription method above pass in your temporary queue object reference.
We are using AWS ECS service where docker containers are running into it. These docker container having application code which continuously polling SQS and gets the single message, process it and kill their self and that's the life cycle of task.
Now we are scaling tasks and EC2 in cluster based on number of messages comes to SQS. We are able to scale up but it's difficult to scale down because we don't know whether any task is still processing any message because message processing time is huge due to some complex logic.
Could anybody suggest what's the based way to scale up and scale down in this case?
Have you considered using AWS Lambda for this use case rather than ECS (provided that your application logic runs in less than 5 mins). You can use SQS as a trigger for the Lambda. AWS Documentation : Using AWS Lambda with Amazon SQS provides a comprehensive guide on how to achieve this using Lambda.
The use case you have mentioned doesn't mean for ECS for EC2 instances. You should consider AWS ECS Fargate or AWS BATCH. On one side fargate will give you more capabilities in term of infrastructure like The task can be run for longer periods or scaling of tasks according to some parameters like CPU or MEM. On another side, you will be paying only the number of tasks running at a moment in your cluster.
Ref: https://aws.amazon.com/fargate/
Is it possible to add custom attributes to queues? Preferably via an API but the console if not. I don't mean messages, I mean the actual queue.
Our ops people require this to track usage and do internal billing.
Thanks
No, there is no way to store "custom attributes" about an SQS queue using the SQS API.
However, this type of data could be stored using a DynamoDB table.
As for tracking usage for billing purposes, CloudWatch provides numerous metrics related to each individual queue that you have.