Batch delete SQS messages after batch receiving in boto3 - amazon-sqs

Question

I am using boto3 to receive_message() in batches. This returns a dict item as specified in the boto3 documentation.
After receiving the batch of message, I process them one by one and want to batch delete them after they are all processed. I do not want to delete them one by one after processing each single item.
The delete_message_batch() function has a slightly different syntax than the response from the receive_message() function, so I need to reformat the response dict from receive_message() into a suitable format (only using Id and ReceiptHandle).
Is there any easier way to directly use the response from receive_message() for batch deletion?

Answer 1

Here is how I handle the messages. I create a list with the ids and the receipt handles. The ids are not mandatory, you could insert the output of range(10). Also, there is no error checking so you need to take care of that.
response = SQS.receive_message(
QueueUrl=queue_url,
AttributeNames=[],
MaxNumberOfMessages=10,
MessageAttributeNames=['All'],
)
delete_batch = []
for message in response.get('Messages') or []:
# do stuff with each message
delete_batch.append(
{'Id': message['MessageId'], 'ReceiptHandle': message['ReceiptHandle']}
)
if delete_batch:
SQS.delete_message_batch(QueueUrl=queue_url, Entries=delete_batch)

Answer 2

In Boto3 you can use the Queue resource to delete batches of messages in one call. Here's some code (from a larger example in GitHub) that shows how:
import logging
import boto3
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
sqs = boto3.resource('sqs')
def delete_messages(queue, messages):
"""
Delete a batch of messages from a queue in a single request.
:param queue: The queue from which to delete the messages.
:param messages: The list of messages to delete.
:return: The response from SQS that contains the list of successful and failed
deletions.
"""
try:
entries = [{
'Id': str(ind),
'ReceiptHandle': msg.receipt_handle
} for ind, msg in enumerate(messages)]
response = queue.delete_messages(Entries=entries)
if 'Successful' in response:
for msg_meta in response['Successful']:
logger.info("Deleted %s", messages[int(msg_meta['Id'])].receipt_handle)
if 'Failed' in response:
for msg_meta in response['Failed']:
logger.warning(
"Could not delete %s",
messages[int(msg_meta['Id'])].receipt_handle
)
except ClientError:
logger.exception("Couldn't delete messages from queue %s", queue)
else:
return response
queue = sqs.get_queue_by_name(QueueName=name)
messages = queue.receive_messages(
MessageAttributeNames=['All'],
MaxNumberOfMessages=max_number,
WaitTimeSeconds=wait_time
)
delete_messages(queue, messages)

Answer 3

There is also an option to purge the queue. Not sure if it will fit your use case but here is the reference:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Queue.purge
and the code example:
import boto3
REGION_NAME = 'us-east-01'
QUEUE_NAME = 'my-queue01-example'
sqs = boto3.resource('sqs', REGION_NAME)
queue = sqs.get_queue_by_name(QueueName=QUEUE_NAME)
response = queue.purge()