Best way to delete messages from SQS during development - amazon-sqs

During development, I'm generating a lot of bogus messages on my Amazon SQS. I was about to write a tiny app to delete all the messages (something I do frequently during development). Does anyone know of a tool to purge the queue?

If you don't want to write script or delete your queue. You can change the queue configuration:
Right click on queue > configure queue
Change Message Retention period to 1 minute (the minimum time it can be set to).
Wait a while for all the messages to disappear.
I found that this way works well for deleting all messages in a queue without deleting the queue.

As of December 2014, the sqs console now has a purge queue option in the queue actions menu.

For anyone who has come here, looking for a way to delete SQS messages en masse in C#...
//C# Console app which deletes all messages from a specified queue
//AWS .NET library required.
using System;
using System.Net;
using System.Configuration;
using System.Collections.Specialized;
using System.IO;
using System.Linq;
using System.Text;
using Amazon;
using Amazon.SQS;
using Amazon.SQS.Model;
using System.Timers;
using System.Collections.Generic;
using System.Text.RegularExpressions;
using System.Diagnostics;
namespace QueueDeleter
{
class Program
{
public static System.Timers.Timer myTimer;
static NameValueCollection appConfig = ConfigurationManager.AppSettings;
static string accessKeyID = appConfig["AWSAccessKey"];
static string secretAccessKeyID = appConfig["AWSSecretKey"];
static private AmazonSQS sqs;
static string myQueueUrl = "https://queue.amazonaws.com/1640634564530223/myQueueUrl";
public static String messageReceiptHandle;
public static void Main(string[] args)
{
sqs = AWSClientFactory.CreateAmazonSQSClient(accessKeyID, secretAccessKeyID);
myTimer = new System.Timers.Timer();
myTimer.Interval = 10;
myTimer.Elapsed += new ElapsedEventHandler(checkQueue);
myTimer.AutoReset = true;
myTimer.Start();
Console.Read();
}
static void checkQueue(object source, ElapsedEventArgs e)
{
myTimer.Stop();
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest();
receiveMessageRequest.QueueUrl = myQueueUrl;
ReceiveMessageResponse receiveMessageResponse = sqs.ReceiveMessage(receiveMessageRequest);
if (receiveMessageResponse.IsSetReceiveMessageResult())
{
ReceiveMessageResult receiveMessageResult = receiveMessageResponse.ReceiveMessageResult;
if (receiveMessageResult.Message.Count < 1)
{
Console.WriteLine("Can't find any visible messages.");
myTimer.Start();
return;
}
foreach (Message message in receiveMessageResult.Message)
{
Console.WriteLine("Printing received message.\n");
messageReceiptHandle = message.ReceiptHandle;
Console.WriteLine("Message Body:");
if (message.IsSetBody())
{
Console.WriteLine(" Body: {0}", message.Body);
}
sqs.DeleteMessage(new DeleteMessageRequest().WithQueueUrl(myQueueUrl).WithReceiptHandle(messageReceiptHandle));
}
}
else
{
Console.WriteLine("No new messages.");
}
myTimer.Start();
}
}
}

Check the first item in queue. Scroll down to last item in queue.
Hold shift, click on item. All will be selected.

I think the best way would be to delete the queue and create it again, just 2 requests.

I think best way is changing Retention period to 1 minute, but here is Python code if someone needs:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import boto.sqs
from boto.sqs.message import Message
import time
import os
startTime = program_start_time = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())
### Lets connect to SQS:
qcon = boto.sqs.connect_to_region(region,aws_access_key_id='xxx',aws_secret_access_key='xxx')
SHQueue = qcon.get_queue('SQS')
m = Message()
### Read file and write to SQS
counter = 0
while counter < 1000: ## For deleting 1000*10 items, change to True if you want delete all
links = SHQueue.get_messages(10)
for link in links:
m = link
SHQueue.delete_message(m)
counter += 1
#### The End
print "\n\nTerminating...\n"
print "Start: ", program_start_time
print "End time: ", time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime())

Option 1: boto sqs has a purge_queue method for python:
purge_queue(queue)
Purge all messages in an SQS Queue.
Parameters: queue (A Queue object) – The SQS queue to be purged
Return type: bool
Returns: True if the command succeeded, False otherwise
Source: http://boto.readthedocs.org/en/latest/ref/sqs.html
Code that works for me:
conn = boto.sqs.connect_to_region('us-east-1',
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
)
q = conn.create_queue("blah")
#add some messages here
#invoke the purge_queue method of the conn, and pass in the
#queue to purge.
conn.purge_queue(self.queue)
For me, it deleted the queue. However, Amazon SQS only lets you run this once every 60 seconds. So I had to use the secondary solution below:
Option 2: Do a purge by consuming all messages in a while loop and throwing them out:
all_messages = []
rs = self.queue.get_messages(10)
while len(rs) > 0:
all_messages.extend(rs)
rs = self.queue.get_messages(10)

If you have access to the AWS console, you can purge a queue using the Web UI.
Steps:
Navigate to Services -> SQS
Filter queues by your "QUEUE_NAME"
Right-click on your queue name -> Purge queue
This will request for the queue to be cleared and this should be completed with 5 or 10 seconds or so.
See below for how to perform this operation:

To purge an SQS from the API see:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_PurgeQueue.html

Related

Unable to get Twilio sms status callbacks when sending proactive message

I'm trying to track the sms delivery status of the messages I send using the bot framework. I'm using Twilio, and sending proactive messages. Right now I'm trying to do so with twilio status callbacks
This is similar to this question, I tried that approach but I couldn't get it to work. I've added my url on the TwiML app and it is not firing. I have double and triple checked, and I suspect this url is somehow ignored or not going through with my current set up. I don't get any callbacks on the proactive message nor on the replies the bot sends to the user. However the flow works fine and I can reply and get proper responses from the bot. Edit: calling this "approach 1"
approach 2: I've also tried this doing some light modifications on Twilio adapter, to be able to add my callback just before create message. (I changed it so it uses a customized client wrapper that adds my callback url when creating the twilio resource) This does work, partially: when I reply a message from my bot, I get the status callbacks. But as the proactive message is sent using the default adapter, I don't get a callback on the initial message.
approach 3: Finally, I also tried using the TwilioAdapter when sending the proactive message but for some reason as soon as I send an activity, the TurnContext is disposed, so I can't save the state or do any subsequent actions. This leads me to believe twilio adapter is not intended to be used this way (can't be used on proactive messages), but I'm willing to explore this path if necessary.
Here is the modified Twilio Adapter:
public class TwilioAdapterWithErrorHandler : TwilioAdapter
{
private const string TwilioNumberKey = "TwilioNumber";
private const string TwilioAccountSidKey = "TwilioAccountSid";
private const string TwilioAuthTokenKey = "TwilioAuthToken";
private const string TwilioValidationUrlKey = "TwilioValidationUrl";
public TwilioAdapterWithErrorHandler(IConfiguration configuration, ILogger<TwilioAdapter> logger, TwilioAdapterOptions adapterOptions = null)
: base(
new TwilioClientWrapperWithCallback(new TwilioClientWrapperOptions(configuration[TwilioNumberKey], configuration[TwilioAccountSidKey], configuration[TwilioAuthTokenKey], new Uri(configuration[TwilioValidationUrlKey]))), adapterOptions, logger)
{
OnTurnError = async (turnContext, exception) =>
{
// Log any leaked exception from the application.
logger.LogError(exception, $"[OnTurnError] unhandled error : {exception.Message}");
Task[] tasks = {
// Send a message to the user
turnContext.SendActivityAsync("We're sorry but this bot encountered an error when processing your answer."),
// Send a trace activity, which will be displayed in the Bot Framework Emulator
turnContext.TraceActivityAsync("OnTurnError Trace", exception.Message, "https://www.botframework.com/schemas/error", "TurnError")
};
Task all = Task.WhenAll(tasks); //task with the long running tasks
await Task.WhenAny(all, Task.Delay(5000)); //wait with a timeout
};
}
}
Modified client Wrapper:
public class TwilioClientWrapperWithCallback : TwilioClientWrapper
{
public TwilioClientWrapperWithCallback(TwilioClientWrapperOptions options) : base(options) { }
public async override Task<string> SendMessageAsync(TwilioMessageOptions messageOptions, CancellationToken cancellationToken)
{
var createMessageOptions = new CreateMessageOptions(messageOptions.To)
{
ApplicationSid = messageOptions.ApplicationSid,
MediaUrl = messageOptions.MediaUrl,
Body = messageOptions.Body,
From = messageOptions.From,
};
createMessageOptions.StatusCallback = new System.Uri("https://myApp.ngrok.io/api/TwilioSms/SmsStatusUpdated");
var messageResource = await MessageResource.CreateAsync(createMessageOptions).ConfigureAwait(false);
return messageResource.Sid;
}
}
Finally, here's my summarized code that sends the proactive message:
[HttpPost("StartConversationWithSuperBill/{superBillId:long}")]
[HttpPost("StartConversationWithSuperBill/{superBillId:long}/Campaign/{campaignId:int}")]
public async Task<IActionResult> StartConversation(long superBillId, int? campaignId)
{
ConversationReference conversationReference = this.GetConversationReference("+17545517768");
//Start a new conversation.
await ((BotAdapter)_adapter).ContinueConversationAsync(_appId, conversationReference, async (turnContext, token) =>
{
await turnContext.SendActivityAsync("proactive message 1");
//this code was edited for brevity. Here I would start a new dialog that would cascade into another, but the end result is the same, as soon as a message is sent, the turn context is disposed.
await turnContext.SendActivityAsync("proactive message 2"); //throws ObjectDisposedException
}, default(CancellationToken));
var result = new { status = "Initialized fine!" };
return new JsonResult(result);
}
private ConversationReference GetConversationReference(string targetNumber)
{
string fromNumber = "+18632704234";
return new ConversationReference
{
User = new ChannelAccount { Id = targetNumber, Role = "user" },
Bot = new ChannelAccount { Id = fromNumber, Role = "bot" },
Conversation = new ConversationAccount { Id = targetNumber },
//ChannelId = "sms",
ChannelId = "twilio-sms", //appparently when using twilio adapter we need to set this. if using TwiML app and not using Twilio Adapter, use the above. Otherwise the frameworks interprets answers from SMS as new conversations instead.
ServiceUrl = "https://sms.botframework.com/",
};
}
(I can see that I could just call create conversation reference and do two callbacks, one for each message, but in my actual code I'm creating a dialog that sends one message and then invokes another dialog that starts another message)
Edit 2:
Some clarifications:
On approach 2, I'm using two adapters, as suggested by code sample and documentation on using twilio adapter. The controller that starts the proactive message uses an instance of a default adapter (similar to this one), and TwilioController (the one that gets the twilio incoming messages) uses TwilioAdapterWithErrorHandler.
On approach 3, I excluded the default adapter, and both controllers use TwilioAdapterWithErrorHandler.
Edit 3:
Here's a small repo with the issue.
I found a fix for this problem, around approach 3, by changing the overload I use for ContinueConversation. Replace this :
//Start a new conversation.
await ((BotAdapter)_adapter).ContinueConversationAsync(_appId, conversationReference, async (turnContext, token) =>
With this:
//Start a new conversation.
var twilioAdapter = (TwilioAdapterWithErrorHandler)_adapter;
await twilioAdapter.ContinueConversationAsync(_appId, conversationReference, async (context, token) =>
This way, the context is not disposed, an I can use the twilio adapter for the proactive message and have status callbacks on all messages.

Grails Quartz plugin not remembering stateful data between job executions?

I need to persist a counter value between executions of a Grails Quartz plugin job. This runs at the correctly timed intervals and I can set the jobDataMap and read the value back correctly (during the same execution run), but it refuses to remember it between between executions.
I've set concurrent = false as the docs advised. Any ideas? I just need to persist and increment a counter. I want to avoid using a DB if at all possible, I think this should just use memory? Or other work arounds?
My TestJob.groovy, in /server/grails-app/jobs:
package myPackage
class MyJob {
static triggers = {
simple repeatInterval: 5000l // execute job every 5 seconds
}
def concurrent = false // Don't run multiple simultaneous instances of this job
def execute(context) {
if(context.jobDetail.jobDataMap['recCounter'] == null) { context.jobDetail.jobDataMap['recCounter'] = 1 }
else { context.jobDetail.jobDataMap['recCounter'] = context.jobDetail.jobDataMap['recCounter'] + 1 }
println(context.jobDetail.jobDataMap['recCounter'])
}
The output when run is a new line with '1' every 5 seconds. It should be incrementing the counter each time.
1
1
1
1
etc..
I'm running Grails 3.3.9 and build.gradle has compile "org.grails.plugins:grails-spring-websocket:2.4.1" in dependencies
Thanks
I have never used context object in my apps, but a counter can be implemented in a straight-forward way:
class MyJob {
//some static stuff
AtomicInteger counter = new AtomicInteger()
def execute(context) {
counter.incrementAndGet()
println counter.intValue()
}
}

Azure function: send notification to specific users

i've written an Azure function and connected the output to a notification-hub to send push notifications using APNS. It works fine as long as i send the notification to all registered devices, but i don't know how to use tags in order to address a specific user. If i try to use a tag, i get an error message saying "Exception while executing function: Functions.SendSinglePushNotification. Microsoft.Azure.WebJobs.Host: Error while handling parameter notification after function returned:. Microsoft.Azure.NotificationHubs: notification.Tag property should be null."
Here's my code so far:
#r "Microsoft.Azure.NotificationHubs"
#r "Newtonsoft.Json"
using System;
using Microsoft.Azure.NotificationHubs;
using Newtonsoft.Json;using
Microsoft.Azure.WebJobs.Host.Bindings.Runtime;
public static void Run(HttpRequestMessage req, TraceWriter log,Binder
binder, out AppleNotification notification)
{
string user = "Test";
string tagExpression = "Test";
string userTag = req.GetQueryNameValuePairs()
.FirstOrDefault(q => string.Compare(q.Key, "userid", true) == 0)
.Value;
string apnsNotificationPayload = "{\"aps\": {\"alert\": \"Test: (" + user + ")\" }}";
notification = new AppleNotification(apnsNotificationPayload);
}
I was trying to use notification = new
AppleNotification(apnsNotificationPayload,tagExpression);
but that does not work. How can i achieve that?
Thanks a lot and best regards
I had similar issue. Eventually, what worked for me was constructing Notification client manually. I am developing functions in Visual Studio, so my code is slightly different from yours.
[FunctionName("MyFunction")]
public static async Task Run([ServiceBusTrigger("queuename", AccessRights.Listen, Connection =
"<connection-settings-name>")] string myQueueItem, TraceWriter log)
{
log.Info($"C# Queue trigger function processed: {myQueueItem}");
var notificationHubSas = "<DefaultFullSharedAccessSignature from Azure portal>";
var notificationHubName = "myhub";
var nhClient = NotificationHubClient.CreateClientFromConnectionString(notificationHubSas, notificationHubName);
var tags = "";
await nhClient.SendTemplateNotificationAsync(<notification payload>, tags);
}

how to make executor service wait until all thread finish

i use executor service to launch multiple thread to sent request to api and get data back. sometimes i see some threads haven't finished their job yet, the service kill that thread already, how can i force the service to wait until the thread finish their job?
here is my code:
ExecutorService pool = Executors.newFixedThreadPool(10);
List<Future<List<Book>>> futures = Lists.newArrayList();
final ObjectMapper mapper1 = new ObjectMapper();
for (final Author a : authors) {
futures.add(pool.submit(new Callable<List<Book>>() {
#Override
public List<Book> call() throws Exception {
String urlStr = "http://localhost/api/book?limit=5000&authorId=" + a.getId();
List<JsonBook> Jsbooks = mapper1.readValue(
new URL(urlStr), BOOK_LIST_TYPE_REFERENCE);
List<Book> books = Lists.newArrayList();
for (JsonBook jsonBook : Jsbooks) {
books.add(jsonBook.toAvro());
}
return books;
}
}));
}
pool.shutdown();
pool.awaitTermination(3, TimeUnit.MINUTES);
List<Book> bookList = Lists.newArrayList();
for (Future<List<Book>> future : futures) {
if (!future.isDone()) {
LogUtil.info("future " + future.toString()); <-- future not finished yet
throw new RuntimeException("Future to retrieve books: " + future + " did not complete");
}
bookList.addAll(future.get());
}
and i saw some excepitons at the (!future.isDone()) block. how can i make sure every future is done when executor service shutdown?
I like to use the countdown latch.
Set the latch to the size that you're iterating and pass that latch into your callables, then in your run / call method have a try/finally block that decrements the countdown latch.
After everything has been enqueued to your executor service, just call your latch's await method, which will block until it's all done. At that time all your callables will be finished, and you can properly shut down your executor service.
This link has an example of how to set it up.
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CountDownLatch.html

can't run the automated project in testcomplete when it calls from jenkins

can't run the automated project in testcomplete when calls from jenkins.
In our continuous integration part ,the project is automated using testcomplete and it is calling through jenkins with the help of bat file.The scripts inside the bat file is
"C:\Program Files\Automated QA\TestComplete 7\Bin\TestComplete.exe " "D:\Test Complete7 Projects\ProjectInput_AllSamples\ProjecInputs.pjs" /r /p:Samples /rt:Main "iexplore" /e
It will open testcomplete and iexplorer ,but it is not filling the data(automation).
It is working perfectly when we directly call the bat file with out jenkins.Is there any solution
From your description it sounds like something in Windows stopping you from allowing your test application to work normally. It might be the fact that the second user could be a problem but I can't confirm that as I was not able find any definite explanations of how it works in Windows XP. I am pretty sure that this won't work on a Windows Vista, 7, 8 or server machine though because of the changes in architecture.
It sounds like the best solution is to make sure that your automated UI tests are started by an interactive user. When I was trying to add automated testing to our builds we used TestComplete 7 on a Windows XP SP2 virtual machine. In order to start our tests as an interactive user we:
Made an user log on when windows started, this way there was always an interactive user which means there was an actual desktop session which has access to the keyboard / mouse. I seem to remember (but can't find any links at the moment) that without an interactive user there is no active desktop that can access the keyboard / mouse.
We wrote a little app that would start when the interactive user logged on. This app would look at a specific file and when that file changed / was created it would read the file and start the application. The code for this app looked somewhat like this:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace ApplicationStarter
{
class Program
{
// The string used to indicate that the application should quit.
private const string ExitString = "exit";
// The path which is being watched for changes.
private static string s_LoadFilePath;
static void Main(string[] args)
{
try
{
{
Debug.Assert(
args != null,
"The arguments array should not be null.");
Debug.Assert(
args.Length == 1,
"There should only be one argument.");
}
s_LoadFilePath = args[0];
{
Console.WriteLine(
string.Format(
CultureInfo.InvariantCulture,
"Watching: {0}",
s_LoadFilePath));
}
if (File.Exists(s_LoadFilePath))
{
RunApplication(s_LoadFilePath);
}
using (var watcher = new FileSystemWatcher())
{
watcher.IncludeSubdirectories = false;
watcher.NotifyFilter =
NotifyFilters.LastAccess
| NotifyFilters.LastWrite
| NotifyFilters.FileName
| NotifyFilters.DirectoryName;
watcher.Path = Path.GetDirectoryName(s_LoadFilePath);
watcher.Filter = Path.GetFileName(s_LoadFilePath);
try
{
watcher.Created += OnConfigFileCreate;
watcher.EnableRaisingEvents = true;
// Now just sit here and wait until hell freezes over
// or until the user tells us that it has
string line = string.Empty;
while (!string.Equals(line, ExitString, StringComparison.OrdinalIgnoreCase))
{
line = Console.ReadLine();
}
}
finally
{
watcher.Created -= OnConfigFileCreate;
}
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
private static void RunApplication(string configFilePath)
{
var appPath = string.Empty;
var arguments = string.Empty;
using (var reader = new StreamReader(configFilePath, Encoding.UTF8))
{
appPath = reader.ReadLine();
arguments = reader.ReadLine();
}
// Run the application
StartProcess(appPath, arguments);
}
private static void StartProcess(string path, string arguments)
{
var startInfo = new ProcessStartInfo();
{
startInfo.FileName = path;
startInfo.Arguments = arguments;
startInfo.ErrorDialog = false;
startInfo.UseShellExecute = true;
startInfo.RedirectStandardOutput = false;
startInfo.RedirectStandardError = false;
}
Console.WriteLine(
string.Format(
CultureInfo.InvariantCulture,
"{0} Starting process {1}",
DateTime.Now,
path));
using (var exec = new Process())
{
exec.StartInfo = startInfo;
exec.Start();
}
}
private static void OnConfigFileCreate(
object sender,
FileSystemEventArgs e)
{
Console.WriteLine(
string.Format(
CultureInfo.InvariantCulture,
"{0} File change event ({1}) for: {2}",
DateTime.Now,
e.ChangeType,
e.FullPath));
// See that the file is there. If so then start the app
if (File.Exists(e.FullPath) &&
string.Equals(s_LoadFilePath, e.FullPath, StringComparison.OrdinalIgnoreCase))
{
// Wait for a bit so that the file is no
// longer locked by other processes
Thread.Sleep(500);
// Now run the application
RunApplication(e.FullPath);
}
}
}
}
This app expects the file to have 2 lines, the first with the app you want to start and the second with the arguments, so in your case something like this:
C:\Program Files\Automated QA\TestComplete 7\Bin\TestComplete.exe
"D:\Test Complete7 Projects\ProjectInput_AllSamples\ProjecInputs.pjs" /r /p:Samples /rt:Main "iexplore" /e
You should be able to generate this file from Jenkins in a build step.
Finally you may need to watch the TestComplete process for exit so that you can grab the results at the end but I'll leave that as an exercise to reader.
If you are running Jenkins (either master or slave) as a windows service, ensure it is running as a user and not as Local System.
We also do the same as Gentlesea's recommends, we run TestExecute on our Jenkins Slaves and keepo the TestComplete licenses for the people designing the TestComplete scripts.

Resources