I am trying to schedule an activity in Amazon SWF. Initially, I used to loop through a list and schedule the activity for each value of the list. But this would invoke the activities in parallel which I did not want. So, I modified my code to do something like this:
Promise<Void> promiseArg = null;
for(Integer i : IntegerList){
Promise<Void> nextArg = activityClient.activity1(i);
promiseArg = nextArg;
}
Though code is working, I am not sure if this is the right way to do it. Any comments would be helpful.
What is the point of using promiseArg if it is unused?
If you want them to be dependent on prev method call, create an Asynchronous method and call that with promise variable.
//Main method of decider.
Promise<Integer> promiseArg = null;
Promise<Integer> nextArg = activityClient.activity1(i, 1);
for(Integer i : IntegerList){
Promise<Integer> nextArg = fun(nextArg, Promise.asPromise(i));
}
#Asynchronous
public Promise<Integer> fun(Promise<int> nextArg, int i) {
System.out.println("Testing with current value: " + Integer.toString(nextArg.get()));
return activityClient.activity1(i, nextArg.get());
}
I haven't tested it but it should work.
Apart from this, you can also try passing prev Promise variable to activity itself with #Wait annotation in the activity declaration.
Something like this,
prevArgs = activityClient.activity1(i, prevArg));
with Activity like,
XYZ activity1(int i,#Wait Promise<int> prevArgs){
//Please check if int should be used instead of Promise<int>
}
Related
I have just started using durable functions and needs some advise for how to do fan out pattern correctly. I have a FTP server where from I read all the files. I want to start an Activity function for each file. As I understand it the orchestrator function will be called everytime an Activity function is being executed. I just want to read the files once. To avoid calling the code that read the files and starts the activity functions multiple times, what is the recommended approach? Is it having an activity function that that add's all the activity functions or is it using the IsReplaying property, or something different?
[FunctionName("OrchestrationMoveFilesToBlob")]
public static async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context)
{
var outputs = new List<string>();
if (!context.IsReplaying)
{
// Do you call your database here and make a call to CallActivityAsync for each row?
}
// doing it here is properly very wrong as it will be called multiple times
var tasks = new Task<string>[7];
for (int i = 0; i < 7; i++)
{
tasks[i] = context.CallActivityAsync<string>("E2_CopyFileToBlob",""); }
await Task.WhenAll(tasks);
return outputs;
}
When looking into the sample in the link below this actually calls it directly in the orchestrator function? Is this not really bad? It continue adding same activities again and again .... ?
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-cloud-backup
Not sure I understand what you try to achieve but your code looks not bad so far. An orchestration is just called once (and maybe some times more for replay but this is not your problem here). From your orchestration you can call in a fan out all your activity functions (gathering a file from an ftp) each activity function one file. await Task.WhenAll(tasks) is your fan in. (you can use a List<Task> instead of the array and call .Add(task) on it if you want. In order to not edit your code I copied it here and added some comments and questions (feel free to edit here):
[FunctionName("OrchestrationMoveFilesToBlob")]
public static async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context)
{
var outputs = new List<string>();
if (!context.IsReplaying)
{
// just needed for things that should not happen twice like logging....
}
// if your work isn't a fixed list just call an activity
// which replies with the list of work here (e.g. list of filenames)
var tasks = new Task<string>[7]; // can be a List<Task> too
for (int i = 0; i < 7; i++)
{
tasks[i] = context.CallActivityAsync<string>("E2_CopyFileToBlob","");
}
await Task.WhenAll(tasks);
return outputs; // currently an empty list. What do you want to give back?
}
I'm making a Future method that lives inside a seperate class, that fetches a bunch XKCD comics, and puts them in a List and returns it.
And that is all fine and dandy, but I would like to notify back when a single comic has been fetched, so I can show a progress dialog, on how far we are.
This is my code:
// This is inside my class ComicManager
Future<List<ComicModel>> generateComicList() async {
List<ComicModel> comicList = new List<ComicModel>();
ComicModel latestComic = await getLatestComic();
for (var i = 1; i <= latestComic.num; i++) {
try {
http.Response response =
await http.get('https://xkcd.com/${i}/info.0.json');
Map comicmap = json.decode(response.body);
var comic = new ComicModel.fromJson(comicmap);
comicList.add(comic);
print(comic.num);
// Notify here that we have fetched a comic
} catch (ex) {
// Comic could apparently not be parsed, skip it.
}
}
return comicList;
}
How should I solve this?
There seems no particularly elegant way to do this. From some flutter code samples, it seems using VoidCallBack listeners is an accepted way.
First register callback functions in a Set
Set<VoidCallBack> listeners
Then define the callback functions you needed. And add them to the set
void fun()
//...
listeners.add(fun);//Or you can define a method to do this or simply pass the function through the constructor of this class.
Finally, write a notifyListeners function or its equivalent and call it wherever you want
void notifyListeners(){
for(final listener in listeners){
listener();
}
}
If you want callback functions to carry an argument, just change the VoidCallBack to whatever function types.
Found a solution.
I just used Streams like so:
Stream<ComicProgressModel> getAllComicsStream() async* {
// Do what you need to do here
// This will respond back when you are listening to the stream
yield stuffToYield; // Can be anything, and you can yield as many times you want
// When you reach the end of the method, the onDone method will be called.
// So if you are running a for loop, and call yield multiple times it onDone is only called the the this method ends
}
Then I can just listen to events like so:
Stream comicStream =
ComicManager().getAllComicsStream().asBroadcastStream();
StreamSubscription comicsub = comicStream.listen((onData) {
// Do what i need
});
Super easy to be honest.
I am using this function to add a timeout callback (repeated) to a specific GMainContext.
guint GstThreadHelper::timeoutAdd(guint delay, GSourceFunc function, gpointer data) {
// See https://developer.gnome.org/programming-guidelines/stable/main-contexts.html.en#implicit-use-of-the-global-default-main-context
// It is important that all thread functions we invoke don't implicitly decide a maincontext.
// We must manually provide one.
GSource *source = NULL;
guint id;
source = g_timeout_source_new(delay);
g_source_set_callback (source, function, data, NULL);
id = g_source_attach (source, priv->mainContext);
g_source_unref (source);
return id;
}
Later, I use the returned id to cancel the callback.
void GstThreadHelper::timeoutRemove(guint id) {
g_source_remove(id);
}
However, the callback still gets called. Here is my callback.
static gboolean position_update (gpointer user_data)
{
Player::PrivateData* priv = (Player::PrivateData*)user_data;
gint64 pos = 0;
if (gst_element_query_position (priv->playbin, GST_FORMAT_TIME, &pos)) {
pos = pos / 1000000;
priv->callback->PositionChanged(pos);
}
// Call me again
return TRUE;
}
I understand I am returning TRUE, but my understanding is that it still should be stopped. If I cancel callbacks by returning FALSE, I wouldn't bother with the g_source_remove call.
Why doesn't g_source_remove stop my callback from being raised?
EDIT
If I replace my timeoutAdd method with this...
guint GstThreadHelper::timeoutAdd(guint delay, GSourceFunc function, gpointer data) {
return g_timeout_add(delay, function, data);
}
...it works. However, I can't use this, because it doesn't trigger the callbacks on a specific GMainContext, as opposed to the default global GMainContext.
EDIT2
I copied the default source for g_timeout_add_seconds_full into my function, and it worked.
However, the moment I changed g_source_attach to use my private GMainContext, it failed.
The issue is something to do with calling g_source_remove for timeouts added on non-default GMainContexts.
It turns out that g_source_remove operates under the assumption that you are using the global/default GMainContext, which in this case, I am not.
I don't remember reading this in the docs.
Anyways, Here is the solution.
void GstThreadHelper::timeoutRemove(guint id) {
GSource* source = g_main_context_find_source_by_id(priv->mainContext, id);
if (source)
g_source_destroy (source);
}
This is essentially what g_source_remove is doing, but instead using our private GMainContext.
public async Task<Utils.TReportReturn> GetReportDataColumnSettings(AdminTechnicianReportInput input)
{
input.CustomerId = m_customerId;
input.UserId = m_userId;
input.TimeOffset = m_timeOffset;
TReportReturn objTReportReturn = new TReportReturn();
objTReportReturn.ReportColumns = await new ReportsEntityContext().GetTechniciansReportColumns<AdminTechnicianReportInput>(input);
objTReportReturn.ReportSettings = await new ReportsEntityContext().GetGeneralSettingsData<AdminTechnicianReportInput>(input);
objTReportReturn.ReportData = await new ReportsDbContext().GetUserListDynamic<AdminTechnicianReportInput>(input);
return objTReportReturn;
}
public class HomeController : Controller
{
public ActionResult Index()
{
AdminTechnicianReportInput objInput = new AdminTechnicianReportInput();
objInput.CustomerId = 1;
objInput.UserId = 5477;
objInput.AddUpdateUserid = 0;
objInput.intRowsPerPage = 200;
objInput.intPageIndex = 0;
objInput.intTarget = 0;
objInput.intStatusId = 0;
objInput.strSearchQuery = "";
objInput.strSortColumn = "Id";
objInput.strSortDirection = "ASC";
var result = new ReportsModel(1, 5477, 0).Administration.AdminTechnicianReport.GetReportDataColumnSettings(objInput).Result;
ViewBag.Title = result.ReportSettings.CustomerName + "__" + result.ReportSettings.PrintedOn;
return View();
}
}
The GetReportDataColumnSettings function calls many async methods which is the the model library dll.
Index() method in HomeController is an mvc method and I want to get the return in the result variable. Itried getawaiter().getmethod(), but it didn't worked. I do not prefer to change the Index() method as asynk Task. Please suggest another way to get the result
I do not prefer to change the Index() method as asynk Task.
Why not? That is by far the easiest and most correct solution.
The second easiest is to just make everything synchronous. That is, if you won't go async all the way, then go sync all the way.
There are a variety of hacks for calling async from sync code that may or may not work in your situation; none of these hacks work everywhere. For more details, see my async brownfield article.
In your case, your initial attempt was to use the Blocking Hack. For this to work appropriately on the pre-Core ASP.NET, you'd have to use ConfigureAwait(false) for every await in that entire method call tree. This is usually possible (not always possible, of course, because library code may or may not use it), but it causes maintenance problems (forget just one, and you've got a race condition that may end in deadlock). I explain why the deadlock happens in detail on my blog.
If you won't go async all the way, and if you can't go sync all the way, then I'd recommend the Boolean Sync Argument Hack, which requires the entire call stack to support both synchronous and asynchronous callers. It would look something like this for the code you posted:
private async Task<Utils.TReportReturn> DoGetReportDataColumnSettingsAsync(AdminTechnicianReportInput input, bool sync)
{
input.CustomerId = m_customerId;
input.UserId = m_userId;
input.TimeOffset = m_timeOffset;
TReportReturn objTReportReturn = new TReportReturn();
objTReportReturn.ReportColumns = await new ReportsEntityContext().GetTechniciansReportColumns<AdminTechnicianReportInput>(input, sync);
objTReportReturn.ReportSettings = await new ReportsEntityContext().GetGeneralSettingsData<AdminTechnicianReportInput>(input, sync);
objTReportReturn.ReportData = await new ReportsDbContext().GetUserListDynamic<AdminTechnicianReportInput>(input, sync);
return objTReportReturn;
}
private Task<Utils.TReportReturn> GetReportDataColumnSettingsAsync(AdminTechnicianReportInput input)
{ return DoGetReportDataColumnSettingsAsync(input, sync: false); }
private Utils.TReportReturn GetReportDataColumnSettings(AdminTechnicianReportInput input)
{ return DoGetReportDataColumnSettingsAsync(input, sync: true).GetAwaiter().GetResult(); }
The key with this pattern is that any method taking a sync parameter must return a completed task when that argument is set to true.
I'm having a little trouble with a method in which I use yield return this doesn't work...
public IEnumerable<MyClass> SomeMethod(int aParam)
{
foreach(DataRow row in GetClassesFromDB(aParam).Rows)
{
yield return new MyClass((int)row["Id"], (string)row["SomeString"]);
}
}
The above code never runs, when the call is made to this method it just steps over it.
However if I change to...
public IEnumerable<MyClass> SomeMethod(int aParam)
{
IList<MyClass> classes = new List<MyClass>();
foreach(DataRow row in GetClassesFromDB(aParam).Rows)
{
classes.Add(new MyClass((int)rows["Id"], (string)row["SomeString"]);
}
return classes;
}
It works just fine.
I don't understand why the first method never runs, could you help me in understanding what is happening here?
The "yield" version is only "run" when the caller actually starts to enumerate the returned collection.
If, for instance, you only get the collection:
var results = SomeObject.SomeMethod (5);
and don't do anything with it, the SomeMethod will not execute.
Only when you start enumerating the results collection, it will hit.
foreach (MyClass c in results)
{
/* Now it strikes */
}
yield return methods are actually converted into state machine classes that retrieve information lazily - only when you actually ask for it. That means that in order to actually pull data, you have to iterate over the result of your method.
// Gives you an iterator object that hasn't done anything yet
IEnumerable<MyClass> list = SomeMethod();
// Enumerate over the object
foreach (var item in list ) {
// Only here will the data be retrieved.
// The method will stop on yield return every time the foreach loops.
}
The reason it runs in the second case is because there's no yield block, and thus the entire method runs in one go.
In this specific case, it's unlikely that you'll have any advantage to use an iterator block over a regular one because your GetClassesFromDb() isn't one either. This means that it will retrieve all the data at the same time first time it runs. Iterator blocks are best used when you can access items one at a time, because that way you can stop if you don't need them anymore.
I had to learn in a near disastrous way how cool/dangerous yield is when I decided to make our company's parser read incoming data lazily. Fortunately only one of the handful of our implementing functions actually used the yield keyword. Took a few days to realize it was quietly not doing any work at all.
The yield keyword it will be as lazy as it possibly can, including skipping over the method altogether if you don't put it to work with something like .ToList() or .FirstOrDefault() or .Any()
Below are two variations, one using the keyword and one returning a straight-up list. One won't even bother to execute, while the other will, even though they seem the same.
public class WhatDoesYieldDo
{
public List<string> YieldTestResults;
public List<string> ListTestResults;
[TestMethod]
public void TestMethod1()
{
ListTest();
Assert.IsTrue(ListTestResults.Any());
YieldTest();
Assert.IsTrue(YieldTestResults.Any());
}
public IEnumerable<string> YieldTest()
{
YieldTestResults = new List<string>();
for (var i = 0; i < 10; i++)
{
YieldTestResults.Add(i.ToString(CultureInfo.InvariantCulture));
yield return i.ToString(CultureInfo.InvariantCulture);
}
}
public IEnumerable<string> ListTest()
{
ListTestResults = new List<string>();
for (var i = 0; i < 10; i++)
{
ListTestResults.Add(i.ToString(CultureInfo.InvariantCulture));
}
return ListTestResults;
}
}
Moral of the story: Make sure that if have a method that returns IEnumerable and you use yield in that method, you have something that will iterate over the results, or the method won't execute at all.