Is Orleans reminder execution interleaved? - orleans

Question

If there are two different reminders on the same grain activation to be fired at the same point, given that grain execution context is single-threaded, will both reminders be executed and interleaved at the same time?
Also, is the reminder execution limited by the default 30s timeout ?

Answer 1

Reminders are invoked using regular grain method calls: the IRemindable interface is a regular grain interface. IRemindable.ReceiveReminder(...) is not marked as [AlwaysInterleave], so it will only be interleaved if your grain class is marked as [Reentrant].
In short: no, reminder calls are not interleaved by default.
Reminders do not override the SiloMessagingOptions.ResponseTimeout value, so the default execution time will be 30s.
If you have a reminder that might need a very long time to execute, you can follow a pattern of starting the long-running work in a background task and ensuring that it is still running (not completed or faulted) whenever the relevant reminder fires.
Here is an example of using that pattern:
public class MyGrain : Grain, IMyGrain
{
private readonly CancellationTokenSource _deactivating = new CancellationTokenSource();
private Task _processQueueTask;
private IGrainReminder _reminder = null;
public Task ReceiveReminder(string reminderName, TickStatus status)
{
// Ensure that the reminder task is running.
if (_processQueueTask is null || _processQueueTask.IsCompleted)
{
if (_processQueueTask?.Exception is Exception exception)
{
// Log that an error occurred.
}
_processQueueTask = DoLongRunningWork();
_processQueueTask.Ignore();
}
return Task.CompletedTask;
}
public override async Task OnActivateAsync()
{
if (_reminder != null)
{
return;
}
_reminder = await RegisterOrUpdateReminder(
"long-running-work",
TimeSpan.FromMinutes(1),
TimeSpan.FromMinutes(1)
);
}
public override async Task OnDeactivateAsync()
{
_deactivating.Cancel(throwOnFirstException: false);
Task processQueueTask = _processQueueTask;
if (processQueueTask != null)
{
// Optionally add some max deactivation timeout here to stop waiting after (eg) 45 seconds
await processQueueTask;
}
}
public async Task StopAsync()
{
if (_reminder == null)
{
return;
}
await UnregisterReminder(_reminder);
_reminder = null;
}
private async Task DoLongRunningWork()
{
// Log that we are starting the long-running work
while (!_deactivating.IsCancellationRequested)
{
try
{
// Do long-running work
}
catch (Exception exception)
{
// Log exception. Potentially wait before retrying loop, since it seems like GetMessageAsync may have failed for us to end up here.
}
}
}
}