I would like to implement an application wide retry system for all entity SaveChanges method calls.
Technologies:
Entity framework 4.0
.Net 4.0
namespace Sample.Data.Store.Entities {
public partial class StoreDB
{
public override int SaveChanges(System.Data.Objects.SaveOptions options)
{
for (Int32 attempt = 1; ; )
{
try
{
return base.SaveChanges(options);
}
catch (SqlException sqlException)
{
// Increment Trys
attempt++;
// Find Maximum Trys
Int32 maxRetryCount = 5;
// Throw Error if we have reach the maximum number of retries
if (attempt == maxRetryCount)
throw;
// Determine if we should retry or abort.
if (!RetryLitmus(sqlException))
throw;
else
Thread.Sleep(ConnectionRetryWaitSeconds(attempt));
}
}
}
static Int32 ConnectionRetryWaitSeconds(Int32 attempt)
{
Int32 connectionRetryWaitSeconds = 2000;
// Backoff Throttling
connectionRetryWaitSeconds = connectionRetryWaitSeconds *
(Int32)Math.Pow(2, attempt);
return (connectionRetryWaitSeconds);
}
/// <summary>
/// Determine from the exception if the execution
/// of the connection should Be attempted again
/// </summary>
/// <param name="exception">Generic Exception</param>
/// <returns>True if a a retry is needed, false if not</returns>
static Boolean RetryLitmus(SqlException sqlException)
{
switch (sqlException.Number)
{
// The service has encountered an error
// processing your request. Please try again.
// Error code %d.
case 40197:
// The service is currently busy. Retry
// the request after 10 seconds. Code: %d.
case 40501:
//A transport-level error has occurred when
// receiving results from the server. (provider:
// TCP Provider, error: 0 - An established connection
// was aborted by the software in your host machine.)
case 10053:
return (true);
}
return (false);
}
}
}
The problem:
How can I run the StoreDB.SaveChanges to retry on a new DB context after an error occured?
Something simular to Detach/Attach might come in handy.
Thanks in advance!
Bart
Check out the Transient Fault Handling Framework for SQL Azure, here. It's pretty generic, and it's easy to modify to include retry logic for your backend.
Related
So I have a redis code like this
public class RedisConnectorHelper
{
static RedisConnectorHelper()
{
RedisConnectorHelper.lazyConnection = new Lazy<ConnectionMultiplexer>(() =>
{
string redisServer = WebConfigurationManager.AppSettings["RedisServer"];
if (redisServer == null)
{
redisServer = "localhost";
}
return ConnectionMultiplexer.Connect(redisServer+ ",allowAdmin=true,abortConnect=false,ssl=True,");
}, System.Threading.LazyThreadSafetyMode.PublicationOnly);
}
Here is the part where the caching has been implemented,
public List<Division> GetDivisons()
{
if (CacheManager.Instance.IsDataSaved(CacheKey.Divisions))
{
return CacheManager.Instance.GetData<List<Division>>(CacheKey.Divisions);
}
else
{
CacheManager.Instance.SaveData<List<Division>>(CacheKey.Divisions, _notebookEntities.Divisions.ToList(), 30, ExpiryTimeUnit.Minutes);
return _notebookEntities.Divisions.ToList();
}
}
And the unit testing part,
[Test]
public void GetDivisons()
{
var result = _notebookService.GetDivisons();
Assert.NotNull(result);
}
but as soon as I uncomment the unit test which is shown below,the exception (shown below) occurs and the azure pipeline build fails, but passes as soon as I comment the unit test.
Here is the Exception,
StackExchange.Redis.RedisConnectionException : It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code. at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(ConfigurationOptions configuration, TextWriter log, Nullable`1 serverType, EndPointCollection endpoints)
at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log)
at StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log)
Is there any way to figure out what is the cause of it. i have tried with abortConnect=true & false and also with ssl=true but nothing works, as soon as I set to abortConnect= true I get a RedisTimeOutException
If there are two different reminders on the same grain activation to be fired at the same point, given that grain execution context is single-threaded, will both reminders be executed and interleaved at the same time?
Also, is the reminder execution limited by the default 30s timeout ?
Reminders are invoked using regular grain method calls: the IRemindable interface is a regular grain interface. IRemindable.ReceiveReminder(...) is not marked as [AlwaysInterleave], so it will only be interleaved if your grain class is marked as [Reentrant].
In short: no, reminder calls are not interleaved by default.
Reminders do not override the SiloMessagingOptions.ResponseTimeout value, so the default execution time will be 30s.
If you have a reminder that might need a very long time to execute, you can follow a pattern of starting the long-running work in a background task and ensuring that it is still running (not completed or faulted) whenever the relevant reminder fires.
Here is an example of using that pattern:
public class MyGrain : Grain, IMyGrain
{
private readonly CancellationTokenSource _deactivating = new CancellationTokenSource();
private Task _processQueueTask;
private IGrainReminder _reminder = null;
public Task ReceiveReminder(string reminderName, TickStatus status)
{
// Ensure that the reminder task is running.
if (_processQueueTask is null || _processQueueTask.IsCompleted)
{
if (_processQueueTask?.Exception is Exception exception)
{
// Log that an error occurred.
}
_processQueueTask = DoLongRunningWork();
_processQueueTask.Ignore();
}
return Task.CompletedTask;
}
public override async Task OnActivateAsync()
{
if (_reminder != null)
{
return;
}
_reminder = await RegisterOrUpdateReminder(
"long-running-work",
TimeSpan.FromMinutes(1),
TimeSpan.FromMinutes(1)
);
}
public override async Task OnDeactivateAsync()
{
_deactivating.Cancel(throwOnFirstException: false);
Task processQueueTask = _processQueueTask;
if (processQueueTask != null)
{
// Optionally add some max deactivation timeout here to stop waiting after (eg) 45 seconds
await processQueueTask;
}
}
public async Task StopAsync()
{
if (_reminder == null)
{
return;
}
await UnregisterReminder(_reminder);
_reminder = null;
}
private async Task DoLongRunningWork()
{
// Log that we are starting the long-running work
while (!_deactivating.IsCancellationRequested)
{
try
{
// Do long-running work
}
catch (Exception exception)
{
// Log exception. Potentially wait before retrying loop, since it seems like GetMessageAsync may have failed for us to end up here.
}
}
}
}
I am using a dbContextwhich is described as the following
public class DbContext: System.Data.Entity.DbContext
{
public DbSet<UserAccount> UserAccounts { get; set; }
public DbSet<Sesison> Sessions { get; set; }
}
And in my login action for some reason after checking with the database that the user is correct i am not able to use another entity of the context to save into the database, whats happens is that i won't get the data saved into the database and it will actually stop executing code after the dc.SaveChanges() but i dont get an exeption on my catch, so what my browser is expiriencing is a 500 internal server error.
public ActionResult Login(LoginViewModel model)
{
using (DbContext dc = new DbContext())
{
var v = dc.UserAccounts.SingleOrDefault(a => a.UserName == model.UserName);
if (v != null)
{
if (GetSHA1(model.Password) == v.Password)
{
Guid sessionGuid =Guid.NewGuid();
var session = dc.Sessions.Add(new Sesison() { SessionID = sessionGuid,StartDateTime=DateTime.UtcNow,UserID = v.UserID,ExpireDateTime=DateTime.UtcNow.AddHours(4)});
System.Web.HttpContext.Current.Session["IsLogin"] = true;
System.Web.HttpContext.Current.Session["Session"] = sessionGuid;
try
{
dc.SaveChanges();
}
catch (System.Data.Entity.Validation.DbEntityValidationException dbEx)
{
Exception raise = dbEx;
foreach (var validationErrors in dbEx.EntityValidationErrors)
{
foreach (var validationError in validationErrors.ValidationErrors)
{
string message = string.Format("{0}:{1}",
validationErrors.Entry.Entity.ToString(),
validationError.ErrorMessage);
// raise a new exception nesting
// the current instance as InnerException
raise = new InvalidOperationException(message, raise);
}
}
throw raise;
}
return Json(new
{
Item1 = "true"
});
}
else
{
return Json(new
{
Item1 = "false",
Item2 = "The username or password you entered is incorrect. Please try again."
});
}
}
else
{
return Json(new
{
Item1 ="false",
Item2 = "The username or password you entered is incorrect. Please try again."
});
}
}
return null;
}
The call to SaveChanges is throwing an exception of a type other than DbEntityValidationException
A quick look at the DbContext SaveChanges call shows that it might throw the following types of exceptions. If you don't want to handle each then you probably want to add a generic Exception handling to find the problem.
// Exceptions:
// T:System.Data.Entity.Infrastructure.DbUpdateException:
// An error occurred sending updates to the database.
//
// T:System.Data.Entity.Infrastructure.DbUpdateConcurrencyException:
// A database command did not affect the expected number of rows. This usually indicates
// an optimistic concurrency violation; that is, a row has been changed in the database
// since it was queried.
//
// T:System.Data.Entity.Validation.DbEntityValidationException:
// The save was aborted because validation of entity property values failed.
//
// T:System.NotSupportedException:
// An attempt was made to use unsupported behavior such as executing multiple asynchronous
// commands concurrently on the same context instance.
//
// T:System.ObjectDisposedException:
// The context or connection have been disposed.
//
// T:System.InvalidOperationException:
// Some error occurred attempting to process entities in the context either before
// or after sending commands to the database.
We are facing an issue in Jetty where on timeout it replays the original request again if we don't complete the request from async context. Here is the behavior, for every request we set a async listener with timeout, so we have 2 threads in play, one (Jetty Thread1) is listening on timeout and other (Thread2) is serving thread. Now let us say write data to client takes longer than timeout, since the request is not completed timeout thread gets triggered, it checks that someone is writing data so it returns silently. Jetty doesn't like returning silently, it replays the request back so another serving and timeout thread gets created and it goes on until data is written and async context is completed.
The code in question is here - In HttpChannelState in expired() method
if (aListeners!=null)
{
for (AsyncListener listener : aListeners)
{
try
{
listener.onTimeout(event);
}
catch(Exception e)
{
LOG.debug(e);
event.setThrowable(e);
_channel.getRequest().setAttribute(RequestDispatcher.ERROR_EXCEPTION,e);
break;
}
}
}
boolean dispatch=false;
synchronized (this)
{
if (_async==Async.EXPIRING)
{
_async=Async.EXPIRED;
if (_state==State.ASYNC_WAIT)
{
_state=State.ASYNC_WOKEN;
dispatch=true;
}
}
}
if (dispatch)
scheduleDispatch(); // <------------ dispatch again why
}
This is normal behaviour. You have put the request into async state and then not handled the timeout, so the request is redispatch with a DispatcherType of ASYNC.
If you add your own timeout listener and within that timeout you either complete or dispatch the asyncContext, then jetty will not redispatch it (unless your listener called dispatch).
You can also protect your async servlet code with a test for the DispatcherType, although that can be confused if you have multiple concerns that might be handled async.
asyncContext.addListener(new AsyncListener()
{
#Override
public void onTimeout(AsyncEvent event) throws IOException
{
event.getAsyncContext().complete();
}
#Override
public void onStartAsync(AsyncEvent event) throws IOException
{
}
#Override
public void onError(AsyncEvent event) throws IOException
{
}
#Override
public void onComplete(AsyncEvent event) throws IOException
{
}
});
I have an Xamarin Android project and was using mono.data.sqlite and had problems with multithreading, so I tried the Zumero component. I'm still having problems. I'm trying to set serialized mode as with the flag SQLITE_CONFIG_SERIALIZED in http://www.sqlite.org/threadsafe.html. I'm still getting random crashes. Can I set the serialized flag with Zumero? Any other suggestions other than recompiling SQLite from the source?
Thanks,
Brian
I used to have this problem. And despite conflicting recommendations here's how I stopped getting the exceptions:
Share a static instance of SQLiteConnection between all threads. This is safe to do as SQLite connection is only a file pointer it's not like a traditional data connection.
Wrapped all my SQLite queries/inserts/updates in a mutex with the statix instance of my SQLiteConnection as the lock. I've been advised that I shouldn't need to do this when using serialized mode however my experience with it begs to differ.
lock(myStaticConnection) {
myStaticConnection.Query<Employee>("....");
}
As a backup I also use some added retry logic to encapsulate every query. Not sure if SQLite does this on its own (I've seen reference to busytimeout and people claiming it is now gone?). Something like this:
public static List<T> Query<T> (string query, params object[] args) where T : new()
{
return Retry.DoWithLock (() => {
return Data.connection.Query<T> (query, args);
}, Data.connection, 0);
}
public static T DoWithLock<T>(
Func<T> action,
object lockable,
long retryIntervalTicks = defaultRetryIntervalTicks,
int retryCount = defaultRetryCount)
{
return Do (() => {
lock (lockable) {
return action();
}
});
}
public static T Do<T>(
Func<T> action,
long retryIntervalTicks = defaultRetryIntervalTicks,
int retryCount = defaultRetryCount)
{
var exceptions = new List<Exception> ();
for (int retry = 0; retry < retryCount; retry++) {
try{
return action();
} catch (Exception ex) {
exceptions.Add (ex);
ManualSleepEvent (new TimeSpan(retryIntervalTicks));
}
}
throw new AggregateException (exceptions);
}