I'm trying to implement comet style features by polling the server for changes in data and holding the connection open untill there is something to response with.
Firstly i have a static variable on my controller which stores the time that the data was last updated:
public static volatile DateTime lastUpdateTime = 0;
So whenever the data i'm polling changes this variable will be changed.
I then have an Action, which takes the last time that the data was retrieved as a parameter:
public ActionResult Push(DateTime lastViewTime)
{
while (lastUpdateTime <= lastViewTime)
{
System.Threading.Thread.Sleep(10000);
}
return Content("testing 1 2 3...");
}
So if lastUpdateTime is less than or equal to the lastViewTime, we know that there is no new data, and we simply hold the request there in a loop, keeping the connection open, untill there is new information, which we could then send back to the client, which would handle the response and then make a new request, so the connection is essentially always open.
This seems to work fine but i'm concerned about thread safety, is this OK? Does lastUpdateTime need to be marked as volatile? Is there a better way?
Thanks
edit: perhaps i should use a lock object when i update the time value
private static object lastUpdateTimeLock = new object();
..
lock (lastUpdateTimeLock)
{
lastUpdateTime = DateTime.Now;
}
Regarding your original question, you do have to be careful with DateTimes, since they're actual objects in the .NET runtime. Only a few data types can be natively accessed (eg ints, bools) without locking (assuming you're not using Interlocked). If you want to avoid any issues with Datetimes, you can get the ticks as a long and use the Interlocked class to manage them.
That said, if you're looking for comet capabilities in a .NET application, you're unfortunately going to have to go a lot further than what you've got here. IIS/ASP.NET won't scale with the approach you've got in place right now; you'll hit limits before you even get to 100 users. Among other things, you will have to switch to using async handlers, and implement a custom bounded thread pool for the incoming requests.
If you really want a tested solution for ASP.NET/IIS, check out WebSync, it's a full comet server designed specifically for that purpose.
Honestly my concern would be with the number of connections kept open and the empty while loop. The connections you're probably fine on, but I'd definitely want to do some load testing to be sure.
The while (lastUpdateTime <= lastViewTime) {} seems like it should have a Thread.Sleep(100) or something in there. Otherwise I'd think it would consume a lot of cpu cycles needlessly.
The lock does not seem necessary to me around lastUpdateTime = DateTime.Now since the previous value does not matter. If it were lastUpdateTime = lastUpdateTime + 1 or something, then maybe it would be.
Related
We're using Realm (Swift binding currently in version 3.12.0) from the earliest days in our project. In some early versions before 1.0 Realm provided change listeners for Results without actually giving changeSets.
We used this a lot in order to find out if a specific Results list changed.
Later the guys at Realm exchanged this API with changeSet providing methods. We had to switch and are now mistreating this API just in order to find out if anything in a specific List changed (inserts, deletions, modifications).
Together with RxSwift we wrote our own implementation of Results change listening which looks like this:
public var observable: Observable<Base> {
return Observable.create { observer in
let token = self.base.observe { changes in
if case .update = changes {
observer.onNext(self.base)
}
}
observer.onNext(self.base)
return Disposables.create(with: {
observer.onCompleted()
token.invalidate()
})
}
}
When we now want to have consecutive updates on a list we subscribe like so:
someRealm.objects(SomeObject.self).filter(<some filter>).rx.observable
.subscribe(<subscription code that gets called on every update>)
//dispose code missing
We wrote the extension on RealmCollection so that we can subscribe to List type as well.
The concept is equal to RxRealm's approach.
So now in our App we have a lot of filtered lists/results that we are subscribing to.
When data gets more and more we notice significant performance losses when it comes to seeing a change visually after writing something into the DB.
For example:
Let's say we have a Car Realm Object class with some properties and some 1-to-n and some 1-to-1 relationships. One of the properties is a Bool, namely isDriving.
Now we have a lot of cars stored in the DB and bunch of change listeners with different filters listing to changes of the cars collection (collection observers listening for changeSets in order to find out if the list was changed).
If I take one car of some list and set the property of isDriving from false to true (important: we do writes in the background) ideally the change listener fires fast and I have the nearly immediate correct response to my write on the main thread.
Added with edit on 2019-06-19:
Let's make the scenario still a little more real:
Let's change something down the hierarchy, let's say the tires manufacturer's name. Let's say a Car has a List<Tire>, a Tire has a Manufacturer and a Manufacturer has aname.
Now we're still listing toResultscollection changes with some more or less complex filters applied.
Then we're changing the name of aManufacturer` which is connected to one of the tires which are connected to one of the cars which is in that filtered list.
Can this still be fast?
Obviously when the length of results/lists where change listeners are attached to gets longer Realm's internal change listener takes longer to calculate the differences and fires later.
So after a write we see the changes - in worst case - much later.
In our case this is not acceptable. So we are thinking through different scenarios.
One scenario would be to not use .observe on lists/results anymore and switch to Realm.observe which fires every time anything did change in the realm, which is not ideal, but it is fast because the change calculation process is skipped.
My question is: What can I do to solve this whole dilemma and make our app fast again?
The crucial thing is the threading stuff. We're always writing in the background due to our design. So the writes itself should be very fast, but then that stuff needs to synchronize to the other threads where Realms are open.
In my understanding that happens after the change detection for all Results has run through, is that right?
So when I read on another thread, the data is only fresh after the thread sync, which happens after all notifications were sent out. But I am not sure currently if the sync happens before, that would be more awesome, did not test it by now.
By the way how do you create a STREAM?
I use AppendToStreamAsync directly, is this right or shall I create a
stream first then append onto this stream?
I also tried performing some tests but using the methods below I can write
events onto EventStore but can't read Events from it.
And most import question is how do I view my saving events in the Admin site of EventStore?
Here are the code:
public async Task AppendEventAsync(IEvent #event)
{
try
{
var eventData = new EventData(#event.EventId,
#event.GetType().AssemblyQualifiedName,
true,
Serializer.Serialize(#event),
Encoding.UTF8.GetBytes("{}"));
var writeResult = await connection.AppendToStreamAsync(
#event.SourceId.ToString(),
#event.AggregateVersion,
eventData);
Console.WriteLine(writeResult);
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
}
public async Task<IEnumerable<IEvent>> ReadEventsAsync(Guid aggregateId)
{
var ret = new List<IEvent>();
StreamEventsSlice currentSlice;
long nextSliceStart = StreamPosition.Start;
do
{
currentSlice = await connection.ReadStreamEventsForwardAsync(aggregateId.ToString(), nextSliceStart, 200, false);
if (currentSlice.Status != SliceReadStatus.Success)
{
throw new Exception($"Aggregate {aggregateId} not found");
}
nextSliceStart = currentSlice.NextEventNumber;
foreach (var resolvedEvent in currentSlice.Events)
{
ret.Add(Serializer.Deserialize(resolvedEvent.Event.EventType, resolvedEvent.Event.Data));
}
} while (!currentSlice.IsEndOfStream);
return ret;
}
Streams are created automatically as you write events. You should follow the recommended naming convention though as it enables a few features out of the box.
await Connection.AppendToStreamAsync("CustomerAggregate-b2c28cc1-2880-4924-b68f-d85cf24389ba", expectedVersion, creds, eventData);
It is recommended to call your streams as "category-id" - (where category in our case is the aggregate name) as we use are using DDD+CQRS pattern
CustomerAggregate-b2c28cc1-2880-4924-b68f-d85cf24389ba
The stream matures as you write more events to the same stream name.
The first events ID becomes the "aggregateID" in our case and then each new
eventID after that is unique. The only way to recreate our aggregate is
to replay the events in sequence. If the sequence fails an exception is thrown
The reason to use this naming convention is that Event Store runs a few default internal projection for your convenience. Here is a very convoluted documentation about it
$by_category
$by_event_type
$stream_by_category
$streams
By Category
By category basically means there is stream created using internal projection which for our CustomerAggregate we subscribe to $ce-CustomerAggregate events - and we will see only those "categories" regardless of their ID's - The event data contains everything we need there after.
We use persistent subscribers (small C# console applications) which are setup to work with $ce-CustomerAggregate. Persistent subscribers are great because they remember the last event your client acknowledged. So if the application crashes, you start it and it starts from the last place that application finished.
This is where event store starts to shine and stand out from the other "event store implementations"
Viewing your events
The example with persistent subscribers is one way to set things up using code.
You cannot really view "all" your data in the admin site. The purpose of the admin site it to manage projections, manage users, see some statistics, create some projections, and have a recent view of streams and events only. (If you know the ID's you can create the URL's as you need them - but you cant search for them)
If you want to see ALL the data then you use the RESTfull API using by using something like Postman. Maybe there is a 3rd party software that can create a grid like data source viewer but I am unaware of this. That would probably also just hook into the REST API and you could create your own visualiser this way quite quickly.
Again back to code, you can also always read all events from 0 using on of the libraries - which incidentally using DDD+CQRS you always read the aggregates stream from 0 to rebuilt its state. But you can do the same for other requirements.
In some cases looking at how to use snapshots makes replaying events allot faster, if you have an extremely large stream to deal with.
Paradigm shift
Event Store has quite a learning curve and is a paradigm shift from conventional transactional databases. Event Stores best friend is CQRS - We use a slightly modified version of the CQRS Lite open source framework
To truly appreciate Event Store you would need to understand DDD concepts and then dig into CQRS/ES - There are a few good YouTube videos and examples.
I have a controller with two actions. One performs a very long computation, and at several steps, stores status in a session container:
public function longAction()
{
$session = new Container('SessionContainer');
$session->finished = 0;
$session->status = "A";
// do something long
$session->status = "B";
// do more long jobs
$session->status = "C";
// ...
}
The second controller:
public function shortAction()
{
$session = new Container('SessionContainer');
return new JsonModel(
array(
'status' => $session->status
)
);
}
These are both called via AJAX, but I can evidence the same behavior in just using browser tabs. I first call /module/long which does its thing. While it completes its tasks, calling /module/short (I thought would just echo JSON) stalls /module/long is done!
Bringing this up, some ZFers felt this was a valid protection against race conditions; but I can't be the only one with this use case that really doesn't care about the latter.
Any cheap tricks that avoid heading towards queues, databases, or memory caches? Trying to keep it lightweight.
this is the expected behavior. this is why:
Sessions are identified using a cookie to store the session id, this allows your browser to pickup the same session on the next request.
As you long process is using sessions, it will not call session_write_close() until the whole process execution is complete, meaning the session is still open while the long process is running.
when you connect with another browser tab the browser will try and pickup the same session (using the same cookie) which is still open and running the long process.
If you open the link using a different browser you will see the page will load fine and not wait around for the session_write_close() to be called, this is because it's opening a separate session (however you will not see the text you want as it's a separate session)
You could try and manually write and close (session_write_close()) the session, but that's probably not the best way to go about things.
It's definitely worth looking at something like Gearman for this, there's not that much extra work, and it's designed especially for this kind of async job processing. Even writing status to the database would be better, but that's still not ideal.
This is the way I am thinking of using DB4o. When I need to query, I would open the file, read and close:
using (IObjectContainer db = Db4oFactory.OpenFile(Db4oFactory.NewConfiguration(), YapFileName))
{
try
{
List<Pilot> pilots = db.Query<Pilot>().ToList<Pilot>();
}
finally
{
try { db.Close(); }
catch (Exception) { };
}
}
At some later time, when I need to insert, then
using (IObjectContainer db = Db4oFactory.OpenFile(Db4oFactory.NewConfiguration(), YapFileName))
{
try
{
Pilot pilot1 = new Pilot("Michael Schumacher", 100);
db.Store(pilot1);
}
finally
{
try { db.Close(); }
catch (Exception) { };
}
}
In this way, I thought I will keep the file more tidy by only having it open when needed, and have it closed most of the time. But I keep getting InvalidCastException
Unable to cast object of type 'Db4objects.Db4o.Reflect.Generic.GenericObject' to type 'Pilot'
What's the correct way to use DB4o?
No, it's not a good idea to work this way. db4o ObjectContainers are intended to be kept open all the time your application runs. A couple of reasons:
db4o maintains a reference system to identify persistent objects, so it can do updates when you call #store() on an object that is already stored (instead of storing new objects) . This reference system is closed when you close the ObjectContainer, so updates won't work.
Class Metadata would have to be read from the database file every time you reopen it. db4o would also have to analyze the structure of all persistent classes again, when they are used. While both operations are quite fast, you probably don't want this overhead every time you store a single object.
db4o has very efficient caches for class and field indexes and for the database file itself. If you close and reopen the file, you take no advantage of them.
The way you have set up your code there could be failures when you work with multiple threads. What if two threads would want to open the database file at exactly the same time? db4o database files can be opened only once. It is possible to run multiple transactions and multiple threads against the same open instance and you can also use Client/Server mode if you need multiple transactions.
Later on you may like to try Transparent Activation and Transparent Persistence. Transparent Activation lazily loads object members when they are first accessed. Transparent Persistence automatically stores all objects that were modified in a transaction. For Transparent Activation (TA) and Transparent Persistence (TP) to work you certainly have to keep the ObjectContainer open.
You don't need to worry about constantly having an open database file. One of the key targets of db4o is embedded use in (mobile) devices. That's why we have written db4o in such a way that you can turn your machine off at any time without risking database corruption, even if the file is still open.
Possible reasons why you are getting a GenericObject back instead of a Pilot object:
This can happen when the assembly name of the assembly that contains the Pilot object has changed between two runs, either because you let VisualStudio autogenerate the name or because you changed it by hand.
Maybe "db4o" is part of your assembly name? One of the recent builds was too agressive at filtering out internal classes. This has been fixed quite some time ago. You may like to download and try the latest release, "development" or "production" should both be fine.
In a presentation I once did I have once seen really weird symptoms when db4o ObjectContainers were opened in a "using" block. You probably want to work without that anyway and keep the db4o ObjectContainer open all the time.
It is ok to reopen the database multiple times. The problem would be performance and loosing the "identity". Also you can't keep a reference to a result of a query and try to iterate it after closing the db (based on you code, looks like you want to do that).
GenericObjects are instantiated when the class cannot be found.
Can you provide a full, minimalist, sample that fails for you?
Also, which db4o version are you using?
Best
i am devloping a site using .net MVC
i have a data access layer which basically consists of static list objects that are created from data within my database.
The method that rebuilds this data first clears all the list objects. Once they are empty it then add the data. Here is an example of one of the lists im using. its a method which generates all the UK postcodes. there are about 50 methods similar to this in my application that return all sorts of information, such as towns, regions, members, emails etc.
public static List<PostCode> AllPostCodes = new List<PostCode>();
when the rebuild method is called it first clears the list.
ListPostCodes.AllPostCodes.Clear();
next it re-bulilds the data, by calling the GetAllPostCodes() method
/// <summary>
/// static method that returns all the UK postcodes
/// </summary>
public static void GetAllPostCodes()
{
using (fab_dataContextDataContext db = new fab_dataContextDataContext())
{
IQueryable AllPostcodeData = from data in db.PostCodeTables select data;
IDbCommand cmd = db.GetCommand(AllPostcodeData);
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.SelectCommand = (SqlCommand)cmd;
DataSet dataSet = new DataSet();
cmd.Connection.Open();
adapter.FillSchema(dataSet, SchemaType.Source);
adapter.Fill(dataSet);
cmd.Connection.Close();
// crete the objects
foreach (DataRow row in dataSet.Tables[0].Rows)
{
PostCode postcode = new PostCode();
postcode.ID = Convert.ToInt32(row["PostcodeID"]);
postcode.Outcode = row["OutCode"].ToString();
postcode.Latitude = Convert.ToDouble(row["Latitude"]);
postcode.Longitude = Convert.ToDouble(row["Longitude"]);
postcode.TownID = Convert.ToInt32(row["TownID"]);
AllPostCodes.Add(postcode);
postcode = null;
}
}
}
The rebuild occurs every 1 hour. this ensures that every 1 hour the site will have fresh set of cached data.
the issue ive got is that occasionally if during a rebuild, the server will be hit by a request and an exception is thrown. The exception is "Index was outside the bounds of the array." it is due to when a list is being cleared.
ListPostCodes.AllPostCodes.Clear(); - // throws exception - although its not always in regard to this list.
Once this exception is thrown application dies, All users are affected. I have to restart the server to fix it.
i have 2 questions...
If i utilise caching instead of static objects would this help ?
Is there any way i can say "while the rebuild is taking place, wait for it to complete until accepting requests"
any help is most appricaiated ;)
truegilly
1 If i utilise caching instead of
static objects would this help ?
Yes, all the things you do are easier done by the caching functionality that is build into ASP.NET
Is there any way i can say "while the
rebuild is taking place, wait for it
to complete until accepting requests"
The common pattern goes like this:
You request data from the Data layer
If the Datlayer sees that there is data in the cache, then it serves the data from cache
If no data is in the cache the data is requested from the db and put into cache. After that it is served to the client
There are rules (CacheDependency and Timeout) when the cache is to be cleared.
The easiest solution would be you stick to this pattern: This way the first request would hit the database and other requests get served from the cache. You trigger the refresh by implementing an SQLCacheDependency
You have to make sure that your list is not modified by one thread while other threads are trying to use it. This would be a problem even if you used the ASP.NET cache since collections are just not thread-safe. One way you can do this is by using a SynchronizedCollection instead of a List. Then make sure to use code like the following when you access the collection:
lock (synchronizedCollection.SyncRoot) {
synchronizedCollection.Clear();
etc...
}
You will also have to use locking when you read the collection. If you are enumerating over it, you should probably make a copy before doing so as you don't want to lock for a long time. For example:
List<whatever> tempCollection;
lock (synchrnonizedCollection.SyncRoot) {
tempCollection = new List<whatever>(synchronizedCollection);
}
//use temp collection to access cached data
The other option would be to create a ThreadSafeList class that uses locking internally to make the list object itself thread-safe.
I agree with Tom, you will have to do synchronization to make this work. One thing that would improve the performance is not clearing the list until you actually receive the new values from the database:
// Modify your function to return a new list instead of filling the existing one.
public static List<PostCode> GetAllPostCodes()
{
List<PostCode> temp = new List<PostCode>();
...
return temp;
}
And when you rebuild the data:
List<PostCode> temp = GetAllPostCodes();
AllPostCodes = temp;
This makes sure that your cached list is still valid while GetAllPostCodes() is executing. It also has the advantage that you can use a read-only list which makes the synchronization a bit easier.
In your case you need to refresh the data every one hour.
1) IT should use cache with absolute expiration set to 1 hour, so it expires after every 1 hour. Check the Cache before using it, by doing a NULL check.If its NULL get the data from DB and populate the Cache.
2) With above approach the disadvantage is that data can be stale by 1 hour. So if u want most updated data at all times, use SQLCacheDependency (PUSH). so whenever there is a change in the select command u r using, cache will be refreshed from the database with updated data.