I am using gridx with dojo 1.8.3 library. My application requires a dojo grid to be refreshed every second with the data which comes from server.
Following is the code to refresh grid with creating a new store for every refresh.
The result comes in xhrPost callback and gets assigned to new grid store.
refreshGrid(){
if(grid.store!=null)
grid.store.close();
var newGridStore = new dojo.data.ItemFileWriteStore({
url:'',
data:result,
urlPreventCache: false
});
grid.setStore(newGridStore);
}
Above function is getting called for every 2 seconds and I could see memory increase in chrome profile.
I had tried using the way to iterate through the store to delete all the items followed by adding new items but when store is updated grid gets locked.
Which is the correct way of grid refresh in dojo?
Nothing you are doing looks sketchy, set store is the way to go unless you are refreshing specific items. The old store should eventually be deleted by the gc, but it does this delete whenever it feels like it and you will see a ramp up in memory usage until that point. In chrome you can force the garbage collector, so if you do this and your memory clears up then everything is running as intended and you may just be dealing with too much data too often.
It might also be useful to let us know what kind of performance decrease (if any) you are seeing, what your memory usage is ranging from, and what amount of data you are dealing with in your store.
Related
the code i'm working on makes heavy usage of TFDMemTables, and clones of those tables using CloneCursor.
Sometimes, under specific conditions which I am unable to identify, the source table and its clone become out of sync: the data between them may be different, the record count as well.
Calling Refresh on the cloned table puts things back in order.
From my understanding, CloneCursor is used to address the same underlying memory where data is stored, meaning alterations to the underlying data from any of the two pointers should reflect on the other table, yet allow the user to have separate filter / record positioning per "view". so how can it possibly go out of sync?
I built a small simulator, where I can insert / delete / filter records in either the table or its clone, and observe the impact on the other one. Changes were reflected correctly.
Another downside of Refresh is that it slows the execution tremendously, if overused.
Has anyone faced similar issues or found explanations / documentation regarding this matter?
Edit:
to clarify what I mean by "out of sync", it means reading a value from the table using FieldByName will return X prior to Refresh, and Y post-refresh. I was not able to reproduce this behavior on the simulator mentioned above.
My question surrounds around one single point - data management in mobile application. I have created a mobile application where data comes from server. The data includes both text and images. Following are the steps I am doing for this :
First launch :
1. Get server data.
2. Save server data in Sqlite database.
3. Show Sqlite data.
Next launches :
1. Show Sqlite data.
2. Get server data in background.
3. Delete previous Sqlite data.
4. Save new server data in Sqlite database.
5. Show Sqlite data.
I have couple of questions on these steps :
1. Is this the right approach ? Other way could be showing data every time from server but that would not display the data on screen immediately (depending on internet speed).
2. I also thought of comparing the Sqlite data with the new server data. But faced a big challenge. The new server data might have new records or deleted records. Also, I could not find an appropriate approach to compare each database field with JSON data.
So what is the best approach to compare local Sqlite data with new server data ?
3. Each time I delete the Sqlite data and insert new data and then refresh the screen (which has a UITableView), it blinks for a second which is obvious. How to avoid this issue if steps 3, 4, 5 are followed ?
4. How should I proceed with data update in case I come back on the screen each time or when the application becomes active ? I am very aware of NSOperationQueues or using GCD for that matter. But what if I am crazy and go back and forth to screen again and again. There will be a number of NSOperations in the queue.
It's a challenge to synchronise server data, I've done that before, and if you can spend time on it I'd say it's the best solution.
You may need creation and modification dates on both server and local objects, to compare them - this will let you decide which objects to add, update and delete.
If the server sends you only the recently updated objects you can save a lot of traffic and improve performance (but deleted objects will be harder to detect).
If the data is only changed in the server it's easier, when the app can change the data too it becomes more complicated (but it seems that it's not your case). It also depends on how complex the database is, of course.
If you don't want to invest some time in doing this, just fetching all data everytime works too, even if it is not ideal! Instead of showing the old data and blinking it, you can just make the user wait 2-3 seconds when entering, while you get the new data. Or instead you can fetch the data only when starting the app, and so when you get to that view controller it will be ready already.
It's a complex problem that everyone faces at some point, so I'm curious to see what other people will suggest :)
This is a good question.
I personally think downloading data, store locally and later try to sync is a dangerous scenario. Easy to introduce bugs, master <-> slave issues (what data should be master, if multiple devices would be used etc.)
I think something like this could be a working approach:
1. I would try to look at possibilities to lazy load the data from the server on-demand. That is when a user have a View that should display data, load that specific data with the creation of that specific View. This ensures the data is allways in sync.
2. Tackling the need to reload data from server from every view, could be done by simply storing the downloaded data as objects in memory (not using SqlLite). The view will try to load the needed data trough your cache manager, and it would serve it from memory, if available. If not in memory simply get the data from your server and add it to your memory cache.
The memory cache could be a home made data manager wrapping a Dictionary stored on you AppDelegate, or some global "Singelton" to wrap the cache management/storing and data loading.
3. With lazy loaded data and memory cache you would need to make sure any updates (changes, new records, deleted records) updates your memory data model, as well as pushing these changes to the server as soon as possible. Depending on data size etc. you could force the user to wait, or do it directly as background process.
4. To ensure the data is in sync, you should make sure that you periodically invalidate (delete) the local memory records in the cache and thereby force data updates from the server. Best approach would probably be to have a last updated timestamp for each record in the memory cache. So the periodical invalidator would only delete "old records" from the memory cache (once again not from the server).
To save server from unnecessary data load, the data should still load on demand when the user needs it in a view, and not as part of "cache invalidation".
5. Depending on the data size you might need to look at "cache invalidation". Could be as simple as when xx records are stored, start deleting old objects from memory cache (not server, only locally on device).
6. If data sync is absolutely critical you might want to look at refreshing your memory cache for a record, just before you allow the user to change data. E.g. when user taps "Edit" or similar, you grab the latest data from server for that record. This is just to make sure the user is not going to update a record using outdated data and thereby accidentally overriding any changes made remote, or on another device etc.
--
My take on it. I do not believe there is a "perfect right way" to do this. But this would be what I would try to do.
Hope this will help with some ideas and inspiration.
How about this:
If data exists in SqlLite, load into "in-memory" copy and show it
In background load new server data
delete old sqlite data if it exists (note that the in-memory copy remains)
save new server data to sqlite
load new sqlite data into "in-memory" copy and show it.
If no data was found in step 1, display a "loading" screen to the user during step 2.
I'm making the assumption that the data from SqlLite is small enough to keep a copy in memory to show in your UITable view (The UITable view would always show data from in-memory).
It may be possible to combine steps 4 and 5 if the data is small enough to hold two copies in memory at the same time (you would create a new in-memory copy and swap with the visible copy when complete).
Note:
I don't talk about error handling here, but I would suggest that you don't delete the sqlite data until you have new data to replace it with.
This approach also eliminates the need to determine if this is the first launch or not. The logic always remains the same which should make it a little easier to implement.
Hope this is useful.
You can do same things more efficiently by MultiVersion Concurrency Control (MVCC), which uses a counter (sort of a very simple "time stamp") for every data record, which is updated whenever the record is changed means you need to get those data which is Updated after last sync call that reduces lots of redundant data and bandwidth.
Source: MultiVersion Concurrency Control
I'm using a UITableView which hooks into a rest API.
On first launch the app retrieves the data the UITableView will display and parses it in to a Core Data database.
This works fine for small datasets. But when the dataset grows to above 300-500 items it does not perform very well. Taking minutes to finish downloading+parsing. The app isn't deadlocked during this time, but the user likely won't wait for the parsing to complete.
I then decided to use paging. So now, I only retrieve the latest 20 items, and then the user can click "Load more" to go back further. The data is cached.
This seems to work well except for one problem.
Because I'm not downloading all the data on each load, I cannot tell when an item has been deleted on the server and I cannot tell when an item has changed (say the title may have changed).
Can anyone provide me with any suggestions to resolve this?
Thanks.
We routinely request a similar number of items and display it in a table view. However in our case the API returns JSON and we store it in model objects, not Core Data. Once the data is downloaded it takes less than a second to go from JSON to appearing in the table. Core Data is a bad idea for anything that isn't actually a database, or that isn't preserved for a past a user session. But you need to identify which part of your transaction is actually taking the most time. In our case it's the backend behind the API, but once it shows up everything is quite fast.
Also, in our case the data is around 700K and we are going to GZIP it soon to minimize the network time even further.
I have an issue I'm hoping someone can help with. I have an application that, for all intents and purposes, is working great. It's basically a picture viewer type of application - for something very specific. What it is is about 500 pictures.
I have all pictures set as Content and I load/unload one at a time. For the 500 pictures, I have a class that is used as data about each picture. So things like "place taken" , "index", "short description", etc. I never need to insert or delete from this list, but I may need to make some changes to each individual one, like "user viewed this picture on..." (date) or "favorite = true" (boolean where user marks a picture as a favorite picture).
When I deploy the app, this "picture metadata" is in xml file. It is then deserialized and saved to IsoStorage upon first run. A copy of it is maintained in memory and that is what is used to run my whole app. I have 3 different pages that all use that data, which is set as a static property in app.xaml.cs. Upon Deactivate/Closing the data is serialized back to xml - upon relaunching it is deserialized. Everything works fine and fast - everywhere. Including tombstoning.
The problem is resuming from Deactivation where the app is not tombstoned. It can take up to 10-15 seconds and definitely is returning from e.IsApplicationInstancePreserved in Application_Activated (i.e. it's not tombstoned).
Activating from brand new it takes about 3-4 seconds for the app to start. Returning from tombstoning it also takes about 3 seconds.
What I'm not understanding is why returning from e.IsApplicationInstancePreserved = true; is taking so long (and it won't allow me to pass certification). I've tested and found that if it is about 10 items in the List its incredibly snappy for FAS. If there are about 50 items in the List then it's not immediate. If there are 100 items, it's the first time you can see "Resuming..." (yep, that word coming from FAS, not tombstone). Where I have it, at 500 in the List, it is painfully slow to watch FAS, which is SAS.
It's interesting that in the emulator, FAS works perfectly fine, even with 1000 objects in memory. It's on an actual device where it is incredibly slow (Samsung Focus) in both debug and release mode.
Now I know the easy answer may be something like "why maintain a class with a list of 500 objects all the time?", but my whole architecture and user experience is based on having data about the pictures available on all three pages all the time. Linq is heavily used to report data everywhere.
Any thoughts or guidance on this situation?
Great to hear you found the problem.
"sounds a serialization/deserialization issue to me. Are you storing the list in a State 'variable' ? were the framework will serialize when leaving the app and deserialize it when you return"
Remember, your PC + Emulator is MUCH faster and more powerful than your phone's 1GHz single core ARM processor with 500MB RAM!
When your app is activated with e.IsApplicationInstancePreserved == true, then you shouldn't need to restore any state and you should be able to pick up where the user left off.
If you're saying that when your list catalogs 500 items, the performance is horribly slow when resuming the app, I wonder whether you're also keeping the 500 images in memory and your phone is having to swap all that data into your app's memory space. This will be further compounded if your app has multiple pages each containing copies of your 500 images!
I strongly encourage you to profile your app to measure its memory footprint and to see where the big memory, storage, IO and perf issues are.
We have a web service iPhone app which fetches short texts (no graphics) in increment of 10 records - user can load more records by pressing 'load more' button. (akin to 'load 25 more' in default 'App Store' App)
Currently, NSXMLParser is used to parse the XML, the records are stored then in NSMutableArray, and new records (from 'load more') are appended to the array.
Now, we are experiencing two problems, which we believe are related to our current method of storing the records into the RAM. These are:
In low memory conditions, when we switch back to the App from app switcher, and then press 'back' button from the detail view, App either crashes or reloads the table view!
After having just 400 records loaded the app crashes!
To combat these issues, will, just storing the records into SQLite (or Core Data) instead of an array, do it? Or we need to do something more/different?
Please guide!
Thanks!
I am able to show more than 12000 records with sizable text & images in each row. I suspect you are having some other problem?
Have you profiled your code? Try Instruments. Also when you launch & come back from App switcher, how are you handling viewWillDisappear and viewDidDisappear? Are you dealloc or release something when you switch views?
You should override didReceiveMemoryWarning: in your view controller if you do not want the view deallocated when a memory warning occurs. It could be due to that. As well as that, your viewDidLoad method should recreate the view state ensuring that no interface element is left deallocated or nil.
If you allocate too much memory in one go, the iOS Watchdog service will kill your app because it suspects the app will cause the amount of memory allocated to reach 100% (after which, bad things happen). It's also possible that if you use the UITableView methods to add extra records onto the end of the list (insertRowsAtIndexPaths:withRowAnimation:) it could be throwing out an NSInconsistencyException under heavy and/or repeated usage. If your loading methods are asynchronous, then maybe it just wasn't ready in time or there was a network failure, and the insert call for your array didn't insert the full number of rows, which will crash the app if the number of rows doesn't match your data source method numberOfRowsInSection:.
resolved it... after reading #Steven's comment... I got convinced that it was not memory issue... so, I digged further, & found the issue.
Actually, I have a deselectRowAtIndexPath method in tableview. And in low memory environment, iPhone releases the records stored in memory... so, in that situation, deselectRowAtIndexPath was pointing to nothing, which was causing the crash... modified & resolved it now!
Thanks!