I'm working with a local SQLITE db in an iOS (obj-c) project where we've chosen to use FMDB. Please note I'm not interested in changing to Core Data or switching from FMDB to a different library.
Everything is working, but when we first run the app data is downloaded from our server, json is parsed, and then sorted into the local db. Due to the amount of info coming in this takes a good 30 - 40 seconds. Ideally this would be done in the background, but FMDB does not allow for this (at least as far as I can tell) as it may cause issues with the db data integrity.
Has anyone successfully used the FMDB library running all queries, inserts, etc on a background thread? I've tried simply wrapping actions in:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
// ... query/insert/update/delete
});
...but I end up with db integrity issues. Any tips or examples would be great. Thanks in advance.
#nawar - Sorry for not posting this sooner. I ended up using a single instace of FMDatabaseQueue (instead of FMDatabase) within the dispatch_async block. Works like a charm. The issues I was running into were due to using FMDatabase.
dispatch_async(dispatch_get_global_queue(0, 0), ^{
[self.dbQueue inTransaction:^(FMDatabase *db, BOOL *rollback) {
BOOL result = YES;
for (int i = 500; i < 1000; i++) {
result = [db executeUpdate:#"insert into testTable (name) values(?)",[NSString stringWithFormat:#"name-%d",i]];
if (!result) {
NSLog(#"break");
*rollback = YES;
break;
}
}
}];
});
Related
My ios application use FMDB framework to work with SQLite database. For updating database from server I use FMDatabaseQueue:
- (BOOL)executeUpdate:(NSString *)update
{
FMDatabaseQueue *dbQueue = [FMDatabaseQueue databaseQueueWithPath:databasePath];
__block BOOL success = NO;
[dbQueue inTransaction:^(FMDatabase *db, BOOL *rollback) {
success = [db executeUpdate: update];
}];
return success;
}
This approach allow me use multiple threads for working with database. But queries will be executed in the order they are received. When I update the big piece of data in database other queries(like getting some data for UI thread) must wait too long.
Is there a way to allow other queries read commited data while updating is in progress?(Like JDBC Isolation level TRANSACTION_READ_COMMITTED)
Reducing the isolation level works only in shared cache mode, which does not make sense on a mobile device.
To increase concurrency, i.e., to allow readers and a writer to access the database at the same time, enable WAL mode.
I want to know how to handle very large DBs in iOS?
Explanation:
I am making an app for the company, it will be holding all daily transactions and operations (this means few hundreds to a couple thousands a day).
The app in fact just displays the daily transactions, but keeps everything in the DB just in case (so I can not delete them).
As there is a need for the app to be highly reactive, I am searching for a way to efficiently load (Core Data Fetch) just the latest operations, without being slowed down in the long run when the DB gets really large.
Knowing that whenever an operation is updated, I refresh the displayed table immediately.
I can't find any article dealing with this on iOS, so I can't find any good practice or a correct way to handle it.
On a MySQL server, I'd do like this:
* Set up a DB for daily "active" operations, when done, delete them and move them to a "finished" operations DB.
* Set up a Cron job to dump/backup daily/monthly DBs by night, and having a fresh lightweight one for the next day.
Sadly, it doesn't seem possible to create a dynamic table on iOS, and even creating a "finished" DB, this will still require accessing a heavy DB at each closed operation, which can lead to a low memory crash.
actually I think you can try sqlite api to match the same operations in your MySQL server. However, you will have to drop the CoreData and rewrite everything instead, i.e., TableView, TableCell, etc.
to deal with the DataSource like this:
sqlite3 * tradeDB;
sqlite3_open_v2([path UTF8String], &tradeDB, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nil);
sqlite3_stmt * recordset;
int res = sqlite3_prepare_v2(tradeDB, "select count(*) from someTable", -1, &recordset, nil);
if (res!=SQLITE_OK) {
return nil;
}//end if
res = sqlite3_step(recordset);
if (res!=SQLITE_ROW) {
_result = nil;
}else{
char * res_char = (char*)sqlite3_column_text(recordset, 0);
if (!res_char)
_result = nil;
else
_result = [NSString stringWithUTF8String:res_char];
//end if
}//end if
sqlite3_finalize(recordset);
if (!_result) {
_result = #"0";
}//end if
return self;
I am using Sqlite as backend in App. I have some confusions. I am using a Class for using Sqlite say "DatabaseManager". In this class I have declared all methods(to access data from DB) as static. In Multithreading is it fine to make all methods static or "DatabaseManager" should be a singleton class or should be a normal class?
I am facing problem like one thread is closing the Database after use while other thread(also accessing the DB) is in between then App Crash with message "Excess Bad"
I am using NSOperationQueue and NSOperation for Multithreading instead of GCD. So that i can cancel the operation if required.
Actually in App there is some calculation part which needs data from more then one table in DB. if i do that in Main thread the it takes time or Table Scrolling become smoothless.
Please Guide me.
SQlite database is thread safe. "threadsafe" we mean that you can use different SQLite database connections in different threads at the same time.
It has never been safe to use the same database connection simultaneously in multiple threads.
One of the good solution is to Test for SQLITE_BUSY
pseudo-code to illustrate a solution:
continueTrying = YES;
while (continueTrying)
{
retval = sqlite_exec(db, sqlQuery, callback, 0, &msg);
switch (retval)
{
case SQLITE_BUSY:
Log("[%s] SQLITE_BUSY: sleeping for a while...", threadName);
sleep a bit... (use something like sleep(), for example) // -> usleep(20);
break;
case SQLITE_OK:
continueTrying = NO; // We're done
break;
default:
Log("[%s] Can't execute \"%s\": %s\n", threadName, sqlQuery, msg);
continueTrying = NO;
break;
}
}
return retval;
Other solution is to use serial queue:
databaseQueue = dispatch_queue_create("com.company.app.yourdatabase", 0);
dispatch_sync(databaseQueue, ^{
// your database activity
});
I'm using two different types of fmdb connections in my app:
FMDatabase for all READ queries and
FMDatabaseQueue for all UPDATE queries.
Both are handled by a singleton, which keeps both types open the whole time while the app is running.
Both, read and update queries, are used in different threads as some tasks in my app are progressed in background; like getting data from a server and inserting it in the db via FMDatabaseQueue in an own background thread - while reading some information from db via FMDatabase and updating an ViewController with it on the main thread.
My problem is that after inserting data into the db via FMDatabaseQueue the second connection (FMDatabase) does not return the updated information as it does not find them. But I know the data was inserted as I have checked the db with an db browser tool + no errors occur while inserting it. To avoid this, I have to close the FMDatabase db connection and reopen it to see the changes made by the other connection. Unfortunately when my app starts up there are a many inserts, updates + reads as a lot of new data is loaded from server which needs to be processed - so closing and opening the db every time an update was made occurs in many "database busy" messages.
I have used one single FMDatabaseQueue for all threads and executes (read, update) before but it was quite slow when using read queries with __block variables to get the resultset out of the callback while another thread does some inserts(between 50-100 in a single transaction).
On top of it the database is encrypted via sqlcipher - not sure if it's important but want to mentioned it. So every time i have to close and open the database I'm doing a setKey.
My question: Is it possible to use a setup with two different connection types on multiple threads and if yes, do I have to close and open the FMDatabase connection? Or is there a better solution for this usecase?
UPDATE
My code to perform an insert / update looks like
-(void) create:(NSArray *)transactions
{
NSMutableString *sqlQuery = [[NSMutableString alloc] initWithString:STANDARD_INSERT_QUERY];
[sqlQuery appendString:#"(transaction_id, name, date) VALUES (?,?,?)"];
FMDBDataSource *ds = [FMDBDataSource sharedManager];
FMDatabaseQueue *queue = [ds getFMDBQ];
[queue inTransaction:^(FMDatabase *db, BOOL *rollback) {
[db setKey:[ds getKey]]; // returns the key to decrypt the database
for (Transaction *transaction in transactions)
{
[db executeUpdate:sqlQuery, transaction.transactionId, transaction.name, transaction.date];
}
}];
}
and a read query
-(Transaction *)read:(NSString *)transactionId
{
NSString *sqlQuery = [[NSString alloc] initWithString:STANDARD_SELECT_QUERY];
Transaction *transaction = nil;
FMDBDataSource *ds = [FMDBDataSource sharedManager];
FMResultSet *rs = [[ds getFMDB] executeQuery:sqlQuery];
while ([rs next]) {
transaction = [[Transaction alloc] init];
[transaction setTransactionId:[rs stringForColumn:#"transaction_id"]];
[transaction setName:[rs stringForColumn:#"name"]];
}
[rs close];
return transaction;
}
The FMDBDataSource is a singleton holding both, FMDatabase and FMDatabaseQueue, connections
- (FMDatabaseQueue *)getFMDBQ
{
if (self.fmdbq == nil)
{
self.fmdbq = [FMDatabaseQueue databaseQueueWithPath:[self getDBPath]];
}
return self.fmdbq;
}
- (FMDatabase *) getFMDB
{
if(self.fmdb == nil)
{
self.fmdb = [FMDatabase databaseWithPath:[self getDBPath]];
[self openAndKeyDatabase]; // opens the db and sets the key as the db is encrypted
}
return self.fmdb;
}
As I said, when using this code the FMDatabase connection does not get the information which was inserted via FMDatabaseQueue.
Personally, I would suggest using the single FMDatabaseQueue for both threads and let the queue coordinate the actions on the two threads. That's what it was created for. It completely eliminates those "database busy" problems.
On your performance update, if doing a bulk update, are you using the FMDatabase method beginTransaction before the update and commit at the end? Or use the inTransaction method. Inserting 10,000 records without transactions in my test takes 36.8 seconds, but with transactions it takes 0.25 seconds.
Or, if your bulk update is slow by necessity (e.g. you're downloading some big data source from a web service using some streaming protocol), you can either:
Load all the results into memory first, with no database interaction, and then use the bulk update with transactions as described in the previous paragraph; or
If your database updates are necessarily constrained by a slow network connection, then use separate inDatabase calls so that it doesn't tie up the FMDatabaseQueue while downloading data from your web service.
Bottom line, through the use of transactions or the judicious use of separate inDatabase calls, you can minimize how long your background operation is ties up the FMDatabaseQueue and you can achieve synchronized multi-threaded interaction with your database without blocking your UI too significantly.
I have spent lots of hours trying to making the same, when I found your post.
I do not know if you have already find a solution.
After a lot of tries and searches, I had to gave up and I'am looking for another solution now.
But, I would like to share my conclusions.
SQLite has a function to define READONLY or READWRITE mode. FMDB implements like openWithFlags.
[db openWithFlags:SQLITE_OPEN_READONLY|SQLITE_OPEN_NOMUTEX];
Setting these flags do not permits reading while wrinting even if we set these flags.
I could accomplish reading+writing (different connections) setting my database to use WAL journal_mode (http://www.sqlite.org/wal.html).
BUT, SQLCipher screw everthing up.
Conclusion READING + WRITING with 2 connections:
FMDB + openWithFlags = BE SAD AND ANGRY
FMDB + openWithFlags + WAL jornal_mode = BE HAPPY
FMDB + SQLCipher + openWithFlags = BE SAD AND ANGRY
FMDB + SQLCipher + openWithFlags + WAL jornal_mode = BE SAD AND ANGRY
As my application needs security, I do not know what to do yet.
Well, I hope it helps.
Best
Hami.
Trying to implement an app which sends offline data stored on local db to web server when connected to internet. I use the code shown below. As far I have tested it works fine, not sure it will work fine for huge number of records. I would like to know whether any tweaking on this code may increase the performance???
NOTE
I know this would be a worst code for offline sync purpose, so trying
to tweak it better.
Its a single way synchronization, from app to server.
-(void)FormatAnswersInJSON {
DMInternetReachability *checkInternet = [[DMInternetReachability alloc] init];
if ([checkInternet isInternetReachable]) {
if ([checkInternet isHostReachable:#"www.apple.com"]) {//Change to domain
responseArray = [[NSMutableArray alloc] init];
dispatch_async(backgroundQueue, ^(void) {
NSArray *auditIDArray = [[NSArray alloc] initWithArray: [self getUnuploadedIDs]];
for (int temp = 0; temp < [auditIDArray count]; temp ++) {
// Code to post JSON to server
NSURLResponse *response;
NSData *urlData=[NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];
if (!error) {
NSString *responseID = [[NSString alloc]initWithData:urlData encoding:NSUTF8StringEncoding];
if ([responseID isEqualToString:#"ERROR"]) {
//Error uploading records
} else {
[responseArray addObject:responseID];
}
} else {
//Error
return;
}
}
dispatch_async( backgroundQueue, ^{
/* Based on return code update local DB */
for (int temp = 0; temp < [responseArray count]; temp ++) {
[self updateRecordsForID:[auditIDArray objectAtIndex:temp] withID:[responseArray objectAtIndex:temp]];
}
});
});
}
}
}
- (void)upload { //Called when internet connection available
if(backgroundQueue){
dispatch_suspend(backgroundQueue);
dispatch_release(backgroundQueue);
backgroundQueue = nil;
}
backgroundQueue = dispatch_queue_create("com.XXXX.TestApp.bgqueue", NULL);
dispatch_async(backgroundQueue, ^(void) {
[self FormatAnswersInJSON];
});
}
If this code were sitting in front of me, my approach would be:
Look at the use cases and define 'huge number of records': Will 50 record updates at a time occur regularly? Or will it be in 1s and 2s? Do my users have wifi connections or is it over the paid network?, etc.
If possible, test in the wild. If my user base was small enough, gather real data and let that guide my decisions, or only release the feature to a subset of users/beta tests and measure.
If the data tells you to, then optimize this code to be more efficient.
My avenue of optimization would be doing group processing. The rough algorithm would be something like:
for records in groups of X
collect
post to server {
on return:
gather records that updated successfully
update locally
}
This assumes you can modify the server code. You could do groups of 10, 20, 50, etc. all depends on the type of data being sent, and the size.
A group algorithm means a bit more pre-processing client side, but has the pro of reducing HTTP requests. If you're only ever going to get a small number of updates, this is YAGNI and pre-mature optimization.
Don't let this decision keep you from shipping!
Your code has a couple of issues. One convention is to always check the return value before you test the error parameter. The error parameter might be set - even though the method succeeded.
When using NSURLConnection for anything else than a quick sample or test, you should also always use the asynchronous style with handling the delegate methods. Since using NSURLConnection properly may become quickly cumbersome and error prone, I would suggest to utilize a third party framework which encapsulates a NSURLConnection object and all connection related state info as a subclass of NSOperation. You can find one example implementation in the Apple samples: QHTTPOperation. Another appropriate third party framework would be AFNetworking (on GitHub).
When you use either the async style with delegates or a third party subclass, you can cancel the connection, retrieve detailed error or progress information, perform authentication and much more - which you can't with the synchronous API.
I think, once you have accomplished this and your approach works correctly, you may test whether the performance is acceptable. But unless you have large data - say >2 MByte - I wouldn't worry too much.
If your data becomes really large, say >10 MByte you need to consider to improve your approach. For example, you could provide the POST data as file stream instead a NSData object (see NSURLRequest's property HTTPBodyStream). Using a stream avoids to load all the POST data into RAM which helps alleviate the limited RAM problem.
If you have instead smaller POST data, but possibly many of them, you might consider to use a NSOperationQueue where you put your NSOperation connection subclass. Set the maximum number of concurrent operations to 2. This then may leverage HTTP pipelining - if the server supports this, which in effect reduces latency.
Of course, there might be other parts in your app, for example you create or retrieve the data which you have to send, which may affect the overall performance. However, if your code is sound and utilizes dispatch queues or NSOperations which let things perform in paralel there aren't many more options to improve the performance of the connection.