I'm using an sqlite database in my iOS app directly, without taking advantage of Core Data. When the app is launched, I need it to insert several sets of table rows of considerable amount of data that will be needed during the usage of the app and that are fixed sets of data (for example, I have a database table for a list of countries the user could choose in the app). I've those data insertions in an .sql file in the app and, since I need this data to be available in the database from the very first time the user installs and launches the app, I wrote a method that firstly copies the database to Documents, and then reads the file and performs the database insertions, and I call this method in application:didFinishLaunchingWithOptions:.
However, I'm finding that performing a lot of insertions is really a slow operation. I proceed this way:
+ (void)insertFileContent:(NSString *)fileContent intoDatabase:(sqlite3 *)database
{
NSArray *lines = [fileContent componentsSeparatedByString:#"\r\n"];
for (NSString *line in lines) {
const char *sqlStatement = [line UTF8String];
sqlite3_stmt *compiledStatement;
if (sqlite3_prepare_v2(database, sqlStatement, -1, &compiledStatement, NULL) == SQLITE_OK) {
if(sqlite3_step(compiledStatement) == SQLITE_DONE) {
// Insertion was ok
}
else {
NSLog(#"%d",sqlite3_step(compiledStatement));
}
sqlite3_finalize(compiledStatement);
}
}
}
Database is opened only once before calling this method, and closed when it has returned. As I said, I have a lot of data I need to insert into database to make the app work properly, and it is taking even minutes to finish. I've been looking for a way to reduce this time significantly, but I didn't succeed.
How could I handle this? Is there a way to make this operations faster, and/or is there a way to insert data into the database at the time the user installs the app, instead of waiting til launch time?
Thanks in advance
You have the transaction overhead once for each statement.
Wrap all statements into a single transaction.
Execute BEGIN/END manually, or just write them into your .sql file.
Related
I've got a JSON file which contains more than 2000 arrays, and in each array there may have an array with 5 strings.
I'm running two for() loops in one function, so I think the 2ed one will be run after the 1st one is done?
and it sometimes appears the following(random data error appear):
database is locked
when I'm executing
INSERT INTO problemTags(problem_IDIndex)VALUES('93E')
And I use the code below to execute my SQL statement
+ (void)execSqliteWithSQL:(NSString *)sql{
sqlite3 *sqlite = nil;
int openResult = sqlite3_open([DBPath UTF8String], &sqlite);
if(openResult != SQLITE_OK){
NSLog(#"open DB error");
}
char *error;
int execResult = sqlite3_exec(sqlite, [sql UTF8String], nil, nil, &error);
if(execResult != SQLITE_OK){
NSString *errorString = [NSString stringWithFormat:#"\n execResult error=>\n %s \n SQL:\n%#\n",error,sql];
NSLog(#"%#",errorString);
}
sqlite3_close(sqlite);
}
I'm thinking about the reason to this issue is that
when I'm running the for loop, it sends several SQL statement to the execSqliteWithSQL function and when it can't run fast enough the data base would be closed?
question added below:
without using threads
the two for loops is like below:
-(void)......{
for(){
//sending SQL to execSqliteWithSQL
}
for(){
//sending SQL to execSqliteWithSQL
}
}
A couple of thoughts:
Do you possibly have multiple threads calling this execSqliteWithSQL concurrently? If so, errors like this are very likely.
You could remedy this by setting up a dedicated serial queue to which execSqliteWithSQL would dispatch its calls, thus eliminating attempts to do this concurrently.
Regardless, I would advise against opening and closing the database for each call. That's horribly inefficient.
In fact, if doing a bunch of update/insert statements, you not only don't want to open and close the database for every call, but rather you want to open, call BEGIN TRANSACTION, do your updates, and then COMMIT those changes. The performance difference is startling.
This execSqliteWithSQL clearly presupposes that you're manually building your SQL (presumably with stringWithFormat). That's problematic at best, dangerous at worst. If a quotation mark appears in a string value being inserted into the database, the SQL will fail. If this feed includes any user input, you are susceptible to SQL injection attacks.
Instead, you should use sqlite3_prepare_v2 to prepare a SQL statement with ? placeholders, and then use sqlite3_bind_xxx to bind values to those placeholders.
I am trying to insert dummy data into my sqlite table. How could I insert this properly and what am I doing wrong? Thanks, all help and advice is appreciated
+(void)setAllValues {
static sqlite3_stmt *insertStatement;
const char *sqlInsert = "insert into session_descriptions (sessionID, sessionTitle, sessionDescription) values (1, 'Programming Title' 'Programming description')";
if(sqlite3_prepare_v2([DBHelper getDB], sqlInsert, -1, &insertStatement, NULL) != SQLITE_OK)
{
NSLog(#"Insert statement not working");
}
sqlite3_finalize(insertStatement);
sqlite3_close([DBHelper getDB]);
}
A couple of issues:
You're checking to see if sqlite3_prepare_v2 failed, but not bothering to check why it failed. This is accomplished by logging sqlite3_errmsg:
if (sqlite3_prepare_v2([DBHelper getDB], sqlInsert, -1, &insertStatement, NULL) != SQLITE_OK) {
NSLog(#"prepare failed: %s", sqlite3_errmsg([DBHelper getDB]));
}
It's important to check sqlite3_prepare_v2 for success, and if not, immediately log sqlite3_errmsg, because most SQLite errors are detected during the preparation of the statement (and if you do any further SQLite calls before examining the error message, any meaningful error message might be replaced with some less-illuminating message).
Even if the prepare succeeded, you're not performing the SQL query. So after the above "prepare" statement, you should "step" it, in order to perform it:
if (sqlite3_step(insertStatement) != SQLITE_DONE) {
NSLog(#"step failed: %s", sqlite3_errmsg([DBHelper getDB]));
}
Do this after sqlite3_prepare_v2 but before sqlite3_finalize.
You are closing the database. This is a bit suspect because you generally want to leave the database in whatever state it was when you called this routine. So, I'd might expect the database to not be open, in which case, this routine would open, prepare, step, finalize, and then close. Or I might expect the database to be open already, in which case you'd merely prepare, step, and finalize, but not close the database, leaving the database in the same state as it was when you called this routine.
As an aside, I would advise opening the database once and keeping it open as long as the app is alive, but either approach works. Just make sure your opens and closes are properly balanced.
Your INSERT statement is missing a comma.
I want to know how to handle very large DBs in iOS?
Explanation:
I am making an app for the company, it will be holding all daily transactions and operations (this means few hundreds to a couple thousands a day).
The app in fact just displays the daily transactions, but keeps everything in the DB just in case (so I can not delete them).
As there is a need for the app to be highly reactive, I am searching for a way to efficiently load (Core Data Fetch) just the latest operations, without being slowed down in the long run when the DB gets really large.
Knowing that whenever an operation is updated, I refresh the displayed table immediately.
I can't find any article dealing with this on iOS, so I can't find any good practice or a correct way to handle it.
On a MySQL server, I'd do like this:
* Set up a DB for daily "active" operations, when done, delete them and move them to a "finished" operations DB.
* Set up a Cron job to dump/backup daily/monthly DBs by night, and having a fresh lightweight one for the next day.
Sadly, it doesn't seem possible to create a dynamic table on iOS, and even creating a "finished" DB, this will still require accessing a heavy DB at each closed operation, which can lead to a low memory crash.
actually I think you can try sqlite api to match the same operations in your MySQL server. However, you will have to drop the CoreData and rewrite everything instead, i.e., TableView, TableCell, etc.
to deal with the DataSource like this:
sqlite3 * tradeDB;
sqlite3_open_v2([path UTF8String], &tradeDB, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nil);
sqlite3_stmt * recordset;
int res = sqlite3_prepare_v2(tradeDB, "select count(*) from someTable", -1, &recordset, nil);
if (res!=SQLITE_OK) {
return nil;
}//end if
res = sqlite3_step(recordset);
if (res!=SQLITE_ROW) {
_result = nil;
}else{
char * res_char = (char*)sqlite3_column_text(recordset, 0);
if (!res_char)
_result = nil;
else
_result = [NSString stringWithUTF8String:res_char];
//end if
}//end if
sqlite3_finalize(recordset);
if (!_result) {
_result = #"0";
}//end if
return self;
I am trying to insert around 500 records into the sqlite database, it is taking 20 sec, which is way too much! Could anyone please tell how to optimize this.
for (int i = 0 ; i< [[[dic objectForKey:#"response" ]objectForKey:#"requete"]count]; i++) {
[sqlManager executeQuery:[[[dic objectForKey:#"response"]objectForKey:#"requetes"]objectAtIndex:i]];
}
with
-(void)executeQuery:(NSString*)_query
{
sqlite3 *database;
if(sqlite3_open([databasePath UTF8String], &database) == SQLITE_OK) {
if(sqlite3_exec(database, [_query UTF8String],NULL, NULL, NULL) == SQLITE_OK){
//NSLog(#"Query Success");
}
else{
// NSLog(#"Query Failed");
}
sqlite3_close(database);
}
sqlite3_close(database);
}
Open once and leave open until the app is closed or you're done with a DB-intense phase. SQLite can easily handle the situation if your app "crashes" with the DB open.
When doing multiple inserts close together, where it's fine if they either all go or none of them go, start an transaction (and commit it when done). When you do not explicitly start a transaction then SQLite does an implicit start/commit around each insert or update, "forcing" the data to "disk". This takes A LOT of time. Doing one start/commit around a bunch of inserts will increase insert performance at least 10-fold.
You are about to Open & closing the database for every Query(insertion) . So it takes time too.
example For inserting a multiple records at once.
I think you must mantein your database open.
What takes more time is to open and close database.
For example, create a method only for that insertion, that opens the database in the firts element and closes it after the last.
I'm using two different types of fmdb connections in my app:
FMDatabase for all READ queries and
FMDatabaseQueue for all UPDATE queries.
Both are handled by a singleton, which keeps both types open the whole time while the app is running.
Both, read and update queries, are used in different threads as some tasks in my app are progressed in background; like getting data from a server and inserting it in the db via FMDatabaseQueue in an own background thread - while reading some information from db via FMDatabase and updating an ViewController with it on the main thread.
My problem is that after inserting data into the db via FMDatabaseQueue the second connection (FMDatabase) does not return the updated information as it does not find them. But I know the data was inserted as I have checked the db with an db browser tool + no errors occur while inserting it. To avoid this, I have to close the FMDatabase db connection and reopen it to see the changes made by the other connection. Unfortunately when my app starts up there are a many inserts, updates + reads as a lot of new data is loaded from server which needs to be processed - so closing and opening the db every time an update was made occurs in many "database busy" messages.
I have used one single FMDatabaseQueue for all threads and executes (read, update) before but it was quite slow when using read queries with __block variables to get the resultset out of the callback while another thread does some inserts(between 50-100 in a single transaction).
On top of it the database is encrypted via sqlcipher - not sure if it's important but want to mentioned it. So every time i have to close and open the database I'm doing a setKey.
My question: Is it possible to use a setup with two different connection types on multiple threads and if yes, do I have to close and open the FMDatabase connection? Or is there a better solution for this usecase?
UPDATE
My code to perform an insert / update looks like
-(void) create:(NSArray *)transactions
{
NSMutableString *sqlQuery = [[NSMutableString alloc] initWithString:STANDARD_INSERT_QUERY];
[sqlQuery appendString:#"(transaction_id, name, date) VALUES (?,?,?)"];
FMDBDataSource *ds = [FMDBDataSource sharedManager];
FMDatabaseQueue *queue = [ds getFMDBQ];
[queue inTransaction:^(FMDatabase *db, BOOL *rollback) {
[db setKey:[ds getKey]]; // returns the key to decrypt the database
for (Transaction *transaction in transactions)
{
[db executeUpdate:sqlQuery, transaction.transactionId, transaction.name, transaction.date];
}
}];
}
and a read query
-(Transaction *)read:(NSString *)transactionId
{
NSString *sqlQuery = [[NSString alloc] initWithString:STANDARD_SELECT_QUERY];
Transaction *transaction = nil;
FMDBDataSource *ds = [FMDBDataSource sharedManager];
FMResultSet *rs = [[ds getFMDB] executeQuery:sqlQuery];
while ([rs next]) {
transaction = [[Transaction alloc] init];
[transaction setTransactionId:[rs stringForColumn:#"transaction_id"]];
[transaction setName:[rs stringForColumn:#"name"]];
}
[rs close];
return transaction;
}
The FMDBDataSource is a singleton holding both, FMDatabase and FMDatabaseQueue, connections
- (FMDatabaseQueue *)getFMDBQ
{
if (self.fmdbq == nil)
{
self.fmdbq = [FMDatabaseQueue databaseQueueWithPath:[self getDBPath]];
}
return self.fmdbq;
}
- (FMDatabase *) getFMDB
{
if(self.fmdb == nil)
{
self.fmdb = [FMDatabase databaseWithPath:[self getDBPath]];
[self openAndKeyDatabase]; // opens the db and sets the key as the db is encrypted
}
return self.fmdb;
}
As I said, when using this code the FMDatabase connection does not get the information which was inserted via FMDatabaseQueue.
Personally, I would suggest using the single FMDatabaseQueue for both threads and let the queue coordinate the actions on the two threads. That's what it was created for. It completely eliminates those "database busy" problems.
On your performance update, if doing a bulk update, are you using the FMDatabase method beginTransaction before the update and commit at the end? Or use the inTransaction method. Inserting 10,000 records without transactions in my test takes 36.8 seconds, but with transactions it takes 0.25 seconds.
Or, if your bulk update is slow by necessity (e.g. you're downloading some big data source from a web service using some streaming protocol), you can either:
Load all the results into memory first, with no database interaction, and then use the bulk update with transactions as described in the previous paragraph; or
If your database updates are necessarily constrained by a slow network connection, then use separate inDatabase calls so that it doesn't tie up the FMDatabaseQueue while downloading data from your web service.
Bottom line, through the use of transactions or the judicious use of separate inDatabase calls, you can minimize how long your background operation is ties up the FMDatabaseQueue and you can achieve synchronized multi-threaded interaction with your database without blocking your UI too significantly.
I have spent lots of hours trying to making the same, when I found your post.
I do not know if you have already find a solution.
After a lot of tries and searches, I had to gave up and I'am looking for another solution now.
But, I would like to share my conclusions.
SQLite has a function to define READONLY or READWRITE mode. FMDB implements like openWithFlags.
[db openWithFlags:SQLITE_OPEN_READONLY|SQLITE_OPEN_NOMUTEX];
Setting these flags do not permits reading while wrinting even if we set these flags.
I could accomplish reading+writing (different connections) setting my database to use WAL journal_mode (http://www.sqlite.org/wal.html).
BUT, SQLCipher screw everthing up.
Conclusion READING + WRITING with 2 connections:
FMDB + openWithFlags = BE SAD AND ANGRY
FMDB + openWithFlags + WAL jornal_mode = BE HAPPY
FMDB + SQLCipher + openWithFlags = BE SAD AND ANGRY
FMDB + SQLCipher + openWithFlags + WAL jornal_mode = BE SAD AND ANGRY
As my application needs security, I do not know what to do yet.
Well, I hope it helps.
Best
Hami.