Recently I have added BEGIN and COMMIT transaction to all the database calls against my sqlcipher sqlite3 database for an iOS application I am in the process of working on.
I am doing this through the following methods:
#define kTRANSACTION_BEGIN #"BEGIN"
#define kTRANSACTION_COMMIT #"COMMIT"
-(void)transactionBegin
{
int status = sqlite3_exec(self.Database, kTRANSACTION_BEGIN.UTF8String, NULL, NULL, NULL);
NSLog(#"BEGIN Status = %i", status);
}
-(void)transactionEnd
{
char* errorMessage;
int status = 0;
status = sqlite3_exec(self.Database, kTRANSACTION_COMMIT.UTF8String, NULL, NULL, &errorMessage);
NSLog(#"COMMIT Status = %i", status);
if (status != SQLITE_OK) {
NSLog(#"ERROR: SQL Error closing transaction (%i): %s\n %s",status, errorMessage ,sqlite3_errmsg(self.Database));
}
}
These methods are called directly after opening (and setting PRAGMAs) and before closing the database connection.
Both of these appear to execute successfully. However, upon inspecting the location of the database, the db.sqlite file remains 0 bytes with a db.sqlite-journal file at 512 bytes. Queries run agains the database continue to execute successfully until the app is closed and then all the data inserted is lost. Why is my COMMIT not persisting my CREATE TABLE or INSERT statements to disk?
After reading the documentation, it would appear that unless explicitly set, the journal_mode of the sqlite database should be set to DELETE. If this were true, I should no longer have a -journal file after the commit has been completed. Would there be any way through the c API that I would be able to test this?
I am also led to believe that the code is in functional order due to how I am using my database wrapper to act upon a few different database files. Upon successful beginning and committing of the transaction, the BEGIN and COMMIT status codes are always 0 which indicates SQLITE_OK. This odd behavior only occurs in situations where I only insert one record before attempting to COMMIT any changes.
The only other option I have changed from default sqlite configuration is the page size (PRAGMA cipher_page_size = 4096). The page size has been adjusted to help increase performance of some of the other larger databases that I also use with this wrapper. Removing my modification to the page size does not resolve the issue.
Lastly, I have noticed that if I remove the transactions all together, I do not experience this issue at all. Preferably, I would like to leave in the transactions for the increase in performance, but having it not persisting my data defeats the purpose of the database.
Would anyone have any ideas of why this might be happening or anything else I can try?
Thanks in advance.
Related
I've got a JSON file which contains more than 2000 arrays, and in each array there may have an array with 5 strings.
I'm running two for() loops in one function, so I think the 2ed one will be run after the 1st one is done?
and it sometimes appears the following(random data error appear):
database is locked
when I'm executing
INSERT INTO problemTags(problem_IDIndex)VALUES('93E')
And I use the code below to execute my SQL statement
+ (void)execSqliteWithSQL:(NSString *)sql{
sqlite3 *sqlite = nil;
int openResult = sqlite3_open([DBPath UTF8String], &sqlite);
if(openResult != SQLITE_OK){
NSLog(#"open DB error");
}
char *error;
int execResult = sqlite3_exec(sqlite, [sql UTF8String], nil, nil, &error);
if(execResult != SQLITE_OK){
NSString *errorString = [NSString stringWithFormat:#"\n execResult error=>\n %s \n SQL:\n%#\n",error,sql];
NSLog(#"%#",errorString);
}
sqlite3_close(sqlite);
}
I'm thinking about the reason to this issue is that
when I'm running the for loop, it sends several SQL statement to the execSqliteWithSQL function and when it can't run fast enough the data base would be closed?
question added below:
without using threads
the two for loops is like below:
-(void)......{
for(){
//sending SQL to execSqliteWithSQL
}
for(){
//sending SQL to execSqliteWithSQL
}
}
A couple of thoughts:
Do you possibly have multiple threads calling this execSqliteWithSQL concurrently? If so, errors like this are very likely.
You could remedy this by setting up a dedicated serial queue to which execSqliteWithSQL would dispatch its calls, thus eliminating attempts to do this concurrently.
Regardless, I would advise against opening and closing the database for each call. That's horribly inefficient.
In fact, if doing a bunch of update/insert statements, you not only don't want to open and close the database for every call, but rather you want to open, call BEGIN TRANSACTION, do your updates, and then COMMIT those changes. The performance difference is startling.
This execSqliteWithSQL clearly presupposes that you're manually building your SQL (presumably with stringWithFormat). That's problematic at best, dangerous at worst. If a quotation mark appears in a string value being inserted into the database, the SQL will fail. If this feed includes any user input, you are susceptible to SQL injection attacks.
Instead, you should use sqlite3_prepare_v2 to prepare a SQL statement with ? placeholders, and then use sqlite3_bind_xxx to bind values to those placeholders.
I want to know how to handle very large DBs in iOS?
Explanation:
I am making an app for the company, it will be holding all daily transactions and operations (this means few hundreds to a couple thousands a day).
The app in fact just displays the daily transactions, but keeps everything in the DB just in case (so I can not delete them).
As there is a need for the app to be highly reactive, I am searching for a way to efficiently load (Core Data Fetch) just the latest operations, without being slowed down in the long run when the DB gets really large.
Knowing that whenever an operation is updated, I refresh the displayed table immediately.
I can't find any article dealing with this on iOS, so I can't find any good practice or a correct way to handle it.
On a MySQL server, I'd do like this:
* Set up a DB for daily "active" operations, when done, delete them and move them to a "finished" operations DB.
* Set up a Cron job to dump/backup daily/monthly DBs by night, and having a fresh lightweight one for the next day.
Sadly, it doesn't seem possible to create a dynamic table on iOS, and even creating a "finished" DB, this will still require accessing a heavy DB at each closed operation, which can lead to a low memory crash.
actually I think you can try sqlite api to match the same operations in your MySQL server. However, you will have to drop the CoreData and rewrite everything instead, i.e., TableView, TableCell, etc.
to deal with the DataSource like this:
sqlite3 * tradeDB;
sqlite3_open_v2([path UTF8String], &tradeDB, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nil);
sqlite3_stmt * recordset;
int res = sqlite3_prepare_v2(tradeDB, "select count(*) from someTable", -1, &recordset, nil);
if (res!=SQLITE_OK) {
return nil;
}//end if
res = sqlite3_step(recordset);
if (res!=SQLITE_ROW) {
_result = nil;
}else{
char * res_char = (char*)sqlite3_column_text(recordset, 0);
if (!res_char)
_result = nil;
else
_result = [NSString stringWithUTF8String:res_char];
//end if
}//end if
sqlite3_finalize(recordset);
if (!_result) {
_result = #"0";
}//end if
return self;
I am trying to insert around 500 records into the sqlite database, it is taking 20 sec, which is way too much! Could anyone please tell how to optimize this.
for (int i = 0 ; i< [[[dic objectForKey:#"response" ]objectForKey:#"requete"]count]; i++) {
[sqlManager executeQuery:[[[dic objectForKey:#"response"]objectForKey:#"requetes"]objectAtIndex:i]];
}
with
-(void)executeQuery:(NSString*)_query
{
sqlite3 *database;
if(sqlite3_open([databasePath UTF8String], &database) == SQLITE_OK) {
if(sqlite3_exec(database, [_query UTF8String],NULL, NULL, NULL) == SQLITE_OK){
//NSLog(#"Query Success");
}
else{
// NSLog(#"Query Failed");
}
sqlite3_close(database);
}
sqlite3_close(database);
}
Open once and leave open until the app is closed or you're done with a DB-intense phase. SQLite can easily handle the situation if your app "crashes" with the DB open.
When doing multiple inserts close together, where it's fine if they either all go or none of them go, start an transaction (and commit it when done). When you do not explicitly start a transaction then SQLite does an implicit start/commit around each insert or update, "forcing" the data to "disk". This takes A LOT of time. Doing one start/commit around a bunch of inserts will increase insert performance at least 10-fold.
You are about to Open & closing the database for every Query(insertion) . So it takes time too.
example For inserting a multiple records at once.
I think you must mantein your database open.
What takes more time is to open and close database.
For example, create a method only for that insertion, that opens the database in the firts element and closes it after the last.
I'm using an sqlite database in my iOS app directly, without taking advantage of Core Data. When the app is launched, I need it to insert several sets of table rows of considerable amount of data that will be needed during the usage of the app and that are fixed sets of data (for example, I have a database table for a list of countries the user could choose in the app). I've those data insertions in an .sql file in the app and, since I need this data to be available in the database from the very first time the user installs and launches the app, I wrote a method that firstly copies the database to Documents, and then reads the file and performs the database insertions, and I call this method in application:didFinishLaunchingWithOptions:.
However, I'm finding that performing a lot of insertions is really a slow operation. I proceed this way:
+ (void)insertFileContent:(NSString *)fileContent intoDatabase:(sqlite3 *)database
{
NSArray *lines = [fileContent componentsSeparatedByString:#"\r\n"];
for (NSString *line in lines) {
const char *sqlStatement = [line UTF8String];
sqlite3_stmt *compiledStatement;
if (sqlite3_prepare_v2(database, sqlStatement, -1, &compiledStatement, NULL) == SQLITE_OK) {
if(sqlite3_step(compiledStatement) == SQLITE_DONE) {
// Insertion was ok
}
else {
NSLog(#"%d",sqlite3_step(compiledStatement));
}
sqlite3_finalize(compiledStatement);
}
}
}
Database is opened only once before calling this method, and closed when it has returned. As I said, I have a lot of data I need to insert into database to make the app work properly, and it is taking even minutes to finish. I've been looking for a way to reduce this time significantly, but I didn't succeed.
How could I handle this? Is there a way to make this operations faster, and/or is there a way to insert data into the database at the time the user installs the app, instead of waiting til launch time?
Thanks in advance
You have the transaction overhead once for each statement.
Wrap all statements into a single transaction.
Execute BEGIN/END manually, or just write them into your .sql file.
I'm writing a C# application that upgrades client machines from one application version to another. The first step is to create a backup of a SQL database. I'm doing this using SMO and it works fine. Next I uninstall a windows service. Then I try to rename the database that I backed up, again, using SMO. This fails because it says it can't gain exclusive access to the database. When I look at the activity monitor, I can see that there are two connections to the database I'm trying to rename. One connection is the one I'm using to try to rename the database, the other is the one I used to backup the database. It's status is sleeping but I'm assuming this is why I can't get exclusive access to rename the database. I was kind of surprised to find the SMO objects didn't implement IDisposable. I tried setting my Server object reference to null incase garbage collection might help, but that didn't work. The connections stay there until I quit the application.
So I have a couple of questions
How do I get rid of the first connection? I know it's possible because it happens when my application shuts down
Can I put the database in single user mode using or force the rename in some other way using SMO?
Thanks
I got it to work if I turn off pooling in my connection string by adding Pooling=false. Then calling Disconnect on the ServerConnection:
ServerConnection svrConn = null;
try
{
string connString = Cryptographer.Decrypt(ConfigurationManager.ConnectionStrings["CS"].ConnectionString);
svrConn = new Microsoft.SqlServer.Management.Common.ServerConnection(new System.Data.SqlClient.SqlConnection(connString));
Server server = new Microsoft.SqlServer.Management.Smo.Server(svrConn);
Backup backup = new Microsoft.SqlServer.Management.Smo.Backup();
...
backup.SqlBackup(server);
}
catch (Exception ex)
{
...
}
finally
{
if (svrConn != null)
svrConn.Disconnect();
}
I think server.ConnectionContext.Disconnect would also work, but haven't tried it.