I am using Sqlite as backend in App. I have some confusions. I am using a Class for using Sqlite say "DatabaseManager". In this class I have declared all methods(to access data from DB) as static. In Multithreading is it fine to make all methods static or "DatabaseManager" should be a singleton class or should be a normal class?
I am facing problem like one thread is closing the Database after use while other thread(also accessing the DB) is in between then App Crash with message "Excess Bad"
I am using NSOperationQueue and NSOperation for Multithreading instead of GCD. So that i can cancel the operation if required.
Actually in App there is some calculation part which needs data from more then one table in DB. if i do that in Main thread the it takes time or Table Scrolling become smoothless.
Please Guide me.
SQlite database is thread safe. "threadsafe" we mean that you can use different SQLite database connections in different threads at the same time.
It has never been safe to use the same database connection simultaneously in multiple threads.
One of the good solution is to Test for SQLITE_BUSY
pseudo-code to illustrate a solution:
continueTrying = YES;
while (continueTrying)
{
retval = sqlite_exec(db, sqlQuery, callback, 0, &msg);
switch (retval)
{
case SQLITE_BUSY:
Log("[%s] SQLITE_BUSY: sleeping for a while...", threadName);
sleep a bit... (use something like sleep(), for example) // -> usleep(20);
break;
case SQLITE_OK:
continueTrying = NO; // We're done
break;
default:
Log("[%s] Can't execute \"%s\": %s\n", threadName, sqlQuery, msg);
continueTrying = NO;
break;
}
}
return retval;
Other solution is to use serial queue:
databaseQueue = dispatch_queue_create("com.company.app.yourdatabase", 0);
dispatch_sync(databaseQueue, ^{
// your database activity
});
Related
I'm working with a local SQLITE db in an iOS (obj-c) project where we've chosen to use FMDB. Please note I'm not interested in changing to Core Data or switching from FMDB to a different library.
Everything is working, but when we first run the app data is downloaded from our server, json is parsed, and then sorted into the local db. Due to the amount of info coming in this takes a good 30 - 40 seconds. Ideally this would be done in the background, but FMDB does not allow for this (at least as far as I can tell) as it may cause issues with the db data integrity.
Has anyone successfully used the FMDB library running all queries, inserts, etc on a background thread? I've tried simply wrapping actions in:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
// ... query/insert/update/delete
});
...but I end up with db integrity issues. Any tips or examples would be great. Thanks in advance.
#nawar - Sorry for not posting this sooner. I ended up using a single instace of FMDatabaseQueue (instead of FMDatabase) within the dispatch_async block. Works like a charm. The issues I was running into were due to using FMDatabase.
dispatch_async(dispatch_get_global_queue(0, 0), ^{
[self.dbQueue inTransaction:^(FMDatabase *db, BOOL *rollback) {
BOOL result = YES;
for (int i = 500; i < 1000; i++) {
result = [db executeUpdate:#"insert into testTable (name) values(?)",[NSString stringWithFormat:#"name-%d",i]];
if (!result) {
NSLog(#"break");
*rollback = YES;
break;
}
}
}];
});
Recently, I had the need for a function that I could use to guarantee synchronous execution of a given block on a particular serial dispatch queue. There was the possibility that this shared function could be called from something already running on that queue, so I needed to check for this case in order to prevent a deadlock from a synchronous dispatch to the same queue.
I used code like the following to do this:
void runSynchronouslyOnVideoProcessingQueue(void (^block)(void))
{
dispatch_queue_t videoProcessingQueue = [GPUImageOpenGLESContext sharedOpenGLESQueue];
if (dispatch_get_current_queue() == videoProcessingQueue)
{
block();
}
else
{
dispatch_sync(videoProcessingQueue, block);
}
}
This function relies on the use of dispatch_get_current_queue() to determine the identity of the queue this function is running on and compares that against the target queue. If there's a match, it knows to just run the block inline without the dispatch to that queue, because the function is already running on it.
I've heard conflicting things about whether or not it was proper to use dispatch_get_current_queue() to do comparisons like this, and I see this wording in the headers:
Recommended for debugging and logging purposes only:
The code must not make any assumptions about the queue returned,
unless it is one of the global queues or a queue the code has itself
created. The code must not assume that synchronous execution onto a
queue is safe from deadlock if that queue is not the one returned by
dispatch_get_current_queue().
Additionally, in iOS 6.0 (but not yet for Mountain Lion), the GCD headers now mark this function as being deprecated.
It sounds like I should not be using this function in this manner, but I'm not sure what I should use in its place. For a function like the above that targeted the main queue, I could use [NSThread isMainThread], but how can I check if I'm running on one of my custom serial queues so that I can prevent a deadlock?
Assign whatever identifier you want using dispatch_queue_set_specific(). You can then check your identifier using dispatch_get_specific().
Remember that dispatch_get_specific() is nice because it'll start at the current queue, and then walk up the target queues if the key isn't set on the current one. This usually doesn't matter, but can be useful in some cases.
This is a very simple solution. It is not as performant as using dispatch_queue_set_specific and dispatch_get_specific manually – I don't have the metrics on that.
#import <libkern/OSAtomic.h>
BOOL dispatch_is_on_queue(dispatch_queue_t queue)
{
int key;
static int32_t incrementer;
CFNumberRef value = CFBridgingRetain(#(OSAtomicIncrement32(&incrementer)));
dispatch_queue_set_specific(queue, &key, value, nil);
BOOL result = dispatch_get_specific(&key) == value;
dispatch_queue_set_specific(queue, &key, nil, nil);
CFRelease(value);
return result;
}
I'm using two different types of fmdb connections in my app:
FMDatabase for all READ queries and
FMDatabaseQueue for all UPDATE queries.
Both are handled by a singleton, which keeps both types open the whole time while the app is running.
Both, read and update queries, are used in different threads as some tasks in my app are progressed in background; like getting data from a server and inserting it in the db via FMDatabaseQueue in an own background thread - while reading some information from db via FMDatabase and updating an ViewController with it on the main thread.
My problem is that after inserting data into the db via FMDatabaseQueue the second connection (FMDatabase) does not return the updated information as it does not find them. But I know the data was inserted as I have checked the db with an db browser tool + no errors occur while inserting it. To avoid this, I have to close the FMDatabase db connection and reopen it to see the changes made by the other connection. Unfortunately when my app starts up there are a many inserts, updates + reads as a lot of new data is loaded from server which needs to be processed - so closing and opening the db every time an update was made occurs in many "database busy" messages.
I have used one single FMDatabaseQueue for all threads and executes (read, update) before but it was quite slow when using read queries with __block variables to get the resultset out of the callback while another thread does some inserts(between 50-100 in a single transaction).
On top of it the database is encrypted via sqlcipher - not sure if it's important but want to mentioned it. So every time i have to close and open the database I'm doing a setKey.
My question: Is it possible to use a setup with two different connection types on multiple threads and if yes, do I have to close and open the FMDatabase connection? Or is there a better solution for this usecase?
UPDATE
My code to perform an insert / update looks like
-(void) create:(NSArray *)transactions
{
NSMutableString *sqlQuery = [[NSMutableString alloc] initWithString:STANDARD_INSERT_QUERY];
[sqlQuery appendString:#"(transaction_id, name, date) VALUES (?,?,?)"];
FMDBDataSource *ds = [FMDBDataSource sharedManager];
FMDatabaseQueue *queue = [ds getFMDBQ];
[queue inTransaction:^(FMDatabase *db, BOOL *rollback) {
[db setKey:[ds getKey]]; // returns the key to decrypt the database
for (Transaction *transaction in transactions)
{
[db executeUpdate:sqlQuery, transaction.transactionId, transaction.name, transaction.date];
}
}];
}
and a read query
-(Transaction *)read:(NSString *)transactionId
{
NSString *sqlQuery = [[NSString alloc] initWithString:STANDARD_SELECT_QUERY];
Transaction *transaction = nil;
FMDBDataSource *ds = [FMDBDataSource sharedManager];
FMResultSet *rs = [[ds getFMDB] executeQuery:sqlQuery];
while ([rs next]) {
transaction = [[Transaction alloc] init];
[transaction setTransactionId:[rs stringForColumn:#"transaction_id"]];
[transaction setName:[rs stringForColumn:#"name"]];
}
[rs close];
return transaction;
}
The FMDBDataSource is a singleton holding both, FMDatabase and FMDatabaseQueue, connections
- (FMDatabaseQueue *)getFMDBQ
{
if (self.fmdbq == nil)
{
self.fmdbq = [FMDatabaseQueue databaseQueueWithPath:[self getDBPath]];
}
return self.fmdbq;
}
- (FMDatabase *) getFMDB
{
if(self.fmdb == nil)
{
self.fmdb = [FMDatabase databaseWithPath:[self getDBPath]];
[self openAndKeyDatabase]; // opens the db and sets the key as the db is encrypted
}
return self.fmdb;
}
As I said, when using this code the FMDatabase connection does not get the information which was inserted via FMDatabaseQueue.
Personally, I would suggest using the single FMDatabaseQueue for both threads and let the queue coordinate the actions on the two threads. That's what it was created for. It completely eliminates those "database busy" problems.
On your performance update, if doing a bulk update, are you using the FMDatabase method beginTransaction before the update and commit at the end? Or use the inTransaction method. Inserting 10,000 records without transactions in my test takes 36.8 seconds, but with transactions it takes 0.25 seconds.
Or, if your bulk update is slow by necessity (e.g. you're downloading some big data source from a web service using some streaming protocol), you can either:
Load all the results into memory first, with no database interaction, and then use the bulk update with transactions as described in the previous paragraph; or
If your database updates are necessarily constrained by a slow network connection, then use separate inDatabase calls so that it doesn't tie up the FMDatabaseQueue while downloading data from your web service.
Bottom line, through the use of transactions or the judicious use of separate inDatabase calls, you can minimize how long your background operation is ties up the FMDatabaseQueue and you can achieve synchronized multi-threaded interaction with your database without blocking your UI too significantly.
I have spent lots of hours trying to making the same, when I found your post.
I do not know if you have already find a solution.
After a lot of tries and searches, I had to gave up and I'am looking for another solution now.
But, I would like to share my conclusions.
SQLite has a function to define READONLY or READWRITE mode. FMDB implements like openWithFlags.
[db openWithFlags:SQLITE_OPEN_READONLY|SQLITE_OPEN_NOMUTEX];
Setting these flags do not permits reading while wrinting even if we set these flags.
I could accomplish reading+writing (different connections) setting my database to use WAL journal_mode (http://www.sqlite.org/wal.html).
BUT, SQLCipher screw everthing up.
Conclusion READING + WRITING with 2 connections:
FMDB + openWithFlags = BE SAD AND ANGRY
FMDB + openWithFlags + WAL jornal_mode = BE HAPPY
FMDB + SQLCipher + openWithFlags = BE SAD AND ANGRY
FMDB + SQLCipher + openWithFlags + WAL jornal_mode = BE SAD AND ANGRY
As my application needs security, I do not know what to do yet.
Well, I hope it helps.
Best
Hami.
With reference to this answer, I am wondering is this correct?
#synchronized does not make any code "thread-safe"
As I tried to find any documentation or link to support this statement, for no success.
Any comments and/or answers will be appreciated on this.
For better thread safety we can go for other tools, this is known to me.
#synchronized does make code thread safe if it is used properly.
For example:
Lets say I have a class that accesses a non thread safe database. I don't want to read and write to the database at the same time as this will likely result in a crash.
So lets say I have two methods. storeData: and readData on a singleton class called LocalStore.
- (void)storeData:(NSData *)data
{
[self writeDataToDisk:data];
}
- (NSData *)readData
{
return [self readDataFromDisk];
}
Now If I were to dispatch each of these methods onto their own thread like so:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[[LocalStore sharedStore] storeData:data];
});
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[[LocalStore sharedStore] readData];
});
Chances are we would get a crash. However if we change our storeData and readData methods to use #synchronized
- (void)storeData:(NSData *)data
{
#synchronized(self) {
[self writeDataToDisk:data];
}
}
- (NSData *)readData
{
#synchronized(self) {
return [self readDataFromDisk];
}
}
Now this code would be thread safe. It is important to note that if I remove one of the #synchronized statements however the code would no longer be thread safe. Or if I were to synchronize different objects instead of self.
#synchronized creates a mutex lock on the object you are syncrhonizing. So in other words if any code wants to access code in a #synchronized(self) { } block it will have to get in line behind all previous code running within in that same block.
If we were to create different localStore objects, the #synchronized(self) would only lock down each object individually. Does that make sense?
Think of it like this. You have a whole bunch of people waiting in separate lines, each line is numbered 1-10. You can choose what line you want each person to wait in (by synchronizing on a per line basis), or if you don't use #synchronized you can jump straight to the front and skip all the lines. A person in line 1 doesn't have to wait for a person in line 2 to finish, but the person in line 1 does have to wait for everyone in front of them in their line to finish.
I think the essence of the question is:
is the proper use of synchronize able to solve any thread-safe
problem?
Technically yes, but in practice it's advisable to learn and use other tools.
I'll answer without assuming previous knowledge.
Correct code is code that conforms to its specification. A good specification defines
invariants constraining the state,
preconditions and postconditions describing the effects of the operations.
Thread-safe code is code that remains correct when executed by multiple threads. Thus,
No sequence of operations can violate the specification.1
Invariants and conditions will hold during multithread execution without requiring additional synchronization by the client2.
The high level takeaway point is: thread-safe requires that the specification holds true during multithread execution. To actually code this, we have to do just one thing: regulate the access to mutable shared state3. And there are three ways to do it:
Prevent the access.
Make the state immutable.
Synchronize the access.
The first two are simple. The third one requires preventing the following thread-safety problems:
liveness
deadlock: two threads block permanently waiting for each other to release a needed resource.
livelock: a thread is busy working but it's unable to make any progress.
starvation: a thread is perpetually denied access to resources it needs in order to make progress.
safe publication: both the reference and the state of the published object must be made visible to other threads at the same time.
race conditions A race condition is a defect where the output is dependent on the timing of uncontrollable events. In other words, a race condition happens when getting the right answer relies on lucky timing. Any compound operation can suffer a race condition, example: “check-then-act”, “put-if-absent”. An example problem would be if (counter) counter--;, and one of several solutions would be #synchronize(self){ if (counter) counter--;}.
To solve these problems we use tools like #synchronize, volatile, memory barriers, atomic operations, specific locks, queues, and synchronizers (semaphores, barriers).
And going back to the question:
is the proper use of #synchronize able to solve any thread-safe
problem?
Technically yes, because any tool mentioned above can be emulated with #synchronize. But it would result in poor performance and increase the chance of liveness related problems. Instead, you need to use the appropriate tool for each situation. Example:
counter++; // wrong, compound operation (fetch,++,set)
#synchronize(self){ counter++; } // correct but slow, thread contention
OSAtomicIncrement32(&count); // correct and fast, lockless atomic hw op
In the case of the linked question you could indeed use #synchronize, or a GCD read-write lock, or create a collection with lock stripping, or whatever the situation calls for. The right answer depend on the usage pattern. Any way you do it, you should document in your class what thread-safe guarantees are you offering.
1 That is, see the object on an invalid state or violate the pre/post conditions.
2 For example, if thread A iterates a collection X, and thread B removes an element, execution crashes. This is non thread-safe because the client will have to synchronize on the intrinsic lock of X (synchronize(X)) to have exclusive access. However, if the iterator returns a copy of the collection, the collection becomes thread-safe.
3 Immutable shared state, or mutable non shared objects are always thread-safe.
Generally, #synchronized guarantees thread safety, but only when used correctly. It is also safe to acquire the lock recursively, albeit with limitations I detail in my answer here.
There are several common ways to use #synchronized wrong. These are the most common:
Using #synchronized to ensure atomic object creation.
- (NSObject *)foo {
#synchronized(_foo) {
if (!_foo) {
_foo = [[NSObject alloc] init];
}
return _foo;
}
}
Because _foo will be nil when the lock is first acquired, no locking will occur and multiple threads can potentially create their own _foo before the first completes.
Using #synchronized to lock on a new object each time.
- (void)foo {
#synchronized([[NSObject alloc] init]) {
[self bar];
}
}
I've seen this code quite a bit, as well as the C# equivalent lock(new object()) {..}. Since it attempts to lock on a new object each time, it will always be allowed into the critical section of code. This is not some kind of code magic. It does absolutely nothing to ensure thread safety.
Lastly, locking on self.
- (void)foo {
#synchronized(self) {
[self bar];
}
}
While not by itself a problem, if your code uses any external code or is itself a library, it can be an issue. While internally the object is known as self, it externally has a variable name. If the external code calls #synchronized(_yourObject) {...} and you call #synchronized(self) {...}, you may find yourself in deadlock. It is best to create an internal object to lock upon that is not exposed outside of your object. Adding _lockObject = [[NSObject alloc] init]; inside your init function is cheap, easy, and safe.
EDIT:
I still get asked questions about this post, so here is an example of why it is a bad idea to use #synchronized(self) in practice.
#interface Foo : NSObject
- (void)doSomething;
#end
#implementation Foo
- (void)doSomething {
sleep(1);
#synchronized(self) {
NSLog(#"Critical Section.");
}
}
// Elsewhere in your code
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
Foo *foo = [[Foo alloc] init];
NSObject *lock = [[NSObject alloc] init];
dispatch_async(queue, ^{
for (int i=0; i<100; i++) {
#synchronized(lock) {
[foo doSomething];
}
NSLog(#"Background pass %d complete.", i);
}
});
for (int i=0; i<100; i++) {
#synchronized(foo) {
#synchronized(lock) {
[foo doSomething];
}
}
NSLog(#"Foreground pass %d complete.", i);
}
It should be obvious to see why this happens. Locking on foo and lock are called in different orders on the foreground VS background threads. It's easy to say that this is bad practice, but if Foo is a library, the user is unlikely to know that the code contains a lock.
#synchronized alone doesn't make code thread safe but it is one of the tools used in writing thread safe code.
With multi-threaded programs, it's often the case of a complex structure that you want to be maintained in a consistent state and you want only one thread to have access at a time. The common pattern is to use a mutex to protect a critical section of code where the structure is accessed and/or modified.
#synchronized is thread safe mechanism. Piece of code written inside this function becomes the part of critical section, to which only one thread can execute at a time.
#synchronize applies the lock implicitly whereas NSLock applies it explicitly.
It only assures the thread safety, not guarantees that. What I mean is you hire an expert driver for you car, still it doesn't guarantees car wont meet an accident. However probability remains the slightest.
It's companion in GCD(grand central dispatch) is dispatch_once. dispatch_once does the same work as to #synchronized.
The #synchronized directive is a convenient way to create mutex locks on the fly in Objective-C code.
side-effects of mutex locks:
deadlocks
starvation
Thread safety will depend on usage of #synchronized block.
I've started experimenting with POSix threads using the ios platform. Coming fro using NSThread it's pretty daunting.
Basically in my sample app I have a big array filled with type mystruct. Every so often (very frequently) I want to perform a task with the contents of one of these structs in the background so I pass it to detachnewthread to kick things off.
I think I have the basics down but Id like to get a professional opinion before I attempt to move on to more complicated stuff.
Does what I have here seem "o.k" and could you point out anything missing that could cause problems? Can you spot any memory management issues etc....
struct mystruct
{
pthread thread;
int a;
long c;
}
void detachnewthread(mystruct *str)
{
// pthread_t thread;
if(str)
{
int rc;
// printf("In detachnewthread: creating thread %d\n", str->soundid);
rc = pthread_create(&str->thread, NULL, DoStuffWithMyStruct, (void *)str);
if (rc){
printf("ERROR; return code from pthread_create() is %d\n", rc);
//exit(-1);
}
}
//
/* Last thing that main() should do */
// pthread_exit(NULL);
}
void *DoStuffWithMyStruct(void *threadid)
{
mystruct *sptr;
dptr = (mystruct *)threadid;
// do stuff with data in my struct
pthread_detach(soundptr->thread);
}
One potential issue would be how the storage for the passed in structure mystruct is created. The lifetime of that variable is very critical to its usage in the thread. For example, if the caller of detachnewthread had that declared on the stack and then returned before the thread finished, it would be undefined behavior. Likewise, if it were dynamically allocated, then it is necessary to make sure it is not freed before the thread is finished.
In response to the comment/question: The necessity of some kind of mutex depends on the usage. For the sake of discussion, I will assume it is dynamically allocated. If the calling thread fills in the contents of the structure prior to creating the "child" thread and can guarantee that it will not be freed until after the child thread exits, and the subsequent access is read/only, then you would not need a mutex to protect it. I can imagine that type of scenario if the structure contains information that the child thread needs for completing its task.
If, however, more than one thread will be accessing the contents of the structure and one or more threads will be changing the data (writing to the structure), then you probably do need a mutex to protect it.
Try using Apple's Grand Central Dispatch (GCD) which will manage the threads for you. GCD provides the capability to dispatch work, via blocks, to various queues that are managed by the system. Some of the queue types are concurrent, serial, and of course the main queue where the UI runs. Based upon the CPU resources at hand, the system will manage the queues and necessary threads to get the work done. A simple example, which shows the how you can nest calls to different queues is like this:
__block MYClass *blockSelf=self;
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
[blockSelf doSomeWork];
dispatch_async(dispatch_get_main_queue(), ^{
[blockSelf.textField setStringValue:#"Some work is done, updating UI"];
});
});
__block MyClass *blockSelf=self is used simply to avoid retain cycles associated with how blocks work.
Apple's docs:
http://developer.apple.com/library/ios/#documentation/Performance/Reference/GCD_libdispatch_Ref/Reference/reference.html
Mike Ash's Q&A blog post:
http://mikeash.com/pyblog/friday-qa-2009-08-28-intro-to-grand-central-dispatch-part-i-basics-and-dispatch-queues.html