OpenCV : Video processing in background queue - ios

Successfully able to detect square object from video stream using powerful OpenCV. Everything was fine, except video lags frame due to calculation burden on main thread.
As you can see below code snippet, I added NSOperationQueue inside CvVideoCameraDelegate delegate method processImage:(cv::Mat&)image When I tried to run findSquaresInImage on background operation queue and UIImageFromCVMat main queue.
- (void)processImage:(cv::Mat&)image
NSOperationQueue *videoProcessQueue = [[NSOperationQueue alloc] init];
[videoProcessQueue addOperationWithBlock:^{
// do some time consuming stuff in the background
cv::Mat matResultImage = [self findSquaresInImage:image]; //---> Method will return the square object and takes more than 0.3 seconds
[[NSOperationQueue mainQueue] addOperationWithBlock:^{
// update the UI here
self.imageView.image = [UtilityClass UIImageFromCVMat:matResultImage]; //---> Updates the UI
}];
}];
}
But I got exception in this line cv::pyrDown()
- (cv::Mat)findSquaresInImage:(cv::Mat)_image
{
std::vector<std::vector<cv::Point> > squares;
cv::Mat pyr, timg, gray0(_image.size(), CV_8U), gray;
int thresh = 20, N = 2;
cv::pyrDown(_image, pyr, cv::Size(_image.cols/2, _image.rows/2)); //----> HERE I GOT EXCEPTION
cv::pyrUp(pyr, timg, _image.size());
std::vector<std::vector<cv::Point> > contours;
//remaining logic goes here........ ...... ...... ...
}
What is wrong here? Any alternative way to improve ?
//UPDATE :
Same issue even with GCD blocks :
dispatch_async( dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
cv::Mat matResultImage = [self findSquaresInImage:image];
dispatch_async( dispatch_get_main_queue(), ^{
self.imageView.image = [JKUtilityClass UIImageFromCVMat:matResultImage];
});
});

Finally able to resolve after 2 major changes :
Directly mapping image from processImage:(cv::Mat&)image to CvVideoCamera object. Never used already initialized parent view self.imageView.
To speedup process, included OpenCVSquares Detection library.

Related

Cocos2D updating UI with threads

I have a "Loading Screen" with a "Loading bar". This loading screen gets images from my server and loads them to CCSprites, so i will be using them in other screens. While the loading screen is downloading images and creating the CCSprites, i want my Loading Bar to update it's UI. LoadingBar is a CCNode subclass.
I know I should use threading to update UI, but the problem is that (as far as I know) Cocos2D nodes are not thread safe so when i use the code below, it does update the UI but then the sprites does not load:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, kNilOptions), ^{
// go do something asynchronous...
dispatch_async(dispatch_get_main_queue(), ^{
// update UI
});
});
I have tried many things, but i can't solve this problem. Please, help :)
Here is the code that makes my LoadingBar update:
-(void)makeStep{
int actualPersentage = 100 * (numSteps - numStepsToGo + 1) / numSteps;
int objectAtIndex = MAX(0, numSteps - numStepsToGo);
persentageLabel.string = [NSString stringWithFormat:#"%i%% %#", actualPersentage, [steps objectAtIndex:objectAtIndex]];
[frontground setTextureRect:CGRectMake(0, 0, frontground.textureRect.size.width + (200/numSteps), 40)];
frontground.position = ccp(initialPosition + (frontground.contentSize.width/2) - (background.contentSize.width/2), frontground.position.y);
numStepsToGo --;
if(numStepsToGo == 0){
[frontground setTextureRect:CGRectMake(0, 0, 200, 40)];
persentageLabel.string = [NSString stringWithFormat:#"All loaded correctly!"];
}
}
Basically what I have is a background gray rectangle as background and a green one (frontground) that "fills" the background each time makeStep is called.
Here is the code example where i parse a JSON and where my LoadingBar should update:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, kNilOptions), ^{
...
CCSprite *ssButtonStart = [self getSpriteFromUrl:[startScreenData objectForKey:#"SSimg_button_start"]];
CCSprite *ssButtonExit = [self getSpriteFromUrl:[startScreenData objectForKey:#"SSimg_button_exit"]];
CCSprite *ssBackground = [self getSpriteFromUrl:[startScreenData objectForKey:#"SSimg_background"]];
self.gameProperties.slBackground = ssBackground;
self.gameProperties.slButtonExit = ssButtonExit;
self.gameProperties.slButtonStart = ssButtonStart;
NSLog(#"Start Screen Loading done!");
dispatch_async(dispatch_get_main_queue(), ^{
[self updateStep];
});
...
dispatch_async(dispatch_get_main_queue(), ^{
[self updateStep];
});
//Etc.
Code where i get the image and convert to CCSprite:
-(CCSprite*)getSpriteFromUrl:(NSString*)stringUrl{
__block NSData *imageData;
__block CCSprite *sprite;
imageData = [[NSData alloc] initWithContentsOfURL: [NSURL URLWithString:stringUrl]];
//NSLog(#"string imageData: %#", imageData);
UIImageView *imView = [[UIImageView alloc] initWithImage:[UIImage imageWithData: imageData]];
dispatch_async(dispatch_get_main_queue(), ^{
CCTexture2D *texture = [[CCTexture2D alloc] initWithCGImage:imView.image.CGImage resolutionType:kCCResolutionUnknown];
if(texture != nil){
sprite = [[CCSprite alloc] init];
sprite = [[CCSprite alloc] initWithTexture:texture];
}else{
[self showError:#"Some textures didn't load correctly. Please try again!"];
}
// update UI
});
return sprite;
}
I am using iOS 7 and Cocos2D 2.0
You should update the bar on the main thread. Instead move the downloading of images to a background thread. NSData or NSURLRequest even has async methods to do so. Then when you have completed downloading the images create the CCSprites on the main thread as a last step. Sprites have to be created on the main thread anyway.
This answer from LearnCocos2D (see code in question) plus:
Solved! I made an sleep after getting every sprite in getSpriteFromUrl. The code is [NSThread sleepForTimeInterval:0.2];. Hope it helps anyone else!
From me, solved my question. Thanks!

Intangible Order of Execution (dispatch_semaphore_t, dispatch_group_async) and the Use of Them in Combination with Different Dispatch Queue Types

I just took some time in the evening to play around with GCD, especially with dispatch_semaphore_t because I never used it. Never had the need to.
So I wrote the following as a test:
- (void)viewDidLoad
{
UIView *firstView = [[UIView alloc] initWithFrame:(CGRect){{0, 0}, self.view.frame.size.width/4, self.view.frame.size.width/5}];
firstView.backgroundColor = [UIColor purpleColor];
[self.view addSubview:firstView];
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^
{
for (long i = 0; i < 1000; i++)
{
sleep(5);
dispatch_async(dispatch_get_main_queue(), ^
{
firstView.layer.opacity = ((i%2) ? 0: 1);
});
}
});
dispatch_queue_t queue1 = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_group_t group1 = dispatch_group_create();
dispatch_group_async(group1, queue1, ^
{
sleep(3);
NSLog(#"dispatch group 1");
});
dispatch_group_notify(group1, queue1, ^
{
NSLog(#"dispatch notify 1");
});
dispatch_async(myQueue, ^
{
for(int z = 0; z < 10; z++)
{
NSLog(#"%i", z);
sleep(1);
}
dispatch_semaphore_signal(mySemaphore);
});
dispatch_semaphore_wait(mySemaphore, DISPATCH_TIME_FOREVER);
NSLog(#"Loop is Done");
}
If I ran the above, the output would be:
0
1
2
dispatch group 1
dispatch notify 1
3
4
5
6
7
8
9
Loop is Done
After the above, firstView appears on the screen (before semaphore the whole screen was black) and finally this gets executed:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^
{
for (long i = 0; i < 1000; i++)
{
sleep(5);
dispatch_async(dispatch_get_main_queue(), ^
{
firstView.layer.opacity = ((i%2) ? 0: 1);
});
}
});
Which only alternates the opacity as the loop runs after semaphore is done.
1.)
So, it seems that I have to wait until dispatch_semaphore finish to do its work before any UI thing takes place.
BUT:
It seems like dispatch_group_t runs concurrently with dispatch_semaphore as shown from the output above (i.e., 1, 2, 3, ....).
???
2.)
And if I change the for loop in the above to using: dispatch_async(dispatch_get_main_queue(), ^
instead of:
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^,
nothing gets shown on the screen even after semaphore is done.
How so???
3.)
Furthermore if I change semaphore to the following as opposed to using a global queue like in the above:
dispatch_async(dispatch_get_main_queue(), ^
{
for(int z = 0; z < 10; z++)
{
NSLog(#"%i", z);
sleep(1);
}
dispatch_semaphore_signal(mySemaphore);
});
Only dispatch_group takes place; nothing else take places / get executed, not the for loop in the above, including UI. Nothing.
4.)
So besides what I pointed out in the above, what can I do, in order to make semaphore not blocking my UI and my other process and just let my UI and the other processes do their thing?
And as mentioned above, why changing the type of queue for semaphore from global to main would cause nothing to be shown on screen and even the loop would not execute except dispatch_group?
5.)
If I change semaphore to:
dispatch_semaphore_t mySemaphore = dispatch_semaphore_create(1);
//1 instead of 0 (zero)
Everything (i.e., both for loop and UI) runs immediately and NSLog(#"Loop is Done"); is displayed also immediately, which tells me that semaphore didn't wait here:
dispatch_semaphore_wait(mySemaphore, DISPATCH_TIME_FOREVER);
NSLog(#"Loop is Done");
???
I spent the whole evening trying to figure this out, but to no avail.
I hope someone with great GCD knowledge can enlighten me on this.
First things first: As a general rule, one should never block the main queue. This rule about not blocking the main queue applies to both dispatch_semaphore_wait() and sleep() (as well as any of the synchronous dispatches, any group wait, etc.). You should never do any of these potentially blocking calls on the main queue. And if you follow this rule, your UI should never become non-responsive.
Your code sample and subsequent questions might seem to suggest a confusion between groups and semaphores. Dispatch groups are a way of keeping track of a group of dispatched blocks. But you're not taking advantage of the features of dispatch groups here, so I might suggest excising them from the discussion as it's irrelevant to the discussion about semaphores.
Dispatch semaphores are, on the other hand, simply a mechanism for one thread to send a signal to another thread that is waiting for the signal. Needless to say, the fact that you've created a semaphore and sent signals via that semaphore will not affect any of your dispatched tasks (whether to group or not) unless the code in question happens to call dispatch_semaphore_wait.
Finally, in some of your later examples you tried have the semaphore send multiple signals or changing the initial count to supplied when creating the semaphore. For each signal, you generally want a corresponding wait. If you have ten signals, you want ten waits.
So, let's illustrate semaphores in a way where your main queue (and thus the UI) will never be blocked. Here, we can send ten signals between two separate concurrently running tasks, having the latter one update the UI:
dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);
// send 10 signals from one background thread
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
for (NSInteger i = 0; i < 10; i++) {
NSLog(#"Sleeping %d", i);
sleep(3);
NSLog(#"Sending signal %d", i);
dispatch_semaphore_signal(semaphore);
}
NSLog(#"Done signaling");
});
// and on another thread, wait for those 10 signals ...
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
for (NSInteger i = 0; i < 10; i++) {
NSLog(#"Waiting for signal %d", i);
dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
NSLog(#"Got signal %d", i);
// if you want to update your UI, then dispatch that back to the main queue
dispatch_async(dispatch_get_main_queue(), ^{
// update your UI here
});
}
NSLog(#"Done waiting");
});
This is, admittedly, not a terribly useful example of semaphores, but it illustrates how theoretically you could use them. In practice, it's rare that you have to use semaphores, as for most business problems, there are other, more elegant coding patterns. If you describe what you're trying to do, we can show you how to best achieve it.
As for an example with non-zero value passed to dispatch_semaphore_create, that's used to control access to some finite resource. In this example, let's assume that you had 100 tasks to run, but you didn't want more than 5 to run at any given time (e.g. you're using network connections (which are limited), or the each operation takes up so much memory that you want to avoid having more than five running at any given time). Then you could do something like:
// we only want five to run at any given time
dispatch_semaphore_t semaphore = dispatch_semaphore_create(5);
// send this to background queue, so that when we wait, it doesn't block main queue
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
for (NSInteger i = 0; i < 100; i++)
{
// wait until one of our five "slots" are available
dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
// when it is, dispatch code to background queue
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
NSLog(#"starting %d", i);
// to simulate something slow happening in the background, we'll just sleep
sleep(5);
NSLog(#"Finishing %d", i);
// when done, signal that this "slot" is free (please note, this is done
// inside the dispatched block of code)
dispatch_semaphore_signal(semaphore);
});
}
});
Again, this isn't a great example of semaphores (in this case, I'd generally use an NSOperationQueue with a maxConcurrentOperationCount), but it illustrates an example of why you'd use a non-zero value for dispatch_source_create.
You've asked a number of questions about groups. I contend that groups are unrelated to your own semaphores. You might use a group, for example, if you want to run a block of code when all of the tasks are complete. So here is a variation of the above example, but using a dispatch_group_notify to do something when all of the other tasks in that group are complete.
dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0); // or create your own concurrent queue
dispatch_semaphore_t semaphore = dispatch_semaphore_create(5);
dispatch_group_t group = dispatch_group_create();
// send this to background queue, so that when we wait, it doesn't block main queue
dispatch_async(queue, ^{
for (NSInteger i = 0; i < 100; i++)
{
// wait until one of our five "slots" are available
dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
// when it is, dispatch code to background queue
dispatch_group_async(group, queue, ^{
NSLog(#"starting %d", i);
// to simulate something slow happening in the background, we'll just sleep
sleep(5);
NSLog(#"Finishing %d", i);
dispatch_semaphore_signal(semaphore);
});
}
dispatch_group_notify(group, queue, ^{
NSLog(#"All done");
});
});

How to illustrate a background task using GCD?

I want to illustrate the progress on MBProgressHUD item, but when i triger this method :
- (IBAction)signInBttn:(id)sender {
MBProgressHUD *hudd = [MBProgressHUD showHUDAddedTo:self.view animated:YES];
hudd.mode = MBProgressHUDModeAnnularDeterminate;
hudd.labelText = #"Loading";
__block float value = 0;
for (int j = 0; j<2000; j++) {
dispatch_async( dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
for (int i = 0; i<20000 ; i++) {
}
value += 0.001;
dispatch_async( dispatch_get_main_queue(), ^{
hudd.progress = value;
});
});
}
}
hud appears fully to 100%. This is only for my information, I dont have idea how to create background task which calculate something and when he done with e.g. 40% the HUD is refreshing to 40% of his progress. I hope I made ​​myself clear, and if anyone has time to help improve my code, thanks a lot for any answers
In this case, you can solve the problem by decoupling the updating of the counter from the updating of your HUD in your UI. Apple refers to this as "updating the state asynchronously" in WWDC 2012 video Asynchronous Design Patterns with Blocks, GCD, and XPC.
Generally this isn't necessary (most of the time the stuff we're doing asynchronously is slow enough that we don't have problems), but if doing something that is running faster than the UI can hope to keep up with, you create a "dispatch source" for this. I'm going to illustrate it with a UIProgressView, but the same applies to pretty much any UI:
// create source for which we'll be incrementing a counter,
// and tell it to run the event handler in the main loop
// (because we're going to be updating the UI)
dispatch_source_t source = dispatch_source_create(DISPATCH_SOURCE_TYPE_DATA_ADD, 0, 0, dispatch_get_main_queue());
// specify what you want the even handler to do (i.e. update the HUD or progress bar)
dispatch_source_set_event_handler(source, ^{
self.iterations += dispatch_source_get_data(source);
[self.progressView setProgress: (float) self.iterations / kMaxIterations];
});
// start the dispatch source
dispatch_resume(source);
// now, initiate the process that will update the source
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
for (long i = 0; i < kMaxIterations; i++)
{
// presumably, do something meaningful here
// now increment counter (and the event handler will take care of the UI)
dispatch_source_merge_data(source, 1);
}
// when all done, cancel the dispatch source
dispatch_source_cancel(source);
});
In my example, iterations is just a long property:
#property (nonatomic) long iterations;
And I defined my kMaxIterations constant as follows:
static long const kMaxIterations = 10000000l;
First off, if you want to delay execution use dispatch_after: Apple Doc since it could be that Clang is optimizing your loop (i.e. by making it not exist).
Within that block call dispatch_sync on the main thread to update the UI, since dispatch_async is not guaranteed to execute 'evenly'. Something like this ought to work...
for (...) {
dispatch_after(<some formula of i>, DEFAULT_PRIORITY, ^{
dispatch_sync(MAIN_QUEUE, ^{ hudd.progress = value });
}
}

Cannot get Grand Central Dispatch implemented correctly on iOS

I'm trying to update and load a UIProgressBar while some code is running on a background thread. When the background thread has finished loading creating an avatar it should update the variable avatarFinishedLoading to TRUE which enables the code within the Open GL ES update method to begin to function.
- (void)viewDidAppear:(BOOL)animated
{
[self performSelectorOnMainThread:#selector(updateProgressBar) withObject:nil waitUntilDone:NO];
dispatch_queue_t loadAvatarAndFrames = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
dispatch_async(loadAvatarAndFrames, ^{
[self createAvatar];
dispatch_async(dispatch_get_main_queue(), ^{
avatarFinishedLoading = TRUE;
});
});
}
- (void)updateProgressBar {
float progress = [threadProgressView progress];
if (progress < 1) {
threadProgressView.progress = progress + (float)0.01;
[NSTimer scheduledTimerWithTimeInterval:0.05 target:self selector:#selector(updateProgressBar) userInfo:nil repeats:NO];
}else{
[threadProgressView setHidden:TRUE];
}
}
- (void)update
{
// Set colour of background
glClearColor(self.backgroundColour.x, self.backgroundColour.y, self.backgroundColour.z, self.backgroundColour.w);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
if (avatarFinishedLoading == TRUE){
//Do Some Code
}
}
The application is crashing with the error message: 0xf5a2503: movzwl (%eax,%ecx,2), %eax
exec_bad_access
I cannot figure out where I am going wrong as it seems ok to me.
EDIT
It appears that I have a GL Error: 1280 when executing this code on the background thread. If I run the 'createAvatar' method on the main thread I do not get any GL Errors and the avatar renders as expected
Thanks
There is nothing wrong with the way you set your ivar since that is done on the main thread.
dispatch_async(dispatch_get_main_queue(), ^{
avatarFinishedLoading = TRUE;
});
Is it possible you touch something in -createAvatar method that you should not touch on a background thread?

NSOperation: addsubview in the main thread and slowness

I have implemented the following NSOperation, to draw N custom views
- (void)main {
for (int i=0; i<N; i++) {
<< Alloc and configure customView #i >>
//(customView is a UIView with some drawing code in drawrect)
[delegate.view addSubview:customView];
}
NSLog(#"Operation completed");
}
in the drawRect method of the customView I have
- (void)drawRect {
<<Drawing code>>
NSLog(#"Drawed");
delegate.drawedViews++;
if (delegate.drawedViews==VIEWS_NUMBER) {
[delegate allViewsDrawn];
}
}
So the delegate get the notification when all the views are drawn.
The problem is that after the "Operation completed" log it takes about 5 seconds before I can see the first "Drawed" log.
Why is this happening? And generally speaking, how should I behave in order to find out which line of code is taking so much time being executed?
------ EDIT ------
Sometimes (like 1 out of 10 times) I was getting crashes doing this because I shouldn't call addsubview from the NSOperation since it is not thread-safe. So I changed it to:
[delegate.view performSelectorOnMainThread:#selector(addSubview:) withObject:customView waitUntilDone:NO];
Now I don't have crashes anymore, but the process takes a very long time to be executed! Like 5 times more than before.
Why is it so slow?
To make things work properly we need to forget about NSOperation and use this "trick"
dispatch_queue_t main_queue = dispatch_get_main_queue();
dispatch_async(main_queue, ^{
[self createCustomViews];
dispatch_async(main_queue, ^{
[self addAnotherCustomViewToView];
});
});

Resources