I'm an undergraduate student and I'm programming some HumanSeg iPhone app. Now I have to read raw frames from the camera, but I found the codes in official guide aren't clear enough for me to understand.
When I get the frames (in CMPixelBuffer format) from the camera, I need to modify it (I mean I have to do some padding and resizing, and turn it into a CVPixelBuffer format, to feed it to a CoreML MobileNet).
I've been searching for the solution for weeks, but unfortunately I got nothing. In the offical guide I was told that these buffers "doen't offer direct access to inner data".
I even tried to use a Context to draw the CMPixelBuffer into a UIImage and draw it back to a CVPixelBuffer，and I found this process terribly slow, just as the offcial guide says. Since I'm doing video processing this method is unacceptable.
What am I supposed to do or read? I really appreciate your help.
Related
I am trying to make a photo editing application for iOS, but am not sure where to start looking. I have attached an image made in Word... that hopefully simply depicts what I am trying to achieve. It will involved manipulating individual pixels of a shape/image and masking/clipping. WHow should I start and what resources are available to me other than the developer docs?
Cheers
If you are not new to programming I would suggest a trial and run kind of approach. If it was me, I would follow a approach like this
Figuring out what to do/ what not to do
Do I need to develop the tech I want from scratch or can I use some pods ?
What are the good reads and example apps - (Try this)
Development approach
Build a photo gallery to pick images from
Build a EDIT mode screen
Get set of template overlay images
Figure out how to overlay them on top of each other
Export the final picture as one picture
The developer documentation is essential when it comes to learning new APIs, but sometimes it can be a little overwhelming. You can try reading raywenderlich.com tutorials on Core Image first to get an idea (link here) or find a book on computer graphics. It is essential to understand at least the underlying techniques to efficiently program image processing code. In many cases you'll find there is a more elegant technique than just looping on pixels and modifying one-by-one.
Then you can continue with reading on image compositing using core image for example.
I have planned to detect an image in a news paper play the video relevant to it. I have seen several news paper reading AR apps include this feature. But i couldn't find how to do so. How can I do it??
I dont expect any code. But like to know what are the steps I should follow to do this. Thank you.
You need to browse through the available marker-based AR SDKs - such SDKs let you defined in advance the database of images you would like to detect and respond to, and once any of these images is detected during runtime, you get some kind of an event with data on the detected image.
Vuforia is considered a good one and it has good samples, so it is supposed to be easier to start with. You should also check out Kudan, and there are more.
Is it possible to process(filter) HDR images through Core Image? I couldn't find much documentation on this, so I was wondering if someone possibly had an answer to it. I do know that it is possible to do the working space computations with RGBAh when you initialize a CIContext, so I figured that if we could do computations with floating point image formats, that it should be possible..
What, if it is not possible, are alternatives if you want to produce HDR effects on iOS?
EDIT: I thought I'd try to be a bit more concise. It is to my understanding that HDR images can be clamped and saved as .jpg, .png, and other image formats by clamping the pixel values. However, I'm more interested in doing tone mapping through Core Image on a HDR image that has not been converted yet. The issue is encoding a CIImage with a HDR image, supposedly with the .hdr extention.
EDIT2: Maybe it would be useful to useful to use CGImageCreate , along with CGDataProviderCreateWithFilename ?
I hope you have basic understanding of how HDR works. An HDR file is generated by capturing 2 or more images at different exposures and combining it. So even if there's something like .HDR file, it would be a container format with more than one jpg in it. Technically you can not give two image files at once as an input to a generic CIFilter.
And in iOS, as I remember, it's not possible to access original set of photos of an HDR but the processed final output. Even if you could, you'd have to manually do the HDR process and generate a single HDR png/jpg anyway before feeding it to a CIFilter.
Since there are people who ask for a CI HDR Algorithm, I decided to share my code on github. See:
https://github.com/schulz0r/CoreImage-HDR
It is the Robertson HDR algorithm, so you cannot use RAW images. Please see the unit tests if you want to know how to get the camera response and obtain the hdr image. CoreImage saturates pixel values outside [0.0 ... 1.0], so the HDR is scaled into said interval.
Coding with metal always causes messy code for me, so I decided to use MetalKitPlus which you have to include in your project. You can find it here:
https://github.com/LRH539/MetalKitPlus
I think you have to check out the dev/v2.0.0 branch. I will merge this into master in the future.
edit: Just clone the master branch of MetalKitPlus. Also, I added a more detailed description to my CI-HDR project.
You can now(iOS 10+) capture Raw images(coded on 12 bits) and then filter them the way you like using CIFilter. You might not get a dynamic range as wide as the one you get by using bracketed captures; nevertheless, it is still wider than capturing 8-bits images.
Check Apple's documentation for capturing and processing RAW images.
I also recommend you watch wwdc2016 video by Apple(move to the raw processing part).
First time I have needed to use the camera sense iOS 6, seems as though some things have been been deprecated. Did some research and couldn't find anything but I know its out there. I just need to take a picture, or get from the camera roll and save it to an Image View. Thanks to anyone that has helpful information.
All the helpful information is in the Apple documentation.
http://developer.apple.com/library/ios/#documentation/uikit/reference/UIImagePickerController_Class/UIImagePickerController/UIImagePickerController.html
Including the deprecated stuff (see Appendix A).
A more high level overview is the Camera Programming Topics guide:
http://developer.apple.com/library/ios/#documentation/AudioVideo/Conceptual/CameraAndPhotoLib_TopicsForIOS/Introduction/Introduction.html#//apple_ref/doc/uid/TP40010400
I need to move large amounts of pixels on the screen on an iOS device. What is the most efficient way of doing this?
So far I'm using glTexSubImage2D(), but I wonder if this can be done any faster. I noticed that OpenGL ES 2.0 does not support pixel buffers, but there seems to be a pixel buffer used by Core Video. Can I use that? Or maybe there's an Apple extension for OpenGL that could help me achieve something similar (I think saw a very vague mention about a client storage extension in one of the WWDC 2012 videos, but I can't find any documentation about it)? Any other way that I can speed this up?
My main concern is that glTexSubImage2D() copies all the pixels that I send. Ideally, I'd like to skip this step of copying the data, since I already have it prepared...
The client storage extension you're probably thinking of is CVOpenGLESTextureCacheCreateTextureFromImage; a full tutorial is here. That's definitely going to be the fastest way to get data to the GPU.
Frustratingly the only mention I can find of it in Apple's documentation is the iOS 4.3 to 5.0 API Differences document — do a quick search for CVOpenGLESTextureCache.h.