Am Captuing video using AVFoundation frame work .With the help of Apple Documentation http://developer.apple.com/library/ios/#documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/03_MediaCapture.html%23//apple_ref/doc/uid/TP40010188-CH5-SW2
Now i did Following things
1.Created videoCaptureDevice
2.Created AVCaptureDeviceInput and set videoCaptureDevice
3.Created AVCaptureVideoDataOutput and implemented Delegate
4.Created AVCaptureSession - set input as AVCaptureDeviceInput and set output as AVCaptureVideoDataOutput
5.In AVCaptureVideoDataOutput Delegate method
-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
i got CMSamplebuffer and Converted into UIImage And tested to print UIImageview using
[self.imageView performSelectorOnMainThread:#selector(setImage:) withObject:image waitUntilDone:YES];
Every thing went well up to this........
MY Problem IS,
I need to send video frames through UDP Socket .even though following one is bad idea i tried ,UIImage to NSData and Send via UDP Pocket. BUt got so Delay in video Processing.Mostly problem because of UIImage to NSDate
So Please GIve me Solution For my problem
1)Any way to convert CMSampleBUffer or CVImageBuffer to NSData ??
2)Like Audio Queue Service and Queue for Video to store UIImage and do UIImage to NSDate
And Sending ???
if am riding behind the Wrong Algorithm Please path me in write direction
Thanks In Advance
Here is code to get at the buffer. This code assumes a flat image (e.g. BGRA).
NSData* imageToBuffer( CMSampleBufferRef source) {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(source);
CVPixelBufferLockBaseAddress(imageBuffer,0);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
void *src_buff = CVPixelBufferGetBaseAddress(imageBuffer);
NSData *data = [NSData dataWithBytes:src_buff length:bytesPerRow * height];
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
return [data autorelease];
}
A more efficient approach would be to use a NSMutableData or a buffer pool.
Sending a 480x360 image every second will require a 4.1Mbps connection assuming 3 color channels.
Use CMSampleBufferGetImageBuffer to get CVImageBufferRef from the sample buffer, then get the bitmap data from it with CVPixelBufferGetBaseAddress. This avoids needlessly copying the image.
Related
I am trying to transmit real time video buffers on one iPhone to another iPhone (called client iPhone) for preview display, and also to accept commands from the client iPhone. I am thinking of a standard way to achieve this. The closest thing I found is AVCaptureMultipeerVideoDataOutput on Github.
However that still uses Multipeer connectivity framework and I think it still requires some setup on both iPhones. The thing I want is there should be ideally no setup required on both iPhones, as long as Wifi (or if possible, bluetooth) is enabled on both iPhones, the peers should recognize each other within the app and prompt user about device discovery. What are the standard ways to achieve this and any links to sample code?
EDIT: I got it working through Multipeer connectivity after writing code from scratch. As of now, I am sending the pixel buffers to peer device by downscaling & compressing the data as jpeg. On the remote device, I have UIImage setup where I display the data every frame time. However I think UIKit may not be the best way to display data, even though images are small. How do I display this data using OpenGLES? Is direct decoding of jpeg possible in Opengles?
Comments:
As of now, I am sending the pixel buffers to peer device by
downscaling & compressing the data as jpeg. On the remote device, I
have UIImage setup where I display the data every frame time. However
I think UIKit may not be the best way to display data, even though
images are small.
Turns out, this is the best way to transmit an image via the Multipeer Connectivity framework. I have tried all the alternatives:
I've compressed frames using VideoToolbox. Too slow.
I've compressed frames using Compression. Too slow, but better.
Let me provide some code for #2:
On the iOS device transmitting image data:
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0);
__block uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
dispatch_async(self.compressionQueue, ^{
uint8_t *compressed = malloc(sizeof(uint8_t) * 1228808);
size_t compressedSize = compression_encode_buffer(compressed, 1228808, baseAddress, 1228808, NULL, COMPRESSION_ZLIB);
NSData *data = [NSData dataWithBytes:compressed length:compressedSize];
NSLog(#"Sending size: %lu", [data length]);
dispatch_async(dispatch_get_main_queue(), ^{
__autoreleasing NSError *err;
[((ViewController *)self.parentViewController).session sendData:data toPeers:((ViewController *)self.parentViewController).session.connectedPeers withMode:MCSessionSendDataReliable error:&err];
});
});
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}
On the iOS device displaying image data:
typedef struct {
size_t length;
void *data;
} ImageCacheDataStruct;
- (void)session:(nonnull MCSession *)session didReceiveData:(nonnull NSData *)data fromPeer:(nonnull MCPeerID *)peerID
{
NSLog(#"Receiving size: %lu", [data length]);
uint8_t *original = malloc(sizeof(uint8_t) * 1228808);
size_t originalSize = compression_decode_buffer(original, 1228808, [data bytes], [data length], NULL, COMPRESSION_ZLIB);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef newContext = CGBitmapContextCreate(original, 640, 480, 8, 2560, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGImageRef newImage = CGBitmapContextCreateImage(newContext);
UIImage *image = [[UIImage alloc] initWithCGImage:newImage scale:1 orientation:UIImageOrientationUp];
CGContextRelease(newContext);
CGColorSpaceRelease(colorSpace);
CGImageRelease(newImage);
if (image) {
dispatch_async(dispatch_get_main_queue(), ^{
[((ViewerViewController *)self.childViewControllers.lastObject).view.layer setContents:(__bridge id)image.CGImage];
});
}
}
Although this code produces original-quality images on the receiving end, you'll find this far too slow for real-time playback.
Here's the best way to do it:
On the iOS device sending the image data:
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0);
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGImageRef newImage = CGBitmapContextCreateImage(newContext);
UIImage *image = [[UIImage alloc] initWithCGImage:newImage scale:1 orientation:UIImageOrientationUp];
CGImageRelease(newImage);
CGContextRelease(newContext);
CGColorSpaceRelease(colorSpace);
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
if (image) {
NSData *data = UIImageJPEGRepresentation(image, 0.7);
NSError *err;
[((ViewController *)self.parentViewController).session sendData:data toPeers:((ViewController *)self.parentViewController).session.connectedPeers withMode:MCSessionSendDataReliable error:&err];
}
}
On the iOS device receiving the image data:
- (void)session:(nonnull MCSession *)session didReceiveData:(nonnull NSData *)data fromPeer:(nonnull MCPeerID *)peerID
{
dispatch_async(self.imageCacheDataQueue, ^{
dispatch_semaphore_wait(self.semaphore, DISPATCH_TIME_FOREVER);
const void *dataBuffer = [data bytes];
size_t dataLength = [data length];
ImageCacheDataStruct *imageCacheDataStruct = calloc(1, sizeof(imageCacheDataStruct));
imageCacheDataStruct->data = (void*)dataBuffer;
imageCacheDataStruct->length = dataLength;
__block const void * kMyKey;
dispatch_queue_set_specific(self.imageDisplayQueue, &kMyKey, (void *)imageCacheDataStruct, NULL);
dispatch_sync(self.imageDisplayQueue, ^{
ImageCacheDataStruct *imageCacheDataStruct = calloc(1, sizeof(imageCacheDataStruct));
imageCacheDataStruct = dispatch_queue_get_specific(self.imageDisplayQueue, &kMyKey);
const void *dataBytes = imageCacheDataStruct->data;
size_t length = imageCacheDataStruct->length;
NSData *imageData = [NSData dataWithBytes:dataBytes length:length];
UIImage *image = [UIImage imageWithData:imageData];
if (image) {
dispatch_async(dispatch_get_main_queue(), ^{
[((ViewerViewController *)self.childViewControllers.lastObject).view.layer setContents:(__bridge id)image.CGImage];
dispatch_semaphore_signal(self.semaphore);
});
}
});
});
}
The reason for the semaphores and the separate GCD queues is simple: you want the frames to display at equal time intervals. Otherwise, the video will seem to slow down at first at times, right before speeding up way past normal in order to catch up. My scheme ensures that each frame plays one after another at the same pace, regardless of network bandwidth bottlenecks.
I'm trying to save full resolution tiff files from the camera. I've seen a bunch of really helpful guides around here on capturing still images in either straight pixel data or using the jpeg hardware compressor. Thus far I'm able to capture straight pixel data in BGRA format, but I can't seem to get a sample buffer larger than 1920x1080 on a 5s. When I switch to the jpeg compressor route I get the full 5MP image.. just in jpeg format.
Here's my setup:
// Create the AVCaptureSession
AVCaptureSession *session = [[AVCaptureSession alloc] init];
[self.session setSessionPreset:AVCaptureSessionPresetPhoto];
and later on for output settings:
NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey,nil];
Or:
[stillImageOutput setOutputSettings:#{AVVideoCodecKey : AVVideoCodecJPEG}];
Before I ask for the sample buffer I set:
setHighResolutionStillImageOutputEnabled:YES
Then I'm using:
-captureStillImageAsynchronouslyFromConnection
to get the sample buffer.
Just to finish up...
Within the completion block of -captureStillImageAsynchronouslyFromConnection:
For Jpegs I use:
NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageDataSampleBuffer];
For BGRA I use:
CFDictionaryRef metadata = CMCopyDictionaryOfAttachments(kCFAllocatorDefault, imageDataSampleBuffer, kCMAttachmentMode_ShouldPropagate);
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(imageDataSampleBuffer);
// >>>>>>>>>> lock buffer address
CVPixelBufferLockBaseAddress(imageBuffer, 0);
//Get information about the image
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
// create suitable color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
//Create suitable context (suitable for camera output setting kCVPixelFormatType_32BGRA)
CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// <<<<<<<<<< unlock buffer address
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
// release color space
CGColorSpaceRelease(colorSpace);
//Create a CGImageRef from the CVImageBufferRef
CGImageRef newImage = CGBitmapContextCreateImage(newContext);
Have I overlooked something? Everything in the pipeline seems to work and I can't find any extra settings in the Apple docs. Is there a better/different way to do this?
TDLR: I can't seem to get more than 1920x1080 from a still image capture session using the BGRA pixel format output settings. Hoping someone can point me in the right direction.
Oops! I set the class var instead of the local one and overwrote the class var later on so no session preset is used.
Code should be:
// Create the AVCaptureSession
AVCaptureSession *session = [[AVCaptureSession alloc] init];
[session setSessionPreset:AVCaptureSessionPresetPhoto];
Super simple mistake.
I need to obtain the UIImage from uncompressed image data from CMSampleBufferRef. I'm using the code:
captureStillImageOutput captureStillImageAsynchronouslyFromConnection:connection
completionHandler:^(CMSampleBufferRef imageSampleBuffer, NSError *error)
{
// that famous function from Apple docs found on a lot of websites
// does NOT work for still images
UIImage *capturedImage = [self imageFromSampleBuffer:imageSampleBuffer];
}
http://developer.apple.com/library/ios/#qa/qa1702/_index.html is a link to imageFromSampleBuffer function.
But it does not work properly. :(
There is a jpegStillImageNSDataRepresentation:imageSampleBuffer method, but it gives the compressed data (well, because JPEG).
How can I get UIImage created with the most raw non-compressed data after capturing Still Image?
Maybe, I should specify some settings to video output? I'm currently using those:
captureStillImageOutput = [[AVCaptureStillImageOutput alloc] init];
captureStillImageOutput.outputSettings = #{ (id)kCVPixelBufferPixelFormatTypeKey : #(kCVPixelFormatType_32BGRA) };
I've noticed, that output has a default value for AVVideoCodecKey, which is AVVideoCodecJPEG. Can it be avoided in any way, or does it even matter when capturing still image?
I found something there: Raw image data from camera like "645 PRO" , but I need just a UIImage, without using OpenCV or OGLES or other 3rd party.
The method imageFromSampleBuffer does work in fact I'm using a changed version of it, but if I remember correctly you need to set the outputSettings right. I think you need to set the key as kCVPixelBufferPixelFormatTypeKey and the value as kCVPixelFormatType_32BGRA.
So for example:
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
NSDictionary* outputSettings = [NSDictionary dictionaryWithObject:value forKey:key];
[newStillImageOutput setOutputSettings:outputSettings];
EDIT
I am using those settings to take stillImages not video.
Is your sessionPreset AVCaptureSessionPresetPhoto? There may be problems with that
AVCaptureSession *newCaptureSession = [[AVCaptureSession alloc] init];
[newCaptureSession setSessionPreset:AVCaptureSessionPresetPhoto];
EDIT 2
The part about saving it to UIImage is identical with the one from the documentation. That's the reason I was asking for other origins of the problem, but I guess that was just grasping for straws.
There is another way I know of, but that requires OpenCV.
- (UIImage *) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
// Get the number of bytes per row for the pixel buffer
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
// Get the pixel buffer width and height
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
// Create a device-dependent RGB color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,
bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Unlock the pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
// Create an image object from the Quartz image
UIImage *image = [UIImage imageWithCGImage:quartzImage];
// Release the Quartz image
CGImageRelease(quartzImage);
return (image);
}
I guess that is of no help to you, sorry. I don't know enough to think of other origins for your problem.
Here's a more efficient way:
UIImage *image = [UIImage imageWithData:[self imageToBuffer:sampleBuffer]];
- (NSData *) imageToBuffer:(CMSampleBufferRef)source {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(source);
CVPixelBufferLockBaseAddress(imageBuffer,0);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
void *src_buff = CVPixelBufferGetBaseAddress(imageBuffer);
NSData *data = [NSData dataWithBytes:src_buff length:bytesPerRow * height];
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
return data;
}
I am trying to create an application which runs a FFT on microphone data, so I can examine e.g. the loudest frequency in the input.
I see that there are many methods of getting audio input (the RemoteIO AudioUnit, AudioQueue services, and AVFoundation) but it seems like AVFoundation is the simplest. I have this setup:
// Configure the audio session
AVAudioSession *session = [AVAudioSession sharedInstance];
[session setCategory:AVAudioSessionCategoryRecord error:NULL];
[session setMode:AVAudioSessionModeMeasurement error:NULL];
[session setActive:YES error:NULL];
// Optional - default gives 1024 samples at 44.1kHz
//[session setPreferredIOBufferDuration:samplesPerSlice/session.sampleRate error:NULL];
// Configure the capture session (strongly-referenced instance variable, otherwise the capture stops after one slice)
_captureSession = [[AVCaptureSession alloc] init];
// Configure audio device input
AVCaptureDevice *device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeAudio];
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:NULL];
[_captureSession addInput:input];
// Configure audio data output
AVCaptureAudioDataOutput *output = [[AVCaptureAudioDataOutput alloc] init];
dispatch_queue_t queue = dispatch_queue_create("My callback", DISPATCH_QUEUE_SERIAL);
[output setSampleBufferDelegate:self queue:queue];
[_captureSession addOutput:output];
// Start the capture session.
[_captureSession startRunning];
(plus error checking, omitted here for readability).
Then I implement the following AVCaptureAudioDataOutputSampleBufferDelegate method:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
NSLog(#"Num samples: %ld", CMSampleBufferGetNumSamples(sampleBuffer));
// Usually gives 1024 (except the first slice)
}
I'm unsure what the next step should be. What exactly does the CMSampleBuffer format describe (and what assumptions can be made about it, if any)? How should I get the raw audio data into vDSP_fft_zrip with the least possible amount of extra preprocessing? (Also, what would you recommend doing to verify that the raw data I see is correct?)
The CMSampleBufferRef is an opaque type that contains 0 or more media samples. There is a bit of blurb in the docs:
http://developer.apple.com/library/ios/#documentation/CoreMedia/Reference/CMSampleBuffer/Reference/reference.html
In this case it will contain an audio buffer, as well as the description of the sample format and timing information and so on. If you are really interested just put a breakpoint in the delegate callback and take a look.
The first step is to get a pointer to the data buffer that has been returned:
// get a pointer to the audio bytes
CMItemCount numSamples = CMSampleBufferGetNumSamples(sampleBuffer);
CMBlockBufferRef audioBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
size_t lengthAtOffset;
size_t totalLength;
char *samples;
CMBlockBufferGetDataPointer(audioBuffer, 0, &lengthAtOffset, &totalLength, &samples);
The default sample format for the iPhone mic is linear PCM, with 16 bit samples. This may be mono or stereo depending on if there is an external mic or not. To calculate the FFT we need to have a float vector. Fortunately there is an accelerate function to do the conversion for us:
// check what sample format we have
// this should always be linear PCM
// but may have 1 or 2 channels
CMAudioFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
const AudioStreamBasicDescription *desc = CMAudioFormatDescriptionGetStreamBasicDescription(format);
assert(desc->mFormatID == kAudioFormatLinearPCM);
if (desc->mChannelsPerFrame == 1 && desc->mBitsPerChannel == 16) {
float *convertedSamples = malloc(numSamples * sizeof(float));
vDSP_vflt16((short *)samples, 1, convertedSamples, 1, numSamples);
} else {
// handle other cases as required
}
Now you have a float vector of the sample buffer which you can use with vDSP_fft_zrip. It doesn't seem possible to change the input format from the microphone to float samples with AVFoundation, so you are stuck with this last conversion step. I would keep around the buffers in practice, reallocing them if necessary when a larger buffer arrives, so that you are not mallocing and freeing buffers with every delegate callback.
As for your last question, I guess the easiest way to do this would be to inject a known input and check that it gives you the correct response. You could play a sine wave into the mic and check that your FFT had a peak in the correct frequency bin, something like that.
I don't suggest to use AVFoundation for 3 reasons:
I used it for some of mine apps (morsedec , irtty), it works well on simulator and in some hardware, but in others totally failed !
you do not have good control of sample rate an format.
latency could be high.
I suggest to start with apple's sample code aurioTouch.
To make FFT you can shift to vDSP framework using a circular buffer (I LOVE https://github.com/michaeltyson/TPCircularBuffer).
Hope this help
A time ago I already asked this question and I also got a good answer:
I've been searching this forum up and down but I couldn't find what I
really need. I want to get raw image data from the camera. Up till now
I tried to get the data out of the imageDataSampleBuffer from that
method
captureStillImageAsynchronouslyFromConnection:completionHandler: and
to write it to an NSData object, but that didn't work. Maybe I'm on
the wrong track or maybe I'm just doing it wrong. What I don't want is
for the image to be compressed in any way.
The easy way is to use jpegStillImageNSDataRepresentation: from
AVCaptureStillImageOutput, but like I said I don't want it to be
compressed.
Thanks!
Raw image data from camera
I thought I could work with this, but I finally noticed that I need to get raw image data more directly in a similar way as it is done in "645 PRO".
645 PRO: RAW Redux
The pictures on that site show that they get the raw data before any jpeg compression is done. That is what I want to do. My guess is that I need to transform imageDataSampleBuffer but I don't see a way to do it completely without compression.
"645 PRO" also saves its pictures in TIFF so I think it uses at least one additional library.
I don't want to make a photo app but I need the best quality I get to check for certain features in a picture.
Thanks!
Edit 1:
So after trying and searching in different directions for a while now I decided to give a status update.
The final goal of this project is to check for certain features in a picture which will happen with the help of opencv. But until the app is able to do it on the phone I'm trying to get mostly uncompressed pictures out of the phone to analyse them on the computer.
Therefore I want to save the "NSData instance containing the uncompressed BGRA bytes returned from the camera" I'm able to get with Brad Larson's code as bmp or TIFF file.
As I said in a comment I tried using opencv for this (it will be needed anyway). But the best I could do was turning it into a UIImage with a function from Computer Vision Talks.
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
cv::Mat frame(height, width, CV_8UC4, (void*)baseAddress);
UIImage *testImag = [UIImage imageWithMat:frame andImageOrientation:UIImageOrientationUp];
//imageWithMat... being the function from Computer Vision Talks which I can post if someone wants to see it
ImageMagick - Approach
Another thing I tried was using ImageMagick as suggested in another post.
But I couldn't find a way to do it without using something like UIImagePNGRepresentationor UIImageJPEGRepresentation.
For now I'm trying to do something with libtiff using this tutorial.
Maybe someone has an idea or knows a much easier way to convert my buffer object into an uncompressed picture.
Thanks in advance again!
Edit 2:
I found something! And I must say I was very blind.
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
cv::Mat frame(height, width, CV_8UC4, (void*)baseAddress);
NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *documentsDirectory = [paths objectAtIndex:0];
NSString *filePath = [documentsDirectory stringByAppendingPathComponent:[NSString stringWithFormat:#"ocv%d.TIFF", picNum]];
const char* cPath = [filePath cStringUsingEncoding:NSMacOSRomanStringEncoding];
const cv::string newPaths = (const cv::string)cPath;
cv::imwrite(newPaths, frame);
I just have to use the imwrite function from opencv. This way I get TIFF-files around 30 MB directly after the beyer-Polarisation!
Wow, that blog post was something special. A whole lot of words to just state that they get the sample buffer bytes that Apple hands you back from a still image. There's nothing particularly innovative about their approach, and I know a number of camera applications that do this.
You can get at the raw bytes returned from a photo taken with a AVCaptureStillImageOutput using code like the following:
[photoOutput captureStillImageAsynchronouslyFromConnection:[[photoOutput connections] objectAtIndex:0] completionHandler:^(CMSampleBufferRef imageSampleBuffer, NSError *error) {
CVImageBufferRef cameraFrame = CMSampleBufferGetImageBuffer(imageSampleBuffer);
CVPixelBufferLockBaseAddress(cameraFrame, 0);
GLubyte *rawImageBytes = CVPixelBufferGetBaseAddress(cameraFrame);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(cameraFrame);
NSData *dataForRawBytes = [NSData dataWithBytes:rawImageBytes length:bytesPerRow * CVPixelBufferGetHeight(cameraFrame)];
// Do whatever with your bytes
CVPixelBufferUnlockBaseAddress(cameraFrame, 0);
}];
This will give you an NSData instance containing the uncompressed BGRA bytes returned from the camera. You can save these to disk or do whatever you want with them. If you really need to process the bytes themselves, I'd avoid the overhead of the NSData creation and just work with the byte array from the pixel buffer.
I could solve it with OpenCV. Thanks to everyone who helped me.
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
cv::Mat frame(height, width, CV_8UC4, (void*)baseAddress);
NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *documentsDirectory = [paths objectAtIndex:0];
NSString *filePath = [documentsDirectory stringByAppendingPathComponent:[NSString stringWithFormat:#"ocv%d.BMP", picNum]];
const char* cPath = [filePath cStringUsingEncoding:NSMacOSRomanStringEncoding];
const cv::string newPaths = (const cv::string)cPath;
cv::imwrite(newPaths, frame);
I just have to use the imwrite function from opencv. This way I get BMP-files around 24 MB directly after the bayer-filter!
While the core of the answer comes from Brad at iOS: Get pixel-by-pixel data from camera, a key element is completely unclear from Brad's reply. It's hidden in "once you have your capture session configured...".
You need to set the correct outputSettings for your AVCaptureStillImageOutput.
For example, setting kCVPixelBufferPixelFormatTypeKey to kCVPixelFormatType_420YpCbCr8BiPlanarFullRange will give you a YCbCr imageDataSampleBuffer in captureStillImageAsynchronouslyFromConnection:completionHandler:, which you can then manipulate to your heart's content.
as #Wildaker mentioned, for a specific code to work you have to be sure which pixel format the camera is sending you. The code from #thomketler will work if it's set for 32-bit RGBA format.
Here is a code for the YUV default from camera, using OpenCV:
cv::Mat convertImage(CMSampleBufferRef sampleBuffer)
{
CVImageBufferRef cameraFrame = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(cameraFrame, 0);
int w = (int)CVPixelBufferGetWidth(cameraFrame);
int h = (int)CVPixelBufferGetHeight(cameraFrame);
void *baseAddress = CVPixelBufferGetBaseAddressOfPlane(cameraFrame, 0);
cv::Mat img_buffer(h+h/2, w, CV_8UC1, (uchar *)baseAddress);
cv::Mat cam_frame;
cv::cvtColor(img_buffer, cam_frame, cv::COLOR_YUV2BGR_NV21);
cam_frame = cam_frame.t();
//End processing
CVPixelBufferUnlockBaseAddress( cameraFrame, 0 );
return cam_frame;
}
cam_frame should have the full BGR frame. I hope that helps.