I don't know which value to use to fetch raw YUV420p data. code below first:
AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init];
output.alwaysDiscardsLateVideoFrames = YES;
output.videoSettings = #{(id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]};
//output.videoSettings = #{(id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA]};
dispatch_queue_t queue;
queue = dispatch_queue_create("CameraQueue", NULL);
[output setSampleBufferDelegate:self queue:queue];
[session addOutput:output];
I noticed that kCVPixelFormatType has some values, does somebody know which value is right to fetch raw YUV420p data?
kCVPixelFormatType_420YpCbCr8Planar
kCVPixelFormatType_420YpCbCr8PlanarFullRange
kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
can be one of them?
It depends on which particular YUV420 you want to get:
Planar/Biplanar refers to the arrangement of the luma and chroma components in memory, Planar meaning that each component comes in a buffer, contiguous or not, and Biplanar pointing to two buffers, one for luma and another for chroma, usually interleaved. An example of Planar is YUV420 format and an example of Biplanar is NV21 or NV12
VideoRange and FullRange refers to the values of the luma component, Video referring to [16,235] accepted levels and FullRange to [0,255]. This confusing agreement comes from the MPEG standard (see here)...
Related
I'm trying to save full resolution tiff files from the camera. I've seen a bunch of really helpful guides around here on capturing still images in either straight pixel data or using the jpeg hardware compressor. Thus far I'm able to capture straight pixel data in BGRA format, but I can't seem to get a sample buffer larger than 1920x1080 on a 5s. When I switch to the jpeg compressor route I get the full 5MP image.. just in jpeg format.
Here's my setup:
// Create the AVCaptureSession
AVCaptureSession *session = [[AVCaptureSession alloc] init];
[self.session setSessionPreset:AVCaptureSessionPresetPhoto];
and later on for output settings:
NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey,nil];
Or:
[stillImageOutput setOutputSettings:#{AVVideoCodecKey : AVVideoCodecJPEG}];
Before I ask for the sample buffer I set:
setHighResolutionStillImageOutputEnabled:YES
Then I'm using:
-captureStillImageAsynchronouslyFromConnection
to get the sample buffer.
Just to finish up...
Within the completion block of -captureStillImageAsynchronouslyFromConnection:
For Jpegs I use:
NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageDataSampleBuffer];
For BGRA I use:
CFDictionaryRef metadata = CMCopyDictionaryOfAttachments(kCFAllocatorDefault, imageDataSampleBuffer, kCMAttachmentMode_ShouldPropagate);
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(imageDataSampleBuffer);
// >>>>>>>>>> lock buffer address
CVPixelBufferLockBaseAddress(imageBuffer, 0);
//Get information about the image
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
// create suitable color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
//Create suitable context (suitable for camera output setting kCVPixelFormatType_32BGRA)
CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// <<<<<<<<<< unlock buffer address
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
// release color space
CGColorSpaceRelease(colorSpace);
//Create a CGImageRef from the CVImageBufferRef
CGImageRef newImage = CGBitmapContextCreateImage(newContext);
Have I overlooked something? Everything in the pipeline seems to work and I can't find any extra settings in the Apple docs. Is there a better/different way to do this?
TDLR: I can't seem to get more than 1920x1080 from a still image capture session using the BGRA pixel format output settings. Hoping someone can point me in the right direction.
Oops! I set the class var instead of the local one and overwrote the class var later on so no session preset is used.
Code should be:
// Create the AVCaptureSession
AVCaptureSession *session = [[AVCaptureSession alloc] init];
[session setSessionPreset:AVCaptureSessionPresetPhoto];
Super simple mistake.
I want to generate a true color animated Gif from a couple of PNG files represented as base64 string. I found this post and did something similar. I have an array with the dataUrls:
NSArray* imageDataUrls; // array with the data urls without data:image/png;base64, prefix
Here is what I did:
NSDictionary *fileProperties = #{
(__bridge id)kCGImagePropertyGIFDictionary: #{
(__bridge id)kCGImagePropertyGIFLoopCount: #0, // 0 means loop forever
}
};
NSDictionary *frameProperties = #{
(__bridge id)kCGImagePropertyGIFDictionary: #{
(__bridge id)kCGImagePropertyGIFDelayTime: #0.4f, // a float (not double!) in seconds, rounded to centiseconds in the GIF data
}
};
NSURL *documentsDirectoryURL = [[NSFileManager defaultManager] URLForDirectory:NSDocumentDirectory inDomain:NSUserDomainMask appropriateForURL:nil create:YES error:nil];
NSURL *fileURL = [documentsDirectoryURL URLByAppendingPathComponent:#"animated.gif"];
CFMutableDataRef destinationData = CFDataCreateMutable(kCFAllocatorDefault, 0);
CGImageDestinationRef destination = CGImageDestinationCreateWithData(destinationData, kUTTypeGIF, kFrameCount, NULL);
CGImageDestinationSetProperties(destination, (__bridge CFDictionaryRef)fileProperties);
NSData* myImageData;
UIImage *myImage = [UIImage alloc];
for (NSUInteger i = 0; i < kFrameCount; i++) {
#autoreleasepool {
myImageData = [NSData dataFromBase64String:[imageDataUrls objectAtIndex:i]];
myImage = [myImage initWithData: myImageData];
CGImageDestinationAddImage(destination, myImage.CGImage, (__bridge CFDictionaryRef)frameProperties);
}
}
myImageData = nil;
myImage = nil;
CFRelease(destination);
NSData* data = nil;
data = (__bridge NSData *)destinationData;
Finally, I send the gif image as base64EncodedString back to the phonegap container.
// send back gif image
CDVPluginResult* pluginResult = [CDVPluginResult resultWithStatus:CDVCommandStatus_OK messageAsString: [data base64EncodedString]];
It works good but the quality of the resulting gif image is bad. This is because it has only 256 colors.
Here is the original png image:
Here is a screenshot of the generated gif image:
How do I get the same quality as I imported, i.e., how can I raise the quality level of the generated gif? How do I generate true color gifs on iOS?
GIFs are not designed to store true-color data, and they are also poorly suited for animations1. Since this is such an unusual use of GIFs, you will have to write a lot of your own code.
Break each frame into rectangular chunks, where each chunk contains at most 256 distinct colors. The easiest way to do this is to use 16x16 chunks.
Convert each chunk to an indexed image.
Add each chunk to the GIF. For the first chunk in a frame, use the frame delay. For other chunks in a frame, use a delay of 0.
Done. You will have to familiarize yourself with the GIF specification, which is freely available online (GIF89a specification at W3.org, see section 23). You will also need to find an LZW compressor, which is not too hard to find. The animation will also use an obscene amount of storage: including base64 conversion, I estimate about 43 bits/pixel, or about 1.2 Gbit/s for 720p video, which is about 400x as much storage as you would use for high-quality MPEG4 or WebM, and probably about 3x as much storage as the PNGs would require. The storage and bandwidth requirements will likely incur undesirable costs for hosts and clients, unless the animations are very short and small.
Note that this will not allow you to use alpha transparency, this is a hard limitation of the GIF format.
Opinion
The idea of putting high quality animations in a GIF is absurd in the extreme, even though it is possible. It is especially absurd given the available alternatives:
If you are targeting modern browsers or mobile devices, MPEG4 (support matrix) and WebM (support matrix) are the obvious choices. Between the two formats, only Opera Mini supports neither.
If you are targeting older browsers or less-capable devices, or if you cannot afford MPEG4 encoding, you can encode the frames as individual JPEG or PNG images. Bundle these with a JSON payload with the timing, and use JavaScript or other client-side scripting to switch between animation frames. This works surprisingly well.
Notes
1 From the GIF 89a specification:
Animation - The Graphics Interchange Format is not intended as a platform for
animation, even though it can be done in a limited way.
I am trying to create an application which runs a FFT on microphone data, so I can examine e.g. the loudest frequency in the input.
I see that there are many methods of getting audio input (the RemoteIO AudioUnit, AudioQueue services, and AVFoundation) but it seems like AVFoundation is the simplest. I have this setup:
// Configure the audio session
AVAudioSession *session = [AVAudioSession sharedInstance];
[session setCategory:AVAudioSessionCategoryRecord error:NULL];
[session setMode:AVAudioSessionModeMeasurement error:NULL];
[session setActive:YES error:NULL];
// Optional - default gives 1024 samples at 44.1kHz
//[session setPreferredIOBufferDuration:samplesPerSlice/session.sampleRate error:NULL];
// Configure the capture session (strongly-referenced instance variable, otherwise the capture stops after one slice)
_captureSession = [[AVCaptureSession alloc] init];
// Configure audio device input
AVCaptureDevice *device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeAudio];
AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:NULL];
[_captureSession addInput:input];
// Configure audio data output
AVCaptureAudioDataOutput *output = [[AVCaptureAudioDataOutput alloc] init];
dispatch_queue_t queue = dispatch_queue_create("My callback", DISPATCH_QUEUE_SERIAL);
[output setSampleBufferDelegate:self queue:queue];
[_captureSession addOutput:output];
// Start the capture session.
[_captureSession startRunning];
(plus error checking, omitted here for readability).
Then I implement the following AVCaptureAudioDataOutputSampleBufferDelegate method:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
NSLog(#"Num samples: %ld", CMSampleBufferGetNumSamples(sampleBuffer));
// Usually gives 1024 (except the first slice)
}
I'm unsure what the next step should be. What exactly does the CMSampleBuffer format describe (and what assumptions can be made about it, if any)? How should I get the raw audio data into vDSP_fft_zrip with the least possible amount of extra preprocessing? (Also, what would you recommend doing to verify that the raw data I see is correct?)
The CMSampleBufferRef is an opaque type that contains 0 or more media samples. There is a bit of blurb in the docs:
http://developer.apple.com/library/ios/#documentation/CoreMedia/Reference/CMSampleBuffer/Reference/reference.html
In this case it will contain an audio buffer, as well as the description of the sample format and timing information and so on. If you are really interested just put a breakpoint in the delegate callback and take a look.
The first step is to get a pointer to the data buffer that has been returned:
// get a pointer to the audio bytes
CMItemCount numSamples = CMSampleBufferGetNumSamples(sampleBuffer);
CMBlockBufferRef audioBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
size_t lengthAtOffset;
size_t totalLength;
char *samples;
CMBlockBufferGetDataPointer(audioBuffer, 0, &lengthAtOffset, &totalLength, &samples);
The default sample format for the iPhone mic is linear PCM, with 16 bit samples. This may be mono or stereo depending on if there is an external mic or not. To calculate the FFT we need to have a float vector. Fortunately there is an accelerate function to do the conversion for us:
// check what sample format we have
// this should always be linear PCM
// but may have 1 or 2 channels
CMAudioFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
const AudioStreamBasicDescription *desc = CMAudioFormatDescriptionGetStreamBasicDescription(format);
assert(desc->mFormatID == kAudioFormatLinearPCM);
if (desc->mChannelsPerFrame == 1 && desc->mBitsPerChannel == 16) {
float *convertedSamples = malloc(numSamples * sizeof(float));
vDSP_vflt16((short *)samples, 1, convertedSamples, 1, numSamples);
} else {
// handle other cases as required
}
Now you have a float vector of the sample buffer which you can use with vDSP_fft_zrip. It doesn't seem possible to change the input format from the microphone to float samples with AVFoundation, so you are stuck with this last conversion step. I would keep around the buffers in practice, reallocing them if necessary when a larger buffer arrives, so that you are not mallocing and freeing buffers with every delegate callback.
As for your last question, I guess the easiest way to do this would be to inject a known input and check that it gives you the correct response. You could play a sine wave into the mic and check that your FFT had a peak in the correct frequency bin, something like that.
I don't suggest to use AVFoundation for 3 reasons:
I used it for some of mine apps (morsedec , irtty), it works well on simulator and in some hardware, but in others totally failed !
you do not have good control of sample rate an format.
latency could be high.
I suggest to start with apple's sample code aurioTouch.
To make FFT you can shift to vDSP framework using a circular buffer (I LOVE https://github.com/michaeltyson/TPCircularBuffer).
Hope this help
I'm trying to get as good an image as possible from the camera, but can only find examples that captureStillImageAsynchronouslyFromConnection and then go straight to:
NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageSampleBuffer];
UIImage *image = [[UIImage alloc] initWithData:imageData];
JPEG being lossy and all, is there any way to get the data as PNG, or even just RGBA (BGRA, what-have-you?). AVCaptureStillImageOutput doesn't seem to have any other NSData* methods....
Actually looking at the CMSampleBufferRef, it seems like it's already locked as JPEG ~
formatDescription = <CMVideoFormatDescription 0xfe5e1f0 [0x3e5ac650]> {
mediaType:'vide'
mediaSubType:'jpeg'
mediaSpecific: {
codecType: 'jpeg' dimensions: 2592 x 1936
}
extensions: {(null)}
}
Is there some other way to take a full-res picture and get the raw data?
You'll need to set the outputSettings with a different pixel format. If you want 32-bit BGRA, for example, you can set:
NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey, nil];
From https://developer.apple.com/library/mac/#documentation/AVFoundation/Reference/AVCaptureStillImageOutput_Class/Reference/Reference.html, the "recommended" pixel formats are:
kCMVideoCodecType_JPEG
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
kCVPixelFormatType_32BGRA
Of course, if you're not using JPEG output, you can't use jpegStillImageNSDataRepresentation:, but there's an example here:
how to convert a CVImageBufferRef to UIImage
Just change the Output settings of your connection:
outputSettings
The compression settings for the output.
#property(nonatomic, copy) NSDictionary *outputSettings
You can retreive an NSDictionary of the supported values with
availableImageDataCodecTypes
The supported image codec formats that can be specified in outputSettings. (read-only)
#property(nonatomic, readonly) NSArray *availableImageDataCodecTypes
Am Captuing video using AVFoundation frame work .With the help of Apple Documentation http://developer.apple.com/library/ios/#documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/03_MediaCapture.html%23//apple_ref/doc/uid/TP40010188-CH5-SW2
Now i did Following things
1.Created videoCaptureDevice
2.Created AVCaptureDeviceInput and set videoCaptureDevice
3.Created AVCaptureVideoDataOutput and implemented Delegate
4.Created AVCaptureSession - set input as AVCaptureDeviceInput and set output as AVCaptureVideoDataOutput
5.In AVCaptureVideoDataOutput Delegate method
-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
i got CMSamplebuffer and Converted into UIImage And tested to print UIImageview using
[self.imageView performSelectorOnMainThread:#selector(setImage:) withObject:image waitUntilDone:YES];
Every thing went well up to this........
MY Problem IS,
I need to send video frames through UDP Socket .even though following one is bad idea i tried ,UIImage to NSData and Send via UDP Pocket. BUt got so Delay in video Processing.Mostly problem because of UIImage to NSDate
So Please GIve me Solution For my problem
1)Any way to convert CMSampleBUffer or CVImageBuffer to NSData ??
2)Like Audio Queue Service and Queue for Video to store UIImage and do UIImage to NSDate
And Sending ???
if am riding behind the Wrong Algorithm Please path me in write direction
Thanks In Advance
Here is code to get at the buffer. This code assumes a flat image (e.g. BGRA).
NSData* imageToBuffer( CMSampleBufferRef source) {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(source);
CVPixelBufferLockBaseAddress(imageBuffer,0);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
void *src_buff = CVPixelBufferGetBaseAddress(imageBuffer);
NSData *data = [NSData dataWithBytes:src_buff length:bytesPerRow * height];
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
return [data autorelease];
}
A more efficient approach would be to use a NSMutableData or a buffer pool.
Sending a 480x360 image every second will require a 4.1Mbps connection assuming 3 color channels.
Use CMSampleBufferGetImageBuffer to get CVImageBufferRef from the sample buffer, then get the bitmap data from it with CVPixelBufferGetBaseAddress. This avoids needlessly copying the image.