AVAssetWriter / AVAssetWriterInputPixelBufferAdaptor - black frames and frame rate - ios

I'm capturing the camera feed and writing it to a movie.
The problem I'm having is that after the export the movie has a couple of black seconds in front of it (relative to the actual recording start time).
I think this is related to [self.assetWriter startSessionAtSourceTime:kCMTimeZero];
I had a half working solution by having a frameStart variable that just counted upwards in the samplebuffer delegate method.
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
frameStart++;
if (self.startRecording == YES) {
static int64_t frameNumber = 0;
if(self.assetWriterInput.readyForMoreMediaData) {
[self.pixelBufferAdaptor appendPixelBuffer:pixelBuffer withPresentationTime:CMTimeMake(frameNumber, 25)];
}
frameNumber++;
}
}
and then call this method when the user pressed a button:
[self.assetWriter startSessionAtSourceTime:CMTimeMake(frameStart,25)];
this works. but only once... if I want to record a second movie the black frames are back again.
Also, when I look at the outputted movie the frame rate is 25fps like I want it to. but the video looks as if it's sped up. as if there is not enough space between the frames. So the movie plays about twice as fast.
NSDictionary *outputSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithInt:640], AVVideoWidthKey, [NSNumber numberWithInt:480], AVVideoHeightKey, AVVideoCodecH264, AVVideoCodecKey, nil];
self.assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:outputSettings];
self.assetWriterInput.expectsMediaDataInRealTime = YES;

You don't need to count frame timestamps on your own. You can get the timestamp of the current sample with
CMTime timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
However, it seems to me you are just passing the pixel buffer of the frame to the adaptor without modifications. Wouldn't it be easier to pass the sample buffer itself directly to the assetWriterInput like the following?
[self.assetWriterInput appendSampleBuffer:sampleBuffer];

First of all, why are you incrementing frameNumber twice for every frame?
Increment once, remove the first one.
This should fix the playback speed.
Second, are you resetting frameNumber to 0 when you finish recording?
If not than this is your problem.
If not I need more explanation about what is going on here..
Regards

Related

Recording 1 second audio ios

How can I record only one second audio sound without using file system and convert it to a byte array to send it by JSON?
NSDictionary *recordSettings = [NSDictionary
dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:AVAudioQualityMin],
AVEncoderAudioQualityKey,
[NSNumber numberWithInt:16],
AVEncoderBitRateKey,
[NSNumber numberWithInt: 1],
AVNumberOfChannelsKey,
[NSNumber numberWithFloat:16000.0], AVSampleRateKey,
[NSNumber numberWithInt:8], AVLinearPCMBitDepthKey,
nil];
recorder = [[AVAudioRecorder alloc] initWithURL:recordedFile settings:recordSettings error:nil];
Else, I wanna know how size is this one second sound recorded and time.
Sample rate = 16,000 Hz (samples per second)
Bit rate = 16 (bits per sample)
Therefore size of 1 second sample = 16K x 16 = 256Kbits
To record one second, you can use
- (BOOL)recordForDuration:(NSTimeInterval)duration
If you want a fast start, you can prime the recording beforehand using
- (void)prepareToRecord
For recording at a specific time, use
- (BOOL)recordAtTime:(NSTimeInterval)time
forDuration:(NSTimeInterval)duration
(which implicitly calls prepareToRecord)
To get an accurate time of recording you will need to use a lower-level API. If you are not worried about accuracy, you can log the time you call the above method or the time it is written to the filesystem.
If you don't want to use the filesystem then you will need to use a lower-level API where you can get at the bitstream using a callback. Apple's Core Audio, or one of the many C++ audio engines (e.g. PortAudio, FMod, Novocaine) will let you do this.

AudioFileReadPackets gives strange data

I'm trying to record audio to a file (working) and then sample the data in that file (giving strange results).
FYI, I am roughly following this code...
Extracting Amplitude Data from Linear PCM on the iPhone
I have noticed a few different results. For simplicity, assume the record time is fixed at 1 second.
when sampling up to 8,000 samples/sec, the mutable array (see code) will list 8,000 entries but only the first 4,000 have real-looking data, the last 4,000 points are the same number value (the exact number value varies from run-to-run).
somewhat related to issue #1. when sampling above 8,000 samples/second, the first half of the samples (ex. 5,000 of a 10,0000 sample set from 10,000 samples/sec for 1 second) will look like real data, while the values of the second half of the set will be fixed to some value (again this exact value varies run to run). See below snippet from my debug window, first number is packetIndex, second number is buffer value.
4996:-137
4997:1043
4998:-405
4999:-641
5000:195notice the switch from random data to constant value at 5k, for 10k sample file
5001:195
5002:195
5003:195
5004:195
3 . when having the mic listen to a speaker playing a 1kHz sinusoidal tone in close proximity and sampling this tone at 40,000 samples per second, the resulting data when plotted in a spreadsheet shows the signal at about 2kHz, or double.
Any ideas what I may be doing wrong here?
Here is my setup work to record the audio from the mic...
-(void) initAudioSession {
// setup av session
AVAudioSession *audioSession = [AVAudioSession sharedInstance];
[audioSession setCategory:AVAudioSessionCategoryPlayAndRecord error:nil];
[audioSession setActive:YES error: nil];
NSLog(#"audio session initiated");
// settings for the recorded file
NSDictionary *recordSettings = [[NSDictionary alloc] initWithObjectsAndKeys:
[NSNumber numberWithFloat:SAMPLERATE],AVSampleRateKey,
[NSNumber numberWithInt:kAudioFormatLinearPCM],AVFormatIDKey,
[NSNumber numberWithInt:1],AVNumberOfChannelsKey,
[NSNumber numberWithInt:16],AVEncoderBitRateKey,
[NSNumber numberWithInt:AVAudioQualityMax],AVEncoderAudioQualityKey, nil];
// setup file name and location
NSString *docDir = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) lastObject];
fileURL = [NSURL fileURLWithPath:[docDir stringByAppendingPathComponent:#"input.caf"]];//caf or aif?
// initialize my new audio recorder
newAudioRecorder = [[AVAudioRecorder alloc] initWithURL:fileURL settings:recordSettings error:nil];
// show file location so i can check it with some player
NSLog(#"file path = %#",fileURL);
// check if the recorder exists, if so prepare the recorder, if not tell me in debug window
if (newAudioRecorder) {
[newAudioRecorder setDelegate:self];
[newAudioRecorder prepareToRecord];
[self.setupStatus setText:[NSString stringWithFormat:#"recorder ready"]];
}else{
NSLog(#"error setting up recorder");
}
}
Here is my code for loading the recorded file and grabbing the data...
//loads file and go thru values, converts data to be put into an NSMutableArray
-(void)readingRainbow{
// get audio file and put into a file ID
AudioFileID fileID;
AudioFileOpenURL((__bridge CFURLRef)fileURL, kAudioFileReadPermission, kAudioFileCAFType /*kAudioFileAIFFType*/ , &fileID);
// get number of packets of audio contained in file
// instead of getting packets, i just set them to the duration times the sample rate i set
// not sure if this is a valid approach
UInt64 totalPacketCount = SAMPLERATE*timer;
// get size of each packet, is this valid?
UInt32 maxPacketSizeInBytes = sizeof(SInt32);
// setup to extract audio data
UInt32 totPack32 = SAMPLERATE*timer;
UInt32 ioNumBytes = totPack32*maxPacketSizeInBytes;
SInt16 *outBuffer = malloc(ioNumBytes);
memset(outBuffer, 0, ioNumBytes);
// setup array to put buffer samples in
readArray = [[NSMutableArray alloc] initWithObjects: nil];
NSNumber *arrayData;
SInt16 data;
int data2;
// this may be where i need help as well....
// process every packet
for (SInt64 packetIndex = 0; packetIndex<totalPacketCount; packetIndex++) {
// method description for reference..
// AudioFileReadPackets(<#AudioFileID inAudioFile#>, <#Boolean inUseCache#>, <#UInt32 *outNumBytes#>,
// <#AudioStreamPacketDescription *outPacketDescriptions#>, <#SInt64 inStartingPacket#>,
// <#UInt32 *ioNumPackets#>, <#void *outBuffer#>)
// extract packet data, not sure if i'm setting this up properly
AudioFileReadPackets(fileID, false, &ioNumBytes, NULL, packetIndex, &totPack32, outBuffer);
// get buffer data and pass into mutable array
data = outBuffer[packetIndex];
data2=data;
arrayData = [[NSNumber alloc] initWithInt:data2];
[readArray addObject:arrayData];
// printf("%lld:%d\n",packetIndex,data);
printf("%d,",data);
}
Also, I'm using this method to start the recorder...
[newAudioRecorder recordForDuration:timer];
Thots? I'm a noob, so any info is greatly appreciated!
You may be recording 16-bit samples, but trying to read 32-bit samples from the file data, thus only finding half as many samples (the rest may be garbage).

AVAssetWriter can merge two video files

Do not tell me to use AVAssetExportSession, thank you.
I tried this, but failed.
for (int i =0; i < count; i++) {
assetWriterInput = nil;
assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:videoSettings];
NSParameterAssert(assetWriterInput);
NSParameterAssert([assetWriter canAddInput:assetWriterInput]);
[assetWriterInput setExpectsMediaDataInRealTime:YES];
[assetWriter addInput:assetWriterInput];
}
[assetWriter startWriting];
The sample app that shows you how to do exactly what you're talking about is AVCompositionDebugVieweriOS:
https://developer.apple.com/library/ios/samplecode/AVCompositionDebugVieweriOS/Introduction/Intro.html#//apple_ref/doc/uid/DTS40013421
I'm sure you can pare down the code to just the parts you need; but, if that's not yet where you're at with your level of understanding, let me know.
One more thing: this app not only contains the code you need, but also draws a graph of your output, showing you where you made the connection between the two clips, and any transition you may have inserted between them.

How build ffmpeg optimized for iOS, using hardware decoding probably?

I make a FFMPEG-based player for ios. It works fine on simulator, but on real-device (iPhone 4) the frame rate is low and make my audio and video out of sync. the player works fine on iPhone 4s, so I guess it's just problem about device's computing power.
So, is there anyway to build FFMPEG optimized for iOS device (armv7, arvm7s arch)? or is there anyway to utilize ios device hardware to decode video stream?
My video stream is encode in H264/AAC.
Those streams should play just fine, I assume since your using ffmpeg you are not using a video protocol that iOS supports directly.
We use ffmpeg to do rtsp/rtmp and we get good performance with h264/aac
There are a number of factors that contribute to av/sync issues, usually some type of pre-buffering of the video is required, also network plays a big part in it.
As to your second question, hardware encoding is only available via avfoundation, you can use avassetwriter to encode your video, but again depends wether or not you need real-time.
see this link https://github.com/mooncatventures-group/FFPlayer-beta1/blob/master/FFAVFrames-test/ViewController.m
-(void) startRecording {
// // create the AVComposition
// [mutableComposition release];
// mutableComposition = [[AVMutableComposition alloc] init];
movieURL = [NSURL fileURLWithPath:[NSString stringWithFormat:#"%#/%llu.mov", NSTemporaryDirectory(), mach_absolute_time()]];
NSError *movieError = nil;
assetWriter = [[AVAssetWriter alloc] initWithURL:movieURL
fileType: AVFileTypeQuickTimeMovie
error: &movieError];
NSDictionary *assetWriterInputSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:FRAME_WIDTH], AVVideoWidthKey,
[NSNumber numberWithInt:FRAME_HEIGHT], AVVideoHeightKey,
nil];
assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType: AVMediaTypeVideo
outputSettings:assetWriterInputSettings];
assetWriterInput.expectsMediaDataInRealTime = YES;
[assetWriter addInput:assetWriterInput];
assetWriterPixelBufferAdaptor = [[AVAssetWriterInputPixelBufferAdaptor alloc]
initWithAssetWriterInput:assetWriterInput
sourcePixelBufferAttributes:nil];
[assetWriter startWriting];
firstFrameWallClockTime = CFAbsoluteTimeGetCurrent();
[assetWriter startSessionAtSourceTime:kCMTimeZero];
startSampleing=YES;
}
The one drawback right now is that a way needs to be determined to read the encoded data as its being written, believe me when I say there are a few of us developers trying to figure out how to do that as we I write this.

decoding h264 in ios

I am a newbie in AVFoundation and decoding process.I need to decode a h264 video file and play it in iphone...can anyone give me guideline to do it.
I dont want to use ffmpeg or any third party library to do that. As far as I know using Avfoundation encoding is possible...here is the code which I thought is used for encoding but not sure at all...
float bitsPerPixel;
CMVideoDimensions dimensions = CMVideoFormatDescriptionGetDimensions(currentFormatDescription);
int numPixels = dimensions.width * dimensions.height;
int bitsPerSecond;
// Assume that lower-than-SD resolutions are intended for streaming, and use a lower bitrate
if ( numPixels < (640 * 480) )
bitsPerPixel = 4.05; // This bitrate matches the quality produced by AVCaptureSessionPresetMedium or Low.
else
bitsPerPixel = 11.4; // This bitrate matches the quality produced by AVCaptureSessionPresetHigh.
bitsPerSecond = numPixels * bitsPerPixel;
NSDictionary *videoCompressionSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInteger:dimensions.width], AVVideoWidthKey,
[NSNumber numberWithInteger:dimensions.height], AVVideoHeightKey,
[NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInteger:bitsPerSecond], AVVideoAverageBitRateKey,
[NSNumber numberWithInteger:30], AVVideoMaxKeyFrameIntervalKey,
nil], AVVideoCompressionPropertiesKey,
nil];
if ([assetWriter canApplyOutputSettings:videoCompressionSettings forMediaType:AVMediaTypeVideo]) {
assetWriterVideoIn = [[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeVideo outputSettings:videoCompressionSettings];
assetWriterVideoIn.expectsMediaDataInRealTime = YES;
assetWriterVideoIn.transform = [self transformFromCurrentVideoOrientationToOrientation:self.referenceOrientation];
if ([assetWriter canAddInput:assetWriterVideoIn])
[assetWriter addInput:assetWriterVideoIn];
else {
NSLog(#"Couldn't add asset writer video input.");
return NO;
}
}
else {
NSLog(#"Couldn't apply video output settings.");
return NO;
}
return YES;
I am completely naive about this, please help...from where to start///
thanks
The simpler solution is to use MPMoviePlayerController. It takes in input a mov or mv4 file (from local file system or through an URL).
The another option is to use AVPlayer class.
Hope it helps,
David
You can refer and build the official sample code: AVPlayerDemo to see how it works. It uses AV Foundation framework, mainly the AVPlayer APIs to play video files and the performance is excellent.
To play a video file by AVPlayerDemo you should copy the video files to your iOS device by itunes and select iPod library on the AVPlayerDemo App.

Resources