I make a FFMPEG-based player for ios. It works fine on simulator, but on real-device (iPhone 4) the frame rate is low and make my audio and video out of sync. the player works fine on iPhone 4s, so I guess it's just problem about device's computing power.
So, is there anyway to build FFMPEG optimized for iOS device (armv7, arvm7s arch)? or is there anyway to utilize ios device hardware to decode video stream?
My video stream is encode in H264/AAC.
Those streams should play just fine, I assume since your using ffmpeg you are not using a video protocol that iOS supports directly.
We use ffmpeg to do rtsp/rtmp and we get good performance with h264/aac
There are a number of factors that contribute to av/sync issues, usually some type of pre-buffering of the video is required, also network plays a big part in it.
As to your second question, hardware encoding is only available via avfoundation, you can use avassetwriter to encode your video, but again depends wether or not you need real-time.
see this link https://github.com/mooncatventures-group/FFPlayer-beta1/blob/master/FFAVFrames-test/ViewController.m
-(void) startRecording {
// // create the AVComposition
// [mutableComposition release];
// mutableComposition = [[AVMutableComposition alloc] init];
movieURL = [NSURL fileURLWithPath:[NSString stringWithFormat:#"%#/%llu.mov", NSTemporaryDirectory(), mach_absolute_time()]];
NSError *movieError = nil;
assetWriter = [[AVAssetWriter alloc] initWithURL:movieURL
fileType: AVFileTypeQuickTimeMovie
error: &movieError];
NSDictionary *assetWriterInputSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInt:FRAME_WIDTH], AVVideoWidthKey,
[NSNumber numberWithInt:FRAME_HEIGHT], AVVideoHeightKey,
nil];
assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType: AVMediaTypeVideo
outputSettings:assetWriterInputSettings];
assetWriterInput.expectsMediaDataInRealTime = YES;
[assetWriter addInput:assetWriterInput];
assetWriterPixelBufferAdaptor = [[AVAssetWriterInputPixelBufferAdaptor alloc]
initWithAssetWriterInput:assetWriterInput
sourcePixelBufferAttributes:nil];
[assetWriter startWriting];
firstFrameWallClockTime = CFAbsoluteTimeGetCurrent();
[assetWriter startSessionAtSourceTime:kCMTimeZero];
startSampleing=YES;
}
The one drawback right now is that a way needs to be determined to read the encoded data as its being written, believe me when I say there are a few of us developers trying to figure out how to do that as we I write this.
Related
I am using AVFoundation's AVCaptureSession to capture video for live streaming ( to send to the server using RTMP protocol).
The callback delegate methods of AVCaptureVideoDataOutputSampleBufferDelegate and AVCaptureAudioDataOutputSampleBufferDelegate returns the CMSampleBufferRef in the following method :
- (void)captureOutput:(AVCaptureOutput *)captureOutputdidOutputSampleBuffer:(CMSampleBufferRef)sampleBufferfromConnection:(AVCaptureConnection *)connection
As per the contents of this link , I can convert this buffer content to a file using THIS LINK.
like
NSURL *url = …;
AVAssetWriter *assetWriter = [AVAssetWriter assetWriterWithURL:url fileType:AVFileTypeMPEG4 error:nil];
AVAssetWriterInput *videoInput = [[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeVideo outputSettings:nil];
videoInput.expectsMediaDataInRealTime = YES;
AVAssetWriterInput *audioInput = [[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeAudio outputSettings:nil];
audioInput.expectsMediaDataInRealTime = YES;
if ([assetWriter canAddInput:videoInput]) {
[assetWriter addInput:videoInput];
}
if ([assetWriter canAddInput:audioInput]) {
[assetWriter addInput:audioInput];
}
But I am not getting the exact logic involved in this file conversion. The didOutputSampleBuffer
is a frame by frame captured (uncompressed )raw data and I want to convert these frames into a single file at certain interval.like say at each 10 seconds or so.And I want to send this file to the server.
Can someone give a brief idea about how AVAssetWriter works?or correct me if I am doing anything wrong.
I use AVAudioPlayer for playing audio file and UISlider to show user current time. Firstly, it looked that everything is fine but I noticed that audio player returns wrong file duration. For example it returns me duration equals to 3.5sec but file durations is equal to 6 sec. in reality.
Do you know What can cause this problem?
Below you can see my code which return file duration:
- (NSTimeInterval)audioDurationAtURL:(NSURL *)url
{
NSError *error;
NSData *data = [NSData dataWithContentsOfURL:url];
_audioPlayer = [[AVAudioPlayer alloc] initWithData:data error:&error];
return _audioPlayer.duration;
}
To add a bit to TonyMkenu's answer, AVAsset is an alternative with the ability to give you a more accurate duration.
https://developer.apple.com/library/mac/documentation/AVFoundation/Reference/AVAsset_Class/Reference/Reference.html#//apple_ref/occ/instp/AVAsset/duration
If you specify providesPreciseDurationAndTiming = YES, then AVAsset will decode the file if needed to determine its duration with accuracy. If the decode time is too long for your use, you can disable it.
In my situation, I use the AVURLAsset subclass:
AVURLAsset *asset = [[AVURLAsset alloc] initWithURL:localURL options:[NSDictionary dictionaryWithObjectsAndKeys: [NSNumber numberWithBool:YES], AVURLAssetPreferPreciseDurationAndTimingKey, nil]];
float t = CMTimeGetSeconds(asset.duration);
AVAudioPlayer appears to only returns the correct duration of a file when it is ready for play it, so try to check the length of the audio file after [_audioPlayer play];
Under certain compression formats, the audio player may change its estimate of the duration as it learns more and more about the song by playing (and hence decoding) more and more of it - https://stackoverflow.com/a/16265186
In my case, having added an audio file to a project then editing (making it longer or shorter) and then deleting and re-adding the file to the Xcode project was the problem.
Essentially the project is caching the old file. To debug this I renamed the audio file to something else, added the new audio file to the project after which the duration reported by the player was always correct, before and after calling play.
Which audio format is small in size for speech recording in ios? The quality is need not to be the best but it should be understandable what user speaks.
Assuming you plan to use AVAudioRecorder class, you should provide the recording settings like so -
NSDictionary *recordSettings = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:AVAudioQualityMin], AVEncoderAudioQualityKey,
[NSNumber numberWithInt:16], AVEncoderBitRateKey,
[NSNumber numberWithInt: 2],AVNumberOfChannelsKey,
[NSNumber numberWithFloat:44100.0], AVSampleRateKey,nil];
NSError* error = nil;
AVAudioRecorder audioRecorder = [[AVAudioRecorder alloc]
initWithURL:soundFileURL
settings:recordSettings
error:&error];
Apple's documentation provides details about the settings constants (specifically AVEncoderAudioQualityKey) you could use in your app.
22.05KHz in mono is more than adequate for speech and is 1/4 the size of 44.1KHz in stereo at the same bit depth. You could likely even try dropping it down to 11.025KHz.
Several iOS apps use the Speex encoder for lower-bit rate speech. It's not built-in, but open source source code is available to do the encoding.
How can I make the process that filtering saved video in photo library in iOS?
I got URLs of videos in the library using AssetsLibrary framework,
then, made a preview for the video.
Next step, I wanna make filtering process for video using CIFilter.
In case of real time issue, I made video filter process using AVCaptureVideoDataOutputSampleBufferDelegate.
But in case of saved video, I don't know how to make filter process.
Do I use AVAsset? If I must use that, how can I filter it? and how to save it?
always thank you.
I hope this will help you
AVAsset *theAVAsset = [[AVURLAsset alloc] initWithURL:mNormalVideoURL options:nil];
NSError *error = nil;
float width = theAVAsset.naturalSize.width;
float height = theAVAsset.naturalSize.height;
AVAssetReader *mAssetReader = [[AVAssetReader alloc] initWithAsset:theAVAsset error:&error];
[theAVAsset release];
NSArray *videoTracks = [theAVAsset tracksWithMediaType:AVMediaTypeVideo];
AVAssetTrack *videoTrack = [videoTracks objectAtIndex:0];
mPrefferdTransform = [videoTrack preferredTransform];
NSDictionary *options = [NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey];
AVAssetReaderTrackOutput* mAssetReaderOutput = [[AVAssetReaderTrackOutput alloc] initWithTrack:videoTrack outputSettings:options];
[mAssetReader addOutput:mAssetReaderOutput];
[mAssetReaderOutput release];
CMSampleBufferRef buffer = NULL;
//CMSampleBufferRef buffer = NULL;
while ( [mAssetReader status]==AVAssetReaderStatusReading ){
buffer = [mAssetReaderOutput copyNextSampleBuffer];//read next image.
}
You should have a look at CVImageBufferRef pixBuf = CMSampleBufferGetImageBuffer(sbuf) then you can have the image pointer first address, so you can add filter to pixBuf, but i find that the performance is not good, If you have any new idea,we can discuss about it further.
I am a newbie in AVFoundation and decoding process.I need to decode a h264 video file and play it in iphone...can anyone give me guideline to do it.
I dont want to use ffmpeg or any third party library to do that. As far as I know using Avfoundation encoding is possible...here is the code which I thought is used for encoding but not sure at all...
float bitsPerPixel;
CMVideoDimensions dimensions = CMVideoFormatDescriptionGetDimensions(currentFormatDescription);
int numPixels = dimensions.width * dimensions.height;
int bitsPerSecond;
// Assume that lower-than-SD resolutions are intended for streaming, and use a lower bitrate
if ( numPixels < (640 * 480) )
bitsPerPixel = 4.05; // This bitrate matches the quality produced by AVCaptureSessionPresetMedium or Low.
else
bitsPerPixel = 11.4; // This bitrate matches the quality produced by AVCaptureSessionPresetHigh.
bitsPerSecond = numPixels * bitsPerPixel;
NSDictionary *videoCompressionSettings = [NSDictionary dictionaryWithObjectsAndKeys:
AVVideoCodecH264, AVVideoCodecKey,
[NSNumber numberWithInteger:dimensions.width], AVVideoWidthKey,
[NSNumber numberWithInteger:dimensions.height], AVVideoHeightKey,
[NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithInteger:bitsPerSecond], AVVideoAverageBitRateKey,
[NSNumber numberWithInteger:30], AVVideoMaxKeyFrameIntervalKey,
nil], AVVideoCompressionPropertiesKey,
nil];
if ([assetWriter canApplyOutputSettings:videoCompressionSettings forMediaType:AVMediaTypeVideo]) {
assetWriterVideoIn = [[AVAssetWriterInput alloc] initWithMediaType:AVMediaTypeVideo outputSettings:videoCompressionSettings];
assetWriterVideoIn.expectsMediaDataInRealTime = YES;
assetWriterVideoIn.transform = [self transformFromCurrentVideoOrientationToOrientation:self.referenceOrientation];
if ([assetWriter canAddInput:assetWriterVideoIn])
[assetWriter addInput:assetWriterVideoIn];
else {
NSLog(#"Couldn't add asset writer video input.");
return NO;
}
}
else {
NSLog(#"Couldn't apply video output settings.");
return NO;
}
return YES;
I am completely naive about this, please help...from where to start///
thanks
The simpler solution is to use MPMoviePlayerController. It takes in input a mov or mv4 file (from local file system or through an URL).
The another option is to use AVPlayer class.
Hope it helps,
David
You can refer and build the official sample code: AVPlayerDemo to see how it works. It uses AV Foundation framework, mainly the AVPlayer APIs to play video files and the performance is excellent.
To play a video file by AVPlayerDemo you should copy the video files to your iOS device by itunes and select iPod library on the AVPlayerDemo App.