In my iOS app, I need to save an image as a short video segment. I have this working using AVAssetWriter and AVAssetWriterPixelBufferAdaptor, thanks to some of the great posts on this site, but I've had to fudge the start and end session times, and presentation times, because I don't really understand them.
The following fragment creates a 2 second video, but I've set the various times by trial and error. I'm not sure why it doesn't create a 3 second video, to be honest.
// start session
videoWriter.movieFragmentInterval = CMTimeMake(1,600);
[videoWriter startWriting];
CMTime startTime = CMTimeMake(0, 600);
[videoWriter startSessionAtSourceTime:startTime];
while (1) {
if (![writerInput isReadyForMoreMediaData]) {
NSLog(#"Not ready for data");
} else {
[avAdaptor appendPixelBuffer:pixelBuffer
withPresentationTime:CMTimeMake(1200,600)];
break;
}
}
//Finish the session:
[writerInput markAsFinished];
CMTime endTime = CMTimeMake(1800, 600);
[videoWriter endSessionAtSourceTime:endTime];
[videoWriter finishWriting];
Can anyone explain the various time settings in this fragment, or point me to a document that will help? I've read the apple docs until I'm cross-eyed, but they assume more knowledge than I currently have, I guess.
TIA: John
Related
How to upload audio clip realtime to a server while its recording? Basically my requirement is upload an audio clip as chucks/packets while its recording.
I already did the recording part with using IQAudioRecorderController https://github.com/hackiftekhar/IQAudioRecorderController. It records the audio and save to TemporaryDirectory.
I wanted to know how to upload realtime without saving the audio clip.
This is the recording part
//Unique recording URL
NSString *fileName = [[NSProcessInfo processInfo] globallyUniqueString];
_recordingFilePath = [NSTemporaryDirectory() stringByAppendingPathComponent:[NSString stringWithFormat:#"%#.m4a",fileName]];
// Initiate and prepare the recorder
_audioRecorder = [[AVAudioRecorder alloc] initWithURL:[NSURL fileURLWithPath:_recordingFilePath] settings:recordSetting error:nil];
_audioRecorder.delegate = self;
_audioRecorder.meteringEnabled = YES;
// Recording start
- (void)recordingButtonAction:(UIBarButtonItem *)item
{
if (_isRecording == NO)
{
_isRecording = YES;
//UI Update
{
[self showNavigationButton:NO];
_recordButton.tintColor = _recordingTintColor;
_playButton.enabled = NO;
_trashButton.enabled = NO;
}
/*
Create the recorder
*/
if ([[NSFileManager defaultManager] fileExistsAtPath:_recordingFilePath])
{
[[NSFileManager defaultManager] removeItemAtPath:_recordingFilePath error:nil];
}
_oldSessionCategory = [[AVAudioSession sharedInstance] category];
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryRecord error:nil];
[_audioRecorder prepareToRecord];
[_audioRecorder record];
}
else
{
_isRecording = NO;
//UI Update
{
[self showNavigationButton:YES];
_recordButton.tintColor = _normalTintColor;
_playButton.enabled = YES;
_trashButton.enabled = YES;
}
[_audioRecorder stop];
[[AVAudioSession sharedInstance] setCategory:_oldSessionCategory error:nil];
}
}
// Recording done
-(void)doneAction:(UIBarButtonItem*)item
{
if ([self.delegate respondsToSelector:#selector(audioRecorderController:didFinishWithAudioAtPath:)])
{
IQAudioRecorderController *controller = (IQAudioRecorderController*)[self navigationController];
[self.delegate audioRecorderController:controller didFinishWithAudioAtPath:_recordingFilePath];
}
[self dismissViewControllerAnimated:YES completion:nil];
}
There are various ways of solving this, one way is to create your own AudioGraph. The AudioGraph can grab samples from microphone or from a file. Then you proceed to an output unit, but install a callback to get the sampled frames. These you then push to your network class which then can upload packet by packet.
A good example that shows you how to write these captured packets to disk is AVCaptureAudioDataOutput .
In that example packets are written suing ExtAudioFileWriteAsync. You have to replace this with your own logic for uploading to a server. Note that while you can do that easily, one problem is that this will give you raw audio samples. If you need them as wave file or similar, you may need to wait until recording is finished, since the header of the file needs an information about contained audio samples.
The code you are currently using will work for you if you want to upload recorded file after recording is done as it will give you the final recorded file.
If you want to upload live audio recording to the server then I think you have to go with combination of,
AudioSession for recording stuff
ffmpeg for uploading your live audio to server.
You can get good help for recording audio and managing Audio Buffers from here
For ffmpeg I think you have to lear a lot. It will be easy to send static/saved audio file to server using ffmpeg but for sending live Audio Buffer to server will be tricky job.
I am trying to mute and unmute the audio for AVCaptureSession. Once I start the sessions I can enable and disable the audio connection, but once I play back the video all of the audio portions are pushed back to back at the front of the video leaving only the end of the video with no sound. This seems to be a time stamp issue but I don't know how it could be. Just in case I have tried to adjust the PTS of the audio sample buffer to match the previous video buffer.
For pausing
if (self.muted) {
[self.session beginConfiguration];
self.audioConnection.enabled = NO;
[self.session commitConfiguration];
} else {
[self.session beginConfiguration];
self.audioConnection.enabled = YES;
[self.session commitConfiguration];
}
For adjusting the time stamp I grab last timestamp
if ( connection == self.videoConnection ) {
// Get timestamp
CMTime timestamp = CMSampleBufferGetPresentationTimeStamp( sampleBuffer );
testPreviousTimeStamp = timestamp;
Then I adjust the timestamp on the sample buffer
CMSampleBufferSetOutputPresentationTimeStamp (sampleBuffer,testPreviousTimeStamp);
if (![self.assetWriterAudioIn appendSampleBuffer:sampleBuffer]) {
[self showError:[self.assetWriter error]];
NSLog(#"Problem writing audio sample buffer");
}
Any ideas what the problem could be and how to fix it?
Rather than adjusting the time stamp which I had no luck with. I just wrote zero to the data buffer. It works and this way I don't need to disable/enable any connections.
// Write audio data to file
if (readyToRecordAudio && readyToRecordVideo) {
if (self.muted) {
CMBlockBufferRef buffRef = CMSampleBufferGetDataBuffer(sampleBuffer);
char fillByte = 0;
CMBlockBufferFillDataBytes(fillByte,buffRef,0,CMBlockBufferGetDataLength(buffRef));
}
[self writeSampleBuffer:sampleBuffer ofType:AVMediaTypeAudio];
}
My app includes the ability for the user to record a brief message; I'd like to trim off any silence (or, to be more precise, any audio whose volume falls below a given threshold) from the beginning and end of the recording.
I'm recording the audio with an AVAudioRecorder, and saving it to an .aif file. I've seen some mention elsewhere of methods by which I could have it wait to start recording until the audio level reaches a threshold; that'd get me halfway there, but won't help with trimming silence off the end.
If there's a simple way to do this, I'll be eternally grateful!
Thanks.
This project takes audio from the microphone, triggers on loud noise and untriggers when quiet. It also trims and fades in/fades out around the ends.
https://github.com/fulldecent/FDSoundActivatedRecorder
Relevant code you are seeking:
- (NSString *)recordedFilePath
{
// Prepare output
NSString *trimmedAudioFileBaseName = [NSString stringWithFormat:#"recordingConverted%x.caf", arc4random()];
NSString *trimmedAudioFilePath = [NSTemporaryDirectory() stringByAppendingPathComponent:trimmedAudioFileBaseName];
NSFileManager *fileManager = [NSFileManager defaultManager];
if ([fileManager fileExistsAtPath:trimmedAudioFilePath]) {
NSError *error;
if ([fileManager removeItemAtPath:trimmedAudioFilePath error:&error] == NO) {
NSLog(#"removeItemAtPath %# error:%#", trimmedAudioFilePath, error);
}
}
NSLog(#"Saving to %#", trimmedAudioFilePath);
AVAsset *avAsset = [AVAsset assetWithURL:self.audioRecorder.url];
NSArray *tracks = [avAsset tracksWithMediaType:AVMediaTypeAudio];
AVAssetTrack *track = [tracks objectAtIndex:0];
AVAssetExportSession *exportSession = [AVAssetExportSession
exportSessionWithAsset:avAsset
presetName:AVAssetExportPresetAppleM4A];
// create trim time range
CMTime startTime = CMTimeMake(self.recordingBeginTime*SAVING_SAMPLES_PER_SECOND, SAVING_SAMPLES_PER_SECOND);
CMTimeRange exportTimeRange = CMTimeRangeFromTimeToTime(startTime, kCMTimePositiveInfinity);
// create fade in time range
CMTime startFadeInTime = startTime;
CMTime endFadeInTime = CMTimeMake(self.recordingBeginTime*SAVING_SAMPLES_PER_SECOND + RISE_TRIGGER_INTERVALS*INTERVAL_SECONDS*SAVING_SAMPLES_PER_SECOND, SAVING_SAMPLES_PER_SECOND);
CMTimeRange fadeInTimeRange = CMTimeRangeFromTimeToTime(startFadeInTime, endFadeInTime);
// setup audio mix
AVMutableAudioMix *exportAudioMix = [AVMutableAudioMix audioMix];
AVMutableAudioMixInputParameters *exportAudioMixInputParameters =
[AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:track];
[exportAudioMixInputParameters setVolumeRampFromStartVolume:0.0 toEndVolume:1.0
timeRange:fadeInTimeRange];
exportAudioMix.inputParameters = [NSArray
arrayWithObject:exportAudioMixInputParameters];
// configure export session output with all our parameters
exportSession.outputURL = [NSURL fileURLWithPath:trimmedAudioFilePath];
exportSession.outputFileType = AVFileTypeAppleM4A;
exportSession.timeRange = exportTimeRange;
exportSession.audioMix = exportAudioMix;
// MAKE THE EXPORT SYNCHRONOUS
dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);
[exportSession exportAsynchronouslyWithCompletionHandler:^{
dispatch_semaphore_signal(semaphore);
}];
dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
if (AVAssetExportSessionStatusCompleted == exportSession.status) {
NSLog(#"AVAssetExportSessionStatusCompleted");
return trimmedAudioFilePath;
} else if (AVAssetExportSessionStatusFailed == exportSession.status) {
// a failure may happen because of an event out of your control
// for example, an interruption like a phone call comming in
// make sure and handle this case appropriately
NSLog(#"AVAssetExportSessionStatusFailed %#", exportSession.error.localizedDescription);
} else {
NSLog(#"Export Session Status: %d", exportSession.status);
}
return nil;
}
I'm recording the audio with an AVAudioRecorder, and saving it to an .aif file. I've seen some mention elsewhere of methods by which I could have it wait to start recording until the audio level reaches a threshold; that'd get me halfway there
Without adequate buffering, that would truncate the start.
I don't know of an easy way. You would have to write a new audio file after recording and analyzing it for the desired start and end points. Modifying the existing file would be straightforward if you knew the AIFF format well (not many people do) and had an easy way to read the file's sample data.
The analysis stage is pretty easy for a basic implementation -- evaluate the average power of sample data, until your threshold is exceeded. Repeat in reverse for end.
I've built an AVMutableComposition and VideoComposition from AVAssets and am able to play it. I am also able to export it using AVAssetExportSession, however AVAssetExportSession does not offer much control of the settings, so I'm exporting it using AVAssetReader/AVAssetWriter, but unfortunately I'm getting an error I don't understand and an only partially written output file.
Here is the code I have so far. I've left out the writer and as much other stuff as possible (including some error checking) that I think is irrelevant to make the code easier to read because it's a lot. Note that I haven't yet dealt with the audio track -- I'm trying to do this one step at a time, but maybe that's my problem?
The variable asset is the AVMutableComposition.
// ------- reader
_assetReader = [AVAssetReader assetReaderWithAsset:asset error:error];
_assetReader.timeRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
_duration = asset.duration;
// --- video reader
NSArray *videoTracks = [asset tracksWithMediaType:AVMediaTypeVideo];
NSDictionary *videoOptions = [NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_32BGRA] forKey:(id)kCVPixelBufferPixelFormatTypeKey];
_assetReaderOutput = [AVAssetReaderVideoCompositionOutput assetReaderVideoCompositionOutputWithVideoTracks:videoTracks
videoSettings:videoOptions];
((AVAssetReaderVideoCompositionOutput *)_assetReaderOutput).videoComposition = composition;
[_assetReader addOutput:_assetReaderOutput];
// -- leaving out the export settings and construction
// of the assetWriter since that doesn't seem to be the issue
// -- here's the actual export:
[self.assetWriter startWriting];
[self.assetWriter startSessionAtSourceTime:kCMTimeZero];
[self.assetReader startReading];
CMSampleBufferRef buffer;
while ( [self.assetReader status]==AVAssetReaderStatusReading )
{
if(![self.assetWriterInput isReadyForMoreMediaData])
continue;
buffer = [self.assetReaderOutput copyNextSampleBuffer];
NSLog(#"READING");
if(buffer)
[self.assetWriterInput appendSampleBuffer:buffer];
NSLog(#"WRITTING...");
}
if( self.assetReader.status == AVAssetReaderStatusFailed ) {
NSLog( #"%#", self.assetReader.error ); //<--------- I get a problem here
}
//Finish the session:
[self.assetWriterInput markAsFinished];
[self.assetWriter finishWriting];
NSLog(#"Write Ended");
The problem is the loop exits after a short time and I get this output:
Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo=0x17f1dfa0 {NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x191617f0 "The operation couldn’t be completed. (OSStatus error -308.)", NSLocalizedFailureReason=An unknown error occurred (-308)}
It seems I was following some bad sample code, rather than correct sample code provided by apple, which is pretty clear, if extremely verbose, about how to do this:
https://developer.apple.com/library/Mac/DOCUMENTATION/AudioVideo/Conceptual/AVFoundationPG/Articles/05_Export.html#//apple_ref/doc/uid/TP40010188-CH9-SW2
Although I've completely rewritten my code, I think the specific problem I was having was not calling CFRelease on my CMSampleBufferRef returned from [self.assetReaderOutput copyNextSampleBuffer]. You need to do that inside the loop.
Do not tell me to use AVAssetExportSession, thank you.
I tried this, but failed.
for (int i =0; i < count; i++) {
assetWriterInput = nil;
assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:videoSettings];
NSParameterAssert(assetWriterInput);
NSParameterAssert([assetWriter canAddInput:assetWriterInput]);
[assetWriterInput setExpectsMediaDataInRealTime:YES];
[assetWriter addInput:assetWriterInput];
}
[assetWriter startWriting];
The sample app that shows you how to do exactly what you're talking about is AVCompositionDebugVieweriOS:
https://developer.apple.com/library/ios/samplecode/AVCompositionDebugVieweriOS/Introduction/Intro.html#//apple_ref/doc/uid/DTS40013421
I'm sure you can pare down the code to just the parts you need; but, if that's not yet where you're at with your level of understanding, let me know.
One more thing: this app not only contains the code you need, but also draws a graph of your output, showing you where you made the connection between the two clips, and any transition you may have inserted between them.