I am developing an app that records a video using AVAssetWriter (the source media are sample buffers output from captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection).
I've noticed that the very first time I instantiate my AVAssetWriter, then call startWriting() and startSession(atSourceTime: CMTime), there is a lag of about 30ms. This causes the video preview on the screen to momentarily hang, which also results in the first few frames of the video not write properly.
Curiously, if I then re-instantiate my AVAssetWriter and take all of the same steps again, everything works fine from that point onward. This only occurs the very first time.
While perhaps too lengthy to post, this is the function in which I am certain the error is occurring;
func insert(pixel buffer: CVPixelBuffer, with time: CMTime) {
// Check for unknown status
if fileWriter.status == .unknown {
guard startingVideoTime == nil else {
print("We've received an error setting the starting video time.")
return
}
startingVideoTime = time
if fileWriter.startWriting() {
fileWriter.startSession(atSourceTime: startingVideoTime!)
isRecording = true
}
}
// Append buffer
if videoInput.isReadyForMoreMediaData {
append(pixel: buffer, with: time)
isRecording = true
}
}
For posterity, I am calling the above function from my captureOutput, only doing so when my self.isRecording = true, which I set when the user taps a "Start Record" button.
let videoCompressionSettings: [String: Any] = [
AVVideoCodecKey: AVVideoCodecType.h264,
AVVideoWidthKey: NSNumber(value: 1080),
AVVideoHeightKey: NSNumber(value: 1920)
]
let audioCompressionSettings: [String: Any] = [
AVNumberOfChannelsKey: NSNumber(value: 1),
AVEncoderAudioQualityForVBRKey: NSNumber(value: 91),
AVEncoderBitRatePerChannelKey:NSNumber(value: 9600),
AVEncoderBitRateStrategyKey: AVAudioBitRateStrategy_Variable,
AVFormatIDKey: NSNumber(value: 1633772320),
AVSampleRateKey: NSNumber(value: 44100)
]
// My own wrapper for AVAssetWriter
movieWriterManager = MovieWriterManager(videoUrl: recordingVideoURL(), audioUrl: recordingAudioURL(), videoCompressionSettings: videoCompressionSettings, audioCompressionSettings: audioCompressionSettings)
movieWriterManager?.warmup()
You may try to buy time by running the same steps at AppDelegate's didFinishLaunchingWithOptions with a default videoCompressionSettings and audioCompressionSettings. It's actually not blocking UI thread but only the video output connection of AVCaptureVideoDataOutput.
Related
I'm trying to use AudioKit.renderToFile() to export short MIDI passages to audio (m4a):
// renderSequencer is an instance of AKSequencer
self.renderSequencer.loadMIDIFile(fromURL: midiURL)
Conductor.sharedInstance.setInstrument(renderItem.soundID, forOfflineRender: true)
// we only have one track with note content
for track in self.renderSequencer.tracks {
if track.isNotEmpty {
track.setMIDIOutput(Conductor.sharedInstance.midiIn)
}
}
let audioCacheDir = self.module.stateManager.audioCacheDirectory
// strip name off midi file
let midiFileName = String(midiURL.lastPathComponent.split(separator: ".")[0])
audioFileName = midiFileName
audioFileURL = audioCacheDir.appendingPathComponent("\(midiFileName).m4a")
if let audioFileURL = audioFileURL {
let settings = [
AVFormatIDKey: Int(kAudioFormatMPEG4AAC),
AVSampleRateKey: 44100,
AVNumberOfChannelsKey: 2,
AVEncoderAudioQualityKey: AVAudioQuality.high.rawValue
]
let audioFile: AVAudioFile = try! AVAudioFile(forWriting: audioFileURL, settings: settings)
// get time in seconds of audio file (with 4-beat tail)
var duration: Float64 = 0.0
MusicSequenceGetSecondsForBeats(seq, (16.0 + 4), &duration)
// render sequence
do { try AudioKit.renderToFile(audioFile, duration: duration) {
self.renderSequencer.setRate(60.0)
self.renderSequencer.play()
}
} catch { print("Error performing offline file render!") }
}
This does produce an audio file of the expected duration, but it is silent. I've also tried logging from my MIDI output and can see that the events "played" from inside the preload closure are actually being sent/handled.
Mostly, I suppose, I'm curious to know whether this is actually expected to work. I've seen a couple of posts suggesting that renderToFile from MIDI is not supported (while others have suggested they have it working).
I did, btw, also post an issue on the audiokit GitHub.
I am making an app which needs to stream audio to a server. What I want to do is to divide the recorded audio into chunks and upload them while recording.
I used two recorders to do that, but it didn't work well; I can hear the difference between the chunks (stops for couple of milliseconds).
How can I do this?
Your problem can be broken into two pieces: recording and chunking (and uploading, but who cares).
For recording from the microphone and writing to the file, you can get started quickly with AVAudioEngine and AVAudioFile. See below for a sample, which records chunks at the device's default input sampling rate (you will probably want to rate convert that).
When you talk about the "difference between the chunks" you are referring to the ability to divide your audio data into pieces in such a way that when you concatenate them you don't hear discontinuities. e.g. LPCM audio data can be divided into chunks at the sample level, but the LPCM bitrate is high, so you're more likely to use a packetised format like adpcm (called ima4 on iOS?), or mp3 or aac. These formats can only be divided on packet boundaries, e.g. 64, 576 or 1024 samples, say. If your chunks are written without a header (usual for mp3 and aac, not sure about ima4), then concatenation is trivial: simply lay the chunks end to end, exactly as the cat command line tool would. Sadly, on iOS there is no mp3 encoder, so that leaves aac as a likely format for you, but that depends on your playback requirements. iOS devices and macs can definitely play it back.
import AVFoundation
class ViewController: UIViewController {
let engine = AVAudioEngine()
struct K {
static let secondsPerChunk: Float64 = 10
}
var chunkFile: AVAudioFile! = nil
var outputFramesPerSecond: Float64 = 0 // aka input sample rate
var chunkFrames: AVAudioFrameCount = 0
var chunkFileNumber: Int = 0
func writeBuffer(_ buffer: AVAudioPCMBuffer) {
let samplesPerSecond = buffer.format.sampleRate
if chunkFile == nil {
createNewChunkFile(numChannels: buffer.format.channelCount, samplesPerSecond: samplesPerSecond)
}
try! chunkFile.write(from: buffer)
chunkFrames += buffer.frameLength
if chunkFrames > AVAudioFrameCount(K.secondsPerChunk * samplesPerSecond) {
chunkFile = nil // close file
}
}
func createNewChunkFile(numChannels: AVAudioChannelCount, samplesPerSecond: Float64) {
let fileUrl = NSURL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("chunk-\(chunkFileNumber).aac")!
print("writing chunk to \(fileUrl)")
let settings: [String: Any] = [
AVFormatIDKey: kAudioFormatMPEG4AAC,
AVEncoderBitRateKey: 64000,
AVNumberOfChannelsKey: numChannels,
AVSampleRateKey: samplesPerSecond
]
chunkFile = try! AVAudioFile(forWriting: fileUrl, settings: settings)
chunkFileNumber += 1
chunkFrames = 0
}
override func viewDidLoad() {
super.viewDidLoad()
let input = engine.inputNode!
let bus = 0
let inputFormat = input.inputFormat(forBus: bus)
input.installTap(onBus: bus, bufferSize: 512, format: inputFormat) { (buffer, time) -> Void in
DispatchQueue.main.async {
self.writeBuffer(buffer)
}
}
try! engine.start()
}
}
I am running on iPad iOS 9.3.4 (latest version as of this writing).
I am running this code:
let settings = [
AVFormatIDKey: NSNumber(unsignedInt: kAudioFormatMPEG4AAC),
AVSampleRateKey: 44100.0,
AVNumberOfChannelsKey: 2 as NSNumber,
AVEncoderAudioQualityKey: AVAudioQuality.High.rawValue
]
do {
audioRecorder = try AVAudioRecorder(URL: audioURL, settings: settings)
audioRecorder.delegate = self
audioRecorder.record()
} catch let error as NSError{
print(error.description)
}
I am catching this error:
Error Domain=NSOSStatusErrorDomain Code=1718449215 "(null)"
When I try to use AVAudioRecorder with Objective-C - I am able to record with no problems. The problem seems to only occur with Swift and only on a device - no issue in simulator.
If I switch out kAudioFormatMPEG4AAC with kAudioFormatLinearPCM, I am able to record - but when I try to play back the recording nothing plays - seems like it hasn't recorded well.
Has anyone been able to record with AVAudioRecorder in Swift lately and have the recording play back on a real iPad? I would like to just have that code.
Output file path extension must be in sync with AVFormatIDKey
For .wav
let recordSettings:[String:Any] = [AVFormatIDKey:kAudioFormatLinearPCM,
AVEncoderAudioQualityKey:AVAudioQuality.max.rawValue,
AVEncoderBitRateKey:320000,
AVNumberOfChannelsKey:2,
AVSampleRateKey:44100.0 ] as [String : Any]
For .m4a
let recordSettings:[String:Any] = [AVFormatIDKey:kAudioFormatAppleLossless,
AVEncoderAudioQualityKey:AVAudioQuality.max.rawValue,
AVEncoderBitRateKey:320000,
AVNumberOfChannelsKey:2,
AVSampleRateKey:44100.0 ] as [String : Any]
Looks like I never set the recording session as being active. I wish the error description was better though.
override init() {
super.init()
recordingSession = AVAudioSession.sharedInstance()
do {
try recordingSession.setCategory(AVAudioSessionCategoryPlayAndRecord)
try recordingSession.setActive(true)
recordingSession.requestRecordPermission() { (allowed: Bool) -> Void in
dispatch_async(dispatch_get_main_queue()) {
if allowed {
// success
} else {
// TBD: Show a message to the user that they need to give permission in settings app to proceed
}
}
}
} catch {
// TBD: Show a message to the user that they need to give permission in settings app to proceed
}
}
I'm trying to record audio using the microphone and AVAudioRecorder.
It works on iOS 8 but my code does not work anymore on iOS 9.
The recordSettings dictionary is set properly, then I give it to the
AVAudioRecorder:URL:settings constructor.
But, just after, recorder.settings is empty, an assertion failure is thrown
let recordSettings: [String: AnyObject] = [
AVNumberOfChannelsKey: NSNumber(integer: 2),
AVFormatIDKey: NSNumber(integer: Int(kAudioFormatMPEG4AAC)),
AVEncoderBitRateKey: NSNumber(integer: 64)]
var recorder: AVAudioRecorder!
do {
recorder = try AVAudioRecorder(URL: tempURL, settings:recordSettings) // recordSettings.count = 3
assert(recorder.settings.count != 0, "Audio Recorder does not provide settings") // assertion failure threw
} catch let error as NSError {
print("error when intitializing recorder: \(error)")
return
}
Anyone can help me ? Is it a bug ?
EDIT : In my entire code I did not test recorder.settings just after. I did instantiate recorder like my code above, then I did that :
recorder.delegate = self
recorder.prepareToRecord()
recorder.meteringEnabled = true
And it crashes in this line :
for i in 1...(recorder.settings[AVNumberOfChannelsKey] as! Int) {
...
}
It crashes because recorder.settings[AVNumberOfChannelsKey] is nil
I'm not sure why you're checking the settings property, but
from the AVAudioRecorder header file, on the settings property:
these settings are fully valid only when prepareToRecord has been called
so you must call prepareToRecord() first BUT it will fail/return false, because your bitrate is way too low! Its unit is bits per second, not kilobits per second:
AVEncoderBitRateKey: NSNumber(integer: 64000)
This worked on iOS 8 because your too-low bitrate was simply discarded. Looks like it became an error in iOS 9.
When using an AVAudioPlayerNode to schedule a short buffer to play immediately on a touch event ("Touch Up Inside"), I've noticed audible glitches / artifacts on playback while testing. The audio does not glitch at all in iOS simulator, however there is audible distortion on playback when I run the app on an actual iOS device. The audible distortion occurs randomly (the triggered sound will sometimes sound great, while other times it sounds distorted)
I've tried using different audio files, file formats, and preparing the buffer for playback using the prepareWithFrameCount method, but unfortunately the result is always the same and I'm stuck wondering what could be going wrong..
I've stripped the code down to globals for clarity and simplicity. Any help or insight would be greatly appreciated. This is my first attempt at developing an iOS app and my first question posted on Stack Overflow.
let filePath = NSBundle.mainBundle().pathForResource("BD_withSilence", ofType: "caf")!
let fileURL: NSURL = NSURL(fileURLWithPath: filePath)!
var error: NSError?
let file = AVAudioFile(forReading: fileURL, error: &error)
let fileFormat = file.processingFormat
let frameCount = UInt32(file.length)
let buffer = AVAudioPCMBuffer(PCMFormat: fileFormat, frameCapacity: frameCount)
let audioEngine = AVAudioEngine()
let playerNode = AVAudioPlayerNode()
func startEngine() {
var error: NSError?
file.readIntoBuffer(buffer, error: &error)
audioEngine.attachNode(playerNode)
audioEngine.connect(playerNode, to: audioEngine.mainMixerNode, format: buffer.format)
audioEngine.prepare()
func start() {
var error: NSError?
audioEngine.startAndReturnError(&error)
}
start()
}
startEngine()
let frameCapacity = AVAudioFramePosition(buffer.frameCapacity)
let frameLength = buffer.frameLength
let sampleRate: Double = 44100.0
func play() {
func scheduleBuffer() {
playerNode.scheduleBuffer(buffer, atTime: nil, options: AVAudioPlayerNodeBufferOptions.Interrupts, completionHandler: nil)
playerNode.prepareWithFrameCount(frameLength)
}
if playerNode.playing == false {
scheduleBuffer()
let time = AVAudioTime(sampleTime: frameCapacity, atRate: sampleRate)
playerNode.playAtTime(time)
}
else {
scheduleBuffer()
}
}
// triggered by a "Touch Up Inside" event on a UIButton in my ViewController
#IBAction func triggerPlay(sender: AnyObject) {
play()
}
Update:
Ok I think I've identified the source of the distortion: the volume of the node(s) is too great at output and causes clipping. By adding these two lines in my startEngine function, the distortion no longer occurred:
playerNode.volume = 0.8
audioEngine.mainMixerNode.volume = 0.8
However, I'm still don't know why I need to lower the output- my audio file itself does not clip. I'm guessing that it might be a result of the way that the AVAudioPlayerNodeBufferOptions.Interrupts is implemented. When a buffer interrupts another buffer, could there be an increase in output volume as a result of the interruption, causing output clipping? I'm still looking for a solid understanding as to why this occurs.. If anyone is willing/able to provide any clarification about this that would be fantastic!
Not sure if this is the problem you experienced in 2015, it may be the same issue that #suthar experienced in 2018.
I experienced a very similar problem and was due to the fact that the sampleRate on the device is different to the simulator. On macOS it is 44100 and on iOS Devices (late model ones) it is 48000.
So when you fill your buffer with 44100 samples on a 48000 device, you get 3900 samples of silence. When played back it doesn't sound like silence, it sounds like a glitch.
I used the mainMixer format when connecting my playerNode and also when creating my pcmBuffer. Don't refer to 48000 or 44100 anywhere in the code.
audioEngine.attach( playerNode)
audioEngine.connect( playerNode, to:mixerNode, format:mixerNode.outputFormat(forBus:0))
let pcmBuffer = AVAudioPCMBuffer( pcmFormat:SynthEngine.shared.audioEngine.mainMixerNode.outputFormat( forBus:0),
frameCapacity:AVAudioFrameCount(bufferSize))