Using AVFoundation and AVAudioPlayerNode to play sound with Xcode Version 9.2 (9C40b) and deploying to iOS 11.2
The problem is that when you change volume, the change is not applied the first time you play the sound, and there are other weird effects.
Start a new project in Xcode, select iOS Game, and give any name. Then replace the code in GameScene.swift with:
import SpriteKit
import GameplayKit
import AVFoundation
class GameScene: SKScene {
var entities = [GKEntity]()
var graphs = [String : GKGraph]()
let audioFilePlayer = AVAudioPlayerNode()
var audioFile:AVAudioFile! = nil
var audioFileBuffer:AVAudioPCMBuffer! = nil
override func sceneDidLoad() {
do {
let path = Bundle.main.path(forResource: "sound", ofType: "caf")!
let url = URL(fileURLWithPath: path)
audioFile = try AVAudioFile(forReading: url)
audioFileBuffer = AVAudioPCMBuffer(pcmFormat: audioFile.processingFormat, frameCapacity: UInt32(audioFile.length))
try audioFile.read(into: audioFileBuffer!)
audioEngine.attach(audioFilePlayer)
audioEngine.connect(audioFilePlayer, to: audioEngine.mainMixerNode, format: audioFileBuffer?.format)
try audioEngine.start()
}
catch {
print(error)
}
let playAction = SKAction.sequence([
SKAction.wait(forDuration: 3.0),
SKAction.run { self.play(volume: 1.0) },
SKAction.wait(forDuration: 1.0),
SKAction.run { self.play(volume: 1.0) },
SKAction.wait(forDuration: 1.0),
SKAction.run { self.play(volume: 0.0) },
SKAction.wait(forDuration: 1.0),
SKAction.run { self.play(volume: 0.0) },
SKAction.wait(forDuration: 1.0),
SKAction.run { self.play(volume: 1.0) },
SKAction.wait(forDuration: 1.0),
SKAction.run { self.play(volume: 1.0) },
])
self.run(playAction)
}
func play(volume:Float) {
print("playing at \(volume)")
audioFilePlayer.stop()
audioFilePlayer.volume = volume
audioFilePlayer.scheduleFile(audioFile, at: nil, completionHandler: nil)
audioFilePlayer.play()
}
}
Apologies for poor optionals... Also, add a sound file called sound.caf to the project.
Console output is:
playing at 1.0
playing at 1.0
playing at 0.0
playing at 0.0
playing at 1.0
playing at 1.0
I would expect to hear: loud, loud, nothing, nothing, loud, loud.
I am actually hearing: loud, loud, loud, nothing, soft, loud
(the 'soft' is particularly weird)
I have also tried changing the master volume with:
audioEngine.mainMixerNode.outputVolume = volume
but the sounds are the same.
In older games, used to use OpenAL, but this requires Obj-C and is pretty messy with bridging headers etc. Also, OpenAL is supposedly not supported any more. SKAction based audio cannot handle lots of sounds repeating fast without glitches and scratches (it's a space shooter game...) Problem is the same in simulator and on device. Any help appreciated!
OK I found the answer. AudioEngine needs to be reset after volume changes in order for the changes to be immediate:
audioEngine.reset()
this play() function works:
func play(volume:Float) {
print("playing at \(volume)")
audioFilePlayer.volume = volume
audioEngine.reset()
audioFilePlayer.stop()
audioFilePlayer.scheduleFile(audioFile, at: nil, completionHandler: nil)
audioFilePlayer.play()
}
Yep, in case anyone else is interested I also solved it by adding an AVAudioMixerNode between the input source and engine.mainMixerNode
Related
I need to calculate the frequency, in Hertz, of a sound recorded with the microphone.
What I'm doing now is using AVAudioRecorder to listen to the mic, with a timer that call a specific function every 0.5 seconds. Here some code :
class ViewController: UIViewController {
var audioRecorder: AVAudioRecorder?
var timer: Timer?
override func viewDidLoad() {
super.viewDidLoad()
// Do any additional setup after loading the view.
let permission = AVAudioSession.sharedInstance().recordPermission
if permission == AVAudioSession.RecordPermission.undetermined {
AVAudioSession.sharedInstance().requestRecordPermission { (granted) in
if granted {
print("Permission granted!")
} else {
print("Permission not granted!")
}
}
} else if permission == AVAudioSession.RecordPermission.granted {
do {
try AVAudioSession.sharedInstance().setCategory(AVAudioSession.Category.record)
let settings = [
AVSampleRateKey: 44100.0,
AVFormatIDKey: kAudioFormatAppleLossless,
AVNumberOfChannelsKey: 1,
AVEncoderAudioQualityKey: AVAudioQuality.max
] as [String : Any]
audioRecorder = try AVAudioRecorder.init(url: NSURL.fileURL(withPath: "dev/null"), settings: settings)
audioRecorder?.prepareToRecord()
audioRecorder?.isMeteringEnabled = true
audioRecorder?.record()
timer = Timer.scheduledTimer(
timeInterval: 0.5,
target: self,
selector: #selector(analyze),
userInfo: nil,
repeats: true
)
} catch (let error) {
print("Error! \(error.localizedDescription)")
}
}
}
#objc func analyze() {
audioRecorder?.updateMeters()
let peak = audioRecorder?.peakPower(forChannel: 0)
print("Peak : \(peak)")
audioRecorder?.updateMeters()
}
}
I do not know how to get the frequency of the sound in hertz. It's fine also to use a 3rd party framework for me.
Thanks.
Any given recorded sound will not have a single frequency. It will have a mix of frequencies at different amplitudes.
You need to do frequency analysis on the input sounds, usually using FFT (Fast Fourier Transforms) on the audio data.
A google search revealed this article on doing frequency analysis using the Accelerate framework:
http://www.myuiviews.com/2016/03/04/visualizing-audio-frequency-spectrum-on-ios-via-accelerate-vdsp-fast-fourier-transform.html
First of all, a great framework. This is singlehandedly allowing me to graduate from my master's program. Also, I'm a sponsor! Any help would be taken with much gratitude. I can also push my repository and share it on GitHub for a closer look.
Anyway, here is my code
import Foundation
import AudioKit
class DrumSounds {
let drums = AKMIDISampler()
var currentBPM = 60
var rideCymbalFile: AKAudioFile?
var snareDrumFile: AKAudioFile?
var bassDrumFile: AKAudioFile?
var hiHatFile: AKAudioFile?
let sequencer = AKAppleSequencer(filename: "4tracks")
var booster = AKBooster()
init() {
do{
try rideCymbalFile = AKAudioFile(readFileName: "rideCymbalSound.wav")
try snareDrumFile = AKAudioFile(readFileName: "snareDrumSound.wav")
try bassDrumFile = AKAudioFile(readFileName: "bassDrumSound.wav")
try hiHatFile = AKAudioFile(readFileName: "hiHatSound.mp3")
try drums.loadAudioFiles([rideCymbalFile!,
snareDrumFile!,
bassDrumFile!,
hiHatFile!])
} catch {
print("error loading samples to drum object")
}
drums.volume = 1
booster = AKBooster(drums)
AudioKit.output = drums
sequencer.clearRange(start: AKDuration(beats: 0), duration: AKDuration(beats: 100))
sequencer.debug()
sequencer.setGlobalMIDIOutput(drums.midiIn)
sequencer.enableLooping(AKDuration(beats: 4))
sequencer.setTempo(Double(currentBPM))
}
func playDrumSounds () {
do {
try AKSettings.setSession(category: .playAndRecord, with: AVAudioSession.CategoryOptions.defaultToSpeaker)
let session = AVAudioSession.sharedInstance()
try session.setCategory(AVAudioSession.Category.playAndRecord)
if !AKSettings.headPhonesPlugged {
try session.overrideOutputAudioPort(AVAudioSession.PortOverride.speaker)
}
}catch {
print("error in settings.setSession")
}
sequencer.tracks[0].add(noteNumber: 0, velocity: 127, position: AKDuration(beats: 0), duration: AKDuration(beats: 1.0))
sequencer.tracks[0].add(noteNumber: 0, velocity: 127, position: AKDuration(beats: 1), duration: AKDuration(beats: 1.0))
sequencer.tracks[0].add(noteNumber: 0, velocity: 127, position: AKDuration(beats: 2), duration: AKDuration(beats: 1.0))
sequencer.tracks[0].add(noteNumber: 0, velocity: 127, position: AKDuration(beats: 3), duration: AKDuration(beats: 1.0))
sequencer.play()
}
}
I figured it out by randomly stumbling upon a comment in another post. The volume is low because you need to enable "Audio, AirPlay, and Picture in Picture" in "Background Modes" under "Signing & Capabilities". Click the "+" button in the top left to add a capability:
As for playing the right drum sound: The right drum sound was, in fact, being played. However I set the MIDI note number too low, so it sounded like harsh static. If you're having this problem and have never worked with MIDI (like me), here is a link to a description of MIDI note numbers: https://www.inspiredacoustics.com/en/MIDI_note_numbers_and_center_frequencies. The higher the number, the higher the frequency. Changing the MIDI note number will change the frequency of your audio file!
First of all, this error only occurs in the latest 12.4 release on iOS. The issue does NOT occur in the simulator and must be run on a device. The issue is that the call to record on the AVAudioRecorder is returning false once the app goes into the background. In all previous versions of iOS, this would not happen. The info.plist is updated with the NSMicrophoneUsageDescription tag, and the capabilities for the app includes Audio background mode.
I have written a small ViewController that shows the issue. Steps to recreate:
1) Update the info.plist file with the NSMicrophoneUsageDescription tag so that the app gets permission to use the microphone
2) Update the app capabilities to set Audio background mode
3) Run the application
4) Send the application to the background
5) The current recording will finish, but the call to start a new recording will fail.
class ViewController: UIViewController, AVAudioRecorderDelegate {
var recordLabel: UILabel!
var recordingSession: AVAudioSession!
var audioRecorder: AVAudioRecorder!
var count: Int = 0
override func viewDidLoad() {
super.viewDidLoad()
recordingSession = AVAudioSession.sharedInstance()
do {
try recordingSession.setCategory(.playAndRecord, mode: .default)
try recordingSession.setActive(true)
recordingSession.requestRecordPermission() { [unowned self] allowed in
DispatchQueue.main.async {
if allowed {
self.loadRecordingUI()
} else {
print("No permissions!!!")
}
}
}
} catch {
print("Exception in viewDidLoad!!!")
}
}
func loadRecordingUI() {
super.viewDidLoad()
recordLabel = UILabel(frame: CGRect(x: 0, y: 0, width: 300, height: 21))
recordLabel.center = CGPoint(x: 160, y: 285)
recordLabel.textAlignment = .center
recordLabel.text = "Waiting...."
self.view.addSubview(recordLabel)
setupRecorder()
startRecording();
}
func setupRecorder() {
let audioFilename = getDocumentsDirectory().appendingPathComponent("recording.m4a")
let settings = [
AVFormatIDKey: Int(kAudioFormatMPEG4AAC),
AVSampleRateKey: 12000,
AVNumberOfChannelsKey: 1,
AVEncoderAudioQualityKey: AVAudioQuality.high.rawValue
]
do {
audioRecorder = try AVAudioRecorder(url: audioFilename, settings: settings)
audioRecorder.delegate = self
} catch {
print("Exception thrown in setupRecorder")
}
}
func startRecording() {
count += 1
let ret = audioRecorder.record(forDuration: 10) //record for 10 seonds
let txt = "Record returned " + ret.description + " for #\(count)"
recordLabel.text = txt
print(txt)
}
func getDocumentsDirectory() -> URL {
let paths = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)
return paths[0]
}
func audioRecorderDidFinishRecording(_ recorder: AVAudioRecorder, successfully flag: Bool) {
startRecording() //immediately start recording again
}
}
Since the capability for the app to record in the background has been set, I expect the call to record on the AVAudioRecorder to return true
I filed a feedback to Apple about this (audioRecorder.record() returns false while in background) and got the answer that it's a new privacy protection restriction, i.e. they will not fix it.
"The behaviour mentioned is a change but is the correct behaviour going forward, attempting to start audio recording in the background which had not been recording previously (and then interrupted by a call or Siri) will not work (essentially trying to start recording from the background randomly will no longer work). This is a new a privacy protection restriction introduced in 12.4."
I have a navigation app that gives direction voice instruction (e.g. " In 200 feet turn left") using AVSpeechUtterance. I have put volume to 1 like so. speechUtteranceInstance.volume = 1, but still the volume is very low compared to the music or podcast coming from the iPhone, especially when the sound is on a Bluetooth or cabled connection (like connected to car with Bluetooth)
Is there any way to boost the volume?
(I know this has been asked before on SO but so far have not found a solution that works for me.)
After a lot more research and playing around, I found a good workaround solution.
First of all I think this is an iOS bug. When all below conditions are true I found that the voice instruction itself is also ducked (or at least it sounds ducked) resulting in the voice instruction playing at the same volume as the DUCKED music (thus way too soft to hear well).
Playing music in the background
Ducking this background music through
the .duckOther audioSessionCategory
Playing a voiceUtterance through AVSpeechSynthesizer
Playing audio over a connected bluetooth
device (like bluetooth headset or bluetooth car speakers)
The workaround solution I found is to feed the speechUtterance to an AVAudioEngine. This can only be done on iOS13 or above, since that adds the .write method to AVSpeechSynthesizer
In short I use AVAudioEngine, AVAudioUnitEQ and AVAudioPlayerNode, setting the globalGain property of the AVAudioUnitEQ to about 10 dB. There are also a few quirks with this, but they can be worked around (see code comments).
Here's the complete code:
import UIKit
import AVFoundation
import MediaPlayer
class ViewController: UIViewController {
// MARK: AVAudio properties
var engine = AVAudioEngine()
var player = AVAudioPlayerNode()
var eqEffect = AVAudioUnitEQ()
var converter = AVAudioConverter(from: AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatInt16, sampleRate: 22050, channels: 1, interleaved: false)!, to: AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatFloat32, sampleRate: 22050, channels: 1, interleaved: false)!)
let synthesizer = AVSpeechSynthesizer()
var bufferCounter: Int = 0
let audioSession = AVAudioSession.sharedInstance()
override func viewDidLoad() {
super.viewDidLoad()
let outputFormat = AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatFloat32, sampleRate: 22050, channels: 1, interleaved: false)!
setupAudio(format: outputFormat, globalGain: 0)
}
func activateAudioSession() {
do {
try audioSession.setCategory(.playback, mode: .voicePrompt, options: [.mixWithOthers, .duckOthers])
try audioSession.setActive(true, options: .notifyOthersOnDeactivation)
} catch {
print("An error has occurred while setting the AVAudioSession.")
}
}
#IBAction func tappedPlayButton(_ sender: Any) {
eqEffect.globalGain = 0
play()
}
#IBAction func tappedPlayLoudButton(_ sender: Any) {
eqEffect.globalGain = 10
play()
}
func play() {
let path = Bundle.main.path(forResource: "voiceStart", ofType: "wav")!
let file = try! AVAudioFile(forReading: URL(fileURLWithPath: path))
self.player.scheduleFile(file, at: nil, completionHandler: nil)
let utterance = AVSpeechUtterance(string: "This is to test if iOS is able to boost the voice output above the 100% limit.")
synthesizer.write(utterance) { buffer in
guard let pcmBuffer = buffer as? AVAudioPCMBuffer, pcmBuffer.frameLength > 0 else {
print("could not create buffer or buffer empty")
return
}
// QUIRCK Need to convert the buffer to different format because AVAudioEngine does not support the format returned from AVSpeechSynthesizer
let convertedBuffer = AVAudioPCMBuffer(pcmFormat: AVAudioFormat(commonFormat: AVAudioCommonFormat.pcmFormatFloat32, sampleRate: pcmBuffer.format.sampleRate, channels: pcmBuffer.format.channelCount, interleaved: false)!, frameCapacity: pcmBuffer.frameCapacity)!
do {
try self.converter!.convert(to: convertedBuffer, from: pcmBuffer)
self.bufferCounter += 1
self.player.scheduleBuffer(convertedBuffer, completionCallbackType: .dataPlayedBack, completionHandler: { (type) -> Void in
DispatchQueue.main.async {
self.bufferCounter -= 1
print(self.bufferCounter)
if self.bufferCounter == 0 {
self.player.stop()
self.engine.stop()
try! self.audioSession.setActive(false, options: [])
}
}
})
self.converter!.reset()
//self.player.prepare(withFrameCount: convertedBuffer.frameLength)
}
catch let error {
print(error.localizedDescription)
}
}
activateAudioSession()
if !self.engine.isRunning {
try! self.engine.start()
}
if !self.player.isPlaying {
self.player.play()
}
}
func setupAudio(format: AVAudioFormat, globalGain: Float) {
// QUIRCK: Connecting the equalizer to the engine somehow starts the shared audioSession, and if that audiosession is not configured with .mixWithOthers and if it's not deactivated afterwards, this will stop any background music that was already playing. So first configure the audio session, then setup the engine and then deactivate the session again.
try? self.audioSession.setCategory(.playback, options: .mixWithOthers)
eqEffect.globalGain = globalGain
engine.attach(player)
engine.attach(eqEffect)
engine.connect(player, to: eqEffect, format: format)
engine.connect(eqEffect, to: engine.mainMixerNode, format: format)
engine.prepare()
try? self.audioSession.setActive(false)
}
}
The docs mention that the default for .volume is 1.0 and that's the loudest. Actual loudness is based on the user volume settings. I didn't really have an issue with the speech being not loud enough if the user has the volume turned up.
Maybe you could consider showing a visual warning if the user volume level is below a certain level. Seems like this answer shows how to do that via AVAudioSession.
AVAudioSession is worth exploring as there are some settings that do impact speech output... like for example does you app's speech interrupt audio from other apps.
Try this:
import Speech
try? AVAudioSession.sharedInstance().setCategory(.playback, mode: .default, options: [])
let utterance = AVSpeechUtterance(string: "Hello world")
utterance.voice = AVSpeechSynthesisVoice(language: "en-GB")
let synthesizer = AVSpeechSynthesizer()
synthesizer.speak(utterance)
I'm using AVMutableComposition to play a video which is split in different continuous files. However, when playing, there is a small jump (short black screen) between 2 consecutive segments.
Is there any way to prevent that?
import UIKit
import AVFoundation
class KLMasterPlayerViewController: KLPlayerViewController {
let comp:AVMutableComposition = AVMutableComposition()
var playerItem:AVPlayerItem!
init() {
super.init(nibName: nil, bundle: nil)
self.initComp()
self.playerItem = AVPlayerItem(asset: self.comp)
self.player = AVPlayer(playerItem: self.playerItem)
}
private func initComp() {
let segments = MasterVideo.sharedInstance.videoSegments
var insertedTime:Double = 0.0
for segment in segments {
do {
let asset = segment.getURLAsset()
try comp.insertTimeRange(CMTimeRangeMake(kCMTimeZero,asset!.duration), ofAsset: asset!, atTime: CMTimeMake(Int64(insertedTime * 10000), 10000))
insertedTime += segment.getDuration()
} catch {
}
}
}
}
I've found the answer: Swift wasn't calculating the duration of my video assets properly. For each one of them, there was a different of 1 image.
By forcing the video segments duration, that was fine and flawless.
Glad it's solved!
Many thanks to vaibhav for trying to help.