I need to record a video and show a video with an AVPlayer at the same time. The result needs to be synchronized: if the video I'm showing is a metronome, or someone clapping, the recorded video and the player should click or clap at the same time.
I play the AVPlayer using preroll and setRate, with a delegate in the camera controller.
Using AVCaptureMovieFileOutput, I've tried calling setRate on the player once fileOutput(output:didStartRecordingTo) is called, but the videos end up desynchronized, with the recorded video going behind the player.
Using an AssetWriter, I've tried calling setRate on the player once captureOutput(captureOutput:sampleBuffer) is called, and the first buffer is appended, but the result is the same.
Is there any other way to do this?
EDIT: Adding some code to show what I'm doing:
Camera with AVCaptureMovieFileOutput:
func startRecordingVideo() {
guard let movieFileOutput = self.movieFileOutput else {
return
}
sessionQueue.async {
if !movieFileOutput.isRecording {
let movieFileOutputConnection = movieFileOutput.connection(with: .video)
movieFileOutput.setOutputSettings([AVVideoCodecKey: AVVideoCodecH264], for: movieFileOutputConnection!)
let outputFileName = NSUUID().uuidString
let outputFilePath = (NSTemporaryDirectory() as NSString).appendingPathComponent((outputFileName as NSString).appendingPathExtension("mov")!)
movieFileOutput.startRecording(to: URL(fileURLWithPath: outputFilePath), recordingDelegate: self)
} else {
movieFileOutput.stopRecording()
}
}
}
func fileOutput(_ output: AVCaptureFileOutput, didStartRecordingTo fileURL: URL, from connections: [AVCaptureConnection]) {
cameraDelegate?.startedRecording()
}
Implementing delegate's startRecording in the viewController:
private func setRateForAll() {
let hostTime = CMClockGetTime(CMClockGetHostTimeClock())
activePlayers.forEach {
$0.setRate(atHostTime: hostTime)
}
}
Once the active player or players end, I call stopRecording on the camera, and show the resulting recorded video in another player, along with the player or players I've been showing.
I ended up using the recording session master clock as the hostTime for the setRate method. The desynchronization is almost imperceptible now!
In fileOutput(output:didStartRecordingTo):
cameraDelegate?.startedRecording(clock: CMClockGetTime(self.session.masterClock!))
In the view with the players:
private func setRateForAll(clock: CMTime) {
activePlayers.forEach {
$0.setRate(atHostTime: clock)
}
}
I am having a couple of problems with player. I know there are many player examples but almost all of them are designed to play just one url and does not have the functionalities that I want.
First of all this is my existing code for the player:
struct PlayerViewController: UIViewControllerRepresentable {
var video: Video
private let player = AVPlayer()
func makeUIViewController(context: Context) -> AVPlayerViewController {
let playerVC = AVPlayerViewController()
playerVC.player = self.player
do {
try AVAudioSession.sharedInstance().setCategory(.playback)
} catch(let error) {
print(error.localizedDescription)
}
playerVC.player?.play()
return playerVC
}
func updateUIViewController(_ uiViewController: AVPlayerViewController, context: Context) {
guard let url = URL(string: video.url) else { return }
resetPlayer(uiViewController)
let item = AVPlayerItem(url: url)
uiViewController.player?.replaceCurrentItem(with: item)
uiViewController.player?.play()
if((uiViewController.view.window) == nil)
{
uiViewController.player?.pause()
}
print("Player::UpdatePlayer: Player updated")
}
func resetPlayer(_ vc: AVPlayerViewController) {
vc.player?.pause()
vc.player?.replaceCurrentItem(with: nil)
print("Player::ResetPlayer: Player reseted.!")
}
}
struct Video: Identifiable, Equatable {
var id: Int
var url: String
}
And I use it as follows. This is just example I replace the str with button action in the actual project.
struct TestTestView: View {
#State var str: String = ""
var body: some View {
PlayerViewController(video: Video(id: 2, url: (str == "") ? chan.getDVR()[0].getHlsStreamURL() : self.str ))
}
}
Now the code above works up to a point. It plays, you can update the url with button click and it plays the new url. But when it enters the fullscreen the video stops, because it updates the whole player although nothing changed. The following lines makes it pause when entering the fullscreen mode:
if((uiViewController.view.window) == nil)
{
uiViewController.player?.pause()
}
However, If I remove those lines when I navigate to another screen it continues to play you keep hearing the sound. I tried to detect the fullscreen to update my code but I was not able to do it. I tried to detect I the player is actually in the screen but it did not work as well. So what is the correct way to implement it. Basically I want to do:
1 - Be able to update the player and play different streams
2 - Starts auto play
3 - Smoothly continue to play when switching in and out of full screen mode
4 - Stop when navigated to another screen.
By the way if it is not in full screen mode I cannot click on play button or progress bar but somehow I can click on the mute or fullscreen button. I need help!!
I am using Fairplay implementation as per Apple's Fairplay Streaming sample code at https://developer.apple.com/streaming/fps/, although I tried to choose only parts that are related to Online Fairplay Streaming, not the persistence/offline playback. In the below code a video without Fairplay plays/pauses/seeks normally, but when I play a Fairplay protected video, only the video track behaves correctly.
Pausing playback won't stop the audio playback, changing audio track won't stop the previous audio track, so both plays together and perhaps the seek also does not work.
Besides this helper class below, I have AssetLoaderDelegate and AssetPlaybackManager from Apple's client sample code of FairPlay Streaming Server SDK https://developer.apple.com/streaming/fps/ and I have updated the code to handle SPC/CKC for our DRM keys provider.
Did I miss to implement some important part of the code to handle audio for FPS Streaming? Can you please point me into right direction? Many thanks.
class PlayHelper {
static let shared = PlayHelper()
fileprivate var playerViewController: PlayerViewController?
init() {
AssetPlaybackManager.sharedManager.delegate = self
}
// Play video without DRM
func playVideo(from urlString: String, at context: UIViewController) {
guard let videoURL = URL(string: urlString) else {
Log.error("Video URL can't be created from string: \(urlString)")
return }
let player = AVPlayer(url: videoURL)
let playerViewController = PlayerViewController()
playerViewController.player = player
context.present(playerViewController, animated: true) {
playerViewController.player?.play()
}
}
// Play FPS video
func playFpsVideo(with asset: AVURLAsset, at context: UIViewController) {
// Cleanup, should be done when playerViewController is actually dismissed
if self.playerViewController != nil {
// The view reappeared as a results of dismissing an AVPlayerViewController.
// Perform cleanup.
AssetPlaybackManager.sharedManager.setAssetForPlayback(nil)
self.playerViewController?.player = nil
self.playerViewController = nil
}
let item = AVPlayerItem(asset: asset)
let player = AVPlayer(playerItem: item)
// Customize player
player.appliesMediaSelectionCriteriaAutomatically = true
let playerViewController = PlayerViewController()
playerViewController.player = player
self.playerViewController = playerViewController
context.present(playerViewController, animated: true) {
playerViewController.player?.play()
}
}
// Stop video
func stop() {
// Cleanup, should be done when playerViewController is dismissed
if self.playerViewController != nil {
// Results of dismissing an AVPlayerViewController, perform cleanup
AssetPlaybackManager.sharedManager.setAssetForPlayback(nil)
self.playerViewController?.player = nil
self.playerViewController = nil
}
}
}
// MARK: - Extend `PlayHelper` to conform to the `AssetPlaybackDelegate` protocol
extension PlayHelper: AssetPlaybackDelegate {
func streamPlaybackManager(_ streamPlaybackManager: AssetPlaybackManager, playerReadyToPlay player: AVPlayer) {
player.play()
}
func streamPlaybackManager(_ streamPlaybackManager: AssetPlaybackManager, playerCurrentItemDidChange player: AVPlayer) {
guard let playerViewController = playerViewController, player.currentItem != nil else { return }
playerViewController.player = player
}
}
I can also provide the code in AssetLoaderDelegate and AssetPlaybackManager if needed.
My bad. I called play() twice in the code above... Grrr.. Once when the presentation of the PlayerViewController finished and second time in the callback from AssetPlaybackDelegate that is triggered by KVO in AssetPlaybackManager. This way the player controls stopped playing the video, but most probably a second (audio) stream was still playing there. I removed the play() in playerReadyToPlay callback and now all the controls in the Player works as expected. I can pause, resume, seek, change audio tracks.
I'm trying to get multiple sounds files to play on an AVAudioPlayer instance, however when one sound plays, the other stops. I can't get more than one sound to play at a time. Here is my code:
import AVFoundation
class GSAudio{
static var instance: GSAudio!
var soundFileNameURL: NSURL = NSURL()
var soundFileName = ""
var soundPlay = AVAudioPlayer()
func playSound (soundFile: String){
GSAudio.instance = self
soundFileName = soundFile
soundFileNameURL = NSURL(fileURLWithPath: NSBundle.mainBundle().pathForResource(soundFileName, ofType: "aif", inDirectory:"Sounds")!)
do{
try soundPlay = AVAudioPlayer(contentsOfURL: soundFileNameURL)
} catch {
print("Could not play sound file!")
}
soundPlay.prepareToPlay()
soundPlay.play ()
}
}
Can anyone please help me by telling me how to get more than one sound file to play at a time? Any help is much appreciated.
Many thanks,
Kai
The reason the audio stops is because you only have one AVAudioPlayer set up, so when you ask the class to play another sound you are currently replacing the old instance with a new instance of AVAudioPlayer. You are overwriting it basically.
You can either create two instances of the GSAudio class, and then call playSound on each of them, or make the class a generic audio manager that uses a dictionary of audioPlayers.
I much prefer the latter option, as it allows for cleaner code and is also more efficient. You can check to see if you have already made a player for the sound before, rather than making a new player for example.
Anyways, I re-made your class for you so that it will play multiple sounds at once. It can also play the same sound over itself (it doesn't replace the previous instance of the sound) Hope it helps!
The class is a singleton, so to access the class use:
GSAudio.sharedInstance
for example, to play a sound you would call:
GSAudio.sharedInstance.playSound("AudioFileName")
and to play a number of sounds at once:
GSAudio.sharedInstance.playSounds("AudioFileName1", "AudioFileName2")
or you could load up the sounds in an array somewhere and call the playSounds function that accepts an array:
let sounds = ["AudioFileName1", "AudioFileName2"]
GSAudio.sharedInstance.playSounds(sounds)
I also added a playSounds function that allows you to delay each sound being played in a cascade kind of format. So:
let soundFileNames = ["SoundFileName1", "SoundFileName2", "SoundFileName3"]
GSAudio.sharedInstance.playSounds(soundFileNames, withDelay: 1.0)
would play sound2 a second after sound1, then sound3 would play a second after sound2 etc.
Here is the class:
class GSAudio: NSObject, AVAudioPlayerDelegate {
static let sharedInstance = GSAudio()
private override init() {}
var players = [NSURL:AVAudioPlayer]()
var duplicatePlayers = [AVAudioPlayer]()
func playSound (soundFileName: String){
let soundFileNameURL = NSURL(fileURLWithPath: NSBundle.mainBundle().pathForResource(soundFileName, ofType: "aif", inDirectory:"Sounds")!)
if let player = players[soundFileNameURL] { //player for sound has been found
if player.playing == false { //player is not in use, so use that one
player.prepareToPlay()
player.play()
} else { // player is in use, create a new, duplicate, player and use that instead
let duplicatePlayer = try! AVAudioPlayer(contentsOfURL: soundFileNameURL)
//use 'try!' because we know the URL worked before.
duplicatePlayer.delegate = self
//assign delegate for duplicatePlayer so delegate can remove the duplicate once it's stopped playing
duplicatePlayers.append(duplicatePlayer)
//add duplicate to array so it doesn't get removed from memory before finishing
duplicatePlayer.prepareToPlay()
duplicatePlayer.play()
}
} else { //player has not been found, create a new player with the URL if possible
do{
let player = try AVAudioPlayer(contentsOfURL: soundFileNameURL)
players[soundFileNameURL] = player
player.prepareToPlay()
player.play()
} catch {
print("Could not play sound file!")
}
}
}
func playSounds(soundFileNames: [String]){
for soundFileName in soundFileNames {
playSound(soundFileName)
}
}
func playSounds(soundFileNames: String...){
for soundFileName in soundFileNames {
playSound(soundFileName)
}
}
func playSounds(soundFileNames: [String], withDelay: Double) { //withDelay is in seconds
for (index, soundFileName) in soundFileNames.enumerate() {
let delay = withDelay*Double(index)
let _ = NSTimer.scheduledTimerWithTimeInterval(delay, target: self, selector: #selector(playSoundNotification(_:)), userInfo: ["fileName":soundFileName], repeats: false)
}
}
func playSoundNotification(notification: NSNotification) {
if let soundFileName = notification.userInfo?["fileName"] as? String {
playSound(soundFileName)
}
}
func audioPlayerDidFinishPlaying(player: AVAudioPlayer, successfully flag: Bool) {
duplicatePlayers.removeAtIndex(duplicatePlayers.indexOf(player)!)
//Remove the duplicate player once it is done
}
}
Here's a Swift 4 version of #Oliver Wilkinson code with some safechecks and improved code formatting:
import Foundation
import AVFoundation
class GSAudio: NSObject, AVAudioPlayerDelegate {
static let sharedInstance = GSAudio()
private override init() { }
var players: [URL: AVAudioPlayer] = [:]
var duplicatePlayers: [AVAudioPlayer] = []
func playSound(soundFileName: String) {
guard let bundle = Bundle.main.path(forResource: soundFileName, ofType: "aac") else { return }
let soundFileNameURL = URL(fileURLWithPath: bundle)
if let player = players[soundFileNameURL] { //player for sound has been found
if !player.isPlaying { //player is not in use, so use that one
player.prepareToPlay()
player.play()
} else { // player is in use, create a new, duplicate, player and use that instead
do {
let duplicatePlayer = try AVAudioPlayer(contentsOf: soundFileNameURL)
duplicatePlayer.delegate = self
//assign delegate for duplicatePlayer so delegate can remove the duplicate once it's stopped playing
duplicatePlayers.append(duplicatePlayer)
//add duplicate to array so it doesn't get removed from memory before finishing
duplicatePlayer.prepareToPlay()
duplicatePlayer.play()
} catch let error {
print(error.localizedDescription)
}
}
} else { //player has not been found, create a new player with the URL if possible
do {
let player = try AVAudioPlayer(contentsOf: soundFileNameURL)
players[soundFileNameURL] = player
player.prepareToPlay()
player.play()
} catch let error {
print(error.localizedDescription)
}
}
}
func playSounds(soundFileNames: [String]) {
for soundFileName in soundFileNames {
playSound(soundFileName: soundFileName)
}
}
func playSounds(soundFileNames: String...) {
for soundFileName in soundFileNames {
playSound(soundFileName: soundFileName)
}
}
func playSounds(soundFileNames: [String], withDelay: Double) { //withDelay is in seconds
for (index, soundFileName) in soundFileNames.enumerated() {
let delay = withDelay * Double(index)
let _ = Timer.scheduledTimer(timeInterval: delay, target: self, selector: #selector(playSoundNotification(_:)), userInfo: ["fileName": soundFileName], repeats: false)
}
}
#objc func playSoundNotification(_ notification: NSNotification) {
if let soundFileName = notification.userInfo?["fileName"] as? String {
playSound(soundFileName: soundFileName)
}
}
func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) {
if let index = duplicatePlayers.index(of: player) {
duplicatePlayers.remove(at: index)
}
}
}
I have created a helper library that simplifies playing sounds in Swift. It creates multiple instances of AVAudioPlayer to allow playing the same sound multiple times concurrently. You can download it from Github or import with Cocoapods.
Here is the link: SwiftySound
The usage is as simple as it can be:
Sound.play(file: "sound.mp3")
All answers are posting pages of code; it doesn't need to be that complicated.
// Create a new player for the sound; it doesn't matter which sound file this is
let soundPlayer = try AVAudioPlayer( contentsOf: url )
soundPlayer.numberOfLoops = 0
soundPlayer.volume = 1
soundPlayer.play()
soundPlayers.append( soundPlayer )
// In an timer based loop or other callback such as display link, prune out players that are done, thus deallocating them
checkSfx: for player in soundPlayers {
if player.isPlaying { continue } else {
if let index = soundPlayers.index(of: player) {
soundPlayers.remove(at: index)
break checkSfx
}
}
}
Swift 5+
Compiling some of the previous answers, improving code style and reusability
I usually avoid loose strings throughout my projects and use, instead, custom protocols for objects that will hold those string properties.
I prefer this to the enum approach simply because enumerations tend to couple your project together quite quickly. Everytime you add a new case you must edit the same file with the enumeration, breaking somewhat the Open-Closed principle from SOLID and increasing chances for error.
In this particular case, you could have a protocol that defines sounds:
protocol Sound {
func getFileName() -> String
func getFileExtension() -> String
func getVolume() -> Float
func isLoop() -> Bool
}
extension Sound {
func getVolume() -> Float { 1 }
func isLoop() -> Bool { false }
}
And when you need a new sound you can simply create a new structure or class that implements this protocol (It will even be suggested on autocomplete if your IDE, just like Xcode, supports it, giving you similar benefits to those of the enumeration... and it works way better in medium to large multi framework projects).
(Usually I leave volume and other configurations with default implementations as they are less frequently customized).
For instance, you could have a coin drop sound:
struct CoinDropSound: Sound {
func getFileName() -> String { "coin_drop" }
func getFileExtension() -> String { "wav" }
}
Then, you could use a singleton SoundManager that would take care of managing playing audio files
import AVFAudio
final class SoundManager: NSObject, AVAudioPlayerDelegate {
static let shared = SoundManager()
private var audioPlayers: [URL: AVAudioPlayer] = [:]
private var duplicateAudioPlayers: [AVAudioPlayer] = []
private override init() {}
func play(sound: Sound) {
let fileName = sound.getFileName()
let fileExtension = sound.getFileExtension()
guard let url = Bundle.main.url(forResource: fileName, withExtension: fileExtension),
let player = getAudioPlayer(for: url) else { return }
player.volume = sound.getVolume()
player.numberOfLoops = numberOfLoops
player.prepareToPlay()
player.play()
}
private func getAudioPlayer(for url: URL) -> AVAudioPlayer? {
guard let player = audioPlayers[url] else {
let player = try? AVAudioPlayer(contentsOf: url)
audioPlayers[url] = player
return player
}
guard player.isPlaying else { return player }
guard let duplicatePlayer = try? AVAudioPlayer(contentsOf: url) else { return nil }
duplicatePlayer.delegate = self
duplicateAudioPlayers.append(duplicatePlayer)
return duplicatePlayer
}
func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool) {
duplicateAudioPlayers.removeAll { $0 == player }
}
}
Here I created a helper getAudioPlayer to be able to return early from code execution and make use of the guard let.
Using guard let more often and preferring less nested code can, most of the time, highly improve readability.
To use this SoundManager from anywhere in your project, simply access its shared instance and pass an object that conforms to Sound.
For example, given the previous CoinDropSound:
SoundManager.shared.play(sound: CoinDropSound())
You could maybe omit the sound parameter as it may improve readability
class SoundManager {
// ...
func play(_ sound: Sound) {
// ...
}
// ...
}
And then:
SoundManager.shared.play(CoinDropSound())
I'm using AVPlayer (I don't need to, but I wanna stream it and start playing as soon as possible) to play an m4a file (it's an iTunes audio preview). Only I only want it to play a part of that file.
I'm able to set a start time but not an end time.
Using a timer is not working because I'm using URL as a http address. I'm playing as it loads, without downloading the file.
I also saw solutions in Objective-C to use KVO to know when music starts playing but I'm thinking this is not the best approach since I'm using swift and also because of glitches that may occur so the song will not stop at the right moment.
You can add a addBoundaryTimeObserverForTimes to your AVPlayer as follow:
update: Xcode 8.3.2 • Swift 3.1
import UIKit
import AVFoundation
class ViewController: UIViewController {
var player: AVPlayer!
var observer: Any!
override func viewDidLoad() {
super.viewDidLoad()
guard let url = URL(string: "https://www.example.com/audio.mp3") else { return }
player = AVPlayer(url: url)
let boundary: TimeInterval = 30
let times = [NSValue(time: CMTimeMake(Int64(boundary), 1))]
observer = player.addBoundaryTimeObserver(forTimes: times, queue: nil) {
[weak self] time in
print("30s reached")
if let observer = self?.observer {
self?.player.removeTimeObserver(observer)
}
self?.player.pause()
}
player.play()
print("started loading")
}
}