I have an AVPlayer which is playing a HLS video stream. My user interface provides a row of buttons, one for each "chapter" in the video (the buttons are labeled "1", "2", "3"). The app downloads some meta-data from a server which contains the list of chapter cut-in points denoted in seconds. For example, one video is 12 minutes in length - the list of chapter cut-in points are 0, 58, 71, 230, 530, etc., etc.
When the user taps one of the "chapter buttons" the button handler code does this:
[self.avPlayer pause];
[self.avPlayer seekToTime: CMTimeMakeWithSeconds(seekTime, 600)
toleranceBefore: kCMTimeZero
toleranceAfter: kCMTimeZero
completionHandler: ^(BOOL finished)
{
[self.avPlayer play];
}];
Where "seekTime" is a local var which contains the cut-in point (as described above).
The problem is that the video does not always start at the correct point. Sometimes it does. But sometimes it is anywhere from a tenth of a second, to 2 seconds BEFORE the requested seekTime. It NEVER starts after the requested seekTime.
Here are some stats on the video encoding:
Encoder: handbrakeCLI
Codec: h.264
Frame rate: 24 (actually, 23.976 - same as how it was shot)
Video Bitrate: multiple bitrates (64/150/300/500/800/1200)
Audio Bitrate: 128k
Keyframes: 23.976 (1 per second)
I am using the Apple mediafilesegmenter tool, of course, and the variantplaylistcreator to generate the playlist.
The files are being served from an Amazon Cloud/S3 bucket.
One area which I remain unclear about is CMTimeMakeWithSeconds - I have tried several variations based on different articles/docs I have read. For example, in the above excerpt I am using:
CMTimeMakeWithSeconds(seekTime, 600)
I have also tried:
CMTimeMakeWithSeconds(seekTime, 1)
I can't tell which is correct, though BOTH seem to produce the same inconsistent results!
I have also tried:
CMTimeMakeWithSeconds(seekTime, 23.967)
Some articles claim this works like a numerator/denomenator, so n/1 should be correct where 'n' is number of seconds (as in CMTimeMakeWithseconds(n, 1)). But, the code was originally created by a different programmer (who is gone now) and he used the 600 number for the preferredTimeScale (ie. CMTimeMakeWithseconds(n, 600)).
Can anyone offer any clues as to what I am doing wrong, or even if the kind of accuracy I am trying to achieve is even possible?
And in case someone is tempted to offer "alternative" solutions, we are already considering breaking the video up into separate streams, one per chapter, but we do not believe that will give us the same performance in the sense that changing chapters will take longer as a new AVPlayerItem will have to be created and loaded, etc., etc., etc. So if you think this is the only solution that will work (and we do expect this will achieve the result we want - ie. each chapter WILL start exactly where we want it to) feel free to say so.
Thanks in advance!
int32_t timeScale = self.player.currentItem.asset.duration.timescale;
CMTime time = CMTimeMakeWithSeconds(77.000000, timeScale);
[self.player seekToTime:time toleranceBefore:kCMTimeZero toleranceAfter:kCMTimeZero];
I had a problem with 'seekToTime'. I solved my problem with this code. 'timescale' is trick for this problem.
Swift version:
let playerTimescale = self.player.currentItem?.asset.duration.timescale ?? 1
let time = CMTime(seconds: 77.000000, preferredTimescale: playerTimescale)
self.player.seek(to: time, toleranceBefore: kCMTimeZero, toleranceAfter: kCMTimeZero) { (finished) in /* Add your completion code here */
}
My suggestion:
1) Don't use [avplayer seekToTime: toleranceBefore: toleranceAfter: ], this will delay your seek time 4-5 seconds.
2) HLS video cut to 10 seconds per segment. Your chapter start postion should fit the value which is multipes of 10. As the segment starts with I frame, on this way, you can get quick seek time and accurate time.
please use function like [player seekToTime:CMTimeMakeWithSeconds(seekTime,1)] .
Because your tolerance value kCMTimeZero will take more time to seek.Instead of using tolerance value of kCMTimeZero you can use kCMTimeIndefinite which is equivalent the function that i specified earlier.
Put this code it may be resolve your problem.
let targetTime = CMTimeMakeWithSeconds(videoLastDuration, 1) // videoLastDuration hold the previous video state.
self.playerController.player?.currentItem?.seekToTime(targetTime, toleranceBefore: kCMTimeZero, toleranceAfter: kCMTimeZero)
Swift5
let seconds = 45.0
let time = CMTimeMake(value: seconds, timescale: 1)
player?.seek(to: time, toleranceBefore: CMTime.zero, toleranceAfter: CMTime.zero)
Related
i'm currently working a musician app. In my app notes should be played with a specific duration. I don't get into detail when the notes are played. Basically there is a ui view (a vertical line) which is moving and when this hits my other ui views (rectangle) it should be played a note. Important here: the note should be played until the line is not hitting the rectangle anymore.
The note playing is no problem but I don't find any duration. Also it should be possible to play the same note multiple times with a delay.
So I tried to make this work with AudioKit cause it's seems like the best greatest solution for audio. But it has so much stuff. I took a look into their examples and found this:
let bundlePath = Bundle.main.bundlePath
let soundPath = ("\(bundlePath)/sounds")
let akSampler = AKAppleSampler()
let mixer = AKMixer(akSampler)
try! akSampler.loadSoundFont(soundPath, preset: 0, bank: 0)
mixer.start()
AudioKit.output = mixer
do {
_ = try AudioKit.engine.start()
} catch {
print("AudioKit wouldn't start!")
}
do {
try akSampler.play(noteNumber: myNote.rawValue, velocity: 100, channel: 1)
} catch let e{
print(e)
}
Unfortunately I can't pass any duration and when I call akSampler.stop(noteNumber: myNote.rawValue) it also stops the other notes with the same type.
I tried to find a solution with AVFoundation like so:
engine = AVAudioEngine()
sampler = AVAudioUnitSampler()
engine.attach(sampler)
engine.connect(sampler, to: engine.mainMixerNode, format: nil)
guard let bankURL = Bundle.main.url(forResource: "sounds", withExtension: "SF2") else {
print("could not load sound font")
return
}
... init engine
sampler.startNote(60, withVelocity: 64, onChannel: 0)
But same result. Also the same case that I can't pass any duration.
I also digged into MIDISequencer's but it seems that they generating a sequence which I can play but this does not fit on my problem.
Does someone has a solution here?
The laziest solution would be to just schedule a stop with asyncAfter when you trigger the note, e.g.,
func makeNote(note: MIDINoteNumber, dur: Double) {
sampler.play(noteNumber: note, velocity: 100, channel: 0)
DispatchQueue.main.asyncAfter(deadline: .now() + dur) {
self.sampler.stop(noteNumber: note)
}
}
A better solution would probably use either AKSequencer or AKAppleSequencer. Both allow you to create sequences on the fly by adding individual notes with a specified duration (in musical time, i.e., number of beats). AKSequencer is considerably more accurate, but AKAppleSequencer has more readily available code examples on the web. A little confusingly, the current AKAppleSequencer used to also be called AKSequencer, but their interfaces are sufficiently different that a quick look at the docs for the two classes will tell you which you're looking at.
Your question is asking about how to schedule MIDI events which is precisely what these classes are designed to do. You haven't really given a clear reason why generating a sequence doesn't fit your problem.
I have an AVPlayer that I want to begin playing at the specific time of 11.593 seconds. I have this number in milliseconds retrieved from a URL string, converted to a Double, then to a CMTime like this:
http://www.mywebsite.com/video?s=11593
Extract the 11593 as a String -> convert to Double 11593.0.
Then I convert to a CMTime:
let time = CMTime(seconds: milliseconds,
preferredTimescale: CMTimeScale(NSEC_PER_MSEC))
Then I tell the AVPlayer to seek:
player.seek(to: time, toleranceBefore: .zero, toleranceAfter: .zero)
But the player always seeks to 25.88 seconds. Why??
Your entire use of CMTime(seconds:preferredTimescale:) is wrong. The first argument should be a number of seconds (hence the name, seconds:), not a number of milliseconds; and the second argument should be a reasonable timescale, such as 600.
First, I called a AKMIDISampler to play an audio file, and then assigned it to AKSequencer. The 'midi' file I used is just a 2 bars long, C3 note, single track midi file, exactly as long as the audio file I wanted to play. But, in calling AKAudioFile, I wanted to choose mp3 file randomly. I temporarily made 1.mp3, 2.mp3 and 3.mp3 as below.
let track = AKMIDISampler()
let sequencer = AKSequencer(filename: "midi")
try? track.loadAudioFile(AKAudioFile(readFileName: String(arc4random_uniform(3)+1) + ".mp3"))
sequencer.tracks[0].setMIDIOutput(track.midiIn)
// Tempo track I had to made to remove sine wave
sequencer.tracks[1].setMIDIOutput(track.midiIn)
And did some sequencer settings,
sequencer.setTempo(128.0)
sequencer.setLength(AKDuration(beats: 8))
sequencer.setLoopInfo(AKDuration(beats: 8), numberOfLoops: 4)
sequencer.preroll()
and assigned AKMIDISampler to AudioKit.output, then did sequencer.play().
The sequencer playback was successful! It loaded among three mp3 files randomly, and played 8 beats (2 bars), looped for 4 times exactly.
But my goal is to load random MP3 files every time the loop repeats. It seems like the sequencer only plays the first assigned mp3 file when looping. I am struggling finding a solution to this.
Perhaps I could use "AKCallbackInstrument"? Since I play audiofile through a midi note in this case, I might reset "loadAudioFile" whenever the midi note is off? In that way I might loop the sequencer and play random a audio file in every loop. This is just an idea, but for me now it is hard to write it properly. I hope I am on the right track. It would be great if I could get an advice here. <3
You're definitely on the right track - you can easily get random audio files to loop at a fixed interval with AKSequencer + AKCallbackInstrument. But I wouldn't worry about trying to reload on the NoteOff message.
I would first load each mp3 into a separate player (e.g., AKAppleSampler) in an array (e.g.,you could call it players) and create a method that will trigger one of these players at random:
func playRandom() {
let playerIndex = Int(arc4random_uniform(UInt32(players.count)))
try? players[playerIndex].play()
}
When you create your sequencer, add a track and assign it to an AKCallbackInstrument. The callback function for this AKCallbackInstrument will call playRandom when it receives a noteOn message.
seq = AKSequencer()
track = seq.newTrack()!
callbackInst = AKCallbackInstrument()
track.setMIDIOutput(callbackInst.midiIn)
callbackInst.callback = { status, note, vel in
guard status == .noteOn else { return }
self.playRandom()
}
It isn't necessary to load the sequencer with a MIDI file. You could just add the triggering MIDI event directly to the track.
track.add(noteNumber: 48, // i.e., C3
velocity: 127,
position: AKDuration(beats: 0), // noteOn message here
duration: AKDuration(beats: 8), // noteOff 8 beats later
channel: 0)
Your problem with the sine wave is probably being caused by an extra track (probably tempo track) in the MIDI file which you created which hasn't been assigned an output. You can avoid the problem altogether by adding the MIDI events directly.
In principle, you could use the callback to check for noteOff events and trigger code from the noteOff, but I wouldn't recommend it in your case. There is no good reason to re-use a single player for multiple audiofiles. Loading the file is where you are most likely to create an error. What happens if your file hasn't finished playing and you try to load another one? The resources needed to keep multiple players in memory is pretty trivial - if you're going to play the same file more than once, it is cleaner and safer to load it once and keep the player in memory.
It was very helpful, c_booth! Thanks to you, I made a huge progress today. Here's what I've written based on your advise. First, I made an array of AKPlayers include 6 mp3 files. They're assigned to AKMixer, and then I called sequencer and callback instrument. I made a track and a note on the sequencer, which calls 'playRandom' function on every noteOn :
let players: [AKPlayer] = {
do {
let filenames = ["a1.mp3", "a2.mp3", "a3.mp3", "b1.mp3", "b2.mp3", "b3.mp3"]
return try filenames.map { AKPlayer(audioFile: try AKAudioFile(readFileName: $0)) }
} catch {
fatalError()
}
}()
func playRandom() {
let playerIndex = Int(arc4random_uniform(UInt32(players.count)))
players[playerIndex].play()
}
func addTracks() {
let track = sequencer.newTrack()!
track.add(noteNumber: 48, velocity: 127, position: AKDuration(beats: 0), duration: AKDuration(beats: 16), channel: 0)
track.setMIDIOutput(callbackInst.midiIn)
callbackInst.callback = { status, note, vel in
guard status == .noteOn else { return }
self.playRandom()
}
}
func sequencerSettings() {
sequencer.setTempo(128.0)
sequencer.setLength(AKDuration(beats: 16))
sequencer.setLoopInfo(AKDuration(beats: 16), numberOfLoops: 4)
sequencer.preroll()
}
func makeConnections() {
players.forEach { $0 >>> mixer }
}
func startAudioEngine() {
AudioKit.output = mixer
do {
try AudioKit.start()
} catch {
print(error)
fatalError()
}
}
func startSequencer() {
sequencer.play()
}
This worked great. It randomly selects one from 6 mp3 files (they are all the same length, 128bpm and 16 beats). What I found strange here is, though, the first playback plays two audio files at once. It works fine after the second loop. I changed the numberOfLoop setting, enableLooping(), etc but still the same - plays two files on the first playback. The trackcount is still 1, and I only called one AKPlayer as you could see. Is there anything I can do about this?
Also, ultimately, I'd like to call hundreds of mp3 files on the array, as what I'm trying to make is a sort of DJing app (something like Ableton Live preset). Do you think it's a good idea to use AKPlayer, assuming this code will load mp3 files from the cloud and stream it to the user? Much appreciated. <3
I have a simple audio file in .wav format (the audio file is cut perfectly to loop). I've tried different methods to loop it. My first attempt was simply using AVPlayer and NSNotification to detect when audioItem ended to seek time at zero and play again. However, there was clearly a gap.
I've been looking at different solutions online, and found people using AVQueuePlayer to do a switching:
Looping AVPlayer seamlessly
However, when implemented, this still produces a gap.
Here's my current notification code:
weak var weakSelf = self
NSNotificationCenter.defaultCenter().addObserverForName(AVPlayerItemDidPlayToEndTimeNotification, object: nil, queue: nil, usingBlock: {(note: NSNotification) -> Void in
if weakSelf?.currentQueuePlayer.currentItem == weakSelf?.currentAudioItemOne {
weakSelf?.currentQueuePlayer.insertItem((weakSelf?.currentAudioItemTwo)!, afterItem: nil)
weakSelf?.currentAudioItemTwo.seekToTime(kCMTimeZero)
} else {
weakSelf?.currentQueuePlayer.insertItem((weakSelf?.currentAudioItemOne)!, afterItem: nil)
weakSelf?.currentAudioItemOne.seekToTime(kCMTimeZero)
}
})
Here's my code to set up the current QueuePlayer.
let audioPlayerItem = AVPlayerItem(URL: url)
currentAudioItemOne = audioPlayerItem
currentAudioItemTwo = audioPlayerItem
currentQueuePlayer = AVQueuePlayer()
currentQueuePlayer.insertItem(currentAudioItemOne, afterItem: nil)
currentQueuePlayer.play()
I've been working at this problem for several days now. Any leads or new things to try would be appreciated. The only thing I haven't tried so far is lower quality audio files. These .wav files are all over 1mb, and had be suspecting that the file size could be affecting the seamless looping.
EDIT:
Using AVPlayerLooper to create the 'Treadmill' effect:
let url = URL(fileURLWithPath: path)
let audioPlayerItem = AVPlayerItem(url: url)
currentAudioItemOne = audioPlayerItem
currentQueuePlayer = AVQueuePlayer()
currentAudioPlayerLayer = AVPlayerLayer(player: currentQueuePlayer)
currentAudioLooper = AVPlayerLooper(player: currentQueuePlayer, templateItem: currentAudioItemOne)
currentQueuePlayer.play()
EDIT 2:
afinfo on one of my wav files:
Num Tracks: 1
----
Data format: 2 ch, 44100 Hz, 'lpcm' (0x0000000C) 16-bit little-endian signed integer
no channel layout.
estimated duration: 11.302336 sec
audio bytes: 1993732
audio packets: 498433
bit rate: 1411200 bits per second
packet size upper bound: 4
maximum packet size: 4
audio data file offset: 44
not optimized
source bit depth: I16
----
You are inserting the item too late in your current solution. You need to queue up more than one initial item, so there's always a primed AVPlayerItem ready to go.
This is called the AVPlayerQueue "treadmill pattern" as better described in this WWDC 2016 session. If you're targeting iOS 10, you can use new AVPlayerLooper class which does it for you (also described in the same link). Apple has also provided a sample project which provides an example of both strategies.
Lower level solutions include queuing up the audio buffers to an AVAudioEngine instance or using an AudioQueue or mashing the buffers together yourself with an AudioUnit.
I am playing a song using AVAudioPlayerNode and I am trying to control its time using a UISlider but I can't figure it out how to seek the time using AVAUdioEngine.
After MUCH trial and error I think I have finally figured this out.
First you need to calculate the sample rate of your file. To do this get the last render time of your AudioNode:
var nodetime: AVAudioTime = self.playerNode.lastRenderTime
var playerTime: AVAudioTime = self.playerNode.playerTimeForNodeTime(nodetime)
var sampleRate = playerTime.sampleRate
Then, multiply your sample rate by the new time in seconds. This will give you the exact frame of the song at which you want to start the player:
var newsampletime = AVAudioFramePosition(sampleRate * Double(Slider.value))
Next, you are going to want to calculate the amount of frames there are left in the audio file:
var length = Float(songDuration!) - Slider.value
var framestoplay = AVAudioFrameCount(Float(playerTime.sampleRate) * length)
Finally, stop your node, schedule the new segment of audio, and start your node again!
playerNode.stop()
if framestoplay > 1000 {
playerNode.scheduleSegment(audioFile, startingFrame: newsampletime, frameCount: framestoplay, atTime: nil,completionHandler: nil)
}
playerNode.play()
If you need further explanation I wrote a short tutorial here: http://swiftexplained.com/?p=9
For future readers, probably better to get the sample rate as :
playerNode.outputFormat(forBus: 0).sampleRate
Also take care when converting to AVAudioFramePosition, as it is an integer, while sample rate is a double. Without rounding the result, you may end up with undesirable results.
P.S. The above answer assumes that the file you are playing has the same sample rate as the output format of the player, which may or may not be true.