I have an AVPlayer which is streaming a live HLS stream.
When the user multitasks the app, I see the play rate drop to 0.0 (paused), when the user comes back it return to 1.0(play), but starts playing from the point it was paused.
What is the best way to force the player back to live without restarting the stream completely? Is there a seekToTime method that handles a closest to live time parameter?
Thanks!
I use:
double time = MAXFLOAT;
[player seekToTime: CMTimeMakeWithSeconds(time, NSEC_PER_SEC)];
Works well in my app.
Assuming player is an AVPlayer instance:
CMTimeRange seekableRange = [player.currentItem.seekableTimeRanges.lastObject CMTimeRangeValue];
CGFloat seekableStart = CMTimeGetSeconds(seekableRange.start);
CGFloat seekableDuration = CMTimeGetSeconds(seekableRange.duration);
CGFloat livePosition = seekableStart + seekableDuration;
[player seekToTime:CMTimeMake(livePosition, 1)];
Swift version of Igor Kulagin answer:
player.seek(to: kCMTimePositiveInfinity)
player.play()
Works perfectly in any condition. Other solutions gave me NaN error calculating livePosition value, or {INVALID} error working directly with CMTime.
Swift 3.0 Version
public func resumeLive() {
guard let livePosition = player.currentItem?.seekableTimeRanges.last as? CMTimeRange else {
return
}
player.seek(to:CMTimeRangeGetEnd(livePosition))
}
Swift version of Karim Mourra's answer:
let seekableRanges = player.currentItem!.seekableTimeRanges
guard seekableRanges.count > 0 else {
return
}
let range = seekableRanges.last!.CMTimeRangeValue
let livePosition = range.start + range.duration
let minus = CMTimeMakeWithSeconds(Float64(timeOffset), Int32(NSEC_PER_SEC))
let time = livePosition - minus
player.seekToTime(time)
Swift 4 version:
if let seekableRange = player.currentItem?.seekableTimeRanges.last?.timeRangeValue {
let seekableStart = seekableRange.start
let seekableDuration = seekableRange.duration
let livePosition = seekableStart + seekableDuration
player.seek(to: livePosition, completionHandler: { [weak self] _ in
self?.player.play()
})
}
No need to convert to floating point if you use Apple's CMTimeRange manipulation functions:
NSValue *value = player.currentItem.seekableTimeRanges.lastObject;
if (value) {
CMTimeRange seekableRange = [value CMTimeRangeValue];
CMTime latestTime = CMTimeRangeGetEnd(seekableRange);
[player seekToTime:latestTime];
} else {
// there are no seekable time ranges
}
Please see #Fabian's comment below.
Related
// My code is below
do{
file = try AKAudioFile(readFileName: "Sound1.mp3", baseDir: .resources)
// file = try AKAudioFile(forReading: SingletonClass.sharedInstance.recordedURLs[SingletonClass.sharedInstance.recordedURL]!)
// AKSettings.defaultToSpeaker = true
}
catch {
}
do {
player = try AKAudioPlayer(file : file)
}
catch {
}
let lfoAmplitude = 1_000
let lfoRate = 1.0 / 3.428
_ = 0.9
//filter section effect below
filterSectionEffect = AKOperationEffect(tracker) { input, _ in
let lfo = AKOperation.sineWave(frequency: lfoRate, amplitude: lfoAmplitude)
return input.moogLadderFilter(cutoffFrequency: lfo + cutoffFrequency,
resonance: resonance)
}
Audiokit.output = filterSectionEffect
Audiokit.start()
And whenever I play the audio using a button with code player.play , the audio gets played properly. And if I connect the headphones, it gets played properly as well but as soon as I disconnect the headphones, I see the error:
It happens in same way for both wired as well as bluetooth headphones.
My app with stuck because of this issue only that too happens only with AKOperationEffect. Any help would be appreciated.
The comment from Kunal Verma that this is fixed is correct, but just for completeness here is the commit that fixed it.
https://github.com/AudioKit/AudioKit/commit/ffac4acbe93553764f6095011e9bf5d71fdc88c2
I am working on an application that plays back video and allows the user to scrub forwards and backwards in the video. The scrubbing has to happen smoothly, so we always re-write the video with SDAVAssetExportSession with the video compression property AVVideoMaxKeyFrameIntervalKey:#1 so that each frame will be a keyframe and allow smooth reverse scrubbing. This works great and provides smooth playback. The application uses video from a variety of sources and can be recorded on android or iOS devices and even downloaded from the web and added to the application, so we end up with quite different encodings, some of which are already suited for scrubbing (each frame is a keyframe). Is there a way to detect the keyframe interval of a video file so I can avoid needless video processing? I have been through much of AVFoundation's docs and don't see an obvious way to get this information. Thanks for any help on this.
If you can quickly parse the file without decoding the images by creating an AVAssetReaderTrackOutput with nil outputSettings. The frame sample buffers you encounter have an attachment array containing a dictionary with useful information, include whether the frame depends on other frames, or whether other frames depend on it. I would interpret that former as indicating a keyframe, although it gives me some low number (4% keyframes in one file?). Anyway, the code:
let asset = AVAsset(url: inputUrl)
let reader = try! AVAssetReader(asset: asset)
let videoTrack = asset.tracks(withMediaType: AVMediaTypeVideo)[0]
let trackReaderOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: nil)
reader.add(trackReaderOutput)
reader.startReading()
var numFrames = 0
var keyFrames = 0
while true {
if let sampleBuffer = trackReaderOutput.copyNextSampleBuffer() {
// NB: not every sample buffer corresponds to a frame!
if CMSampleBufferGetNumSamples(sampleBuffer) > 0 {
numFrames += 1
if let attachmentArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, false) as? NSArray {
let attachment = attachmentArray[0] as! NSDictionary
// print("attach on frame \(frame): \(attachment)")
if let depends = attachment[kCMSampleAttachmentKey_DependsOnOthers] as? NSNumber {
if !depends.boolValue {
keyFrames += 1
}
}
}
}
} else {
break
}
}
print("\(keyFrames) on \(numFrames)")
N.B. This only works for local file assets.
p.s. you don't say how you're scrubbing or playing. An AVPlayerViewController and an AVPlayer?
Here is the Objective C version of the same answer. After implementing this and using it, Videos that should have all keyframes are returning about 96% keyframes from this code. I'm not sure why, so I am using that number as a determining factor even though I would like it to be more accurate. I am also only looking through the first 600 frames or the end of the video (whichever comes first) since I don't need to read through a whole 20 minute video to make this determination.
+ (BOOL)videoNeedsProcessingForSlomo:(NSURL*)fileUrl {
BOOL needsProcessing = YES;
AVAsset* anAsset = [AVAsset assetWithURL:fileUrl];
NSError *error;
AVAssetReader *assetReader = [AVAssetReader assetReaderWithAsset:anAsset error:&error];
if (error) {
DLog(#"Error:%#", error.localizedDescription);
return YES;
}
AVAssetTrack *videoTrack = [[anAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
AVAssetReaderTrackOutput *trackOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:videoTrack outputSettings:nil];
[assetReader addOutput:trackOutput];
[assetReader startReading];
float numFrames = 0;
float keyFrames = 0;
while (numFrames < 600) { // If the video is long - only parse through 20 seconds worth.
CMSampleBufferRef sampleBuffer = [trackOutput copyNextSampleBuffer];
if (sampleBuffer) {
// NB: not every sample buffer corresponds to a frame!
if (CMSampleBufferGetNumSamples(sampleBuffer) > 0) {
numFrames += 1;
NSArray *attachmentArray = ((NSArray*)CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, false));
if (attachmentArray) {
NSDictionary *attachment = attachmentArray[0];
NSNumber *depends = attachment[(__bridge NSNumber*)kCMSampleAttachmentKey_DependsOnOthers];
if (depends) {
if (depends.boolValue) {
keyFrames += 1;
}
}
}
}
}
else {
break;
}
}
needsProcessing = keyFrames / numFrames < 0.95f; // If more than 95% of the frames are keyframes - don't decompress.
return needsProcessing;
}
Using kCMSampleAttachmentKey_DependsOnOthers was giving me 0 key frames in some cases, when ffprobe would return key frames.
To get the same number of key frames as ffprobe shows, I used:
if attachment[CMSampleBuffer.PerSampleAttachmentsDictionary.Key.notSync] == nil {
keyFrames += 1
}
In the CoreMedia header it says:
/// Boolean (absence of this key implies Sync)
public static let notSync: CMSampleBuffer.PerSampleAttachmentsDictionary.Key
for dependsOnOthers key it says:
/// `true` (e.g., non-I-frame), `false` (e.g. I-frame), or absent if
/// unknown
public static let dependsOnOthers: CMSampleBuffer.PerSampleAttachmentsDictionary.Key
I’m looking for a way to maintain a seamless audio track while flipping between front and back camera. Many apps in the market can do this, one example is SnapChat…
Solutions should use AVCaptureSession and AVAssetWriter. Also it should explicitly not use AVMutableComposition since there is a bug between AVMutableComposition and AVCaptureSession ATM. Also, I can't afford post processing time.
Currently when I change the video input the audio recording skips and becomes out of sync.
I’m including the code that could be relevant.
Flip Camera
-(void) updateCameraDirection:(CamDirection)vCameraDirection {
if(session) {
AVCaptureDeviceInput* currentInput;
AVCaptureDeviceInput* newInput;
BOOL videoMirrored = NO;
switch (vCameraDirection) {
case CamDirection_Front:
currentInput = input_Back;
newInput = input_Front;
videoMirrored = NO;
break;
case CamDirection_Back:
currentInput = input_Front;
newInput = input_Back;
videoMirrored = YES;
break;
default:
break;
}
[session beginConfiguration];
//disconnect old input
[session removeInput:currentInput];
//connect new input
[session addInput:newInput];
//get new data connection and config
dataOutputVideoConnection = [dataOutputVideo connectionWithMediaType:AVMediaTypeVideo];
dataOutputVideoConnection.videoOrientation = AVCaptureVideoOrientationPortrait;
dataOutputVideoConnection.videoMirrored = videoMirrored;
//finish
[session commitConfiguration];
}
}
Sample Buffer
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
//not active
if(!recordingVideo)
return;
//start session if not started
if(!startedSession) {
startedSession = YES;
[assetWriter startSessionAtSourceTime:CMSampleBufferGetPresentationTimeStamp(sampleBuffer)];
}
//Process sample buffers
if (connection == dataOutputAudioConnection) {
if([assetWriterInputAudio isReadyForMoreMediaData]) {
BOOL success = [assetWriterInputAudio appendSampleBuffer:sampleBuffer];
//…
}
} else if (connection == dataOutputVideoConnection) {
if([assetWriterInputVideo isReadyForMoreMediaData]) {
BOOL success = [assetWriterInputVideo appendSampleBuffer:sampleBuffer];
//…
}
}
}
Perhaps adjust audio sample timeStamp?
Hey I was facing the same issue and discovered that after switching cameras the next frame was pushed far out of place. This seemed to shift every frame after that thus causing the the video and audio to be out of sync. My solution was to shift every misplaced frame to it's correct position after switching cameras.
Sorry my answer will be in Swift 4.2
You'll have to use AVAssetWriterInputPixelBufferAdaptor in order to append the sample buffers at a specify presentation timestamp.
previousPresentationTimeStamp is the presentation timestamp of the previous frame and currentPresentationTimestamp is as you guessed the presentation timestamp of the current. maxFrameDistance worked every well when testing but you can change this to your liking.
let currentFramePosition = (Double(self.frameRate) * Double(currentPresentationTimestamp.value)) / Double(currentPresentationTimestamp.timescale)
let previousFramePosition = (Double(self.frameRate) * Double(previousPresentationTimeStamp.value)) / Double(previousPresentationTimeStamp.timescale)
var presentationTimeStamp = currentPresentationTimestamp
let maxFrameDistance = 1.1
let frameDistance = currentFramePosition - previousFramePosition
if frameDistance > maxFrameDistance {
let expectedFramePosition = previousFramePosition + 1.0
//print("[mwCamera]: Frame at incorrect position moving from \(currentFramePosition) to \(expectedFramePosition)")
let newFramePosition = ((expectedFramePosition) * Double(currentPresentationTimestamp.timescale)) / Double(self.frameRate)
let newPresentationTimeStamp = CMTime.init(value: CMTimeValue(newFramePosition), timescale: currentPresentationTimestamp.timescale)
presentationTimeStamp = newPresentationTimeStamp
}
let success = assetWriterInputPixelBufferAdator.append(pixelBuffer, withPresentationTime: presentationTimeStamp)
if !success, let error = assetWriter.error {
fatalError(error.localizedDescription)
}
Also please note - This worked because I kept the frame rate consistent, so make sure that you have total control of the capture device's frame rate throughout this process.
I have a repo using this logic here
I did manage to find an intermediate solution for the sync problem I found on the Woody Jean-louis solution using is repo.
The results are similar to what instagram does but it seems to work a little bit better. Basically what I do is to prevent the assetWriterAudioInput to append new samples when switching cameras. There is no way to know exactly when this happens so I figured out that before and after the switch the captureOutput method was sending video samples every 0.02 seconds +- (max 0.04 seconds).
Knowing this I created a self.lastVideoSampleDate that is updated every time a video sample is appended to assetWriterInputPixelBufferAdator and I only allow the audio sample to be appended to assetWriterAudioInput is that date is lower than 0.05.
if let assetWriterAudioInput = self.assetWriterAudioInput,
output == self.audioOutput, assetWriterAudioInput.isReadyForMoreMediaData {
let since = Date().timeIntervalSince(self.lastVideoSampleDate)
if since < 0.05 {
let success = assetWriterAudioInput.append(sampleBuffer)
if !success, let error = assetWriter.error {
print(error)
fatalError(error.localizedDescription)
}
}
}
let success = assetWriterInputPixelBufferAdator.append(pixelBuffer, withPresentationTime: presentationTimeStamp)
if !success, let error = assetWriter.error {
print(error)
fatalError(error.localizedDescription)
}
self.lastVideoSampleDate = Date()
The most 'stable way' to fix this problem - is to 'pause' recording when switching sources.
But also you can 'fill the gap' with blank video and silent audio frames.
This is what I have implemented in my project.
So, create boolean to block ability to append new CMSampleBuffer's while switching cameras/microphones and reset it after some delay:
let idleTime = 1.0
self.recordingPaused = true
DispatchQueue.main.asyncAfter(deadline: .now() + idleTime) {
self.recordingPaused = false
}
writeAllIdleFrames()
In writeAllIdleFrames method you need to calculate how many frames you need to write:
func writeAllIdleFrames() {
let framesPerSecond = 1.0 / self.videoConfig.fps
let samplesPerSecond = 1024 / self.audioConfig.sampleRate
let videoFramesCount = Int(ceil(self.switchInputDelay / framesPerSecond))
let audioFramesCount = Int(ceil(self.switchInputDelay / samplesPerSecond))
for index in 0..<max(videoFramesCount, audioFramesCount) {
// creation synthetic buffers
recordingQueue.async {
if index < videoFramesCount {
let pts = self.nextVideoPTS()
self.writeBlankVideo(pts: pts)
}
if index < audioFramesCount {
let pts = self.nextAudioPTS()
self.writeSilentAudio(pts: pts)
}
}
}
}
How to calculate next PTS?
func nextVideoPTS() -> CMTime {
guard var pts = self.lastVideoRawPTS else { return CMTime.invalid }
let framesPerSecond = 1.0 / self.videoConfig.fps
let delta = CMTime(value: Int64(framesPerSecond * Double(pts.timescale)),
timescale: pts.timescale, flags: pts.flags, epoch: pts.epoch)
pts = CMTimeAdd(pts, delta)
return pts
}
Tell me, if you also need code that creates blank/silent video/audio buffers :)
I'm really out of ideas so I'll have to ask you guys again...
I'm building an iPhone application which uses three instances of AVPlayer. They all play at the same time and it's very important that they do so. I used to run this code:
CMClockRef syncTime = CMClockGetHostTimeClock();
CMTime hostTime = CMClockGetTime(hostTime);
[self.playerOne setRate:1.0f time:kCMTimeInvalid atHostTime:hostTime];
[self.playerTwo setRate:1.0f time:kCMTimeInvalid atHostTime:hostTime];
[self.playerThree setRate:1.0f time:kCMTimeInvalid atHostTime:hostTime];
which worked perfectly. But a few days ago it just stopped working, the three players are delayed by about 300-400ms (which is way to much, everything under 100ms would be okay). Two of these AVPlayer have some Audio processing, which takes some time more than the "normal" AVPlayer, but it used to work before and the currentTime property tells me, that these players are delayed, so the syncing seems to fail.
I have no idea why it stopped working, I didn't really changed something, but I'm using an observer where i can ask the self.playerX.currentTime property, which gives me a delay of about .3-.4 seconds... I already tried to resync the players if delay>.1f but the delay is still there. So I think the audio processing of player1 and 2 can't be responsable for the delay, as the currentTime property does know they are delayed (i hope you know what I mean). Maybe someone of you guys know why I'm having such a horrible delay, or is able to provide me another idea.
Thanks in advance!
So, I found the solution. I forgot to [self.playerX prerollAtRate:]. I thought if the observer is AVPlayerReadyToPlay it means, that the player is "really" ready. In fact, it does not. After AVPlayer is readyToPlay, it has to be pre rolled. Once that is done you can sync your placer. The delay is now somewhere at 0.000006 seconds.
Full func to sync avplayer's across multiple iOS devices
private func startTribePlayer() {
let dateFormatterGet = DateFormatter()
dateFormatterGet.dateFormat = "yyyy-MM-dd"
guard let refDate = dateFormatterGet.date(from: "2019-01-01") else { return }
let tsRef = Date().timeIntervalSince(refDate)
//currentDuration is avplayeritem.duration().seconds
let remainder = tsRef.truncatingRemainder(dividingBy: currentDuration)
let ratio = remainder / currentDuration
let seekTime = ratio * currentDuration
let bufferTime = 0.5
let bufferSeekTime = seekTime + bufferTime
let mulFactor = 10000.0
let timeScale = CMTimeScale(mulFactor)
let seekCMTime = CMTime(value: CMTimeValue(CGFloat(bufferSeekTime * mulFactor)), timescale: timeScale)
let syncTime = CMClockGetHostTimeClock()
let hostTime = CMClockGetTime(syncTime)
tribeMusicPlayer?.seek(to: seekCMTime, toleranceBefore: .zero, toleranceAfter: .zero, completionHandler: { [weak self] (successSeek) in
guard let tvc = self, tvc.tribeMusicPlayer?.currentItem?.status == .readyToPlay else { return }
tvc.tribeMusicPlayer?.preroll(atRate: 1.0, completionHandler: { [tvc] (successPreroll) in
tvc.tribePlayerDidPlay = true
tvc.tribeMusicPlayer?.setRate(1.0, time: seekCMTime, atHostTime: hostTime)
})
})
}
NSDictionary* fileAttributes =
[[NSFileManager defaultManager] attributesOfItemAtPath:filename
error:nil]
From the file attribute keys, you can get the date, size, etc. But how do you get the duration?
In the 'File Attribute Keys' of the NSFileManager class reference you can see that there is no key to use that will return the duration of a song. All the information that the NSFileManager instance gets about a file is to do with the properties of the actual file itself within the operating system, such as its file-size. The NSFileManager doesn't actually interpret the file.
In order to get the duration of the file, you need to use a class that knows how to interpret the file. The AVFoundation framework provides the exact class you need, AVAsset. You can instantiate an instance of this abstract class using the concrete subclass AVURLAsset, and then provide it an NSURL which points to the audio file you wish to get the duration. You can then get the duration from the AVAsset instance by querying its duration property.
For example:
AVURLAsset* audioAsset = [AVURLAsset URLAssetWithURL:audioFileURL options:nil];
CMTime audioDuration = audioAsset.duration;
float audioDurationSeconds = CMTimeGetSeconds(audioDuration);
Note that AVFoundation is designed as a heavily asynchronous framework in order to improve performance and the overall user experience. Even performing simple tasks such as querying a media file's duration can potentially take a long period of time and can cause your application to hang. You should use the AVAsynchronousKeyValueLoading protocol to asynchronously load the duration of the song, and then update your UI in a completion handler block. You should take a look at the 'Block Programming Guide' as well as the WWDC2010 video titled, 'Discovering AV Foundation', which is available free at https://developer.apple.com/videos/wwdc/2010.
For anyone still looking for this.
Based on the answer, the code for Swift 4 (including the async loading of values taken from Apple's documentation):
let audioAsset = AVURLAsset.init(url: yourURL, options: nil)
audioAsset.loadValuesAsynchronously(forKeys: ["duration"]) {
var error: NSError? = nil
let status = audioAsset.statusOfValue(forKey: "duration", error: &error)
switch status {
case .loaded: // Sucessfully loaded. Continue processing.
let duration = audioAsset.duration
let durationInSeconds = CMTimeGetSeconds(duration)
print(Int(durationInSeconds))
break
case .failed: break // Handle error
case .cancelled: break // Terminate processing
default: break // Handle all other cases
}
}
You can achieve the same in Swift using :
let audioAsset = AVURLAsset.init(url: audioFileURL, options: nil)
let duration = audioAsset.duration
let durationInSeconds = CMTimeGetSeconds(duration)
For completeness - There is another way to get the duration for a mp3 file:
NSURL * pathToMp3File = ...
NSError *error = nil;
AVAudioPlayer* avAudioPlayer = [[AVAudioPlayer alloc]initWithContentsOfURL:pathToMp3File error:&error];
double duration = avAudioPlayer.duration;
avAudioPlayer = nil;
I have used this with no discernible delay.
Swift 5.0 + iOS 13: This is the only way it worked for me (#John Goodstadt solution in Swift). Currently I'm not sure why, but the there is a difference of average 0.2 seconds between a recorded audio file (in my case a voice memo) and the received audio file using the following code.
do {
let audioPlayer = try AVAudioPlayer(contentsOf: fileURL)
return CGFloat(audioPlayer.duration)
} catch {
assertionFailure("Failed crating audio player: \(error).")
return nil
}
If you can, try this newer method:
let asset = AVURLAsset(url: fileURL, options: nil)
// Returns a CMTime value.
let duration = try await asset.load(.duration)
// An array of AVMetadataItem for the asset.
let metadata = try await asset.load(.metadata)
// A CMTime value and an array of AVMetadataItem.
let (duration, metadata) = try await asset.load(.duration, .metadata)
Note: Loading several properties at the same time enables AVFoundation to optimize performance by batch-loading requests.
Here's a neat guide from Apple documentation Loading media data asynchronously.
I record a linear PCM file (.pcm) through AVAudioRecorder. I get the duration with the help of Farhad Malekpour. Maybe this can help you :
iPhone: get duration of an audio file
NSURL *fileUrl = [NSURL fileURLWithPath:yourFilePath];
AudioFileID fileID;
OSStatus result = AudioFileOpenURL((__bridge CFURLRef)fileUrl,kAudioFileReadPermission, 0, &fileID);
Float64 duration = 0; //seconds. the duration of the audio.
UInt32 ioDataSize = sizeof(Float64);
result = AudioFileGetProperty(fileID, kAudioFilePropertyEstimatedDuration,
&ioDataSize, &duration);
AudioFileClose(fileID);
if(0 == result) {
//success
}
else {
switch (result) {
case kAudioFileUnspecifiedError:{
//
} break;
// other cases...
default:
break;
}
}
Use AVAssetReader to get the duration of the audio file
guard let assetReader = try? AVAssetReader(asset: audioAsset) {
return nil
}
let duration = Double(assetReader.asset.duration.value)
let timescale = Double(assetReader.asset.duration.timescale)
let totalDuration = duration / timescale
print(totalDuration)
I actually found that you can just use the “get details from music” block and set it to “get duration of ” where can be an mp3 input from a prompt, a link to an mp3, an mp3 from the share sheet, etc