AVPlayer/AVAudioMix fade-in effect clicks in the beginning - ios

I'm trying to implement a fade-in effect based on AVPlayer + AVAudioMix + AVAudioMixInputParameters. It basically works except when playing the audio for the first time after starting my app there is a click in the beginning. Subsequent plays work perfect though, but the first-time glitch is pretty stable and reproducible.
My Play button is enabled only after the AVPlayerItem's status is set to ready, so it's impossible to fire a play method while the player is not ready. In fact it doesn't matter how long I wait after loading the audio file and constructing all the objects.
This happens on OS X, I haven't tested it on iOS (yet).
Note that for this test you need an audio file that starts with sound and not silence. Here is my stripped down code without the GUI part (testFadeIn is the entry point):
static AVPlayer* player;
static void* PlayerItemStatusObserverContext = &PlayerItemStatusObserverContext;
- (void)testFadeIn
{
AVURLAsset* asset = [AVURLAsset.alloc initWithURL:[NSURL fileURLWithPath:#"Helicopter.m4a"] options:#{AVURLAssetPreferPreciseDurationAndTimingKey: #YES}];
AVPlayerItem* item = [AVPlayerItem playerItemWithAsset:asset];
player = [AVPlayer playerWithPlayerItem:item];
[item addObserver:self forKeyPath:#"status" options:NSKeyValueObservingOptionInitial | NSKeyValueObservingOptionNew context:PlayerItemStatusObserverContext];
}
- (void)observeValueForKeyPath:(NSString*)keyPath ofObject:(id)object change:(NSDictionary*)change context:(void*)context
{
if (context == PlayerItemStatusObserverContext)
{
AVPlayerStatus status = (AVPlayerStatus)[[change objectForKey:NSKeyValueChangeNewKey] integerValue];
if (status == AVPlayerStatusReadyToPlay)
{
[self applyFadeIn];
[self performSelector:#selector(play:) withObject:nil afterDelay:1.0];
}
}
}
- (void)applyFadeIn
{
assert(player.currentItem.tracks.firstObject);
AVMutableAudioMixInputParameters* fadeIn = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:player.currentItem.tracks.firstObject];
[fadeIn setVolume:0 atTime:kCMTimeZero];
[fadeIn setVolume:1 atTime:CMTimeMake(2, 1)];
NSMutableArray* paramsArray = [NSMutableArray new];
[paramsArray addObject:fadeIn];
AVMutableAudioMix* audioMix = [AVMutableAudioMix audioMix];
audioMix.inputParameters = paramsArray;
player.currentItem.audioMix = audioMix;
}
- (void)play:(id)unused
{
[player play];
}
Click! What is wrong with this?
Edit:
An obvious workaround that I use at the moment is: when the player reports it's ready, I do a short 100ms playback with volume=0, then restore currentTime and volume and only then I report to the main app that the player is ready. This way there are no clicks. Interestingly, anything less than 100ms still gives the click.
This seems like an issue with something that's being cached by AVFoundation after the first playback. It's neither the tracks, as they are available when I set the fade in params, nor the seek status.

Related

iOS AVPlayer play a lot videos and cause memory too high

First of all, thank you for your help.
The Questions:
1.I need to test a lot videos.I wrote two methods to test videos.
2.When playing a video status change, and I set avPlayer = nil and avPlayerItem = nil, i also add the #autoreleasepool {}, but the memory is slowly increased, I used the instruments but I did not find memory leaks, I find a lot questions and did not find a solution.
3.Here is my code:
-(void)CheckVideoURlCanPlay{
#autoreleasepool {
VideoPlayDataObj *videoPlay = [self.needCheckArr objectAtIndex:0];
NSString *str = videoPlay.videoURL;
NSURL *url = [NSURL URLWithString:str];
self.playItem = [AVPlayerItem playerItemWithURL:url];
self.player = [AVPlayer playerWithPlayerItem:self.playItem];
self.playLayer = [AVPlayerLayer playerLayerWithPlayer:self.player];
self.playLayer.videoGravity = AVLayerVideoGravityResizeAspect;
self.playLayer.frame=CGRectMake(100, 80, 100, 100);
[self.view.layer addSublayer:self.playLayer];
[self.player play];
[self.playItem addObserver:self forKeyPath:#"status" options:NSKeyValueObservingOptionNew context:nil];
}}
-(void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary<NSString *,id> *)change context:(void *)context{
#autoreleasepool {
if ([keyPath isEqualToString:#"status"]) {
if (self.player.currentItem.status == AVPlayerItemStatusReadyToPlay) {
#autoreleasepool {
[self.player.currentItem.asset cancelLoading];
[self.player.currentItem cancelPendingSeeks];
[self.player cancelPendingPrerolls];
[self.playItem removeObserver:self forKeyPath:#"status"];
self.playItem = nil;
self.player = nil;
}
[self.needCheckArr removeObjectAtIndex:0];
[NSThread sleepForTimeInterval:5];
[self CheckVideoURlCanPlay];
}
}
}
}
4.I also tried to relieve the current view, but it does not relieve all. If you play a large number of video memory will be high, the final app will be killed.
5.I wonder if after loading a video, it will produce dirty memory?
this my app start memory:13M
this test some videos and i pop the view :
enter image description here
this test some videos again and pop the view :
enter image description here
6.Finally, I hope you can help me, thank you very much.
It does not look like you are ever removing the AVPlayerLayer from the view:
[self.playerLayer removeFromSuperlayer];
Also I cannot see the point of #autoreleasepool here, especially if you have ARC turned on (which you almost certainly do!). #autoreleasepool is usually only useful when you have ARC turned off, or when you are churning through a lot of memory (as in, many megabytes) in a single main-loop invocation and need to control when they get cleaned up.

iOS objective C low-latency audio playback like SoundPool

Hey on my Android apps I can preload my sounds in a SoundPool and then play them with almost no latency at all. Now I am looking for the same thing on iOS/obj-c, but I just can't find anything similar.
I followed a couple of tutorials but eventually there was a bigger lag than I expected and most of the tutorials are advising you to convert your audio to an uncompressed format like wav or caf but my MP3's are already at 14 mb and converting them to lossless audio leads to 81 mb of data which is way too much for me.
The most promising thing I tried was preloading the file (just like I did in Android's SoundPool) like shown in this OAL example:
- (bool) preloadUrl:(NSURL*) url seekTime:(NSTimeInterval)seekTime
{
if(nil == url)
{
OAL_LOG_ERROR(#"%#: Cannot open NULL file / url", self);
return NO;
}
OPTIONALLY_SYNCHRONIZED(self)
{
// Bug: No longer re-using AVAudioPlayer because of bugs when using multiple players.
// Playing two tracks, then stopping one and starting it again will cause prepareToPlay to fail.
bool wasPlaying = playing;
[self stopActions];
if(playing || paused)
{
[player stop];
}
as_release(player);
if(wasPlaying)
{
[[NSNotificationCenter defaultCenter] performSelectorOnMainThread:#selector(postNotification:) withObject:[NSNotification notificationWithName:OALAudioTrackStoppedPlayingNotification object:self] waitUntilDone:NO];
}
NSError* error;
player = [[AVAudioPlayer alloc] initWithContentsOfURL:url error:&error];
if(nil == player)
{
OAL_LOG_ERROR(#"%#: Could not load URL %#: %#", self, url, [error localizedDescription]);
return NO;
}
player.volume = muted ? 0 : gain;
player.numberOfLoops = numberOfLoops;
player.meteringEnabled = meteringEnabled;
player.delegate = self;
player.pan = pan;
as_release(currentlyLoadedUrl);
currentlyLoadedUrl = as_retain(url);
self.currentTime = seekTime;
playing = NO;
paused = NO;
BOOL allOK = [player prepareToPlay];
if(!allOK)
{
OAL_LOG_ERROR(#"%#: Failed to prepareToPlay: %#", self, url);
}
else
{
[[NSNotificationCenter defaultCenter] performSelectorOnMainThread:#selector(postNotification:) withObject:[NSNotification notificationWithName:OALAudioTrackSourceChangedNotification object:self] waitUntilDone:NO];
}
preloaded = allOK;
return allOK;
}
}
But this still makes a quite considerable delay of about ~60ms which is way too much for an audio app like mine. My audio files don't have any delay in the beginning so it must have to do something with the code.
I tried all that stuff on an iPhone 5c.
You should be able to create several AVAudioPlayers and call prepareToPlay on them, but personally I like to use AVAssetReader to keep a buffer of LPCM audio ready to play at a moment's notice.

Replaying AVPlayerItem / AVPlayer without re-downloading

I have an AVPlayer class all set up that streams an audio file. It's a bit long, so I can't post the whole thing here. What I am stuck on is how to allow the user to replay the audio file after they have finished listening to it once. When it finishes the first time, I correctly receive a notification AVPlayerItemDidPlayToEndTimeNotification. When I go to replay it, I immediately receive the same notification, which blocks me from replaying it.
How can I reset this such that the AVPlayerItem doesn't think that it has already played the audio file? I could deallocate everything and set it up again, but I believe that would force the user to download the audio file again, which is pointless and slow.
Here are some parts of the class that I think are relevant. The output that I get when attempting to replay the file looks like this. The first two lines are exactly what I would expect, but the third is a surprise.
is playing no timer audio player has finished playing audio
- (id) initWithURL : (NSString *) urlString
{
self = [super init];
if (self) {
self.isPlaying = NO;
self.verbose = YES;
if (self.verbose) NSLog(#"url: %#", urlString);
NSURL *url = [NSURL URLWithString:urlString];
self.playerItem = [AVPlayerItem playerItemWithURL:url];
self.player = [[AVPlayer alloc] initWithPlayerItem:self.playerItem];
[self determineAudioPlayTime : self.playerItem];
self.lengthOfAudioInSeconds = #0.0f;
[self.player addObserver:self forKeyPath:#"status" options:0 context:nil];
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(itemDidFinishPlaying:) name:AVPlayerItemDidPlayToEndTimeNotification object:self.playerItem];
}
return self;
}
// this is what gets called when the user clicks the play button after they have
// listened to the file and the AVPlayerItemDidPlayToEndTimeNotification has been received
- (void) playAgain {
[self.playerItem seekToTime:kCMTimeZero];
[self toggleState];
}
- (void) toggleState {
self.isPlaying = !self.isPlaying;
if (self.isPlaying) {
if (self.verbose) NSLog(#"is playing");
[self.player play];
if (!timer) {
NSLog(#"no timer");
CMTime audioTimer = CMTimeMake(0, 1);
[self.player seekToTime:audioTimer];
timer = [NSTimer scheduledTimerWithTimeInterval:1.0
target:self
selector:#selector(updateProgress)
userInfo:nil
repeats:YES];
}
} else {
if (self.verbose) NSLog(#"paused");
[self.player pause];
}
}
-(void)itemDidFinishPlaying:(NSNotification *) notification {
if (self.verbose) NSLog(#"audio player has finished playing audio");
[[NSNotificationCenter defaultCenter] postNotificationName:#"audioFinished" object:self];
[timer invalidate];
timer = nil;
self.totalSecondsPlayed = [NSNumber numberWithInt:0];
self.isPlaying = NO;
}
You can call the seekToTime method when your player received AVPlayerItemDidPlayToEndTimeNotification
func itemDidFinishPlaying() {
self.player.seek(to: CMTime.zero)
self.player.play()
}
Apple recommends using AVQueueplayer with an AVPlayerLooper.
Here's Apple's (slightly revised) sample code:
AVQueuePlayer *queuePlayer = [[AVQueuePlayer alloc] init];
AVAsset *asset = // AVAsset with its 'duration' property value loaded
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];
// Create a new player looper with the queue player and template item
self.playerLooper = [AVPlayerLooper playerLooperWithPlayer:queuePlayer
templateItem:playerItem];
// Begin looping playback
[queuePlayer play];
The AVPlayerLooper does all the event listening and playing for you, and the queue player is used to create what they call a "treadmill pattern". This pattern is essentially chaining multiple instances of the same AVAssetItem in a queue player and moving each finished asset back to the beginning of the queue.
The advantage of this approach is that it enables the framework to preroll the next asset (which is the same asset in this case, but its start still needs prerolling) before it arrives, reducing latency between the asset's end and looped start.
This is described in greater detail at ~15:00 in the video here: https://developer.apple.com/videos/play/wwdc2016/503/

forwardPlaybackEndTime does not work

AVPlayerItem has this property forwardPlaybackEndTime
The value indicated the time at which playback should end when the
playback rate is positive (see AVPlayer’s rate property).
The default value is kCMTimeInvalid, which indicates that no end time
for forward playback is specified. In this case, the effective end
time for forward playback is the item’s duration.
But I don't know why it does not work. I tried to set it in AVPlayerItemStatusReadyToPlay, duration available callback, ... but it does not have any effect, it just plays to the end
I think that forwardPlaybackEndTime is used to restrict the playhead, right?
In my app, I want to play from the beginning to the half of the movie only
My code looks like this
- (void)playURL:(NSURL *)URL
{
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithURL:URL];
if (self.avPlayer) {
if (self.avPlayer.currentItem && self.avPlayer.currentItem != playerItem) {
[self.avPlayer replaceCurrentItemWithPlayerItem:playerItem];
}
} else {
[self setupAVPlayerWithPlayerItem:playerItem];
}
playerItem.forwardPlaybackEndTime = CMTimeMake(5, 1);
// Play
[self.avPlayer play];
}
How to make forwardPlaybackEndTime work?
Try this
AVPlayerItem.forwardPlaybackEndTime = CMTimeMake(5, 1);
Here 5 is the time till the AVPlayerItem will play.
Set the following on your AVPlayer
AVPlayer.actionAtItemEnd = AVPlayerActionAtItemEndNone;
Then set your notification
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(playerItemDidPlayToEndTime:) name:AVPlayerItemDidPlayToEndTimeNotification object:nil];
with the method obviously
- (void) playerItemDidPlayToEndTime:(NSNotification*)notification
{
// do something here
}
then set your forwardPlaybackEndTime
AVPlayer.currentItem.forwardPlaybackEndTime = CMTimeAdd(AVPlayer.currentItem.currentTime, CMTimeMake(5.0, 1));
and start your avplayer playing
AVPlayer.rate = 1.0;
the notification will be triggered and your track will continue playing. In your handler you can stop it and do a seekToTime or whatever.
Alternatively you can just set a boundary observer
NSArray *array = [NSArray arrayWithObject:[NSValue valueWithCMTime:CMTimeMakeWithSeconds(5.0, 1)]];
__weak OTHD_AVPlayer* weakself = self;
self.observer_End = [self addBoundaryTimeObserverForTimes:array queue:NULL usingBlock:^{ if( weakself.rate >= 0.0 ) [weakself endBoundaryHit]; }];
I have checked below code its running and streaming stops at specified time:
- (void)playURL
{
NSURL *url = [NSURL URLWithString:#"http://clips.vorwaerts-gmbh.de/VfE_html5.mp4"];
self.playerItem = [AVPlayerItem playerItemWithURL:url];
self.playerItem.forwardPlaybackEndTime = CMTimeMake(10, 1);
self.avPlayer = [AVPlayer playerWithPlayerItem:self.playerItem];
[videoView setPlayer:self.avPlayer];
// Play
[self.avPlayer play];
}
I hope this will help you.
Also please check these tutorials: AVFoundation Framework
I think that you have to update avPlayer's current playerItem.
So,
playerItem.forwardPlaybackEndTime = CMTimeMake(5, 1);
it should be:
self.avPlayer.currentItem.forwardPlaybackEndTime = CMTimeMake(5, 1);

tracks in AVComposition losing time when paused

I've created an AVMutableComposition that consists of a bunch of audio tracks that start at specific times. From there, following Apple recommendations, i turned it into an AVComposition before playing it with AVPlayer.
It all works fine playing this AVPlayer item, but if I pause it and then continue, all the tracks in the composition appear to slip back about 0.2 seconds relative to each other (i.e., they bunch up). Hitting pause and continuing several times compounds the effect and the overlap is more significant (basically if I hit it enough, I will end up with all 8 tracks playing simultaneously).
if (self.player.rate > 0.0) {
//if player is playing, pause
[self.player pause];
} else {
if (self.player) {
[self.player play];
return;
}
*/CODE CREATING COMPOSITION - missed out big chunk of code relating to finding the track and retrieving its position and scale/*
NSDictionary *options = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:YES]
forKey:AVURLAssetPreferPreciseDurationAndTimingKey];
AVURLAsset *sourceAsset = [[AVURLAsset alloc] initWithURL:url options:options];
//calculate times
NSNumber *time = [soundArray1 objectAtIndex:1]; //this is the time scale - e.g. 96 or 120 etc.
double timenow = [time doubleValue];
double insertTime = (240*y);
AVMutableCompositionTrack *track =
[composition addMutableTrackWithMediaType:AVMediaTypeAudio
preferredTrackID:kCMPersistentTrackID_Invalid];
//insert the audio track from the asset into the track added to the mutable composition
AVAssetTrack *myTrack = [[sourceAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];
CMTimeRange myTrackRange = myTrack.timeRange;
NSError *error = nil;
[track insertTimeRange:myTrackRange
ofTrack:myTrack
atTime:CMTimeMake(insertTime, timenow)
error:&error];
[sourceAsset release];
}
}
AVComposition *immutableSnapshotOfMyComposition = [composition copy];
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:immutableSnapshotOfMyComposition];
self.player = [[AVPlayer alloc] initWithPlayerItem:playerItem];
NSLog(#"here");
[self.player play];
Thanks
OK, this feels a little hacky, but it definitely works if anybody is stuck. If someone has a better answer, do let me know!
Basically, I just save the player.currentTime of the track when I hit pause and remake the track when i hit play, just starting from the point at which i paused it. No discernible delay, but I'd still be happier without wasting this extra processing.
Make sure you properly release your player item after you hit pause, otherwise you'll end up with a giant stack of AVPlayers!
I have a solution that is a bit less hacky but still hacky.
The solution comes from the fact that I noticed that if you seeked on the player, the latency between audio and video introduced by pausing disappeared.
Hence: just save the player.currentTime just before pausing and, player seekToTime just before playing again. It works pretty well on iOS 6, haven't tested on other versions yet.

Resources