Im trying to figure out how to retrieve a videos frame rate via AVPlayer. AVPlayerItem has a rate variable but it only returns a value between 0 and 2 (usually 1 when playing). Anybody have an idea how to get the video frame rate?
Cheers
Use AVAssetTrack's nominalFrameRate property.
Below method to get FrameRate : Here queuePlayer is AVPlayer
-(float)getFrameRateFromAVPlayer
{
float fps=0.00;
if (self.queuePlayer.currentItem.asset) {
AVAssetTrack * videoATrack = [[videoAsset tracksWithMediaType:AVMediaTypeVideo] lastObject];
if(videoATrack)
{
fps = videoATrack.nominalFrameRate;
}
}
return fps;
}
Swift 4 version of the answer:
let asset = avplayer.currentItem.asset
let tracks = asset.tracks(withMediaType: .video)
let fps = tracks?.first?.nominalFrameRate
Remember to handle nil checking.
There seems to be a discrepancy in this nominalFrameRate returned for the same media played on different versions of iOS. I have a video I encoded with ffmpeg at 1 frame per second (125 frames) with keyframes every 25 frames and when loading in an app on iOS 7.x the (nominal) frame rate is 1.0, while on iOS 8.x the (nominal) frame rate is 0.99. This seems like a very small difference, however in my case I need to navigate precisely to a given frame in the movie and this difference screws up such navigation (the movie is an encoding of a sequence of presentation slides). Given that I already know the frame rate of the videos my app needs to play (e.g. 1 fps) I can simply rely on this value instead of determining the frame rate dynamically (via nominalFrameRate value), however I wonder WHY there is such discrepancy between iOS versions as far as this nominalFrameRate goes. Any ideas?
The rate value on AVPlayer is the speed relative to real time to which it's playing, eg 0.5 is slow motion, 2 is double speed.
As Paresh Navadiya points out a track also has a nominalFrameRate variable however this seems to sometimes give strange results. the best solution I've found so far is to use the following:
CMTime frameDuration = [myAsset tracksWithMediaType:AVMediaTypeVideo][0].minFrameDuration;
float fps = frameDuration.timescale/(float)frameDuration.value;
The above gives slightly unexpected results for variable frame rate but variable frame rate has slightly odd behavior anyway. Other than that it matches ffmpeg -i in my tests.
EDIT ----
I've found sometimes the above gives time kCMTimeZero. The workaround I've used for this is to create an AVAssetReader with a track output,get the pts of the first frame and second frame then do a subtraction of the two.
I don't know anything in AVPlayer that can help you to calculate the frame rate.
AVPlayerItem rate property is the playback rate, nothing to do with the frame rate.
The easier options is to obtain a AVAssetTrack and read its nominalFrameRate property. Just create an AVAsset and you'll get an array of tracks.
Or use AVAssetReader to read the video frame by frame, get its presentation time and count how many frames are in the same second, then average for a few seconds or the whole video.
This is not gonna work anymore, API has changed, and this post is old. :(
The swift 4 answer is also cool, this is answer is similar.
You get the video track from the AVPlayerItem, and you check the FPS there. :)
private var numberOfRenderingFailures = 0
func isVideoRendering() -> Bool {
guard let currentItem = player.currentItem else { return false }
// Check if we are playing video tracks
let isRendering = currentItem.tracks.contains { ($0.assetTrack?.mediaType == .video) && ($0.currentVideoFrameRate > 5) }
if isRendering {
numberOfRenderingFailures = 0
return true
}
numberOfRenderingFailures += 1
if numberOfRenderingFailures < 5 {
return true
}
return false
}
Related
I use multiple AVAudioPlayerNode in AVAudioEngine to mix audio files for playback.
Once all the setup is done (engine prepared, started, audio file segments scheduled), I'm calling play() method on each player node to start playback.
Because it takes times to loop through all player nodes, I take a snapshot of the first nodes's lastRenderTime value and use it to compute a start time for the nodes play(at:) method, to keep playback in sync between nodes :
let delay = 0.0
let startSampleTime = time.sampleTime // time is the snapshot value
let sampleRate = player.outputFormat(forBus: 0).sampleRate
let startTime = AVAudioTime(
sampleTime: startSampleTime + AVAudioFramePosition(delay * sampleRate),
atRate: sampleRate)
player.play(at: startTime)
The problem is with the current playback time.
I use this computation to get the value, where seekTime is a value I keep track of in case we seek the player. It's 0.0 at start :
private var _currentTime: TimeInterval {
guard player.engine != nil,
let lastRenderTime = player.lastRenderTime,
lastRenderTime.isSampleTimeValid,
lastRenderTime.isHostTimeValid else {
return seekTime
}
let sampleRate = player.outputFormat(forBus: 0).sampleRate
let sampleTime = player.playerTime(forNodeTime: lastRenderTime)?.sampleTime ?? 0
if sampleTime > 0 && sampleRate != 0 {
return seekTime + (Double(sampleTime) / sampleRate)
}
return seekTime
}
While this produces a relatively correct value, I can hear a delay between the time I play, and the first sound I hear. Because the lastRenderTime immediately starts to advance once I call play(at:), and there must be some kind of processing/buffering time offset.
The noticeable delay is around 100ms, which is very big, and I need a precise current time value to do visual rendering in parallel.
It probably doesn't matter, but every audio file is AAC audio, and I schedule segments of them in player nodes, I don't use buffers directly.
Segments length may vary. I also call prepare(withFrameCount:) on each player node once I have scheduled audio data.
So my question is, is the delay I observe is a buffering issue ? (I mean should I schedule shorter segments for example), is there a way to compute precisely this value so I can adjust my current playback time computation ?
When I install a tap block on one AVAudioPlayerNode, the block is called with a buffer of length 4410, and the sample rate is 44100 Hz, this means 0.1s of audio data. Should I rely on this to compute the latency ?
I'm wondering if I can trust the length of the buffer I get in the tap block. Alternatively, I'm trying to compute the total latency for my audio graph. Can someone provide insights on how to determine this value precisely ?
From a post on Apple's developer forums by theanalogkid:
On the system, latency is measured by:
Audio Device I/O Buffer Frame Size + Output Safety Offset + Output Stream Latency + Output Device Latency
If you're trying to calculate total roundtrip latency you can add:
Input Latency + Input Safety Offset to the above.
The timestamp you see at the render proc. account for the buffer frame size and the safety offset but the stream and device latencies are not accounted for.
iOS gives you access to the most important of the above information via AVAudioSession and as mentioned you can also use the "preferred" session settings - setPreferredIOBufferDuration and preferredIOBufferDuration for further control.
/ The current hardware input latency in seconds. */
#property(readonly) NSTimeInterval inputLatency NS_AVAILABLE_IOS(6_0);
/ The current hardware output latency in seconds. */
#property(readonly) NSTimeInterval outputLatency NS_AVAILABLE_IOS(6_0);
/ The current hardware IO buffer duration in seconds. */
#property(readonly) NSTimeInterval IOBufferDuration NS_AVAILABLE_IOS(6_0);
Audio Units also have the kAudioUnitProperty_Latency property you can query.
I'm using AVPlayer to play a live streaming. This stream supports one hour catch-up which means user can seek to one hour ago and play. But I have one question how do I know the accurate position that the player is playing. I need to display current position on the player view. For example,if user is playing half an hour ago then display -30:00; if user is playing the latest content, the player will show 00:00 or live. Thanks
Swift solution :
override func getLiveDuration() -> Float {
var result : Float = 0.0;
if let items = player.currentItem?.seekableTimeRanges {
if(!items.isEmpty) {
let range = items[items.count - 1]
let timeRange = range.timeRangeValue
let startSeconds = CMTimeGetSeconds(timeRange.start)
let durationSeconds = CMTimeGetSeconds(timeRange.duration)
result = Float(startSeconds + durationSeconds)
}
}
return result;
}
To get a live position poison and seek to it you can by using seekableTimeRanges of AVPlayerItem:
CMTimeRange seekableRange = [player.currentItem.seekableTimeRanges.lastObject CMTimeRangeValue];
CGFloat seekableStart = CMTimeGetSeconds(seekableRange.start);
CGFloat seekableDuration = CMTimeGetSeconds(seekableRange.duration);
CGFloat livePosition = seekableStart + seekableDuration;
[player seekToTime:CMTimeMake(livePosition, 1)];
Also when you seek some time back, you can get current playing position by calling currentTime method
CGFloat current = CMTimeGetSeconds([self.player.currentItem currentTime]);
CGFloat diff = livePosition - current;
I know this question is old, but I had the same requirement and I believe the solutions aren't addressing properly the intent of the question.
What I did for this same requirement was to gather the current point in time, the starting time, and the length of the total duration of the stream.
I'll explain something before going further, the current point in time could surpass the (starting time + total duration) this is due to the way hls is structured as ts segments. Ts segments are small chucks of playable video, you could have on your seekable range 5 ts segments of 10 seconds each. This doesn't mean that 50 secs is the full length of the live stream, there is around a full segment more (so 60 seconds of playtime total) but it isn't categorized as seekable since you shouldn't seek to that segment. If you were to do this you'll notice in most instances rebuffering (cause the source may be still creating the next ts segment when you already reached the end of playback).
What I did was checking if the current stream time is further than the seekable rage, if so this would mean were are live on stream. If it isn't you could easily calculate how far behind you are from live if you subtract the current time, starting time, and total duration.
let timeRange:CMTimeRange = player.currentItem?.seekableTimeRanges.last
let start = timeRange.start.seconds
let totalDuration = timeRange.duration.seconds
let currentTime = player.currentTime().seconds
let secondsBehindLive = currentTime - totalDuration - start
The code above will give you a negative number with the number of seconds behind "live" or more specifically the start of the lastest ts segment. Or a positive number or zero when it's playing the latest ts segment.
Tbh I don't really know when does the seekableTimeRanges will have more than 1 value, it has always been just one for the streams I have tested with, but if you find in your streams more than 1 value you may have to figure if you want to add all the ranges duration, which time range to use as the start value, etc. At least for my use case, this was enough.
I can see the nominalFrameRate for some video tracks, but not current frame in AVFoundation docs. How can I get the current frame number of the track as it is played in an AVPlayer? I know frame rates will vary, and nominalFrameRate will always be 0.0 in .m3u8 streams, but surely there must be a way to get the frame number of the currently playing track without having to multiply nominalFrameRate by currentTime?
Thanks.
For iOS 7+ you can use the currentVideoFrameRate property of AVPlayerItemTrack. Its the only consistent property that I've seen measure FPS. The nominalFrameRate property seems to be broken in HLS streams and always returns 0.0 as you mentioned.
AVPlayerItem *item = AVPlayer.currentItem; // Your current item
float fps = 0.00;
for (AVPlayerItemTrack *track in item.tracks) {
if ([track.assetTrack.mediaType isEqualToString:AVMediaTypeVideo]) {
fps = track.currentVideoFrameRate;
}
}
I'm using openCV to split a video into frames. For that I need the fps and duration. Both of these value return 1 when asking them via cvGetCaptureProperty.
I've made a hack where I use AVURLAsset to get the fps and duration, but when I combine that with openCV I get only a partial video. It seems like it's missing frames.
This is my code right now:
while (cvGrabFrame(capture)) {
frameCounter++;
if (frameCounter % (int)(videoFPS / MyDesiredFramesPerSecond) == 0) {
IplImage *frame = cvCloneImage(cvRetrieveFrame(capture));
// Do Stuff
}
if (frameCounter > duration*fps)
break; // this is here because the loop never stops on its own
}
How can I get all the frames of a video using openCV on iOS? (opencv 2.3.2)
According to the documentation you should check the value returned by cvRetrieveFrame(), if a null pointer is returned you're at the end of the video sequence. Then you break the loop when that happens, instead of relying on the accuracy of FPS*frame_number.
All QTKit examples use seconds for making ranges. I, unfortunately, have frame numbers and need to be frame accurate. I suppose I could multiply up by the frame rate if I could figure out how to get that out of my movie.
You should be able to calculate the frame rate of a given video media by querying the following mediaAttributes of a QTMedia:
QTMediaDurationAttribute
QTMediaSampleCountAttribute
(they are described in the QTKit docs here)
and use the following formula for calculation:
QTTime duration = ... // value get from mediaAttribute
NSNumber sample_count = ... // value get from mediaAttribute
double fps = (sample_count.longValue * duration.timeScale) / duration.timeValue;
Disclaimer:
Note that I have not tried if this works, but that it is how I expect it to work based on my experience the QuickTime C APIs and the QuickTime File Format.
Good Luck!
Multiplying by the frame rate is not frame accurate because many of the file containers and codecs that Quicktime uses make use of variable frame rates to get better compression. You'll notice this in any kind of movie that has freezes frame for any length of time. See macbreak's The Road to 1080p, part1 as an example.
You can do frame accurate ranges with the QTMovie methods frameStartTime:atTime and frameEndTime:atTime introduced in OSX 10.6. These will give you the start and end of a frame respectively without doing frame decoding.
For example to count all the frames in a movie:
// Initialize QTMovie object called 'movie', disable looping, etc
[movie gotoEnd];
QTTime endTime = [movie currentTime];
[movie gotoBeginning];
QTTime curTime = [movie currentTime];
unsigned long numFrames = 0;
while (true)
{
% get the end time of the current frame
[movie frameEndTime:&curTime];
numFrames++;
% If we get to the last frame, stop counting
if (QTTimeCompare(curTime, endTime) == NSOrderedSame)
{
break;
}
}