How do I get current playing time, CMTime in milliseconds in AVPlayer? - ios

I am getting current playing value in seconds but i need in milliseconds. I tried to currentTime.value/currentTime.scale. But it didn't get exact value.
CMTime currentTime = vPlayer.currentItem.currentTime; //playing time
CMTimeValue tValue=currentTime.value;
CMTimeScale tScale=currentTime.timescale;
NSTimeInterval time = CMTimeGetSeconds(currentTime);
NSLog(#"Time :%f",time);//This is in seconds, it misses decimal value double shot=(float)tValue/(float)tScale;
shotTimeVideo=[NSString stringWithFormat:#"%.2f",(float)tValue/(float)tScale];
CMTime currentTime = vPlayer.currentItem.currentTime; //playing time
CMTimeValue tValue=currentTime.value;
CMTimeScale tScale=currentTime.timescale;
NSTimeInterval time = CMTimeGetSeconds(currentTime);
NSLog(#"Time :%f",time);//This is in seconds, it misses decimal value
double shot=(float)tValue/(float)tScale;
shotTimeVideo=[NSString stringWithFormat:#"%.2f", (float)tValue/(float)tScale];

okay,first of all, the value you want is millisecond not seconds
So,you can just use CMTimeGetSeconds(<#CMTime time#>) to get Secondsthen,if you want millisecond , use seconds / 1000.f for float or double value
for CMTime calculating use CMTime method CMTimeMultiplyByRatio(<#CMTime time#>, <#int32_t multiplier#>, <#int32_t divisor#>)just do this --> CMTimeMultiplyByRatio(yourCMTimeValue, 1, 1000)
Apple's Doc
#function CMTimeMultiplyByRatio
#abstract Returns the result of multiplying a CMTime by an integer, then dividing by another integer.
#discussion The exact rational value will be preserved, if possible without overflow. If an overflow
would occur, a new timescale will be chosen so as to minimize the rounding error.
Default rounding will be applied when converting the result to this timescale. If the
result value still overflows when timescale == 1, then the result will be either positive
or negative infinity, depending on the direction of the overflow.
If any rounding occurs for any reason, the result's kCMTimeFlags_HasBeenRounded flag will be
set. This flag will also be set if the CMTime operand has kCMTimeFlags_HasBeenRounded set.
If the denominator, and either the time or the numerator, are zero, the result will be
kCMTimeInvalid. If only the denominator is zero, the result will be either kCMTimePositiveInfinity
or kCMTimeNegativeInfinity, depending on the signs of the other arguments.
If time is invalid, the result will be invalid. If time is infinite, the result will be
similarly infinite. If time is indefinite, the result will be indefinite.
#result (time * multiplier) / divisor

A little late, but may be useful for others.
You can get the timestamp in milliseconds from the CMTime object by
CMTimeConvertScale(yourCMTime, timescale:1000, method:
CMTimeRoundingMethod.roundHalfAwayFromZero
An example:
var yourtime:CMTime = CMTimeMake(value:1234567, timescale: 10)
var timemilli:CMTime = CMTimeConvertScale(yourtime, timescale:1000, method:
CMTimeRoundingMethod.roundHalfAwayFromZero)
var timemillival:Int64 = timemilli.value
print("timemilli \(timemillival)")
which will produce
timemilli 123456700

Related

iOS: Synchronizing frames from camera and motion data

I'm trying to capture frames from camera and associated motion data.
For synchronization I'm using timestamps. Video and motion is written to a file and then processed. In that process I can calculate motion-frames offset for every video.
Turns out motion data and video data for same timestamp is offset from each other by different time from 0.2 sec up to 0.3 sec.
This offset is constant for one video but varies from video to video.
If it was same offset every time I would be able to subtract some calibrated value but it's not.
Is there a good way to synchronize timestamps?
Maybe I'm not recording them correctly?
Is there a better way to bring them to the same frame of reference?
CoreMotion returns timestamps relative to system uptime so I add offset to get unix time:
uptimeOffset = [[NSDate date] timeIntervalSince1970] -
[NSProcessInfo processInfo].systemUptime;
CMDeviceMotionHandler blk =
^(CMDeviceMotion * _Nullable motion, NSError * _Nullable error){
if(!error){
motionTimestamp = motion.timestamp + uptimeOffset;
...
}
};
[motionManager startDeviceMotionUpdatesUsingReferenceFrame:CMAttitudeReferenceFrameXTrueNorthZVertical
toQueue:[NSOperationQueue currentQueue]
withHandler:blk];
To get frames timestamps with high precision I'm using AVCaptureVideoDataOutputSampleBufferDelegate. It is offset to unix time also:
-(void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
CMTime frameTime = CMSampleBufferGetOutputPresentationTimeStamp(sampleBuffer);
if(firstFrame)
{
firstFrameTime = CMTimeMake(frameTime.value, frameTime.timescale);
startOfRecording = [[NSDate date] timeIntervalSince1970];
}
CMTime presentationTime = CMTimeSubtract(frameTime, firstFrameTime);
float seconds = CMTimeGetSeconds(presentationTime);
frameTimestamp = seconds + startOfRecording;
...
}
It is actually pretty simple to correlate these timestamps - although it's not clearly documented, both camera frame and motion data timestamps are based on the mach_absolute_time() timebase.
This is a monotonic timer that is reset at boot, but importantly also stops counting when the device is asleep. So there's no easy way to convert it to a standard "wall clock" time.
Thankfully you don't need to as the timestamps are directly comparable - motion.timestamp is in seconds, you can log out mach_absolute_time() in the callback to see it is the same timebase. My quick test shows the motion timestamp is typically about 2ms before mach_absolute_time in the handler, which seems about right for how long it might take for the data to get reported to the app.
Note mach_absolute_time() is in tick units that need conversion to nanoseconds; on iOS 10 and later you can just use the equivalent clock_gettime_nsec_np(CLOCK_UPTIME_RAW); which does the same thing.
[_motionManager
startDeviceMotionUpdatesUsingReferenceFrame:CMAttitudeReferenceFrameXArbitraryZVertical
toQueue:[NSOperationQueue currentQueue]
withHandler:^(CMDeviceMotion * _Nullable motion, NSError * _Nullable error) {
// motion.timestamp is in seconds; convert to nanoseconds
uint64_t motionTimestampNs = (uint64_t)(motion.timestamp * 1e9);
// Get conversion factors from ticks to nanoseconds
struct mach_timebase_info timebase;
mach_timebase_info(&timebase);
// mach_absolute_time in nanoseconds
uint64_t ticks = mach_absolute_time();
uint64_t machTimeNs = (ticks * timebase.numer) / timebase.denom;
int64_t difference = machTimeNs - motionTimestampNs;
NSLog(#"Motion timestamp: %llu, machTime: %llu, difference %lli", motionTimestampNs, machTimeNs, difference);
}];
For the camera, the timebase is also the same:
// In practice gives the same value as the CMSampleBufferGetOutputPresentationTimeStamp
// but this is the media's "source" timestamp which feels more correct
CMTime frameTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
uint64_t frameTimestampNs = (uint64_t)(CMTimeGetSeconds(frameTime) * 1e9);
The delay between the timestamp and the handler being called is a bit larger here, usually in the 10s of milliseconds.
We now need to consider what a timestamp on a camera frame actually means - there are two issues here; finite exposure time, and rolling shutter.
Rolling shutter means that not all scanlines of the image are actually captured at the same time - the top row is captured first and the bottom row last. This rolling readout of the data is spread over the entire frame time, so in 30 FPS camera mode the final scanline's exposure start/end time is almost exactly 1/30 second after the respective start/end time of the first scanline.
My tests indicate the presentation timestamp in the AVFoundation frames is the start of the readout of the frame - ie the end of the exposure of the first scanline. So the end of the exposure of the final scanline is frameDuration seconds after this, and the start of the exposure of the first scanline was exposureTime seconds before this. So a timestamp right in the centre of the frame exposure (the midpoint of the exposure of the middle scanline of the image) can be calculated as:
const double frameDuration = 1.0/30; // rolling shutter effect, depends on camera mode
const double exposure = avCaptureDevice.exposureDuration;
CMTime frameTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
double midFrameTime = CMTimeGetSeconds(frameTime) - exposure * 0.5 + frameDuration * 0.5;
In indoor settings, the exposure usually ends up the full frame time anyway, so the midFrameTime from above ends up identical to the frameTime. The difference is noticeable (under extremely fast motion) with short exposures that you typically get from brightly lit outdoor scenes.
Why the original approach had different offsets
I think the main cause of your offset is that you assume the timestamp of the first frame is the time that the handler runs - ie it doesn't account for any delay between capturing the data and it being delivered to your app. Especially if you're using the main queue for these handlers I can imagine the callback for that first frame being delayed by the 0.2-0.3s you mention.
The best solution I was able to find to this problem was
to run a feature tracker over the recorded video, pick one of the strong features and plot the the speed of it's movement along say X axis and then correlate this plot to the accelerometer Y data.
When there's 2 similar plots that are offset of each other along abscissa there's a technique called cross-correlation that allows to find the offset.
There's an obvious drawback of this approach - it's slow as it requires some video processing.

Inaccurate DispatchTime.now() in Swift 3.0

I'm using DispatchTime.now() to measure elapsed time between events. Sometimes it works correctly, but occasionally it generates values that are far below what I would expect.
My current usage:
var t = DispatchTime.now()
var pt = DispatchTime.now()
// called when an event happens
t = DispatchTime.now()
var elapsed = Double(t.uptimeNanoseconds - pt.uptimeNanoseconds)
elapsed *= 32768/1000000000
pt = t
Such that t and pt are current and previous times, elapsed takes the difference in nanoseconds, converts to double, and scales such that 1 second = 32768. When this technique fails the data recorded is about 100 times smaller than what is expected. The scaling is not the issue, I've checked the rawValue of t and pt. My assumption would be that the clock that runs DispatchTime is running at a slower speed, maybe because of debugging, but in general I would think iOS would compensate for something like this.
As #AlexanderMomchliov suggested NSDate is a better approach than DispatchTime.
Implemented as:
var t: TimeInterval = 0
var pt: TimeInterval = NSDate().timeIntervalSinceReferenceDate
// called when an event happens
t = NSDate().timeIntervalSinceReferenceDate
var elapsed: Double = t - pt
elapsed *= 32768
pt = t
You are performing integer division which will result in inaccurate elapsed time:
elapsed *= 32768/1000000000
You should either wrap these as Double or end them with a decimal (i.e. 32768.0/1000000000.0):
elapsed *= Double(32768)/Double(1000000000)
Additionally, NSEC_PER_SEC is defined as a global variable as part of the Dispatch framework:
elapsed *= Double(32768)/Double(NSEC_PER_SEC)
NSDate or Date may be adjusted by the system and should not be used for reliably measuring elapsed time. DispatchTime or CACurrentMediaTime would be good solutions and are both based on mach absolute time. However, if the app is put in the background during measurement, then using Date would be a good fallback.
Also recommend checking out this question for further discussion.

AVAsset video adjust duration

Given a list of CMSampleBuffers that have been read in from an asset, I want to adjust the duration of the asset so that it's half length (twice the speed) of the original.
Currently my function for generating new time stamps looks like:
func adjustTimeStampsForBuffers(buffers: [CMSampleBuffer]) -> [CMTime] {
let frameCount = buffers.count
// self.duration is CMTimeGetSeconds(asset.duration)
let increment = Float(self.duration / 2) / Float(frameCount)
return Array(0.stride(to: frameCount, by: 1)).enumerate().map {
let seconds: Float64 = Float64(increment) * Float64($0.index)
return CMTimeMakeWithSeconds(seconds, self.asset.duration.timescale)
}
}
however this doesn't seem to work and the outputted assets are in fact twice the length, not half. Can anybody point out where I'm going wrong?
Edit:
Thanks to #sschale, here's my final answer:
func adjustTimeStampsForBuffers(buffers: [CMSampleBuffer]) -> [CMTime] {
return buffers.map {
let time = CMSampleBufferGetPresentationTimeStamp($0)
return CMTimeMake(time.value, time.timescale * 2)
}
}
Instead of calculating new values, the timestamp is adjusted instead.
Based on my reading of the docs, it looks like that self.asset.duration.timescale may be the key here, as changing it will influence the whole file (if I'm understanding the reference you're making that that timescale is for the whole file, or maybe you need to adjust it in each of the buffers).
See here for more info as well.
Relevant section:
A CMTime is represented as a rational number, with a numerator (an
int64_t value), and a denominator (an int32_t timescale).
Conceptually, the timescale specifies the fraction of a second each
unit in the numerator occupies. Thus if the timescale is 4, each unit
represents a quarter of a second; if the timescale is 10, each unit
represents a tenth of a second, and so on. In addition to a simple
time value, a CMTime can represent non-numeric values: +infinity,
-infinity, and indefinite. Using a flag CMTime indicates whether the time been rounded at some point.
CMTimes contain an epoch number, which is usually set to 0, but can be
used to distinguish unrelated timelines: for example, it could be
incremented each time through a presentation loop, to differentiate
between time N in loop 0 from time N in loop 1

CMTimeMake(duration, fps). What is this mean?

I am making video file on my Mac programmatically.
I want to know that "CMtimeMake" is what's mean.
For example:
AVAssetWriterInputPixelBufferAdaptor * avAdaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerInput sourcePixelBufferAttributes:NULL];
[avAdaptor appendPixelBufferixelBuffer withPresentationTime:CMTimeMake(1, 10)];
CMTimeMake makes a valid CMTime with value and timescale.
Here in your code:
CMTimeMake(1,10)
A CMTime is represented as a rational number, with a numerator (an int64_t value), and a denominator (an int32_t timescale).
CMTimeMake is called to create CMTime representing times(either durations or timestamps).
1 is the value field of the resulting CMTime and 10 means each unit represents a tenth of a second.

Trying to understand CMTime

I have seen some examples of CMTime (Three separate links), but I still don't get it. I'm using an AVCaptureSession with AVCaptureVideoDataOutput and I want to set the max and min frame rate of the the output. My problem is I just don't understand the CMTime struct.
Apparently CMTimeMake(value, timeScale) should give me value frames every 1/timeScale seconds for a total of value/timeScale seconds, or am I getting that wrong?
Why isn't this documented anywhere in order to explain what this does?
If it does truly work like that, how would I get it to have an indefinite number of frames?
If its really simple, I'm sorry, but nothing has clicked just yet.
A CMTime struct represents a length of time that is stored as rational number (see CMTime Reference). CMTime has a value and a timescale field, and represents the time value/timescale seconds .
CMTimeMake is a function that returns a CMTime structure, for example:
CMTime t1 = CMTimeMake(1, 10); // 1/10 second = 0.1 second
CMTime t2 = CMTimeMake(2, 1); // 2 seconds
CMTime t3 = CMTimeMake(3, 4); // 3/4 second = 0.75 second
CMTime t4 = CMTimeMake(6, 8); // 6/8 second = 0.75 second
The last two time values t3 and t4 represent the same time value, therefore
CMTimeCompare(t3, t4) == 0
If you set the videoMinFrameDuration of a AVCaptureSession is does not make a difference if you set
connection.videoMinFrameDuration = CMTimeMake(1, 20); // or
connection.videoMinFrameDuration = CMTimeMake(2, 40);
In both cases the minimum time interval between frames is set to 1/20 = 0.05 seconds.
My experience differs.
For let testTime = CMTime(seconds: 3.83, preferredTimescale: 100)
If you set a breakpoint and look in the debugger side window it says:
"383 100ths of a second"
Testing by seeking to a fixed offset in a video in AVPlayer has confirmed this.
So put the actual number of seconds in the seconds field, and the precision in the preferredTimescale field. So 100 means precision of hundredths of a second.
Doing
let testTime = CMTime(seconds: 3.83, preferredTimescale: 100)
Still seeks to the same place in the video, but it displays in the debugger side window as "3833 1000ths of a second"
Doing
let testTime = CMTime(seconds: 3.83, preferredTimescale: 1)
Does not seek to the same place in the video, because it's been truncated, and it displays in the debugger side window as "3 seconds". Notice that the .833 part has been lost due to the preferredTimescale.
CMTime(seconds: value, timescale: scale)
means value/scale in a just one second

Resources