I've created a scrubber in my app that allows the user to scrub forwards/backwards through a video via [AVPlayer seekToTime:toleranceBefore:toleranceAfter].
The video that is being scrubbed is captured via an AVCaptureSession that uses AVCaptureMovieFileOutput. I've ffprobed the resulting .MOV and the results are as expected (e.g., on my iPhone 5s I'm recording at 120fps at approx 23000 kb/s with approx 1 keyframe per second).
Since there is only approximately 1 keyframe per second, it is difficult to scrubber backward through the video with any precision and without any lag (since it would have to go back to the closest keyframe and the compute the frame at my current scrubbing position).
So I'm wondering if there is a better strategy for smooth scrubbing? There are apps out there that do this really well (e.g., I've examined the Coach's Eye app and it records video precisely the same way I do and yet its scrubbing performance is quite good).
I'd be very appreciative of any suggestions.
Related
Is it possible to apply slow motion effect while recording video?
This means that the recording has not finished yet, the file has not been saved, but the user sees the recording process in slow motion.
I think it is important to understand what slow-motion actually means. To "slow down motion" in a movie, you need to film more images per second than usually and play this movie afterwards in normal speed, that's making the slow motion effect.
Example: Videos are often shot in 30 frames per second (fps), so for one second of movie you're creating 30 single images. If you want a motion to be half as fast, you need to shoot 60 fps (60 images per second). If you play those 60 images at half-speed (the normal 30fps), it will result in a movie of 2 second lengths showing the slow-motion effect.
As you can see, you cannot record and show a slow-motion effect at the same time. You'll need to save it first and then play it slower than recorded.
The perfect example of what I am trying to achieve can be seen in the Flow ● Slow and Fast Motion app .
One can change the playback rate of the video by dragging points on the curve up or down. The video can also be saved in this state.
I am looking for a way to dynamically speed up/down a video , so that the playback rate can be changed while the video is being played.
Video explanation
WHAT I'VE TRIED
The playback rate property of AVPlayer .But it Only works with a few values for playback Rate(0.50, 0.67, 0.80, 1.0, 1.25, 1.50, and 2.0 ) and one cannot save the video
The scaleTimeRange(..) property of AVMutableComposition. But it doesn't work when you want to ramp the video for gradually decreasing slow/fast motion.
Display video frames on screen using CAEAGLLayer and CADisplayLink. But my many attempts on trying to achieve Slow/Fast motion with this have been unsuccessful .
All this has taken me months and I'm starting to doubt if I'll be able to accomplish this at all.
Thus any suggestion , would be immensely valuable.
In IOS, the MPNowPlayingInfoCenter object contains a 'nowPlayingInfo' dictionary whose contents describe the item being played. It is advised that you start the playback at the 'currentplaybackrate' and then set the speed. See this thread on the developer's forum.
You might possibly end up with something like this (but this is javascript) where the playback rate of the video has been sped up by 4.
document.querySelector('video').playbackRate = 4.0;
document.querySelector('video').play();
video{width:400px;
height:auto;}
<video controls preload="true" autoplay>
<source src="http://www.rachelgallen.com/nature.mp4" type="video/mp4" >
</video>
So I'm not sure I fully understand the use case you're going for, but I think
func setRate(_ rate: Float,
time itemTime: CMTime,
atHostTime hostClockTime: CMTime)
[Apple Documentation Source]
Is something that you're looking for. While this may not be exactly what you need, I'm also not sure where in the docs there is exactly what you're looking for, but with the above method alone, you could do the following to save videos at a variable rate:
Use the above method to play the video throughout (assuming it's not too long, otherwise this will be computationally impossible/timeout-worthy on some devices) at the desired rates each second. Design UI to adjust this per second rate.
under the hood you can actually play the video at that speed "frame by frame" and capture the frames you want (in the right # which will give you the rate you desire) and voila -- saving the right number of frames together (skipping/duplicating as needed to increase/lower desired rate based on "picker" UI) you've now accomplished what you desire
To be clear, what I'm talking about here is a video output # 60FPS has 60 frames per second. You would literally "cut and paste" frames together from the source video into the "destination" video based on whatever UI steppers values you receive from your user (in the screenshot-ed example the question contains, as my basis), and pick up that many frames. AKA if the user says seconds 2-10 of their 20 second video should be at 2X, only put in 30 frames for each of those seconds (if filmed at 60 FPS) alternating frames. The output will, at 60FPS, seem like 2X speed (since there are now 30 frames per 1 second of original video, which is 0.5 seconds at 60 FPS). Similarly, any value can appropriately be factored into:
(desired consistent FPS) = (source video FPS) = (destination video FPS) (ie 60 or 90)
(rate) = (rate from UI steppers/graph UI to pick rate # each time interval) (ie 1X/2x/0.25X)
(desired consistent FPS) * (rate) = (# frames kept in destination video)
(destination video frames) = (source video) * (desired consistent FPS) ~modulated by~ (per custom time interval rate)
The exact mechanisms for ^^ might actually be built into AVPlayer and I didn't find the details, but this alone should be a good start to get you going in that direction.
I am using scaleTimeRange:toDuration: to produce a fast-motion effect of upto 10x the original video speed.But I noticed that videos start to stutter when played through an AVPlayer at 10x.
I also noticed that on OSX's QuickTime the same composition plays smoothly.
Another question states that the reason for this is hardware limitation , but I want to know if there is a way around this , so that the fast motion effect occurs smoothly over the length of the entire video.
Video Specs
Format : H.264 , 1280x544
FPS : 25
Data Size : 26MB
Data Rate : 1.17 Mbit/s
I have a feeling that by playing your videos at 10x using scaleTimeRange:toDuration simply has the effect of multiplying your data rate by 10, bringing it up to 10Mbit/s, which osx machines can handle, but iOS devices cannot.
In other words, you're creating videos that need to play back at 300 frames per second, which is pushing AVPlayer too hard.
If I didn't know about your other question, I would have said that the solution is to export your AVComposition using AVAssetExportSession, which should result in your high FPS video being down sampled to an easier to handle 30fps, and then play that with AVPlayer.
If AVAssetExportSession isn't working, you could try applying the speedup effect yourself, by reading the frames from the source video using AVAssetReader and writing every tenth frame to the output file using AVAssetWriter (don't forget to set the correct presentation timestamps).
I am trying do slow motion for my video file along with audio. In my case, I have to do Ramped Slow motion(Gradually slowing down and speeding up
like parabola not a "Linear Slow Motion".
Ref:Linear slow motion :
Ref : Ramped Slow Motion :
What have i done so far:
Used AVFoundation for first three bullets
From video files, separated audio and video.
Did slow motion for video using AVFoundation api (scaletimeRange).Its really working fine.
The same is not working for audio. Seems there's a bug in apple api itself (Bug ID : 14616144). The relevant question is scaleTimeRange has no effect on audio type AVMutableCompositionTrack
So i switched to Dirac. later found there is a limitation with Dirac's open source edition that it doesn't support Dynamic Time Stretching.
Finally trying to do with OpenAL.
I've taken a sample OpenAL program from Apple developer forum and executed it.
Here are my questions:
Can i store/save the processed audio in OpenAl?if its directly not possible with "OpenAl", can it be done with AVFoundation + OpenAL?
Very importantly, how to do slow motion or stretch the time scale with OpenAL? If i know time stretching, i can apply logic for Ramp Slow Motion.
Is there any other way?
I can't really speak to 1 or 2, but time scaling audio can be as easy as resampling. If you have RAW/PCM audio sampled at 48 kHz and want to playback at half speed, resample to 96 kHz and play the audio at 48 kHz. Since you have twice the number of samples it will take twice as long to play. Generally:
scaledSampleRate = (orignalSampleRate / playRate);
or
playRate = (originalSampleRate / scaledSampleRate);
This will effect the pitch of the track, however that may be the desired effect since that behavior is somewhat is expected in "slow motion" audio. There are more advanced techniques that preserve pitch while scaling time. The open source software Audacity implements these algorithms. You could find inspiration there. There are many resources on the web that explain the tradeoffs of pitch shifting vs time stretching.
http://en.wikipedia.org/wiki/Audio_time-scale/pitch_modification
http://www.dspdimension.com/admin/time-pitch-overview/
Another option you may not have considered is muting the audio during slow motion. That seems to be the technique employed by most AV playback utilities. However, depending on your use case, distorted audio does indicate time is being manipulated.
I have applied slow motion on complete video including audio this might help You check this link : How to do Slow Motion video in IOS
I have an AVMutableComposition with 2 audio tracks and one video track. I'm using the composition to string about 40 different video clips from .mov files, putting the video content of each clip in the video track of my composition and the audio in the audio track. The second audio track I use for music.
I also have a synchronized layer for titles graphics.
When I play this composition using an AVPlayer, the audio slowly gets out of sync. It takes about 4 minutes to start becoming noticeable. It seems like if I only string together a handfull of longer clips the problem is not as apparent, it is when there are many clips shorter clips (~40 in my test) that it gets really bad.
Pausing and Playing doesn't re-sync the audio, however seeking does. In other words, if I let the video play to the end, towards the end the lip sync gets noticeably off even if I pause and play throughout, however, if I seek to a time towards the end the audio gets back in sync.
My hacky solution for now is to seek to the currentTime + 1 frame every minute or so. This creates an unpleasant jump in the video caused by a lag in the seek operation, so not a good solution.
Exporting with an ExportSession doesn't present this problem, audio remains in sync in the output movie.
I'm wondering if the new masterClock property in the AVPlayer is the answer to this, and if it is, how is it used?
I had the same issue and fixed it, among many other audio and video things, by specifying times timescales in the following manner:
CMTime(seconds: my_seconds, preferredTimescale: CMTimeScale(600))
Before, my time scale was CMTimeScale(NSEC_PER_SEC). That caused me jittery when composing clips at a different frame rate, plus this audio out-of-sync that Eddy mentions here.
In spite of looking like a magic number, 600 is a common multiple of 24, 30, 60 and 120. These are usual frame rates for different purposes. The common multiple avoids dragging around rounding problems when composing multiple clips.