I have used the following method iOS4: how do I use video file as an OpenGL texture? to get video frames rendering in openGL successfully.
This method however seems to fall down when you want to scrub (jump to a certain point in the playback) as it only supplies you with video frames sequentially.
Does anyone know a way this behaviour can successfully be achieved?
One easy way to implement this is to export the video to a series of frames, store each frame as a PNG, and then "scrub" by seeing to a PNG at a specific offset. That gives you random access in the image stream at the cost of decoding the entire video first and holding all the data on disk. This would also involve decoding each frame as it is accessed, that would eat up CPU but modern iPhones and iPads can handle it as long as you are not doing too much else.
Related
I'm trying to create a camera capture session that has the following features:
Preview
Photo capture
Video capture
Realtime frame processing (AI)
While the first two things are not a problem, I haven't found a way to make the last two separately.
Currently, I use a single AVCaptureVideoDataOutput and run the video recording first, then the frame processing in the same function, in the same queue. (see code here and here)
The only problem with this is that the video capture captures 4k video, and I don't really want the frame processor to receive 4k buffers as that is going to be very slow and blocks the video recording (frame drops).
Ideally I want to create one AVCaptureDataOutput for 4k video recording, and another one that receives frames in a lower (preview?) resolution - but you cannot use two AVCaptureDataOutputs in the same capture session.
I thought maybe I could "hook into" the Preview layer to receive the CMSampleBuffers from there, just like the captureOutput(...) func, since those are in preview-sized resolutions, does anyone know if that is somehow possible?
For such thing I recommended implement custom renderer flow.
You need just one AVCaptureDataOutput without system PreviewLayer which provided by iOS.
Setup Color scheme to YUV (it's more compact then BGRA)
Get CMSampleBuffer in AVCaptureDataOutput
Send CMSampleBuffer to Metal texture.
Create resized low resolution texture in Metal
Hi resolution texture send to Renderer in draw it in MTKLView
Low resolution texture send to CVPixelBuffer, then you can convert it to CGImage, CGImage, Data.
Send low resolution image to Neural network
I have article on Medium: link. You can use it as some example.
I'm trying to decode a video in real time (30 fps) and display /modify it with OpenGL. On an iPod touch, if I decode a video that I took with the camera, decoding a frame can take over 1s, while 30 fps should be 0.03s max. Thus the result is not very good..
Is it possible to achieve that with AVAssetReader ? For example Instagram applies filters (I think GLSL shaders) in real time on a video, and they can even navigate in the video. Instagram works fine on the ipod touch.
The code to decode can be found in the answer here :
Best way to access all movie frames in iOS
And more specifically here : Hardware accelerated h.264 decoding to texture, overlay or similar in iOS
Thank you in advance
Due to the very limited information you provided, I have to assume that your video sequence are compressed in the format of YUV and you set settings of AVAssetReader with other format like kCVPixelFormatType_32BGRA which forces iOS use hardware acceleration to convert colour space for you, then you feel it slowly. I suggest that no settings to set, just use its original pixel format.
Actually my app was just doing too much work on the CPU, I had another process analyzing images. When I removed it, the decoding was really fast.
I am using GPUImage's GPUImageVideoCamera initWithSessionPreset:cameraPosition: in order to display video from the rear facing camera on an iOS device (targeting iOS 7). This is filtered and displayed on a GPUImageView. Will not exceed AVCaptureSessionPreset640x480.
At any given moment in the app, I need to recall the past 5 seconds of unfiltered video captured from the rear-facing camera and instantly play this back on another (or the same) GPUImageView.
I can access CMSampleBufferRef via GPUImageVideoCamera's willOutputSampleBuffer: which is passed through from but I'm not sure how one goes about getting the most recent frames into memory in an efficient way such that they can be instantly, seamlessly played back.
I believe the solution is a Circular Buffer using something like TPCircularBuffer but I'm not sure that will work with a video stream. Also wanted to reference unanswered Buffering CMSampleBufferRef into a CFArray and Hold multiple Frames in Memory before sending them to AVAssetWriter as they closely resembled my original plan of attack until I started researching this.
I'm looking for a tips to develop an application for iPhone/iPad that will be able to process video (let's consider only local files stored on the device for simplicity) and play it in real-time. For example you can choose any movie and choose "Old movie" filter and want it like on old lamp TV.
In order to make this idea real i need to implement two key features:
1) Grab frames and audio stream from a movie file and get access to separate frames (I'm interested in raw pixel buffer in BGRA or at least YUV color space).
2) Display processed frames somehow. I know it's possible to render processed frame to OpenGL texture, but i would like to have more powerful component with playback controls. Is there any class of media player that supports playing custom image and audio buffers?
The processing function is done and it's fast (less than duration on one frame).
I'm not asking for ready solution, but any tips are welcome!
Answer
Frame grabbing.
It seems the only way to grab video and audio frames is to use AVAssetReader class. Although it's not recommended to use for real-time grabbing it does the job. In my tests on iPad2 grabbing single frame needs about 7-8 ms. Seeking across the video is a tricky. Maybe someone can point to more efficient solution?
Video playback. I've done this with custom view and GLES to render a rectangle texture with a video frame inside of it. As far as i know it's the fastest way to draw bitmaps.
Problems
Need to manually play a sound samples
AVAssetReader grabbing should be synchronized with a movie frame rate. Otherwise movie will go too fast or too slow.
AVAssetReader allows only continuous frame access. You can't seek forward and backward. Only proposed solution is to delete old reader and create a new with trimmed time range.
This is how you would load a video from the camera roll..
This is a way to start processing video. Brad Larson did a great job..
How to grab video frames..
You can use AVPlayer+ AVPlayerItem, it provide you a chance to apply a filter on the display image.
I'm trying to display the contents of a video file (let's just say without the audio for now) onto a UV mapped 3D object in OpenGL. I've done a fair bit in OpenGL but have no idea where to begin in video file handling, and most of the examples out there seems to be for getting video frames from cameras, which is not what I'm after.
At the moment I feel if I can get individual frames of the video as CGImageRef I'd be set, so I'm wondering how to do this? Perhaps there are even be better ways to do this? Where should I start and what's the most straight forward file format for video playback on iOS? .mov?
Apologies; typing on an iPhone so I'll be a little brief.
Create an AVURLAsset with the URL of your video - which can be a local file URL if you like. Anything QuickTime can do is fine, so MOV or M4V in H.264 is probably the best source.
Query the asset for tracks of type AVMediaTypeVideo. You should get just one unless your source video has multiple camera angles of something like that, so just taking objectAtIndex:0 should give you the AVAssetTrack you want.
Use that to create an AVAssetReaderTrackOutput. Probably you want to specify kCVPixelFormatType_32BGRA.
Create an AVAssetReader using the asset; attach the asset reader track output as an output. And call startReading.
Henceforth you can call copyNextSampleBuffer on the track output to get new CMSampleBuffers, putting you in the same position as if you were taking input from the camera. So you can lock that to get at pixel contents and push those to OpenGL via Apple's BGRA extension.
You're probably going to have to use a player layer and flatten its contents into a bitmap context. See the documentation for AVPlayerLayer. The performance might be very poor though.