How to draw into CIImage (or maybe into CVPixelBuffer, but I guess it easier to add text to CIImage)? not to UIImage
I record video (.mp4 file) using AVAssetWriter and CMSampleBuffer (from video, audio inputs). While recording I want to add text on the video frames, I'm already converting CMSampleBuffer to CIImage, now I need somehow add text to CIImage and convert back to CVPixelBuffer. I didn't really find any simple examples in Swift how to add (draw) text to CIImage or add anything else (like lines, rects, not just text).
/// encode video buffer
func appendVideoSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
// before encoding video frames into a file I would like process it
// 1. convert CMSampleBuffer to ciImage
let cvPixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let ciimage : CIImage = CIImage(cvPixelBuffer: cvPixelBuffer)
// 2. do something with CIImage
// 3. convert CIImage back to CMSampleBuffer
let sampleBufferWithText == ...
appendSampleBuffer(sampleBufferWithText, ofMediaType: AVMediaType.video.rawValue)
}
/// encode audio buffer
func appendAudioSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
appendSampleBuffer(sampleBuffer, ofMediaType: AVMediaType.audio.rawValue)
}
You can create NSAttributedString for text -> draw this string in to cgContext -> convert UIImage/CGImage -> CIImage
Related
I am trying to modify some sample code from Apple Developer codes for my own purpose (I am very new to programming for iOS). I am trying to get images from a camera and apply some detection and just show the detections.
Currently, I am using the AVCaptureVideoPreviewLayer and basically the camera feed gets displayed on the screen. I actually want to zero out the camera feed and draw some detections only. So, I am basically trying to handle this in the captureOutputfunction. So something like:
extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
// Grab the pixelbuffer frame from the camera output
guard let pixelBuffer = sampleBuffer.imageBuffer else { return }
// Here now I should be able to set it 0s (all black)
}
}
I am trying to do something basic like setting this CVImageBuffer to a black background but have not been able to figure that out in the last hours!
EDIT
So, I discovered that I can do something like:
var image: CGImage?
// Create a Core Graphics bitmap image from the buffer.
VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &image)
This copies the buffer data to the CGImage, which I can then use for my purposes. Now, is there an API that can basically make an all black image with the same size as one represented by the input image buffer?
While trying to apply a simple vignette filter to the raw camera feed of an iPhone6, with the help of Metal and Core Image, I see a lot of lag between the frames being processed and rendered in an MTKView
The approach which I have followed is (MetalViewController.swift):
Get raw camera output using AVCaptureVideoDataOutputSampleBufferDelegate
Convert CMSampleBuffer > CVPixelBuffer > CGImage
Create an MTLTexture with this CGImage.
Point no. 2 and 3 are inside the method named: fillMTLTextureToStoreTheImageData
Apply a CIFilter to the CIImage fetched from the MTLTexture in the MTKViewDelegate
func draw(in view: MTKView) {
if let currentDrawable = view.currentDrawable {
let commandBuffer = self.commandQueue.makeCommandBuffer()
if let myTexture = self.sourceTexture{
let inputImage = CIImage(mtlTexture: myTexture, options: nil)
self.vignetteEffect.setValue(inputImage, forKey: kCIInputImageKey)
self.coreImageContext.render(self.vignetteEffect.outputImage!, to: currentDrawable.texture, commandBuffer: commandBuffer, bounds: inputImage!.extent, colorSpace: self.colorSpace)
commandBuffer?.present(currentDrawable)
commandBuffer?.commit()
}
}
}
The performance is not at all what Apple mentioned in this doc: https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_tasks/ci_tasks.html#//apple_ref/doc/uid/TP30001185-CH3-TPXREF101
Am I missing something?
Your step 2 is way too slow to support real-time rendering... and it looks like you're missing a couple of steps. For your purpose, you would typically:
Setup:
create a pool of CVPixelBuffer - using CVPixelBufferPoolCreate
create a pool of metal textures using CVMetalTextureCacheCreate
For each frame:
convert CMSampleBuffer > CVPixelBuffer > CIImage
Pass that CIImage through your filter pipeline
render the output image into a CVPixelBuffer from the pool created in step 1
use CVMetalTextureCacheCreateTextureFromImage to create a metal texture with your filtered CVPixelBuffer
If setup correctly, all these steps will make sure your image data stays on the GPU, as opposed to travelling from GPU to CPU and back to GPU for display.
The good news is all this is demoed in the AVCamPhotoFilter sample code from Apple https://developer.apple.com/library/archive/samplecode/AVCamPhotoFilter/Introduction/Intro.html#//apple_ref/doc/uid/TP40017556. In particular see the RosyCIRenderer class and its superclass FilterRenderer.
So I'm trying to get a jpeg/png representation of the grayscale depth maps that are typically used in iOS image depth examples. The depth data is stored in each jpeg as aux data. I've followed some tutorials and I have no problem rendering this gray scale data to the screen, but I can find no way to actually save it as a jpeg/png representation. I'm pretty much using this code: https://www.raywenderlich.com/168312/image-depth-maps-tutorial-ios-getting-started
The depth data is put into a cvpixelbuffer and manipulated accordingly. I believe it's in the format kCVPixelFormatType_DisparityFloat32.
While I'm able to see this data represented accordingly on the screen, I'm unable to use UIImagePNGRepresentation or UIImageJPGRepresentation. Sure, I could manually capture a screenshot, but that's not really ideal.
I have a suspicion that the cvpixelbuffer data format is not compatible with these UIImage functions and that's why I can't get them to spit out an image.
Does anyone have any suggestions?
// CVPixelBuffer to UIImage
let ciImageDepth = CIImage(cvPixelBuffer: cvPixelBufferDepth)
let contextDepth:CIContext = CIContext.init(options: nil)
let cgImageDepth:CGImage = contextDepth.createCGImage(ciImageDepth, from: ciImageDepth.extent)!
let uiImageDepth:UIImage = UIImage(cgImage: cgImageDepth, scale: 1, orientation: UIImage.Orientation.up)
// Save UIImage to Photos Album
UIImageWriteToSavedPhotosAlbum(uiImageDepth, nil, nil, nil)
I figured it out. I had to convert to a CGImage first. Ended up doing cvPixelBuffer to CIImage to CGImage to UIImage.
Posting a swift code sample in case anyone wants to use it.
let ciimage = CIImage(cvPixelBuffer: depthBuffer) // depth cvPixelBuffer
let depthUIImage = UIImage(ciImage: ciimage)
What is the best way to record a video with augmented reality? (adding text, images logo to frames from iPhone/iPad camera)
Previously I was trying to figure out how to draw into CIImage (How to draw text into CIImage?) and convert CIImage back to CMSampleBuffer (CIImage back to CMSampleBuffer)
I almost did everything, only have problem with recording video using new CMSampleBuffer in AVAssetWriterInput
But this solution anyway isn't good at all, it eats a lot of CPU while converting CIImage to CVPixelBuffer (ciContext.render(ciImage!, to: aBuffer))
So I want to stop here and find some other ways to record a video with augmented reality (for example dynamically add (draw) text inside frames while encoding video into mp4 file)
Here what I've tried and don't want to use anymore...
// convert original CMSampleBuffer to CIImage,
// combine multiple `CIImage`s into one (adding augmented reality -
// text or some additional images)
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let ciimage : CIImage = CIImage(cvPixelBuffer: pixelBuffer)
var outputImage: CIImage?
let images : Array<CIImage> = [ciimage, ciimageSec!] // add all your CIImages that you'd like to combine
for image in images {
outputImage = outputImage == nil ? image : image.composited(over: outputImage!)
}
// allocate this class variable once
if pixelBufferNew == nil {
CVPixelBufferCreate(kCFAllocatorSystemDefault, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer), kCVPixelFormatType_32BGRA, nil, &pixelBufferNew)
}
// convert CIImage to CVPixelBuffer
let ciContext = CIContext(options: nil)
if let aBuffer = pixelBufferNew {
ciContext.render(outputImage!, to: aBuffer) // >>> IT EATS A LOT OF <<< CPU
}
// convert new CVPixelBuffer to new CMSampleBuffer
var sampleTime = CMSampleTimingInfo()
sampleTime.duration = CMSampleBufferGetDuration(sampleBuffer)
sampleTime.presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
sampleTime.decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer)
var videoInfo: CMVideoFormatDescription? = nil
CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBufferNew!, &videoInfo)
var oBuf: CMSampleBuffer?
CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, pixelBufferNew!, true, nil, nil, videoInfo!, &sampleTime, &oBuf)
/*
try to append new CMSampleBuffer into a file (.mp4) using
AVAssetWriter & AVAssetWriterInput... (I met errors with it, original buffer works ok
- "from func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection)")
*/*
Is there any better solution?
now I answer my own question
the best would be to use Objective-C++ class (.mm) where we can use OpenCV and easily/fast convert from CMSampleBuffer to cv::Mat and back to CMSampleBuffer after processing
we can easily call Objective-C++ functions from Swift
I am converting the CMSampleBuffer argument in the captureOutput function of my AVCaptureVideoDataOuput delegate into a MTLTexture like so (side note, I have set the pixel format of the video output to kCVPixelFormatType_32BGRA):
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
var outTexture: CVMetalTexture? = nil
var textCache : CVMetalTextureCache?
CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice, nil, &textCache)
var textureRef : CVMetalTexture?
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textCache!, imageBuffer, nil, MTLPixelFormat.bgra8Unorm, width, height, 0, &textureRef)
let texture = CVMetalTextureGetTexture(textureRef!)!
print(texture.bufferBytesPerRow)
}
The issue is when I print the bytes per row of the texture, it always prints 0, which is problematic because I later try to convert the texture back into a UIImage using the methodology in this article: https://www.invasivecode.com/weblog/metal-image-processing. Why is the texture I receive seemingly empty? I know the CMSampleBuffer property is fine because I can convert it into a UIIMage and draw it like so:
let myPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let myCIimage = CIImage(cvPixelBuffer: myPixelBuffer!)
let image = UIImage(ciImage: myCIimage)
self.imageView.image = image
The bufferBytesPerRow property is only meaningful for a texture that was created using the makeTexture(descriptor:offset:bytesPerRow:) method of a MTLBuffer. As you can see, the bytes-per-row is an input to that method to tell Metal how to interpret the data in the buffer. (The texture descriptor provides additional information, too, of course.) This method is only a means to get that back out.
Note that textures created from buffers can also report which buffer they were created from and the offset supplied to the above method.
Textures created in other ways don't have that information. These textures have no intrinsic bytes-per-row. Their data is not necessarily organized internally in a simple raster buffer.
If/when you want to get the data from a texture to either a Metal buffer or a plain old byte array, you have the freedom to choose a bytes-per-row value that's useful for your purposes, so long as it's at least the bytes-per-pixel of the texture pixel format times the texture's width. (It's more complicated for compressed formats.) The docs for getBytes(_:bytesPerRow:from:mipmapLevel:) and copy(from:sourceSlice:sourceLevel:sourceOrigin:sourceSize:to:destinationOffset:destinationBytesPerRow:destinationBytesPerImage:) explain further.