I would like to use ARKit to obtain a light estimate from an image. I was able to retrieve a light estimate from a frames in a video.
var SceneView = new ARSCNView();
var arConfig = new ARKit.ARWorldTrackingConfiguration { PlaneDetection = ARPlaneDetection.Horizontal };
SceneView.Session.Run(arConfig, ARSessionRunOptions.ResetTracking);
var frame = SceneView.Session.CurrentFrame;
float light = frame.LightEstimate.AmbientIntensity;
However is it possible to instantiate an ARFrame using a CGImage?
Like
CGImage img = new CGImage("my file.jpg");
ARFrame frame = new ARFrame(img);
float light = frame.LightEstimate.AmbientIntensity;
Solutions using swift or Xamarin are welcome
Sorry, but ARFrame wraps CVPixelBuffer, which represents a video frame and depending on the device is likely in a different format than CGImage. Also ARFrame has no public initializer and the var capturedImage: CVPixelBuffer property is read only. However if you are getting the CGImage from the camera then why no get the light estimate at the time of capture and save it along with the image?
Related
Within our iOS app, we are using custom filters using Metal (CIKernel/CIColorKernel wrappers).
Let's assume we have a 4K video and a custom video composition with a 1080p output size, that applies an advanced filter on the video buffers.
Obviously, we don't need to filter the video in its original size, doing so we'll probably terminate the app with a memory warning (true story).
This is the video-filtering pipeline:
Getting the buffer in 4K (as CIImage) -->
Apply filter on the CIImage -->
the filter applies the CIKernel Metal filter function on the CIImage-->
Return the filtered CIImage to the composition
The only two places I can think of applying the resize is before we send it into the filter process or within the Metal function.
public class VHSFilter: CIFilter {
public override var outputImage: CIImage? {
// InputImage size is 4K
guard let inputImage = self.inputImage else { return nil }
// Manipulate the image here
let roiCallback: CIKernelROICallback = { _, rect -> CGRect in
return inputImage.extent
}
// Or inside the Kernel Metal function
let outputImage = self.kernel.apply(extent: inputExtent,
roiCallback: roiCallback,
arguments: [inputImage])
return outputImage
}
}
I'm sure I'm not the first one to encounter this issue
What does one do when the incoming video-buffer are too large (memory-wise) to filter, and they need to resize on-the-fly efficiently? Without re-encoding the video before?
As warrenm says, you could use a CILanczosScaleTransform filter to downsample the video frames before processing. However, this would still cause AVFoundation to allocate buffers in full resolution.
I assume you use a AVMutableVideoComposition to do the filtering? In this case you can just set the renderSize of the composition to the target size. From the docs:
The size at which the video composition should render.
This will tell AVFoundation to resample the frames (efficiently, fast) before handing them to your filter pipeline.
While trying to apply a simple vignette filter to the raw camera feed of an iPhone6, with the help of Metal and Core Image, I see a lot of lag between the frames being processed and rendered in an MTKView
The approach which I have followed is (MetalViewController.swift):
Get raw camera output using AVCaptureVideoDataOutputSampleBufferDelegate
Convert CMSampleBuffer > CVPixelBuffer > CGImage
Create an MTLTexture with this CGImage.
Point no. 2 and 3 are inside the method named: fillMTLTextureToStoreTheImageData
Apply a CIFilter to the CIImage fetched from the MTLTexture in the MTKViewDelegate
func draw(in view: MTKView) {
if let currentDrawable = view.currentDrawable {
let commandBuffer = self.commandQueue.makeCommandBuffer()
if let myTexture = self.sourceTexture{
let inputImage = CIImage(mtlTexture: myTexture, options: nil)
self.vignetteEffect.setValue(inputImage, forKey: kCIInputImageKey)
self.coreImageContext.render(self.vignetteEffect.outputImage!, to: currentDrawable.texture, commandBuffer: commandBuffer, bounds: inputImage!.extent, colorSpace: self.colorSpace)
commandBuffer?.present(currentDrawable)
commandBuffer?.commit()
}
}
}
The performance is not at all what Apple mentioned in this doc: https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_tasks/ci_tasks.html#//apple_ref/doc/uid/TP30001185-CH3-TPXREF101
Am I missing something?
Your step 2 is way too slow to support real-time rendering... and it looks like you're missing a couple of steps. For your purpose, you would typically:
Setup:
create a pool of CVPixelBuffer - using CVPixelBufferPoolCreate
create a pool of metal textures using CVMetalTextureCacheCreate
For each frame:
convert CMSampleBuffer > CVPixelBuffer > CIImage
Pass that CIImage through your filter pipeline
render the output image into a CVPixelBuffer from the pool created in step 1
use CVMetalTextureCacheCreateTextureFromImage to create a metal texture with your filtered CVPixelBuffer
If setup correctly, all these steps will make sure your image data stays on the GPU, as opposed to travelling from GPU to CPU and back to GPU for display.
The good news is all this is demoed in the AVCamPhotoFilter sample code from Apple https://developer.apple.com/library/archive/samplecode/AVCamPhotoFilter/Introduction/Intro.html#//apple_ref/doc/uid/TP40017556. In particular see the RosyCIRenderer class and its superclass FilterRenderer.
I am converting the CMSampleBuffer argument in the captureOutput function of my AVCaptureVideoDataOuput delegate into a MTLTexture like so (side note, I have set the pixel format of the video output to kCVPixelFormatType_32BGRA):
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
var outTexture: CVMetalTexture? = nil
var textCache : CVMetalTextureCache?
CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice, nil, &textCache)
var textureRef : CVMetalTexture?
CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textCache!, imageBuffer, nil, MTLPixelFormat.bgra8Unorm, width, height, 0, &textureRef)
let texture = CVMetalTextureGetTexture(textureRef!)!
print(texture.bufferBytesPerRow)
}
The issue is when I print the bytes per row of the texture, it always prints 0, which is problematic because I later try to convert the texture back into a UIImage using the methodology in this article: https://www.invasivecode.com/weblog/metal-image-processing. Why is the texture I receive seemingly empty? I know the CMSampleBuffer property is fine because I can convert it into a UIIMage and draw it like so:
let myPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
let myCIimage = CIImage(cvPixelBuffer: myPixelBuffer!)
let image = UIImage(ciImage: myCIimage)
self.imageView.image = image
The bufferBytesPerRow property is only meaningful for a texture that was created using the makeTexture(descriptor:offset:bytesPerRow:) method of a MTLBuffer. As you can see, the bytes-per-row is an input to that method to tell Metal how to interpret the data in the buffer. (The texture descriptor provides additional information, too, of course.) This method is only a means to get that back out.
Note that textures created from buffers can also report which buffer they were created from and the offset supplied to the above method.
Textures created in other ways don't have that information. These textures have no intrinsic bytes-per-row. Their data is not necessarily organized internally in a simple raster buffer.
If/when you want to get the data from a texture to either a Metal buffer or a plain old byte array, you have the freedom to choose a bytes-per-row value that's useful for your purposes, so long as it's at least the bytes-per-pixel of the texture pixel format times the texture's width. (It's more complicated for compressed formats.) The docs for getBytes(_:bytesPerRow:from:mipmapLevel:) and copy(from:sourceSlice:sourceLevel:sourceOrigin:sourceSize:to:destinationOffset:destinationBytesPerRow:destinationBytesPerImage:) explain further.
I want to manipulate 2D textures in a 3D SceneKit scene.
Therefore i used this code to get local coordinates:
#IBAction func tap(sender: UITapGestureRecognizer) {
var arr:NSArray = my3dView.hitTest(sender.locationInView(my3dView), options: NSDictionary(dictionary: [SCNHitTestFirstFoundOnlyKey:true]))
var res:SCNHitTestResult = arr.firstObject as SCNHitTestResult
var vect:SCNVector3 = res.localCoordinates}
I have the texture read out from my scene with:
var mat:SCNNode = myscene.rootNode.childNodes[0] as SCNNode
var child:SCNNode = mat.childNodeWithName("ID12", recursively: false)
var geo:SCNMaterial = child.geometry.firstMaterial
var channel = geo.diffuse.mappingChannel
var textureimg:UIImage = geo.diffuse.contents as UIImage
and now i want to draw at the touchpoint to the texture...
how can i do that? how can i transform my coordinate from touch to the texture image?
Sounds like you have two problems. (Without even having used regular expressions. :))
First, you need to get the texture coordinates of the tapped point -- that is, the point in 2D texture space on the surface of the object. You've almost got that right already. SCNHitTestResult provides those with the textureCoordinatesWithMappingChannel method. (You're using localCoordinates, which gets you a point in the 3D space owned by the node in the hit-test result.) And you already seem to have found the business about mapping channels, so you know what to pass to that method.
Problem #2 is how to draw.
You're doing the right thing to get the material's contents as a UIImage. Once you've got that, you could look into drawing with UIGraphics and CGContext functions -- create an image with UIGraphicsBeginImageContext, draw the existing image into it, then draw whatever new content you want to add at the tapped point. After that, you can get the image you were drawing with UIGraphicsGetImageFromCurrentImageContext and set it as the new diffuse.contents of your material. However, that's probably not the best way -- you're schlepping a bunch of image data around on the CPU, and the code is a bit unwieldy, too.
A better approach might be to take advantage of the integration between SceneKit and SpriteKit. This way, all your 2D drawing is happening in the same GPU context as the 3D drawing -- and the code's a bit simpler.
You can set your material's diffuse.contents to a SpriteKit scene. (To use the UIImage you currently have for that texture, just stick it on an SKSpriteNode that fills the scene.) Once you have the texture coordinates, you can add a sprite to the scene at that point.
var nodeToDrawOn: SCNNode!
var skScene: SKScene!
func mySetup() { // or viewDidLoad, or wherever you do setup
// whatever else you're doing for setup, plus:
// 1. remember which node we want to draw on
nodeToDrawOn = myScene.rootNode.childNodeWithName("ID12", recursively: true)
// 2. set up that node's texture as a SpriteKit scene
let currentImage = nodeToDrawOn.geometry!.firstMaterial!.diffuse.contents as UIImage
skScene = SKScene(size: currentImage.size)
nodeToDrawOn.geometry!.firstMaterial!.diffuse.contents = skScene
// 3. put the currentImage into a background sprite for the skScene
let background = SKSpriteNode(texture: SKTexture(image: currentImage))
background.position = CGPoint(x: skScene.frame.midX, y: skScene.frame.midY)
skScene.addChild(background)
}
#IBAction func tap(sender: UITapGestureRecognizer) {
let results = my3dView.hitTest(sender.locationInView(my3dView), options: [SCNHitTestFirstFoundOnlyKey: true]) as [SCNHitTestResult]
if let result = results.first {
if result.node === nodeToDrawOn {
// 1. get the texture coordinates
let channel = nodeToDrawOn.geometry!.firstMaterial!.diffuse.mappingChannel
let texcoord = result.textureCoordinatesWithMappingChannel(channel)
// 2. place a sprite there
let sprite = SKSpriteNode(color: SKColor.greenColor(), size: CGSize(width: 10, height: 10))
// scale coords: texcoords go 0.0-1.0, skScene space is is pixels
sprite.position.x = texcoord.x * skScene.size.width
sprite.position.y = texcoord.y * skScene.size.height
skScene.addChild(sprite)
}
}
}
For more details on the SpriteKit approach (in Objective-C) see the SceneKit State of the Union Demo from WWDC14. That shows a SpriteKit scene used as the texture map for a torus, with spheres of paint getting thrown at it -- whenever a sphere collides with the torus, it gets a SCNHitTestResult and uses its texcoords to create a paint splatter in the SpriteKit scene.
Finally, some Swift style comments on your code (unrelated to the question and answer):
Use let instead of var wherever you don't need to reassign a value, and the optimizer will make your code go faster.
Explicit type annotations (res: SCNHitTestResult) are rarely necessary.
Swift dictionaries are bridged to NSDictionary, so you can pass them directly to an API that takes NSDictionary.
Casting to a Swift typed array (hitTest(...) as [SCNHitTestResult]) saves you from having to cast the contents.
Using Flashdevelop, I managed to add a a jpg in my starling project:
[Embed(source = "../../../../lib/table_org_img_retouched_900.png")]
private static const Graphic:Class;
...
// create a Bitmap object out of the embedded image
var sausageBitmap:Bitmap = new Sausage();
// create a Texture object to feed the Image object
var texture:Texture = Texture.fromBitmap(sausageBitmap);
// create a Image object with our one texture
var image:Image = new Image(texture);
//image.width = 1000;
// show it
addChild(image);
what I get at the end is this:
https://www.dropbox.com/s/wvqws29tg3sxwzv/starling.png
Why is my png cut off?
It is possible that when you started starling your stage size was not accurate ,
_starling = new Starling(Game, stage);
_starling.start();
I suggest you trace the size of the stage you pass to Starling in the creation, if it is not aligned with your device size then you should delay a bit the creation of Starling.