I want to stitch multiple background images provided by ARKit (ARFrame.capturedImage). (I know there are better ways to do this task, but I am using my custom algorithm.)
The issue is that the live stream does not have locked exposure and thus the color of an object in the scene depends on how I orient my iPhone. This, for example, leads to a wall having very different color in each frame (from white through gray to brown-ish), which creates visible banding when stitching the images together.
I noticed ARKit provides lightEstimate for each ARFrame with the ambientIntensity and ambientColorTemperature properties. There is also the ARFrame.camera.exposureOffset property.
Can these properties be used to "normalize" captured images so that colors of the objects in the scene stay roughly the same throughout time and I don't end up with severe banding?
P.S. I do need to use ARKit, otherwise I would set-up my own session based on the AVFoundation API with my own settings (e.g. locked exposure).
Since all mentioned properties are not settable you can't use them directly to fix an intensity of every stitched image in panorama-360.
But you can calculate a difference of intensity and exposure of each frame and then use that multipliers for CoreImage filters. For instance, exposure difference is as simple as that:
Frame_02_Exposure / Frame_01_Exposure = 0.37
Then use the result as input multiplier for CIExposureAdjust filter.
Related
Is there a way to change the appearance (size, color, etc) of the feature points in ARKit easily? (After setting debugOptions in the sceneView to ARSCNDebugOptions.showFeaturePoints I'm thinking I might have to iterate over the rawFeaturePoints and manually add custom objects into the scene at those points.
As its name suggests, ARSCNDebugOptions.showFeaturePoints is a tool to aid in debugging your app. Because the size and color of feature point indicators aren't essential to knowing where feature points are (for the sake of making sure your app is behavior correctly), Apple doesn't offer API to change their appearance. (Any more than they offer APIs for changing the colors of bounding boxes, physics shapes, and other indicators available in SceneKit debug options.)
If you want to create your own visualization for feature points, you'll need to do exactly as you suggest: read the rawFeaturePoints from the current ARFrame and use those to position content in the SceneKit scene. You might do this by creating a bunch of nodes with geometry and setting their positions. You might also look into whether it's easy to pass the entire buffer of points to create an SCNGeometry that renders in point-cloud mode.
I am using the warpGeometry feature of Sprite Kit extensively to display deformed images.
I would also like to use this distortion effect off screen, without instantiating a SKScene and so on, by configuring a corresponding CIFilter. However, I cannot find the corresponding filter among all available ones.
The closest equivalent CIFilter is CIPerspectiveTransform. Alternatively, you could also consider using CIAffineTransform.
I'm trying to recreate the barrel effect that can be seen on the camera mode picker below:
(source: androidnova.org)
Do I have to use OpenGL in order to achieve this effect? What is the best approach?
I found a great library on GitHub that can be used to achieve this effect (https://github.com/Ciechan/BCMeshTransformView), but unfortunately it doesn't support animation and is therefore not usable.
I bet Apple used CGMeshTransform. It's just like BCMeshTransform, except it is a private API and fully integrates with Core Graphics. BCMeshTransformView was born when a developer discovered this.
The only easy option I see is:
Use CALayer.transform, which is a CATransform3D. You can use this to simulate the barrel effect you want by adjusting the z position and y rotation of each item on the barrel. Also add a semitransparent dark gradient (CAGradientLayer) to the wheel to simulate the effect of choices getting darker towards the edges. This will be simple to do, but won't look as smooth and realistic as an actual 3D barrel. Maybe it will look good enough to create a convincing illusion though? (To enable 3D transforms, you need to enable depth by using view.layer.transform.m34 = 1/500.f or similar)
http://www.thinkandbuild.it/introduction-to-3d-drawing-in-core-animation-part-1/
The hardest option is using a custom OpenGL view that makes a barrel shape and applies your contents on top of it as a texture. I would expect that you run into most of the complexities behind creating BCMeshTransformView, and have difficulty supporting animations just like BCMeshTransformView did.
You may still be able to use BCMeshTransformView though. BCMeshTransformView is slow at processing content animations such as color changes, but is very fast at processing geometry changes. So you could use it to do a barrel effect, as long as you define the barrel effect entirely in terms of mesh geometry changes (instead of as content changes like using a scroll view or adjusting subview positions). You would need to do gesture handling + scrolling yourself instead of using UIScrollView though, which is tricky and tedious to get right.
Considering the options, I would want to fudge it by using 3D transforms, then move to other options only if I can't create a convincing illusion using 3D transforms.
I'm trying to achieve the "Ghost" effect from http://webcamtoy.com/ using GPUImage.
My understanding is that it would be a two-input filter, with a given time delay between the two frames used. I'd then just add the two frames with 0.5 alpha each.
I've seen how to use the current and previous frames with GPUImage using GPUImageBuffer (example of that in the GPUImageLowPassFilter) but I'm not sure how to set up a time delay between the two frames I want to use.
Any ideas or pointers? I was thinking of creating a custom filter and overriding newFrameReadyAtTime:atIndex: to delay the propagation downstream for the first x frames (where x is the delay in terms of number of frames). Maybe a clean way to do this would be to subclass GPUImageBuffer to automatically stack x frames before piping them out into a 2-input filter.
Thanks!
I think you're on the right track with keeping old frames. For the color effects, you're looking at something like extracting the color channels, using them as input to combine in a blend filter. The key is that the input's values have to add up to the natural color values in the non-changing portions of the video.
I am creating a custom button that needs to be able to glow to a varying degree
How would I use these pictures to make a button that 'glows' the diamond when it is pressed, and have this glow gradually fade back to inert state?
I want to churn out several different colours of diamond as well... I am hoping to generate all different coloured diamonds from the same stock images presented here.
I would like to get my head around the basic methods available, in enough detail that I can see each one through and make a decision which path to take...
My tangled efforts so far... ( I will delete all of this, or move it into possibly several answers as a solution unfolds... )
I can see 3 potential solution paths:
GL
it looks as though GL has everything it takes to get complete fine-grained control over the process, although functions exposed by core graphics come tantalisingly close, and that would save several hundred lines of code spread over a bunch of source files, which seems a bit ridiculous for such a basic task.
core graphics, and core animation to accomplish the blending
documentation goes on to say
Anything underneath the unpainted samples, such as the current fill color or other drawing, shows through.
so I can chroma-key mask the left image, setting {0,0,0} ie Black as the key.
this at least secures a transparent background, now I have to work on making it yellow instead of grey.
so maybe I could have started instead with setting a yellow back colour for my image context, then use some CGContextSetBlendMode(...) to imprint the diamond on the yellow, THEN use chroma-key masking to get a transparent background
ok, this covers at least getting the basic unlit image on-screen
now I could overlay the sparkly image, using some blend mode, maybe I could keep it in its current greyscale state, and that would just boost the colours of the original
only problem with this is that it is a lot of heavy real-time blending
so maybe I could pre-calculate every image in the animation... this is looking increasingly mucky...
Cocos2D
if this allows me to set the blend mode to additive blending then I could just composite the glowing image over the original image with an appropriate Alpha setting.
After digging through a lot of documentation, the optimal solution seems to be to use core graphics functions to get the source images into a single 2-component GL texture, and then use GL to blend between them.
I will need to pass a uniform value glow_factor into the shader
The obvious solution might seem to simply use
r,g,b = in_r,g,b * { (1 - glow_factor) * inertPixel + glow_factor * shinyPixel }
(where inertPixel is the appropriate pixel of the inert diamond etc)...
it looks like I would also do well to manufacture my own sparkles and add them over the top; a gem should sparkle white irrespective of its characteristic colour.
After having looked at this problem a little more, I can see several solutions
Solution A -- store the transition from glow=0 to glow=1 as 60 frames in memory, then load the appropriate frame into a GL texture every time it is required.
this has an obvious benefit that a graphic designer could construct the entire sequence and I could load it in as a bunch of PNG files.
another advantage is that these frames wouldn't need to be played in sequence... the appropriate frame can be chosen on-the-fly
however, it has a potential drawback of a lot of sending data RAM->VRAM
this can be optimised by using glTexSubImage2D; several frames can be sent simultaneously and then unpacked from within GL... in fact maybe the entire sequence. if this is so, then it would make sense to use PVRT texture compression.
iOS: playing a frame-by-frame greyscale animation in a custom colour
Solution B -- load glow=0 and glow=1 images as GL textures, and manually write shader code that takes in the glow factor as a uniform and performs the blend
this has an advantage that it is close to the wire and can be tweaked in all sorts of ways. Also it is going to be very efficient. This advantage is that it is a big extra slice of code to maintain.
Solution C -- set glBlendMode to perform additive blending.
then draw the glow=0 image image, setting eg alpha=0.2 on each vertex.
then draw the glow=1 image image, setting eg alpha=0.8 on each vertex.
this has an advantage that it can be achieved with a more generic code structure -- ie a very general ' draw textured quad / sprite ' class.
disadvantage is that without some sort of wrapper it is a bit messy... in my game I have a couple of dozen diamonds -- at any one time maybe 2 or 3 are likely to be glowing. so first-pass I would render EVERYTHING ( just need to set Alpha appropriately for everything that is glowing ) and then on the second pass I could draw the glowing sprite again with appropriate Alpha for everything that IS glowing.
it is worth noting that if I pursue solution A, this would involve creating some sort of real-time movie player object, which could be a very useful reusable code component.