I want to make an application filtering videos like Apple's photo booth app
How can I make that??
Using OpenCV, OpenGL ES or anything else?
OpenCV and OpenGL have very different purposes:
OpenCV is a cross-platform computer vision library. It allows you to easily work with image and video files and presents several tools and methods to handle them and execute filters and several other image processing techniques and some more cool stuff in images.
OpenGL is a cross-platform API to produce 2D/3D computer graphics. It is used to draw complex three-dimensional scenes from simple primitives.
If you want to perform cool effects on images OpenCV is the way to go since it provides tools/effects that can be easily used together to achieve the desired effect you are looking for. And this approach doesn't stop you from processing the image with OpenCV and then render the result in a OpenGL window (if you have to). Remember, they have different purposes and every now and then somebody uses them together.
The point is that the effects you want to perform in the image should be done with OpenCV or any other image processing library.
Actually karlphillip, all though what you have said is correct, OpenGL can also be used to perform hardware accelerated image processing.
Apple even has an OpenGL sample project called GLImageProcessing that has hw accelerated brightness, contrast, saturation, hue and sharpness.
Related
Is it possible to import a virtual lamp object into the AR scene, that projects a light cone, which illuminates the surrounding space in the room and the real objects in it, e.g. a table, floor, walls?
For ARKit, I found this SO post.
For ARCore, there is an example of relighting technique. And this source code.
I have also been suggested that post-processing can be used to brighten the whole scene.
However, these examples are from a while ago and perhaps threre is a newer or a more straight forward solution to this problem?
At the low level, RealityKit is only responsible for rendering virtual objects and overlaying them on top of the camera frame.
If you want to illuminate the real scene, you need to post-process the camera frame.
Here are some tutorials on how to do post-processing:
Tutorial1⃣️
Tutorial2⃣️
If all you need is an effect like This , then all you need to do is add a CGImage-based post-processing effect for the virtual object (lights).
More specifically, add a bloom filter to the rendered image(You can also simulate bloom filters with Gaussian blur).
In this way, the code is all around UIImage and CGImage, so it's pretty simple😎
If you want to be more realistic, consider using the depth map provided by LiDAR to calculate which areas can be illuminated for a more detailed brightness.
Or If you're a true explorer, you can use Metal to create a real world Digital Twin point cloud in real time to simulate occlusion of light.
There's nothing new in relighting techniques based on 3D compositing principles in 2021. At the moment, when you're working with RealityKit or SceneKit, you have to personally implement the relighting functionality with the help of two additional render passes (RGB pass is always needed) - Normals pass and PointPosition pass. Both AOVs must be 32-bit.
However, in the near future, when Apple engineers finally implement texture capturing in Scene Reconstruction – any inexperienced AR developer will be able to apply a relighting procedure.
Watch this Vimeo Video to find out how relighting can be achieved in The Foundry NUKE.
A crucial point here, when implementing the Relighting effect, is the presence of a LiDAR scanner (or iToF sensor if you're using ARCore). In other words, today's relighting solution for iOS is Metal + RealityKit.
I am not an ios developer but my client wants me to make an iphone app like
https://itunes.apple.com/us/app/trippy-booth-amazing-filterswarps/id448037560?mt=8
I have seen some custom library like
https://github.com/BradLarson/GPUImage
but do not find any camera lens customization example.
any kind of suggestions would be helpful
Thanks in advance
You can do it through some custom shader written in OpenGL(or metal just for iOS), then you can apply your shader to do interesting stuff like the image in above link.
I suggest you take a look at how to use the OpenGL framework in iOS.
Basically the flow would like:
Use whatever framework to capture(even in real time) a image.
Use some framework to modify the image. (The magic occur here)
Use another stuff to present the image.
You should learn how to obtain a OpenGL context, draw a image on it, write a custom shader, apply the shader, get the output, to "distort the image". For real, the hardest part is how to create that "effect" in your mind by describing it using a formula.
This is quite similar to the photoshop mesh warp (Edit->Transform->Warp). Basically you treat your image as a texture and then you render it on to a mesh (Bezier Patch) that is a grid that has been distorted into bezier curves, but you leave the texture coordinates as if it was still a grid. This has the effect of "pulling" the image towards the nodes of the patch. You can use OpenGL (GL_PATCHES) for this; I imagine metal or sceneKit might work as well.
I can't tell from the screen shots but its possible that the examples you reference are actually placing their mesh based on facial recognition. CoreImage has basic facial recognition to give youth out and eye positions which you could use to control some of the nodes in your mesh.
What are the major differences between the Core Image and GPUImage frameworks (besides GPUImage being open source)? At a glance their interfaces seem pretty similar... Applying a series of filters to an input to create an output. I see a few small differences, such as the easy to use LookupFilter that GPUImage has. I am trying to figure out why someone would choose one over the other for a photo filtering application.
As the author of GPUImage, you may want to take what I say with a grain of salt. I should first say that I have a tremendous amount of respect for the Core Image team and how they continue to update the framework. I was a heavy Core Image user before I wrote GPUImage, and I patterned many of its design elements based on how Core Image worked on the Mac.
Both frameworks are constantly evolving, so a comparison made today might not be true in a few months. I can point to current capabilities and benchmarks, but there's no guarantee that won't flip when either of us update things.
My philosophy with GPUImage was to create a lightweight wrapper around OpenGL (ES) quads rendered with shaders, and to do so with as simple an interface as possible. As I stated earlier, I pulled in aspects of Core Image that I really liked, but I also changed portions of their interface that had tripped me up in the past. I also extended things a bit, in that Core Image only deals with image processing, while I hook in movie playback, camera input, video recording, and image capture.
When I originally was kicking around the idea for this, Core Image had not yet come to iOS. By the time I released it, Core Image had just been added to iOS. However, the number of filters supported on iOS at that time was fairly limited (no blurs, for example), and Core Image on iOS did not allow you to create custom kernels as it did on the Mac.
GPUImage provided the means to do custom GPU-accelerated operations on images and video on iOS, where Core Image did not. Most people who started using it did so for that reason, because they had some effect that they could not do with stock Core Image filters.
Initially, GPUImage also had significant performance advantages for many common operations. However, the Core Image team has made significant improvements in processing speed with each iOS version and things are very close right now. For some operations, GPUImage is faster, and for others, Core Image is faster. They look to employ some pretty clever optimizations for things like blurs, which I've started to replicate in things like my GPUImageiOSBlurFilter. They also combine multi-stage operations intelligently, where I treat filter steps as discrete and separate items. In some cases on iOS, this gives me an advantage, and I've tried to reduce the memory consequences of this recently, but they handle many types of filter chains better than I do.
iOS 8 introduces the custom kernel support in Core Image on iOS that it has always had on the Mac. This makes it possible to write your own custom filters and other operations in Core Image on iOS, so that will no longer be as much of an advantage for GPUImage. Of course, anyone wanting to target an older iOS version will still be limited by what Core Image can do there, where GPUImage can target back to iOS 4.0.
Core Image also has some neat capabilities in terms of being able to do filtering while an iOS application is in the background (CPU-based at first, but iOS 8 adds GPU-side support for this now), where GPUImage's reliance on OpenGL ES prevents it from running when an application is in the background. There might be ways around this limitation in iOS 8, but I haven't worked through all the documentation yet.
My interests with GPUImage are in the field of machine vision. The image filters are a fun distraction, but I want to use this framework to explore what's possible with GPU-accelerated image analysis. I'm working on arbitrary object recognition and tracking operations, and that's the direction I'll continually evolve the framework toward. However, you have the code to the framework, so you don't have to rely on me.
This is an old thread, but I think it's worth noting that GPUImage also has some features that are not present in Core Image: notably hough transform and several edge detection filters.
Core Image seems all about applying filters and effects — and it's good to see GPUImage explores more on image/video analysis, kind of becoming more like openCV, but in a more efficient way.
I am working on an app in iOS that will occur an event if camera detects some changes in image or we can say motion in image. Here I am not asking about face recognition or a particular colored image motion, And I got all result for OpenCV when I searched, And I also found that we can achieve this by using gyroscope and accelerometer both , but how??
I am beginner in iOS.So my question is , Is there any framework or any easy way to detect motion or motion sensing by camera.And How to achieve?
For Example if I move my hand before camera then it will show some message or alert.
And plz give me some useful and easy to understand links about this.
Thanx
If all you want is some kind of crude motion detection, my open source GPUImage framework has a GPUImageMotionDetector within it.
This admittedly simple motion detector does frame-to-frame comparisons, based on a low-pass filter, and can identify the number of pixels that have changed between frames and the centroid of the changed area. It operates on live video and I know some people who've used it for motion activation of functions in their iOS applications.
Because it relies on pixel differences and not optical flow or feature matching, it can be prone to false positives and can't track discrete objects as they move in a frame. However, if all you need is basic motion sensing, this is pretty easy to drop into your application. Look at the FilterShowcase example to see how it works in practice.
I don't exactly understand what you mean here:
Here I am not asking about face recognition or a particular colored
image motion, because I got all result for OpenCV when I searched
But I would suggest to go for opencv as you can use opencv in IOS. Here is a good link which helps you to setup opencv in ios.
There are lot of opencv motion detection codes online and here is one among them, which you can make use of.
You need to convert the UIImage ( image type in IOS ) to cv::Mat or IplImage and pass it to the opencv algorithms. You can convert using this link or this.
Are these classes supported in Stage3D? Or are there equivalents or similar classes that exist?
flash.display.BitmapData;
flash.display.GraphicsSolidFill;
flash.display.GraphicsStroke;
flash.display.GraphicsPath;
flash.display.IGraphicsData;
flash.display.Shape;
flash.filters.BlurFilter;
flash.geom.ColorTransform;
Stage3D is an entirely different, fairly low-level beast. Those classes you list there are all related to the traditional Flash DisplayList, which is a CPU-driven rendering engine, so no, they don't exist, per se. But there's much more to it than that:
If you're using the raw Stage3D APIs (example tutorial here), then it feels very much like OpenGL programming. You're loading Vertex buffers, Index buffers, and textures into the GPU, and defining Vertex and fragment shader programs in an assembly language called AGAL. All this gets you a cross-platform, hardware accelerated application that's probably very fast, but it's very different than the traditional Flash DisplayList. Can you get gradients, filters and vector shapes - sure, but probably with custom shaders and such, not using those classes.
In some applications, it makes sense to use the traditional DisplayList for interactive UI controls on top of the Stage3D hardware accelerated backdrop. The DisplayList sits on top of the Stage3D plane, so this is entirely possible.
However, if such low-level 3D programming is not what you're interested in, you can choose to build on top of a framework. There are many Stage3D frameworks - some are intended for creating 3D applications, others are intended for 2D (but using the underlying 3D acceleration for speed). Adobe has a list of these frameworks here.
For example, Starling is a Stage3D framework that's intended to mimic the traditional Flash DisplayList, so it'll get you close to some of the classes you've mentioned above - check out their demo and API docs for specifics.
Another technique that Flash enables is blitting, which generates Bitmaps for 3D acceleration on the fly. You can draw into Bitmaps (aka blit) any Flash DisplayObjects you like (Shapes, drawn gradients, with filters, whatever), then push those Bitmaps into the 3D acceleration framework. You can blit individual objects separately, or blit the entire stage into one full-screen texture using this technique. But you have to be careful how often and how much you upload new textures into the GPU, because this can affect performance significantly. In fact, a significant performance consideration in GPU programming is the ability to batch several bitmaps into a single texture.
So there are many facets to consider when thinking about transitioning from the traditional DisplayList to Stage3D. Hope this helps. :)