I know a similar question has been asked before, but I have a very specific use case and thought I might put more details here.
I'm not an iOS dev, but very curious about the ARKit -- specifically want to test out the facial capture (tracking facial expressions) using ARKit ARFaceAnchor.
I want to know if it is possible to feed pre-recorded videos to ARKit instead of the camera feed.
I did find an article with sample code here about using "sensor replay".
https://github.com/ittybittyapps/ARRecorder/tree/master/ARRecorder
https://www.ittybittyapps.com/blog/2018-09-24-recording-arkit-sessions/
Unfortunately it is not possible ship an app this way.
I know that the facial capture doesn't necessarily require depth sensor data (I've tried it by holding up the camera to a pre-recorded face on a monitor).
So I'm curious to know if anyone knows of a way to feed a static, pre-recorded video to the ARKit?
Thanks in advance for the help :)
Related
I'm creating motion detect app for ios. when camera on live any object passes the camera like person , animal. than i want detect motion feature. how's it possible?
I suggest you get familiar with the AVFoundation framework to understand how to get live video frames using the camera of an iOS device. A good starting point is Apple's famous sample AVCam, which should get you familiar with all the camera concepts.
As the next step, figure out how to do the movement detection. The simplest algorithm for that would be the background subtraction. The idea is to subtract two consecutive frames one from another. The areas without movement just cancel each other and become black, while the areas with movements show some nonzero values.
Here's an example of background subtraction in the OpenCV framework.
If in the end, you decide to use OpenCV (which is a classic Computer Vision framework which I definitely recommend), then you'll need to integrate OpenCV into your iOS app. You can see a short tutorial here.
I tried to show you some pointers which could get you going. The problem (how you presented it) is definitely not an easy one, so good luck!
Is it possible to use iphone X faceID data to create a 3D model of the user face? If yes, can you please give tell me where should I look? I was not reallw able to found something related to this. I found a video on the WWDC about true depth and ARKit but I am not sure that it would help.
Edit:
I just watched a WWDC video and its says that ARKit provides a detailed 3D geometry face. Do you think it's precise enough to create a 3D representation of a person face? Maybe combined with an image? Any idea?
Yes and no.
Yes, there are APIs for getting depth maps captured with the TrueDepth camera, for face tracking and modeling, and for using Face ID to authenticate in your own app:
You implement Face ID support using the LocalAuthentication framework. It's the same API you use for Touch ID support on other devices — you don't get any access to the internals of how the authentication works or the biometric data involved, just a simple yes-or-no answer about whether the user passed authentication.
For simple depth map capture with photos and video, see AVFoundation > Cameras and Media Capture, or the WWDC17 session on such — everything about capturing depth with the iPhone 7 Plus dual back camera also applies to the iPhone X and 8 Plus dual back camera, and to the front TrueDepth camera on iPhone X.
For face tracking and modeling, see ARKit, specifically ARFaceTrackingConfiguration and related API. There's sample code showing the various basic things you can do here, as well as the Face Tracking with ARKit video you found.
Yes, indeed, you can create a 3D representation of a user's face with ARKit. The wireframe you see in that video is exactly that, and is provided by ARKit. With ARKit's SceneKit integration you can easily display that model, add textures to it, add other 3D content anchored to it, etc. ARKit also provides another form of face modeling called blend shapes — this is the more abstract representation of facial parameters, tracking 50 or so muscle movements, that gets used for driving avatar characters like Animoji.
All of this works with a generalized face model, so there's not really anything in there about identifying a specific user's face (and you're forbidden from trying to use it that way in the App Store — see §3.3.52 "If your application accesses face data..." in the developer program license agreement).
No, Apple provides no access to the data or analysis used for enrolling or authenticating Face ID. Gaze tracking / attention detection and whatever parts of Apple's face modeling have to do with identifying a unique user's face aren't parts of the SDK Apple provides.
Does anyone know how to reproduce the new Notes new scanning feature in iOS 11??
Is AVFoundation used for the camera?
How is the camera detecting the shape of the paper/document/card?
How do they place the overlay over in real time?
How does the camera know when to take the photo?
What's that animated overlay and how can we achieve this?
Does anyone know how to reproduce this?
Not exactly :P
Is AVFoundation used for the camera? Yes
How is the camera detecting the shape of the paper/document/card?
They are using the Vision Framework to do rectangle detection.
It's stated in this WWDC session by one of the demonstrators
How do they place the overlay over in real time?
You Should check out the above video for this as he talks about doing something similar in one of the demos
How does the camera know when to take the photo?
I'm not familiar with this app but it's surely triggered in the capture session, no?
Whats that animated overlay and how can we achieve this?
Not sure about this but I'd imagine it's some kind of CALayer with animation
Is Tesseract framework used for the image afterwards?
Isn't Tesseract OCR for text?
If you're looking for handwriting recognition, you might want to look for a MNIST model
Use Apple’s rectangle detection SDK, which provides an easy-to-use API that can identify rectangles in still images or video sequences in near-realtime. The algorithm works very well in simple scenes with a single prominent rectangle in a clean background, but is less accurate in more complicated scenes, such as capturing small receipts or business cards in cluttered backgrounds, which are essential use-cases for our scanning feature.
An image processor that identifies notable features (such as faces and barcodes) in a still image or video.
https://developer.apple.com/documentation/coreimage/cidetector
I planned to develop a software that can takes attendant (work , school) by face recognition as my final year project(FYP).(Just an Idea)
I have search through the net about the image processing library and i found out OpenCv is more well known as i found a lot of video for face recognition using OpenCv in youtube which will definitely help me a lot.(I'm totally new to image processing). Also, i will be using Visual Studio.
Here come the first problem, which is is it possible to detect that it is a photo or a real person is standing in front of the camera while taking the attending?
If yes, can you provide me some link or tutorial link for how image processing can detect 'photograph' and 'real person'?
As i said, I'm totally new to image processing and this is just an idea for my FYP
Or is the any open sources library that you recommend?
Eulerian Video Magnification can detect that it is a photo or a real person is standing in front of the camera but it may not detect that it is a video or real person is standing in front of the camera. Thus, the Face Recognition Authentication System which is based Eulerian Video Magnification can not be successfull when malicious user uses face video to rather than a real person face.
Here are my ideas to develop robust Face Recogition Authentication system;
You can use Multi-View Face Recognition to develop robust face authentication system. Here is a demo video of this technique and here are papers to get theoritical background. Also you can benefit from this, this, this and this when you start coding.
You can use RANDOM directions to detect that it is a photo/video or real person for example blink your eyes 3 times, move your eyebrow, look at the left side or look at the right side (multi-view face recognition will be used to recognize user's face when user look at the right or left) etc.
You should use these 2 ideas in your project for developing robust Face
Recognition Authentication system.
Here is the scnario;
I am aware of how to add text / image overlay into a video on iOS with AVFoundation.
Is there some way to do this using position/motion tracking of certain objects / areas in the video?
What exactly is this type of video editing feature called?
Let's say I have a video of a car moving from left to right. I want to place an image of another car at the position of the original car so that as the car in the video is moving from left to right, my image follows on top of that car. I would also want this to be properly skewed as the car moves from left to right.
Another example would be a video of a monitor. And me placing an image on the screen of that monitor.
Please let me know if I need to explain further.
Other than iOS is there some other library which is able to do this? Like ffmpeg?
What you're broadly looking for is Object Recognition, which is a fairly complex topic in it's own right and part of the field of Computer Vision.
AVFoundation includes support for Face Detection and does a fairly reasonable job of it https://developer.apple.com/reference/avfoundation/avmetadatafaceobject but that's about it.
To do what you're trying to do, I'd start with OpenCV (which includes support for iOS) and investigate from there http://opencv.org/
You're not going to find a literal "find me a car" API, what you will find is lots of different algorithms which are implemented that allow you to train them and detect the objects they are trained for. One potential algorithm is using Haar Cascades. There's more detail on working with those and training your own classifier here https://github.com/andrewssobral/vehicle_detection_haarcascades