I am using an XBox Kinect with the Kinect for Windows SDK. I want to make an application that will augment a 3D mask (a 3D model of a mask made in 3DS Max) onto the face of anyone using the application. The application will be used in an exhibit locally. I have not tried much because I don't know where to start. So what I want to know is, is it currently possible to augment a 3DS Max model onto a live video stream using the facial recognition and skeletal tracking features in the newest Kinect for Windows SDK, and if so, how/where should I start trying to do/implement this? Any point in the right direction would be great. Thank you! PS And yes, I have read the UI guidelines and the facial documentation. My problem is one of not knowing where to start programming, not one of not understanding the fundamental concepts. Thanks!
If you are serious about getting into developing for the Kinect I would recommend getting this book:
http://www.amazon.com/Programming-Kinect-Windows-Software-Development/dp/0735666814
This goes through developing with the Kinect for Windows SDK from the ground up. There is a face tracking and an augmented reality example so I'm pretty sure you will be able to achieve your goal quite easily.
All the code from the book is here:
http://kinecttoolbox.codeplex.com/
Alternatively, there is an example here which pretty much is what you want to achieve:
http://www.codeproject.com/Articles/213034/Kinect-Getting-Started-Become-The-Incredible-Hulk
It is developed using the Beta version of the SDK, but the same priciples apply.
You can also check out the quick start videos here:
http://channel9.msdn.com/Series/KinectQuickstart
In summary, based on my own experience, I would spend some time going through the beginner examples either in the vides or the book (I found the book very good) just to get familiar with how to setup a simple Kinect project and how the different parts of the SDK work.
When you have developed some throwaway apps with the Kinect, I would then try tackling your project (although, the Incredible Hulk project above should get you most the way there!)
Best of luck with your project
Related
Does anyone work on object detection/tracking on Hololens? I'm expecting to automatically detect a physical object using a marker or without a marker.
As per my knowledge we have a library like OpenCV based on raw images. But, HoloLens is a powerful computer vision device. Is there a possibility Microsoft might expose some high-level object detection API?
With the actual version of Hololens it not could be done native. Rumors says that the next version will come with a IA chipset for this porpoise.
In the meanwhile, you can use thinks like Vuforia Object Detection which works smoothly.
Anyway, there is a version of the OpenCV in the Unity Asset store which come with severals samples of what you want to do.
I'm developing this project where I'm trying to create a distributed version of Tensorflow (the actual open source version is single node) and where the cluster is entirely composed by mobile devices (e.g. smartphones).
In your opinion, what is a possible application or use case where this could be useful? Can you give me some example please?
I know that this is not a "standard" Stack Overflow question, but I didn't know where to post it (if you know a better place where to post it, please let me know it). Thanks so much for your help!
http://www.google.com.hk/search?q=teonsoflow+android
TensorFlow can be used for image identification and there is an example using the camera for Android.
There could be many distributed uses for this. Face recognition, 3D space construction from 2D images.
TensorFlow can be used for a chat bot. I am working towards using it for a personal assistant. The Ai on one phone could communicate with the Ai on other phones.
It could use vision and GPS to 'reserve' a lane for you on the road. Intelligent crowd planned roads and intersections would be safer.
I am also interested in using it for distributed mobile. Please contact me with my user name at gmail or Skype.
https://boinc.berkeley.edu
I think all my answers could run on individual phones with communication between them. If you want them to act like a cluster as #Yaroslav pointed out there is Seti#home and other projects running in the BOINC client.
TensorFlow could be combined with a game engine. You could have a proceduraly generated Ai learning augumented reality game generating the story as multiple players interact with it. I have seen research papers for each of these components.
Nowadays, I wanna do some research of augmented reality technology.Especially, I would like to match a 2d image and a 3d model.And then, I will see the 3d model if scanning the 2d image. What's more, I know that there are a lot of SDKs(like metaio,and wikitude) and software can realize this in mobile app. However, what I want to do is realizing this in a website. I hope the people who use this don't need to download a particular mobile app, but just open a website and then scan a picture.
So, until now, I's like to know that,as the tile asked, can AR be realized in a website? If yes, how can I do it or is there any software like Metaio Creator to do this? If no, why?
Thank you for anyone who would like to answer my naive question.
May I recommend you our completely webbased AR & VR tool holobuilder.com by bitstars.com?
It supports 360 degree photospheres that can be enhanced with custom 3D models and then directly be embedded into your website as iframe, it has native support for stereoscopic view mode and much more.
For your use case you could have a look at the lower part of this blog post where you find information and an embedded example presentation with photosphere imagery containing 3D elements:
http://heyholo.com/google-pushes-vr-great-for-tools-like-holobuilder/
If you want to start creating I recommend the beginners guide:
https://medium.com/#maxspeicher/the-definite-guide-to-holobuilder-3b62a54d303e
The cv feature tracking you requested can not yet be realized without any apps/browser. But what you can do is realizing perspectively correct displaying 3D elements into the camera image and move with sensors. Should be as performant as within the player app.
We hope that it can somehow help you in pushing your research and we would love to read your feedback. In case of any questions please do not hesitate to ask, here or on any other contact channel!
I have been working Augmented Reality for quite a few months. I have used third party tools like Unity/Vuforia to create augmented reality applications for android.
I would like to create my own framework in which I will create my own AR apps. Can someone guide me to right tutorials/links to achieve my target. On a higher level, my plan is to create an application which can recognize multiple markers and match it with cloud stored models.
That seems like a massive undertaking: model recognition is not an easy task. I recommend looking at OpenCV (which has some standard algorithms you can use as a starting point) and then looking at a good computer vision book (e.g., Richard Szeliski's book or Hartley and Zisserman).
But you are going to run into a host of practical problems. Consider that systems like Vuforia provide camera calibration data for most Android devices, and it's hard to do computer vision without it. Then, of course, there's efficiently managing the whole pipeline which (again) companies like Qualcomm and Metaio invest huge amounts of $$ in.
I'm working on a project that does framemarker tracking and I've started exporting bits of it out to a project I'm calling OpenAR. Right now I'm in the process of pulling out unpublishable pieces and making Vuforia and the OpenCV versions of marker tracking interchangeable. You're certainly welcome to check out the work as it progresses. You can see videos of some of the early work on my YouTube channel.
The hard work is improving performance to be as good as Vuforia.
I'm searching for a face tracking system to use in an augmented reality project. I'm trying to find an open source and multi-platform application for it. The goal is to return the direction where the face is looking to interact with the virtual environment, (something like this video).
I've downloaded the sources of the above Johnny Lee's application and tried to use Free Track too, making my own headset (some kind of monster, hehe). But it's not good to be limited to infrared points in your head.
These days I've download FaceTrackNoIR, but when I launch the program I get "No DLL was found in the Waterfall procedure." that I'm actually trying to solve.
Anyone knows a good application, library, code, lecture, anything that could help me to find a good path for this?
Thank you all!
I'll try to post results someday :-)
I would take a look at OpenCV. It is a general purpose machine-learning and computer vision C++ library. One of the examples in the download is a real-time face tracker that connects to a video camera connected to your computer and draws squares around any faces in the camera view.