I am interested in the Visual Intertial SLAM algorithm that is implemented in the ARKit SDK for motion tracking, that performs visual SLAM and fuses it with intertial data. I understand the algorithm and how tracking is performed.
Since I want to use my custom camera, and not an iphone, I was wondering if there is an equivalent open source implementation available already that performs the VI-SLAM + inertial data for tracking the object, with a comparable performance? I am not looking for SDKs that I can use as APIs, rather algorithm implementations that I can edit myself.
Apologies if this question should belong in another forum.
You can try a popular ARToolKit5. It is fast, intuitive and cross-platform. You can run it on macOS, iOS, Linux, Android or Windows. It was released in 2015 as a completely open source platform as LGPLv3 and later. There's also a link to the latest release of ARToolKitX.
There are many open source VISLAM on github. I recommend you to try VINS-Mono(https://github.com/HKUST-Aerial-Robotics/VINS-Mono). You can use your own camera to collect images and IMU data, or you can use public datasets.
Related
A-Frame's immersive-ar functionality will work on some Android devices I've tested with, but I haven't had success with iOS.
It is possible to use an A-Frame scene for markerless AR on iOS using a commercial external library. Example: this demo from Zapworks using their A-Frame SDK. https://zappar-xr.github.io/aframe-example-instant-tracking-3d-model/
The tracking seems to be no where near as good as A-Frame's hit test demo (https://github.com/stspanho/aframe-hit-test), but it does seem to work on virtually any device and browser I've tried, and it is good enough for the intended purpose.
I would be more than happy to fallback to lower quality AR mode in order to have AR at all in devices that don't support immersive-ar in browser. I have not been able to find an A-Frame compatible solution for using only free/open source components for doing this, only commercial products like Zapworks and 8th Wall.
Is there a free / open source plugin for A-Frame that allows a scene to be rendered with markerless AR across a very broad range of devices, similar to Zapworks?
I ended up rolling my own solution which wasn't complete, but good enough for the project. Strictly speaking, there's three problems to overcome with getting a markerless AR experience on mobile without relying on WebXR:
Webcam display
Orientation
Position
Webcam display is fairly trivial to implement in HTML5 without any libraries.
Orientation is already handled nicely by A-FRAME's "magic window" functionality, including on iOS.
Position was tricky and I wasn't able to solve it. I attempted to use the FULLTILT library's accelerometer functions, and even using the readings with gravity filtered out I wasn't able to get a high enough level of accuracy. (It happened that this particular project did not need it)
I'm looking to implement face recognition feature and I see OpenCV is capable of it: https://github.com/Mjrovai/OpenCV-Face-Recognition
At the same time, I see many 3rd party face verification SDKs, like
http://kairos.com, http://www.neurotechnology.com/face-verification.html, http://ever.ai, etc. In general practise, what's the difference between OpenCV and 3rd party ones, if you only need offline face-recognition with no fancy addons, and which shall be used?
The example you linked with OpenCV uses a method (LBP) to perform the face-recognition that is outdated at the state of the art and that I think can hardly lead you to excellent results.
The SDKs you talk about are paid, and since they are obviously not open-source and I can not know what technologies they use.
If you prefer to implement a good face-recognition yourself, you have to use OpenCV only for the image capture / video stream part and then use something like TensorFlow/Keras/PyTorch for the deep learning part.
Can OpenCV seamlessly interact with all cameras that comply with these standards
No it cannot. You need something called a GenTLProducer in order to interact with your camera. Normally your vendor's SDK comes with it. Alternatively, you can use the one from Baumer Here or from Stemmer Imaging Here.
Another option is to use harvesters, which is an open source project that aims to do this. Although you need a GenTLProducer for that as well.
I have been working Augmented Reality for quite a few months. I have used third party tools like Unity/Vuforia to create augmented reality applications for android.
I would like to create my own framework in which I will create my own AR apps. Can someone guide me to right tutorials/links to achieve my target. On a higher level, my plan is to create an application which can recognize multiple markers and match it with cloud stored models.
That seems like a massive undertaking: model recognition is not an easy task. I recommend looking at OpenCV (which has some standard algorithms you can use as a starting point) and then looking at a good computer vision book (e.g., Richard Szeliski's book or Hartley and Zisserman).
But you are going to run into a host of practical problems. Consider that systems like Vuforia provide camera calibration data for most Android devices, and it's hard to do computer vision without it. Then, of course, there's efficiently managing the whole pipeline which (again) companies like Qualcomm and Metaio invest huge amounts of $$ in.
I'm working on a project that does framemarker tracking and I've started exporting bits of it out to a project I'm calling OpenAR. Right now I'm in the process of pulling out unpublishable pieces and making Vuforia and the OpenCV versions of marker tracking interchangeable. You're certainly welcome to check out the work as it progresses. You can see videos of some of the early work on my YouTube channel.
The hard work is improving performance to be as good as Vuforia.
I've been reading about Andrew Russell's ExEn project and I'm wondering what the flow would be like for creating a WP7 accelerometer-based game and then porting it to another platform, say iOS. Here's what I hope would happen:
Create fully functional game in XNA, avoiding dependance on device
specific items like the 'back' button.
Run the project through ExEn (I have no idea on how this would
happen), creating fully functional iOS game.
Run game on iPhone.
Sorry for that pitiful outline, but I just don't have a solid high-level view after reading about it.
Also, being software conversion, surely it wouldn't totally work. How would you iron out the wrinkles? I assume you'd have to know iOS or Android fairly well to pin it down.
Anyway, if anyone can move me one step closer I would appreciate it.
ExEn is an implementation of a subset of the XNA API that runs on different platforms (including iOS and Android). Put simply, it makes the classes and methods that you use when writing XNA code available to you on these other platforms. (Plus appropriate instructions, examples, etc.)
When using ExEn, the bulk of your code should simply "just work". However in most real-world cases you will need to write at least some platform-specific code (and probably provide some platform-specific assets). In particular to support different device resolutions, and also in cases where you use XNA features not available in ExEn.
At time of writing, ExEn does not implement the XNA/WP7 APIs for accelerometer support. At some time in the future they may be added (either by me or anyone who wants to contribute a patch). ExEn is distributed as source code, so you could even add the necessary support yourself.
The alternative would be to write platform-specific code for the parts of your game that query the accelerometer. Using ExEn does not prevent you from also using the APIs of the underlying platform.
ExEn (on iOS and Android) runs on top of Xamarin's MonoTouch and Mono for Android products. These two products provide C# bindings for the underlying platform APIs. Also, much like ExEn implements the XNA APIs, Mono implements the .NET APIs. These products also provide you with the tools you need (IDE, compiler, debugger, etc).
So the iOS API that you would use is UIAccelerometer (doc). This is exposed in C# via MonoTouch.UIKit.UIAccelerometer (doc). I'll leave looking up the Android equivalents as an exercise.
You can't expect:
porting a game to other platform and don't modify it.
porting a game with special platform inherent abilities to other platform that lacks this abilities, or vice versa