We're looking to share AR experiences (ARWorldMap) over the web (not necessarily to devices nearby, I'm referring to data that can be stored to some server and then retrieved by another user).
Right now we're looking into ARWorldMap which is pretty awesome, but I think it only works on the same device AND with devices nearby. We want to be able to delete these constraints (and therefore save the experience over the web on some server) so that everyone else can automatically see 3D things with their devices (not necessarily at the same time) exactly where they were places.
Do you know if it's possible to send the archived data (ARWorldMap) to some server in some kind of format so that another user can later retrieve that data and load the AR experience on their device?
The ARWorldMap contains the feature points in the enviroment around the user. For example, the exact position and size of a table but including all the other points found by the camera. You can visualize those with the debugOptions.
It does not make sense to share those to a user that is not in the same room.
What you want to do is share the interactions between the users, eg when an object was placed or rotated.
This is something you would need to implement yourself anyway since ARWorldMap only contains anchors and features.
I can recommend Inside Swift Shot from last years WWDC as a starting point.
Yep technically it’s possible as according to docs here. You will have to use the nsssecure coding protocol, might have to extend if you aren’t using a swift backend though. It’s still possible as the arkit anchors are defined by a mix of point maps, rotational data , and image algos. Just implement portions of codable to JSON and voila it’s viable for most backends. Of course this is easier said then done.
Related
It's called a memory palace (Read: 'Moonwalking with Einstein') it's an ancient tool used to memorize, in my case coding concepts and Spanish and Indonesian phrases.
I'm learning python now, but I'm not really sure what direction to move in and what stack should be used to build a project like this. it wouldn't be too complex, I just want to store and save "text files" in a virtual space like my bedroom or on my favorite hikes.
If anyone has insights or suggestions it'd be much appreciated.
Probably the two most common AR frameworks, on mobile devices anyway, at the moment are ARKit for iOS devices and ARCore for Android devices.
I am sure you can find comparisons of the strengths and weaknesses of each one but it is likely your choice will be determined by the type of device you have.
In either case, it sounds like you want to have 'places' you can return to over time and see your stored content. For this you could build on some common techniques:
link the AR object to some sort of image in the real world and when this image is recognised by the AR app, launch your AR object, in your case a text file.
use 'Cloud Anchors' - these are essentially anchors for AR objects that can persist over time, when you close the app and come back to it later, and even be shared between users on different devices.
You can find more information on cloud Anchors at the link below, including information on using them with iOS and on Android:
https://developers.google.com/ar/develop/java/cloud-anchors/overview-android
I'm trying to write an app for detecting "where you are" in a building use ARCore. I'd like to use previously learnt and then saved feature points to provide the initial sync position as well as then helping to continuously update position accurately. But this feature does not currently appear to be supported in ARCore.
Currently I'm using tracked images as a way to do an initial sync. It works, but not brilliantly - alignment is often a few degrees off and you have to approach the image pretty slowly and deliberately. And then once synced there is drift... Yes, loop closing works pretty well when it gets back to somewhere it recognises, but it needs to build up that map every time you start the session.
So, obvious solution: are there any plans for Google to implement "Area Learning" as it was back in Google Tango? It looks like Cloud Anchors might be some attempt to do this, but clearly that's all hosted on Google, and it strictly limited as to how long that data is stored. Currently that's just not a possible solution. OTOH, Apple's ARKit seems to now provide just what is needed:
https://developer.apple.com/documentation/arkit/saving_and_loading_world_data
Does this mean that Apple / ARKit is the only way to go for the app? Hope not...
You might want to check out persistent cloud anchors that is still in development.
From documentation:
Note: We’re currently developing persistent Cloud Anchors, which can
be resolved for much longer. Before making the feature broadly
available, we’re looking for more developers to help us explore and
test persistent Cloud Anchors in real world apps at scale. See here if
you’re interested.
What is the best way, if any, to use Apple's new ARKit with multiple users/devices?
It seems that each devices gets its own scene understanding individually. My best guess so far is to use raw features points positions and try to match them across devices to glue together the different points of views since ARKit doesn't offer any absolute referential reference.
===Edit1, Things I've tried===
1) Feature points
I've played around and with the exposed raw features points and I'm now convinced that in their current state they are a dead end:
they are not raw feature points, they only expose positions but none of the attributes typically found in tracked feature points
their instantiation doesn't carry over from frame to frame, nor are the positions exactly the same
it often happens that reported feature points change by a lot when the camera input is almost not changing, with either a lot appearing or disappearing.
So overall I think it's unreasonable to try to use them in some meaningful way, not being able to make any kind of good point matching within one device, let alone several.
Alternative would to implement my own feature point detection and matching, but that'd be more replacing ARKit than leveraging it.
2) QR code
As #Rickster suggested, I've also tried identifying an easily identifiable object like a QR code and getting the relative referential change from that fixed point (see this question) It's a bit difficult and implied me using some openCV to estimate camera pose. But more importantly very limiting
As some newer answers have added, multiuser AR is a headline feature of ARKit 2 (aka ARKit on iOS 12). The WWDC18 talk on ARKit 2 has a nice overview, and Apple has two developer sample code projects to help you get started: a basic example that just gets 2+ devices into a shared experience, and SwiftShot, a real multiplayer game built for AR.
The major points:
ARWorldMap wraps up everything ARKit knows about the local environment into a serializable object, so you can save it for later or send it to another device. In the latter case, "relocalizing" to a world map saved by another device in the same local environment gives both devices the same frame of reference (world coordinate system).
Use the networking technology of your choice to send the ARWorldMap between devices: AirDrop, cloud shares, carrier pigeon, etc all work, but Apple's Multipeer Connectivity framework is one good, easy, and secure option, so it's what Apple uses in their example projects.
All of this gives you only the basis for creating a shared experience — multiple copies on your app on multiple devices all using a world coordinate system that lines up with the same real-world environment. That's all you need to get multiple users experiencing the same static AR content, but if you want them to interact in AR, you'll need to use your favorite networking technology some more.
Apple's basic multiuser AR demo shows encoding an ARAnchor
and sending it to peers, so that one user can tap to place a 3D
model in the world and all others can see it. The SwiftShot game example builds a whole networking protocol so that all users get the same gameplay actions (like firing slingshots at each other) and synchronized physics results (like blocks falling down after being struck). Both use Multipeer Connectivity.
(BTW, the second and third points above are where you get the "2 to 6" figure from #andy's answer — there's no limit on the ARKit side, because ARKit has no idea how many people may have received the world map you saved. However, Multipeer Connectivity has an 8 peer limit. And whatever game / app / experience you build on top of this may have latency / performance scaling issues as you add more peers, but that depends on your technology and design.)
Original answer below for historical interest...
This seems to be an area of active research in the iOS developer community — I met several teams trying to figure it out at WWDC last week, and nobody had even begun to crack it yet. So I'm not sure there's a "best way" yet, if even a feasible way at all.
Feature points are positioned relative to the session, and aren't individually identified, so I'd imagine correlating them between multiple users would be tricky.
The session alignment mode gravityAndHeading might prove helpful: that fixes all the directions to a (presumed/estimated to be) absolute reference frame, but positions are still relative to where the device was when the session started. If you could find a way to relate that position to something absolute — a lat/long, or an iBeacon maybe — and do so reliably, with enough precision... Well, then you'd not only have a reference frame that could be shared by multiple users, you'd also have the main ingredients for location based AR. (You know, like a floating virtual arrow that says turn right there to get to Gate A113 at the airport, or whatever.)
Another avenue I've heard discussed is image analysis. If you could place some real markers — easily machine recognizable things like QR codes — in view of multiple users, you could maybe use some form of object recognition or tracking (a ML model, perhaps?) to precisely identify the markers' positions and orientations relative to each user, and work back from there to calculate a shared frame of reference. Dunno how feasible that might be. (But if you go that route, or similar, note that ARKit exposes a pixel buffer for each captured camera frame.)
Good luck!
Now, after releasing ARKit 2.0 at WWDC 2018, it's possible to make games for 2....6 users.
For this, you need to use ARWorldMap class. By saving world maps and using them to start new sessions, your iOS application can now add new Augmented Reality capabilities: multiuser and persistent AR experiences.
AR Multiuser experiences. Now you may create a shared frame of a reference by sending archived ARWorldMap objects to a nearby iPhone or iPad. With several devices simultaneously tracking the same world map, you may build an experience where all users (up to 6) can share and see the same virtual 3D content (use Pixar's USDZ file format for 3D in Xcode 10+ and iOS 12+).
session.getCurrentWorldMap { worldMap, error in
guard let worldMap = worldMap else {
showAlert(error)
return
}
}
let configuration = ARWorldTrackingConfiguration()
configuration.initialWorldMap = worldMap
session.run(configuration)
AR Persistent experiences. If you save a world map and then your iOS application becomes inactive, you can easily restore it in the next launch of app and in the same physical environment. You can use ARAnchors from the resumed world map to place the same virtual 3D content (in USDZ or DAE format) at the same positions from the previous saved session.
Not bulletproof answers more like workarounds but maybe you'll find these helpful.
All assume the players are in the same place.
DIY ARKit sets up it's world coordinate system quickly after the AR session has been started. So if you can have all players, one after another, put and align their devices to the same physical location and let them start the session there, there you go. Imagine the inside edges of an L square ruler fixed to whatever available. Or any flat surface with a hole: hold phone agains surface looking through the hole with camera, (re)init session.
Medium Save the player aligning phone manually, instead detect a real world marker with image analysis just like #Rickster described.
Involved Train an Core ML model to recognize iPhones and iPads and their camera location. Like it's done with human face and eyes. Aggregate data on a server, then turn off ML to save power. Note: make sure your model is cover-proof. :)
I'm in the process of updating my game controller framework (https://github.com/robreuss/VirtualGameController) to support a shared controller capability, so all devices would receive input from the control elements on the screens of all devices. The purpose of this enhancement is to support ARKit-based multiplayer functionality. I'm assuming developers will use the first approach mentioned by diviaki, where the general positioning of the virtual space is defined by starting the session on each device from a common point in physical space, a shared reference, and specifically I have in mind being on opposite sides of a table. All the devices would launch the game at the same time and utilize a common coordinate space relative to physical size, and using the inputs from all the controllers, the game would remain theoretically in sync on all devices. Still testing. The obvious potential problem is latency or disruption in the network and the sync falls apart, and it would be difficult to recover except by restarting the game. The approach and framework may work for some types of games fairly well - for example, straightforward arcade-style games, but certainly not for many others - for example, any game with significant randomness that cannot be coordinated across devices.
This is a hugely difficult problem - the most prominent startup that is working on it is 6D.ai.
"Multiplayer AR" is the same problem as persistent SLAM, where you need to position yourself in a map that you may not have built yourself. It is the problem that most self driving car companies are actively working on.
I'm still in the very early stages of a game, and I am saving information using a KeyChain wrapper class someone made. I wanted to ask for some advice early on, since I have time to change my approach.
My game has the potential to persist a fair amount of data about the player and what they've done, such as:
how much gold the player has
what items they've acquired (you can get about 50 items)
what skills, spells, and abilities they've chosen for their character
what experience level they are, max health, stats, etc
The reason I decided to store this in KeyChain was that I was told it's encrypted and much more difficult to tamper with. I felt there were other solutions such as the ones below, but I wrote some potential reasons why that might not be good:
Make everything web-based, and stored in a database somewhere on my server - I want my game to be playable offline
Use a local database (FMDB, let's say) - I could use a tool to edit the values directly, and give myself more health, etc.
Use Core Data - Never used this before, not sure if this is the same ease of tampering as #3?
GameCenter - Never used this before, so not sure what the lift is
NSPreferences - Preferences are more easily tampered with (i've used a tool to change the values pretty quickly)
So I am not sure if i'm completely wrong above, but let's say there is some degree of truth there and KeyChain is a good approach. The problem is now what if I want to then somehow allow the player to pick up a new device and pick up where they left off? How on earth would I serialize all that data I'm saving in keychain? I don't mind creating a giant JSON document of the values, and sending it along somewhere (where? to GameCenter?)
Any advice / pointers in the right direction would be good, especially now since i'm in the early stages and can make changes to step back.
Thanks so much everyone appreciate your time!
A few thoughts based on lessons learned (usually, "the hard way") that may (or may not) be helpful. :)
I see three requirements in your post: offline play (requiring local storage), data security (which is a massive topic in and of itself) and synchronization.
Playable offline:
Thus you need some sort of local storage. Keychain, Core Data, SQL, NSPreferences are all options. I don't know the limitations of the Keychain, so not really sure how suitable it is for continuous read/write of large chunks of data.
Data security:
They keychain protects your secrets when you're not logged in, and partitions them between apps. https://developer.apple.com/library/content/documentation/Security/Conceptual/keychainServConcepts/02concepts/concepts.html and http://evgenii.com/blog/sharing-keychain-in-ios/ give more details. That should prevent other tools from modifying other app's content on non-jailbroken devices.
Core Data, SQL, et al will be inside your app's container, which makes it harder for other tools to access on non-jailbroken devices. There's a good description here: https://developer.apple.com/library/content/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html
NSPreferences doesn't offer any security that I'm aware of. Plus, if they jailbreak the device, they basically have root access on a linux machine and can probably do anything they want.
In today's security world, the mantra is generally, "assume breach." As in, assume if they want in bad enough, they're going to find a way in. Thus you need to think about other layers of defense: partitioning your secrets so compromised secrets have limited value, encryption at rest and encryption in transit. Obfuscating the data before you write it/transmit is therefore another layer of defense (although they may still edit it and you'll have to handle potential garbage values).
You can chase security pretty far down the rabbit hole; you'll have to decide where the cost outweighs the benefit, and/or what risks you intend to defend against.
Synchronization across devices:
Thus you need a syncing solution. the iCloud Keychain syncs between devices, so if the keychain meets your storage needs, this probably will meet your sync needs, too. Again, I'm not sure if there are size or frequency-of-update constraints. I haven't used this, but this SO answer gives more info: https://stackoverflow.com/a/32606371/1641444. Based on Apple's docs (https://support.apple.com/en-us/HT204085) it looks like the user does have to enable syncing for this to work.
Otherwise, Apple provides GameCenter and CloudKit. Or you could explore 3rd party options. I've used GameCenter and CloudKit to sync data across devices. CloudKit wins over GameCenter, IMO, no contest.
With GameCenter, you gain multiplayer matchmaking and channels to share data between users. But, you have to adhere specifically to its structure which is IMO both extremely frustrating to work with and limited in functionality. Check the GameCenter and GKTurnBasedMatch tags here on SO for a taste of the problems.
CloudKit is a lower-level solution. It allows you to store a wide variety of data objects in iCloud and "subscribe" to change notifications. All data is contained within a container for the app, and has a "private" (user-specific) section and a public (shared by all users of the app) section. After watching the introductory WWDC videos on CloudKit, I was successfully sync'ing user settings between devices AND different users on different devices in no time. However, if you want some of the multiplayer features of GameCenter, you'll have to build the data model/subscriptions yourself. Since you support offline play, you'd need to save your data to a local storage solution and periodically upload it to iCloud for sync.
Conclusion (aka TL;DR)
So, my opinion is: none of the tools individually meet all three of your requirements, and you'll end up rolling your own solution for at least 1 req, regardless of which option you go for. In my multiplayer game, I'm trusting in Apple's filesystem (containers) to provide sufficient data security, I'm using a local storage option within the app's container, and I'm periodically writing NSData objects to cloudKit for synchronization across devices. I hit my frustration limit with GameCenter and pulled it from my app.
Star for a good question!!
I'm doing something similar with this. Right now, I'm using UserDefaults for primary storage. When I need to transfer data, I can save it to iCloudKit, or I could export it as a plist or json to use with AlamoFire, email, etc.
As for actually storing the data, I saw you mentioned CoreData.. that is kind of overkill! Look at NSCoder and NSKeyedArchiver.
Personally, I make my own save/load functions that manipulate dictionaries, then toss them into a UserDefaults key.
An example hierarchy of how I store dicts in UserDefaults (there are many ways to go about this)
Key:
Items->Equipment->Weapons->WeaponName
Value:
{ dictionary of data like AP, Cost, etc }
This to me is easier than using NSCoder, and certainly easier than CoreData!! The goal is to turn everything into a dictionary, which you can then easily put into UserDefaults, which allows you to easily create plists or save to the cloud.
With the above key system, you basically just do for loops to find what you need, parsing out each section.. so a function to display all equipment, you just load the entire UserDefaults.standard.dictionaryRepresentation, then search for keys that only have 'Items->Equipment' and so on.
Hope this helps!
I have some functions from my game I could share if you need more tips.
UPDATE:
If you are making this an online game, I would focus on learning the essentials of persistence and cloud usage first, then port it to a more secure platform that uses encryption.
Your best bet will be to create your own servers and transfer data that way. That will give you exactly what you want, when you want it, and with the security that you want.
If you want to use GameCenter, I would use GC as an in-between layer of something custom you created, so that way you can filter out the cheaters' scores / have more flexibility.
I'm working on the requirements & specifications for a new iOS app intended for use by certain professionals working "in the field". All day long for weeks on end, these folks have a sizable reporting burden to their superiors using standardized forms that track all different kinds of information. Traditionally, those forms are in PDF, and are simply printed and filled out in ink and then shared with the dozens to hundreds of others working the same operation. Sometimes they'll use a PDF with form fields so the data can be typed and then printed as part of the form. Either way, given their workflow, time and stress pressures, and other factors, it's not a very productive way to get the standardized reporting forms done.
The app we're spec'ing would offer an iOS (and Android, if possible -- but secondary or even tertiary requirement at this point) user interface for tracking the data they enter in the field, organizing it in a logical manner for each individual user, and with the press of a button, take all that data and automatically create a PDF file of it using the standardized form.
Of course, the forms are STRICTLY and rigidly standardized in this industry, and any deviation in format, structure, or presentation is simply not tolerable.
So I was approaching the project by thinking the app would maintain an internal repository of the original standardized forms from the accrediting organization, with each possible data area defined as a field. The app would:
open the necessary PDF form for the task at hand;
parse its dictionary to identity the specific data fields;
for every single field, identify the relevant data from the iOS app's own user interface and data tables, and assign that data to the corresponding field from the PDF/dictionary
export the PDF to a NEW PDF file, which the app would either email or store through iCloud, Dropbox, or some other form of file sharing.
The catch with #4 is that that PDF file must remain editable by standard PDF applications on Windows and Mac (Acrobat, Preview, etc.), so all the fields need to remain. And the PDF should be viewable just the same on either Windows or Mac.
Now, at NO time will the PDF (neither the original nor the exported final document) EVER need to be displayed inside the iOS app, nor would it make much sense to be able to do so.
I don't know if any of this is possible. This is our first iOS project, and we've been leaning towards building the app using Moai or Corona or some other framework to save development time and make porting across platforms easier. That said, if it cannot be done using Lua and one of these frameworks (I remain skeptical...they seem HIGHLY geared towards games), we're not opposed to doing it directly in Objective C and building an Android version some time down the road.
But either way, I'm at a loss in assessing whether this is even a practical undertaking. Our requirements are clear, and frankly if this can't be done, the project won't be pursued any further. But I could definitely use some help from you folks in identifying what my options are, whether I can do it in Lua, and what SDK(s) would be most useful in accomplishing this.
Based on what you've said, it seems that there is little reason to do the PDF-based part of the work on the mobile device itself since:
you don't need to display it on the ipad
you plan to email it or store it in the cloud
if you write this for iOS you will have to write again for Android as you've mentioned
Can you simplify the mobile part of your requirement by focusing on the data-collection and validation, then firing off to a server to do the document production? That will give you a lot more flexibility in the tools that you can use to merge the data into PDF docs. If so you could look at creating PDFs or populating the fields from code using something like iText (C# or Java). If you don't want to build your own back end server you could try something like Docmosis Cloud - but that might not allow you to get your precise layouts.
Certainly the catch you mentioned - needing to keep the PDFs editable with their fields is a significant gotcha in all cases. If you could convince the stakeholders that it is better to generate the final documents from your system (generate draft, review, update data, generate again etc) - rather than generating editable documents that you then lose control and tracability over, then you will be miles ahead.
Hope that helps.
Did you consider just generating a new pdf using an image of the form as the background to the pdf and just writing the user's data into the required areas over the form image. Would reduce the complexity of trying to parse the original form PDFs.
That's a point of worthwhile discussion, but one we don't have an ideal answer on. I tend to think of that as the almost perfect scenario -- it'd be considerably easier to develop. There are two key issues with this approach that have made us table it except as a very last resort:
The users of this product would be working in the field. That field could be quite literally anywhere--the streets of Manhattan, a disaster-stricken area with infrastructure that's been severely damaged or even destroyed, or the most war-ravaged third world country. If it were the streets of, say, Manhattan, there's no problem--their iOS or Android device will have 3G or Wi-Fi access just about anywhere they go. In the latter two scenarios (which are arguably more common in this industry), that connectivity may be very limited. The concern is whether the end user's ability to be productive or to see and share data with their colleagues will be too greatly restricted if they don't have a decent signal. To be fair though, even today they often aren't even using mobile devices, forcing them to go back to a headquarters type location or use radios to share information, effectively negating my point here. But if we're not going to significantly increase their productivity in the field, it just gives us pause to think through whether or not we have enough of a value proposition to ask them to fairly significantly change their methods of doing things.
To your latter point, no there's no convincing the stakeholders that this new system is the better approach. Even if there were, it would take years to do so. These forms are a part of a well-defined, decades-old standard used by literally thousands of organizations.