Adding reference images to ARKit from the app - ios

Question

I was learning to use ARKIT and I was wondering if there is a way to add reference images(images to be recognised) from within the app(based on the user's choice). As per the documentation, this can be done by adding the reference images to the Assets(during the development phase) which limits the usability of the app. I was wondering if there is a way where we can download/add these images based on the user's choice and use these images as reference image(within the app).

Answer 1

If you have a look at the documentation for: ARReferenceImage you will note that there are two methods of generating ARReferenceImages manually:
init(CGImage, orientation: CGImagePropertyOrientation, physicalWidth: CGFloat)
init(CVPixelBuffer, orientation: CGImagePropertyOrientation, physicalWidth: CGFloat)
The one that you will need if you are downloading from a Server is the first one, which requires the use of a CGImage.
So any image(s) which you download will need to be converted using this method.
To download images from a Server first you will need to use a URLSession to download these to a location on your device e.g. the Documents Directory.
A simple example of this would look like so:
/// Downloads An Image From A Remote URL
func downloadImageTask(){
//1. Get The URL Of The Image
guard let url = URL(string: "http://www.blackmirrorz.tech/images/BlackMirrorz/blackMirrorzLogo.png") else { return }
//2. Create The Download Session
let downloadSession = URLSession(configuration: URLSession.shared.configuration, delegate: self, delegateQueue: nil)
//3. Create The Download Task & Run It
let downloadTask = downloadSession.downloadTask(with: url)
downloadTask.resume()
}
}
Having created the URLSession you would then need to register for the URLSessionDownloadDelegate and the following method:
func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL)
Whereby the location parameter refers to the:
A file URL for the temporary file. Because the file is temporary, you
must either open the file for reading or move it to a permanent
location in your app’s sandbox container directory before returning
from this delegate method.
As such your callback might look like so:
func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
//1. Create The Filename
let fileURL = getDocumentsDirectory().appendingPathComponent("image.png")
//2. Copy It To The Documents Directory
do {
try FileManager.default.copyItem(at: location, to: fileURL)
print("Successfuly Saved File \(fileURL)")
} catch {
print("Error Saving: \(error)")
}
}
Whereby I use the following function to get the users Documents Directory:
/// Returns The Documents Directory
///
/// - Returns: URL
func getDocumentsDirectory() -> URL {
let paths = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)
let documentsDirectory = paths[0]
return documentsDirectory
}
Now we have downloaded the images we would then create a function to retrieve these and return a Set of ARReferenceImage which is needed by ARWorldTrackingConfiguration.
/// Creates A Set Of ARReferenceImages From All PNG Content In The Documents Directory
///
/// - Returns: Set<ARReferenceImage>
func loadedImagesFromDirectoryContents() -> Set<ARReferenceImage>?{
var index = 0
var customReferenceSet = Set<ARReferenceImage>()
let documentsDirectory = getDocumentsDirectory()
do {
let directoryContents = try FileManager.default.contentsOfDirectory(at: documentsDirectory, includingPropertiesForKeys: nil, options: [])
let filteredContents = directoryContents.filter{ $0.pathExtension == "png" }
filteredContents.forEach { (url) in
do{
//1. Create A Data Object From Our URL
let imageData = try Data(contentsOf: url)
guard let image = UIImage(data: imageData) else { return }
//2. Convert The UIImage To A CGImage
guard let cgImage = image.cgImage else { return }
//3. Get The Width Of The Image
let imageWidth = CGFloat(cgImage.width)
//4. Create A Custom AR Reference Image With A Unique Name
let customARReferenceImage = ARReferenceImage(cgImage, orientation: CGImagePropertyOrientation.up, physicalWidth: imageWidth)
customARReferenceImage.name = "MyCustomARImage\(index)"
//4. Insert The Reference Image Into Our Set
customReferenceSet.insert(customARReferenceImage)
print("ARReference Image == \(customARReferenceImage)")
index += 1
}catch{
print("Error Generating Images == \(error)")
}
}
} catch {
print("Error Reading Directory Contents == \(error)")
}
//5. Return The Set
return customReferenceSet
}
So to put this last function into place you would do the following:
let detectionImages = loadedImagesFromDirectoryContents()
configuration.detectionImages = detectionImages
augmentedRealitySession.run(configuration, options: [.resetTracking, .removeExistingAnchors])
Hope it helps...