Create a model using Create ML and find classes in metaData - ios

I am using Create ML to create a .mlmodel file using a Image Classifier project.
Create ML, creates this file for 2 classes.
After dragging the created ML Model file , and adding it to compile sources, below code is able to find the file and create the model.
But the meta data does not show classes, any reason why? How would I know the classes in the mlmodel file?
let error: NSError! = nil
guard let modelURL = Bundle.main.url(forResource: "ObjectDetector", withExtension: "mlmodelc") else {
return
}
do {
let model = try MLModel(contentsOf: modelURL)
let visionModel = try VNCoreMLModel(for: model)
let metaData = model.modelDescription.metadata[.creatorDefinedKey] as! [String:String]
// ERROR: It could not find metaData["classes"]
let allClasses = metaData["classes"]!.components(separatedBy: ",")
let objectRecognition = VNCoreMLRequest(model: visionModel, completionHandler: { (request, error) in
DispatchQueue.main.async(execute: {
// perform all the UI updates on the main queue
if let results = request.results {
self.drawVisionRequestResults(results)
}
})
})
self.requests = [objectRecognition]
} catch let error as NSError {
print("Model loading went wrong: \(error)")
}
Please note I am not getting "Model class has not been generated yet.".
https://stackoverflow.com/questions/462476…
Clicking on ML Model file shows Automatically generated Swift model class.

The class data is part of the mlmodel file but not in the metadata. You can add it to the metadata if you want, but you'll have to write a short Python script for that.
As of iOS 14, there is MLModelDescription.classLabels that also lets you access the list of class names.

Related

Pre-heat CoreML MLModel

I'm seeing up to 3x-4x slower results on my first prediction on my MLModel. Seems like once you run the first prediction it's heating the model. Couldn't find any official information about it.
What is the right way to pre-heat my model? Call it on an async thread on app launch with dummy data?
As #Alladinian mentioned, starting from iOS 14 you can use the MLModel.load(contentsOf:...) function.
Here is a usage example for pre-loading a local .mlmodel file:
if let url = Bundle.main.url(forResource: "myModel", withExtension: "mlmodelc") {
let config = MLModelConfiguration()
MLModel.load(contentsOf: url, configuration: config) { [weak self] result in
switch result {
case .success(let model):
print("Model loaded and ready.")
let modelWrapper = MyModelWrapper(model: model)
case .failure(let error):
print("Error loading model: \(error)")
}
}
}
If you want to pre-load an externally fetched model, make sure it's compiled using MLModel.compileModel(at:).

Why does my MLKit model always returns an error when processing an image?

I have a Google MLKit model for labeling an Image after capturing the image, but everytime I tried to process the Image, it always give me this error:
label process error:: Pipeline failed to fully start: Calculator::Open() for node "ClassifierClientCalculator" failed: #vk The TFLite Model Metadata must not contain label maps when text_label_map_file is used.
Here's my MLKit image labeler configuration code (this code is based on MLKit's documentation):
private func configureModelSource() { // Called in viewDidLoad()
guard let manifestPath = Bundle.main.path(forResource: "filename", ofType: "json") else { return }
guard let localModel = LocalModel(manifestPath: manifestPath) else { return }
let options = CustomImageLabelerOptions(localModel: localModel)
options.confidenceThreshold = NSNumber(value: 0.0)
imageLabeler = ImageLabeler.imageLabeler(options: options)
}
private func processImage(with image: UIImage) { // Called after capturing an Image
guard imageLabeler != nil else { return }
let visionImage = VisionImage(image: image)
visionImage.orientation = image.imageOrientation
imageLabeler?.process(visionImage) { labels, error in
guard error == nil, let labels = labels, !labels.isEmpty else {
print("label process error:: \(error?.localizedDescription ?? "nil")")
return
}
for label in labels {
// Do something...
}
}
}
Is there anyway to solve this? For context, the model.tflite file was updated. The file before the one that gives me this error works as expected. But the new model.tflite file always gives me this error everytime I run my app. Is this a file-related error or did I do something wrong with my code that I have to also update it?
Here's my understanding based on the error message:
Given you are using the LocalModel(manifestPath: manifestPath) API, it is expecting a legacy TFLite model format where the label map is provided through a separate text file and the model.tflite itself does not contain the label map. That's why your file before your model update works.
To use your updated model.tflite (which seems to contain the lab map inside its metadata), I think you can try the following to use the model.tflite file directly with the custom models API without going through the filename.json manifest:
guard let modelPath = Bundle.main.path(forResource: "model", ofType: "tflite") else { return }
guard let localModel = LocalModel(path: modelPath) else { return }
You can check out the documentation about custom models here: https://developers.google.com/ml-kit/vision/image-labeling/custom-models/ios

Update Plist data without erasing old data

I'm trying to make a file downloader application using swift and cocoa. I am using plist for the download history. Reading the data works however, writing the data will erase the previous data and replace it for the new data.
Here is the code
let newdownloaditem = downloadList(root: [downloadListt(downloadURL: response.url!.absoluteString, fileName: response.suggestedFilename!)])
// This is a codeable method
let encoder = PropertyListEncoder()
encoder.outputFormat = .xml
let pListFilURL = uniqueDataDir()?.appendingPathComponent("downloads.plist")
do {
let data = try encoder.encode(newdownloaditem)
try data.write(to: pListFilURL!)
// Here is the problem
} catch {
print(error)
}
// Here is the codeable
public struct downloadList: Codable {
let root: [downloadListt]
}
public struct downloadListt: Codable {
let downloadURL: String
let fileName: String
}
Here is an image of what happens
The contents got erased
Thanks!
You are indeed replacing the previous data with your new one.
You need to retrieve the previous data.
Append your new data to it.
Save that combination
let newItem = downloadListt(downloadURL: response.url!.absoluteString,
fileName: response.suggestedFilename!)
var allItems: [downloadListt] = []
allItems.append(contentsOf: previousList.root)
allitems.append(newItem)
let newList = downloadList(root: allItems)
...
let data = try encoder.encode(newList)
try data.write(to: pListFilURL!)
Unrelated but recommended (it's convention):
You should starting naming your struct/classes with an uppercase: downloadList => DownloadList
I would avoid naming downloadListt, it's unredable it's hard at first look to make a difference between downloadListt and downloadList. Instead, name it maybe DownloadItem. More readable.

Error 13010 "Object does not exist" while downloading jpeg image from Firebase storage using getData()

Language : Swift 5
iOS: 13.2
macOS: Catalina 10.15.4
Firebase Storage Rules:
service firebase.storage {
match /b/{bucket}/o {
match /{allPaths=**} {
allow read, write: if request.auth!=null;
}
}
}
The code to upload image and save download URL: (Which works fine, because I can see images uploaded to storage and their respective download URLs stored to real-time database.)
let storageRef = Storage.storage().reference()
//Let's upload all workout pictures
let uploadPicsRef =
storageRef.child("WORKOUTDATA/USERS/"+self.UID!).child("WHITEBOARDWORKOUTS")
let uploadNumberRef = uploadPicsRef.child("\(String(describing: workoutNum))")
let workoutPicturesRef = uploadNumberRef.child("WORKOUTPICTURES")
let workoutPicURLRef = workoutRef.child("WORKOUTPICTURESURL")
var count = 0
var picNumber = 0
//workoutPictures list/array contains images selected from iPhone Gallery, using
//UIImagePickerController
for workoutPic in self.workoutPictures
{
let workoutPicData = workoutPic.jpegData(compressionQuality: 1.0)!
count = count + 1
let pictureName = "Picture\(count).jpg"
// Upload the file to the path in pictureRef
let pictureRef = workoutPicturesRef.child("\(pictureName)")
let metaData = StorageMetadata()
metaData.contentType = "image/jpg"
pictureRef.putData(workoutPicData, metadata: metaData) { (metadata, error) in
if error != nil {
print("Error while uploading image")
}
else
{
pictureRef.downloadURL { (url, err) in
picNumber = picNumber + 1
workoutPicURLRef.child("Picture\(picNumber)").setValue(url?.absoluteString)
}
}
}
}
The code to download image:
let myGroup = DispatchGroup()
let workoutPicUrls = snapshot.childSnapshot(forPath: "WORKOUTPICTURESURL")
for url in workoutPicUrls.children
{
myGroup.enter()
let snap = url as! DataSnapshot
let link = snap.value as? String
let storageRef = Storage.storage().reference()
let pictureRef = storageRef.root().child(link!)
DispatchQueue.main.async {
pictureRef.getData(maxSize: 1*2000000*2000000) { (data, err) in
if (err != nil) {
print(err!)
print(err!.localizedDescription)
} else {
let pic = UIImage(data: data!)
workoutPicsArray.append(pic!)
myGroup.leave()
}
}
}
}
Error:
Error Domain=FIRStorageErrorDomain Code=-13010 "Object https:/firebasestorage.googleapis.com/v0/b/trainer-8cb52.appspot.com/o/WORKOUTDATA%2FUSERS%2F1K7WV1alYIeWPAsFC6YMoJKPFSj1%2FWHITEBOARDWORKOUTS%2F5%2FWORKOUTPICTURES%2FPicture1.jpg?alt=media&token=785ab8c7-1e08-4ad3-a542-c9e6313eb547 does not exist." UserInfo={object=https:/firebasestorage.googleapis.com/v0/b/trainer-8cb52.appspot.com/o/WORKOUTDATA%2FUSERS%2F1K7WV1alYIeWPAsFC6YMoJKPFSj1%2FWHITEBOARDWORKOUTS%2F5%2FWORKOUTPICTURES%2FPicture1.jpg?alt=media&token=785ab8c7-1e08-4ad3-a542-c9e6313eb547, ResponseBody={
"error": {
"code": 404,
"message": "Not Found. Could not get object",
"status": "GET_OBJECT"
}
}, bucket=trainer-8cb52.appspot.com, data={length = 115, bytes = 0x7b0a2020 22657272 6f72223a 207b0a20 ... 54220a20 207d0a7d }, data_content_type=application/json; charset=UTF-8, NSLocalizedDescription=Object https:/firebasestorage.googleapis.com/v0/b/trainer-8cb52.appspot.com/o/WORKOUTDATA%2FUSERS%2F1K7WV1alYIeWPAsFC6YMoJKPFSj1%2FWHITEBOARDWORKOUTS%2F5%2FWORKOUTPICTURES%2FPicture1.jpg?alt=media&token=785ab8c7-1e08-4ad3-a542-c9e6313eb547 does not exist., ResponseErrorDomain=com.google.HTTPStatus, ResponseErrorCode=404}
What I have tried so far:
Checked firebase storage rules.
When I paste the path https:/firebasestorage.googleapis.com/v0/b/trainer8cb52.appspot.com/o/WORKOUTDATA%2FUSERS%2F1K7WV1alYIeWPAsFC6YMoJKPFSj1%2FWHITEBOARDWORKOUTS%2F5%2FWORKOUTPICTURES%2FPicture1.jpg?alt=media&token=785ab8c7-1e08-4ad3-a542-c9e6313eb547 in chrome browser window, the expected image opens.
Set the maxSize to a ridiculously high number 1*2000000*2000000.
Thank you!
Is it possible that you are storing the full https URL in the database and are trying to create a reference by adding the full https url as a child to the storage reference?
I think you should try to either store just the path and name in your database or you change your download code to use the https URL.
// Create a reference from an HTTPS URL
// Note that in the URL, characters are URL escaped!
let httpsReference = storage.reference(forURL: "https://firebasestorage.googleapis.com/b/bucket/o/images%20stars.jpg")
httpsReference.getData(maxSize: ...
Also you're running your getData method inside DispatchQueue.main.async. getData has itself a completion handler and might take some time, when you run that inside of DispatchQueue.main.async it will block your code until the download is done. Only put code that update the UI inside DispatchQueue.main.async. In your case as soon as you do something with your workoutPicsArray or the UIImage to update your view.
Have a look here to see if you can figure out how you are actually trying to get the data. It might be helpful to put a print() after each line to see what you are creating and using at what point.
Download Files on iOS

getMetadata() not returning metadata from file in Firebase Storage iOS

I try to get metadata of files from Firebase Storage, in particular date of create (because I want to compare date of local file and date of Cloud file and change file, if needed). I use getMetadata { (metadata, error) in ...} method, but I don't get result in my completion. Control doesn't go to completion! My code is below
let metadataURL = Storage.storage().reference().child("tutorials/how_to/use_masking/cover.jpg")
metadataURL.getMetadata { (metadata, error) in
if let error = error {
print(error)
} else {
if let data = metadata {
let dict = data.dictionaryRepresentation()
print(dict)
}
}
}
And I want to say now, that image is really in this path, because I get this image with getData(...) method in the next step.

Resources