Couchbase lite, Search query taking very long time - ios

When I try to search the couchbase documents of size around 10K, the searching is taking very long time. Below are the code snippet. Can anyone optimize it or suggest me any alternative approach. Thank you.
1) Search function
func search(keyword:String) -> [[String:AnyObject]] {
var results:[[String:AnyObject]]=[]
let searchView = database.viewNamed(AppConstants().SEARCH)
if searchView.mapBlock == nil {
startIndexing()
}
let query = searchView.createQuery()
var docIds = Set<String>()
let result = try query.run()
while let row = result.nextRow() {
let key = "\(row.key)"
let keyArr = keyword.characters.split(" ")
for (index, element) in keyArr.enumerate() {
let keyItem = String(element)
if key.lowercaseString.containsString(keyItem.lowercaseString) {
let value = row.value as! [String:AnyObject]
let id = value["_id"] as? String
if id != nil && !docIds.contains(id!) {
results.append(value)
docIds.insert(id!)
}
}
}
}
}
2) Indexing
func startIndexing() {
let searchView = database.viewNamed(AppConstants().SEARCH)
if searchView.mapBlock == nil {
searchView.setMapBlock({ (doc, emit) in
let docType = doc[AppConstants().DOC_TYPE] as! String
if AppConstants().DOC_TYPE_CONTACT.isEqual(docType) {
self.parseJsonToKeyValues(doc)
for value in self.fields.values {
emit(value, doc)
}
self.fields.removeAll()
}
}, version: "1")
}
}
self.parseJsonToKeyValues(doc) will return me the key value store of my documents to index.

You're emitting the entire document along with every field for your view. This could easily cause your queries to be slow. It also seems unlikely you want to do this, unless you really need to be able to query against every field in your document.
It's considered best practice to set your map function right after opening the database. Waiting until right before you query may or may not slow you down.
See https://developer.couchbase.com/documentation/mobile/current/guides/couchbase-lite/native-api/view/index.html for more, especially the section labeled "Development Considerations".

Related

Pagination in firebase with identical child values

My data structure is as follows:
users:
user1:
-carModel: evo x
-username: importguy
-region: north east
user2:
-carModel: evo x
-username: evoguy
-region: north east
user3:
-carModel: mustang gt
-username: muscleguy
-region: south east
I want the user to be able to search for a car, say evo, and display results of users who own those particular cars. I need to paginate these results for my app. The problem is I can't figure out how to properly query this. Here is what i have so far.
func fetchUsersBy(car: String) {
if self.carCurrentKey == nil {
let ref = USER_REF.queryOrdered(byChild: "carModel").queryStarting(atValue: car).queryLimited(toFirst: 3)
ref.observeSingleEvent(of: .value, with: { (snapshot) in
guard let snap = snapshot.children.allObjects as? [FIRDataSnapshot] else { return }
guard let last = snapshot.children.allObjects.last as? FIRDataSnapshot else { return }
snap.forEach({ (snapshot) in
guard let userDict = snapshot.value as? Dictionary<String, AnyObject> else { return }
guard let carModel = userDict["carModel"] as? String else { return }
if carModel.contains(car) {
print(snapshot)
}
})
self.carCurrentKey = last.key
self.carCurrentValue = last.childSnapshot(forPath: "carModel").value as? String
})
} else {
// where to start next query?
let ref = USER_REF.queryOrder(byChild: "carModel").queryStarting(atValue: self.carCurrentValue)
}
}
I have to order the query by carModel, in order to group all of the users with that particular car type together in a snapshot. Since all the car models are the same value, I cannot figure out where to start or end the next query for the pagination. Using the reference i have in the else block starts the query at the same place as the block above. Any help or advice would be much appreciated.
I considered doing a fan out, and making a separate structure for car types. This would be difficult though.
For both startAt() and endAt(), you can pass a second value, childKey as shown here.
So your query will look something like this:
let ref = USER_REF.queryOrdered(byChild: "carModel").queryStarting(atValue: self.carCurrentValue, childKey: self.carCurrentKey).queryLimited(toFirst: 3+1)
Note that I used toFirst: 3+1. That's because, annoyingly, startAt() is inclusive and there's no way to skip the first record. So, since we started from the last record retrieved on the previous page, you will want to query for one extra record and discard the first result.
Here's a more complete example in JavaScript. Not familiar enough to translate this to Swift, but it should give you the algorithm in completion.
class Cursor {
constructor(baseRef, pageSize) {
this.baseRef = baseRef;
this.lastKey = null;
this.lastValue = null;
this.pageSize = pageSize;
}
next() {
let ref = this.baseRef;
if( this.lastValue !== null ) {
// a previous page has been loaded so get the next one using the previous value/key
// we have to start from the current cursor so add one to page size
ref = ref.startAt(this.lastValue, this.lastKey).limitToFirst(this.pageSize+1);
}
else {
// this is the first page
ref = ref.limitToFirst(this.pageSize);
}
return ref.once('value').then(snap => {
const keys = [];
const data = []; // store data in array so it's ordered
snap.forEach(ss => {
data.push(ss.val());
keys.push(ss.key);
});
if( this.lastValue !== null ) {
// skip the first value, which is actually the cursor
keys.shift();
data.shift();
}
// store the last loaded record
if( data.length ) {
const last = data.length - 1;
this.lastKey = keys[last];
this.lastValue = data[last].author;
}
return data;
});
}
}
And here's a working fiddle.
Keep in mind that this is a realtime data stream. So pagination is tricky. It's generally easier to just do infinite scroll than to try and maintain a realistic cursor on a moving data set (records can reorder when data changes, get deleted, added in the middle, etc).

Faster way to check if entry exists in Core Data

In my app when data is synced i can get 20k entries (from given timestamp) from the server that should be synced to the local device. For every entry i try to fetch it (if it exist already) and if doesn't i create new. The problem is that the whole operation is too slow - for 20k on iphone 5 is 10+ mins. Another solution that i though is to delete all entries from the given timestamp and create new entries for all returned entries and there will be no need to perform fetch for every single entry ? If someone have any advice will be nice. Here is sample code for the current state:
var logEntryToUpdate:LogEntry!
if let savedEntry = CoreDataRequestHelper.getLogEntryByID(inputID: inputID, fetchAsync: true) {
logEntryToUpdate = savedEntry
} else {
logEntryToUpdate = LogEntry(entity: logEntryEntity!, insertInto: CoreDataStack.sharedInstance.saveManagedObjectContext)
}
logEntryToUpdate.populateWithSyncedData(data: row, startCol: 1)
Here is the actual request method:
class func getLogEntryByID(inputID:Int64, fetchAsync:Bool) ->LogEntry? {
let logEntryRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "LogEntry")
logEntryRequest.predicate = NSPredicate(format: "inputId == %#", NSNumber(value: inputID as Int64))
logEntryRequest.fetchLimit = 1
do {
let mocToFetch = fetchAsync ? CoreDataStack.sharedInstance.saveManagedObjectContext : CoreDataStack.sharedInstance.managedObjectContext
if let fetchResults = try mocToFetch.fetch(logEntryRequest) as? [LogEntry] {
if ( fetchResults.count > 0 ) {
return fetchResults[0]
}
return nil
}
} catch let error as NSError {
NSLog("Error fetching Log Entries by inputID from core data !!! \(error.localizedDescription)")
}
return nil
}
Another thing that i tried is to check the count for specific request but again is too slow.
class func doesLogEntryExist(inputID:Int64, fetchAsync:Bool) ->Bool {
let logEntryRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "LogEntry")
logEntryRequest.predicate = NSPredicate(format: "inputId == %#", NSNumber(value: inputID as Int64))
//logEntryRequest.resultType = .countResultType
logEntryRequest.fetchLimit = 1
do {
let mocToFetch = fetchAsync ? CoreDataStack.sharedInstance.saveManagedObjectContext : CoreDataStack.sharedInstance.managedObjectContext
let count = try mocToFetch.count(for: logEntryRequest)
if ( count > 0 ) {
return true
}
return false
} catch let error as NSError {
NSLog("Error fetching Log Entries by inputID from core data !!! \(error.localizedDescription)")
}
return false
}
Whether fetching the instance or getting the count, you're still doing one fetch request per incoming record. That's going to be slow, and your code will be spending almost all of its time performing fetches.
One improvement is to batch up the records to reduce the number of fetches. Get multiple record IDs into an array, and then fetch all of them at once with a predicate like
NSPredicate(format: "inputId IN %#", inputIdArray)
Then go through the results of the fetch to see which IDs were found. Accumulate 50 or 100 IDs in the array, and you'll reduce the number of fetches by 50x or 100x.
Deleting all the entries for the timestamp and then re-inserting them might be good, but it's hard to predict. You'll have to insert all 20,000. Is that faster or slower than reducing the number of fetches? It's impossible to say for sure.
Based on Paulw11's comment, I came up with the following method to evaluate Structs being imported into Core Data.
In my example, I have a class where I store search terms. Within the search class, create a predicate which describes the values of the stuff within my array of structs.
func importToCoreData(dataToEvaluateArray: [YourDataStruct]) {
// This is what Paul described in his comment
let newDataToEvaluate = Set(dataToEvaluateArray.map{$0.id})
let recordsInCoreData = getIdSetForCurrentPredicate()
let newRecords = newDataToEvaluate.subtracting(recordsInCoreData)
// create an empty array
var itemsToImportArray: [YourDataStruct] = []
// and dump records with ids contained in newRecords into it
dataToEvaluateArray.forEach{ record in
if newRecords.contains(record.id) {
itemsToImportArray.append(record)
}
}
// THEN, import if you need to
itemsToImportArray.forEach { struct in
// set up your entity, properties, etc.
}
// Once it's imported, save
// You can save each time you import a record, but it'll go faster if you do it once.
do {
try self.managedObjectContext.save()
} catch let error {
self.delegate?.errorAlert(error.localizedDescription, sender: self)
}
self.delegate?.updateFetchedResultsController()
}
To instantiate recordsInCoreData, I created this method, which returns a Set of unique identifiers that exist in the managedObjectContext:
func getIdSetForCurrentPredicate() -> Set<String> {
let fetchRequest = NSFetchRequest<NSFetchRequestResult>(entityName: "YourEntity")
// searchQuery is a class I created with a computed property for a creating a predicate. You'll need to write your own predicate
fetchRequest.predicate = searchQuery.predicate
var existingIds: [YourEntity] = []
do {
existingIds = try managedObjectContext.fetch(fetchRequest) as! [YourEntity]
} catch let error {
delegate?.errorAlert(error.localizedDescription, sender: self)
}
return Set<String>(existingIds.map{$0.id})
}

Retrieving data from Firebase takes long on Swift

I'm currently working on a project which let users to rent extra spaces. You can think clone of airbnb. In this iOS app, I'm using Firebase for Back-end as a Service.
I want to retrieve the flats that satisfy the filtering options like, bed count, bedroom count, Smoking etc. Logically, I'm grouping all flats by time slots first, then fetching the flats satisfies filtered city, then i'm fetching the flats which satisfies the filters. Then I'm iterating all the flats one by one to fetch their images.
My Json tree is like:
filter_flats/FlatCity/FlatID
time_slots/timestamp/flatID:true
flat_images/FlatID
Here is what I'm trying:
import Foundation
import Firebase
import FirebaseStorage
protocol QuerymasterDelegate :class
{
func getFilteredFlats(filter:FilterModel, completion: #escaping ([FilteredFlat]) -> ())
}
class Querymaster:QuerymasterDelegate
{
var flatEndpoint = FIRFlat()
var returningFlats = [FilteredFlat]()
/////////////////////////////////////////////////////////////
//MARK: This method pulls all flats with required timeslot.//
/////////////////////////////////////////////////////////////
internal func getFilteredFlats(filter: FilterModel, completion: #escaping ([FilteredFlat]) -> ()) {
var zamanAraliginaUygunFlatlar = [String]()
FIRREF.instance().getRef().child("time_slots").queryOrderedByKey().queryStarting(atValue: filter.fromDate?.toTimeStamp()).queryEnding(atValue: filter.toDate?.toTimeStamp()).observeSingleEvent(of: .value, with: { (ss) in
for ts in ss.children.allObjects
{
let timeslotFlatObj = ts as! FIRDataSnapshot
let timeslotForFlat = timeslotFlatObj.value as! [String:Bool]
for x in timeslotForFlat
{
if (x.value == true && !zamanAraliginaUygunFlatlar.contains(x.key))
{
zamanAraliginaUygunFlatlar.append(x.key)
}
}
}
/////////////////////////////////////////////////////////
//MARK: This method pulls all flats with filtered city.//
/////////////////////////////////////////////////////////
FIRREF.instance().getRef().child("filter_flats/" + filter.city!).observe(.value, with: { (ss) in
var sehirdekiFlatler = [String:Flat]()
for i in ss.children.allObjects
{
let flt = Flat()
let flatObject = i as! FIRDataSnapshot
let mainDict = flatObject.value as! [String:Any]
flt.userID = (mainDict["userId"] as? String)!
flt.id = flatObject.key
flt.city = filter.city
flt.title = mainDict["title"] as? String
flt.bathroomCount = mainDict["bathroomCount"] as? Int
flt.bedCount = mainDict["bedCount"] as? Int
flt.bedroomCount = mainDict["bedroomCount"] as? Int
flt.internet = mainDict["internet"] as? Bool
flt.elevator = mainDict["elevator"] as? Bool
flt.heating = mainDict["heating"] as? Bool
flt.gateKeeper = mainDict["gateKeeper"] as?Bool
flt.parking = mainDict["parking"] as? Bool
flt.pool = mainDict["pool"] as? Bool
flt.smoking = mainDict["smoking"] as? Bool
flt.tv = mainDict["tv"] as? Bool
flt.flatCapacity = mainDict["capacity"] as? Int
flt.cooling = mainDict["cooling"] as? Bool
flt.price = mainDict["price"] as? Double
flt.washingMachine = mainDict["washingMachine"] as? Bool
sehirdekiFlatler[flt.id] = flt
if(!(zamanAraliginaUygunFlatlar).contains(flt.id))
{
if(filter.bathroomCount == nil || filter.bathroomCount! <= flt.bathroomCount!)
{
if(filter.bedCount == nil || filter.bedCount! <= flt.bedCount!)
{
if(filter.bedroomCount == nil || filter.bedroomCount! <= flt.bedroomCount!)
{
if(filter.internet == false || filter.internet! == flt.internet!)
{
if(filter.elevator == false || filter.elevator! == flt.elevator!)
{
if(filter.heating == false || filter.heating! == flt.heating!)
{
if(filter.gateKeeper == false || filter.gateKeeper! == flt.gateKeeper!)
{
if(filter.parking == false || filter.parking! == flt.parking!)
{
if(filter.pool == false || filter.pool! == flt.pool!)
{
if(filter.smoking! == false || filter.smoking! == flt.smoking!)
{
if(filter.tv! == false || filter.tv! == flt.tv!)
{
if(filter.capacity == nil || filter.capacity! <= flt.flatCapacity!)
{
if(filter.cooling == false || filter.cooling! == flt.cooling!)
{
if(filter.priceFrom == nil || filter.priceFrom! <= flt.price!)
{
if(filter.priceTo == nil || filter.priceTo! >= flt.price!)
{
if(filter.washingMachine == false || filter.washingMachine! == flt.washingMachine!)
{
let filteredFlat = FilteredFlat()
filteredFlat.flatCity = flt.city
filteredFlat.flatID = flt.id
filteredFlat.flatPrice = flt.price
filteredFlat.flatTitle = flt.title
filteredFlat.userID = flt.userID
self.returningFlats.append(filteredFlat)
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
////////////////////////////////////////////////////////////////
//MARK: This method pulls thumbnail image for returning Flats.//
////////////////////////////////////////////////////////////////
if self.returningFlats.count > 0
{
for a in self.returningFlats
{
FIRREF.instance().getRef().child("flat_images/" + a.flatID!).observeSingleEvent(of: .value, with: { (ss) in
let dict = ss.children.allObjects[0] as! FIRDataSnapshot
let obj = dict.value as! [String:String]
let flatImageDownloaded = FlatImageDownloaded(imageID: dict.key, imageDownloadURL: obj["downloadURL"]!)
a.flatThumbnailImage = flatImageDownloaded
completion(self.returningFlats)
})
}
}
})
})
}
}
The marked code This method pulls thumbnail image for returning Flats. is usually fetching data slower because of jumping among nodes of flats.
I didn't realized that what i'm doing wrong. Can you suggest more efficient method?
Thanks in advance.
This sounds like you are working with a significant amount of data, and we don't know your current UI but that may be the place to start to help reduce the bandwidth and speed up performance.
For example, in the UI there are three buttons that activate as the user selects the prior one.
Select City <- highlighted and ready to be tapped
Select Bedrooms <-Dimmed and not available yet
Select Internet <-Dimmed and not available yet
When the user taps Select City, they are presented a list of Cities available. this is easy and a fast Firebase query.
They tap Select Bedrooms and select 2. No real query needed here yet since it's a number between 1 and 5 (for example).
They tap Select Internet and this is a Yes/No or maybe None/Dial up/Broadband.
Now here's the key - the Structure.
City0
name: "some city"
Flats
flat_0
location: "City0"
bedrooms: "2"
internet: "Broadband"
filter: "City0_2_Broadband"
flat_1
location: "City0"
bedrooms: "4"
internet: "Dial up"
filter "City0_4_Dial up"
with this, you can search for any city, or any number of bedrooms or any internet as separate queries.
The real power is having the ability to perform an SQL-like 'and' query when you want to get very specific data; which flats are available and 4 bedrooms and dial up and in city0
query for: "City0_4_Dial up"
very fast and flexible and you can build the criteria for the query within the UI which limits the amount of data flowing into the app from Firebase, which makes it very fast.
If this is too far off base, let me know and I will edit and provide another option.
Firebase uses NoSQL database, so the typical querying would be very inefficient.
As far as I understand, your app requires a lot of query for the data, so Firebase itself does not the perfect solution for your app I guess..
Anyway, in the code, you can just request images in Dispatch async block, and that would make the process faster.
Also, fyi, storing images in FirebaseStorage and store the url in the FirebaseDatabase will help you reduce a lot of money per data.

iOS Xcode Swift Combining Input FirstName and LastName into a FullName Issue

I am having trouble combining a first and last name for this code I am running to update it to a tableviewcell. It seems as if it is not saving either first or last name and merely replacing it as I type new text in. I am not sure what is going on and I am certain the row numbers that I am using in my .plist match with the code. Any help with this or other alternatives are appreciated.
func textfieldTextWasChanged(newText: String, parentCell: CustomCell) {
let parentCellIndexPath = tblExpandable.indexPathForCell(parentCell)
let currentFullname = cellDescriptors[0][12]["primaryTitle"] as! String
let fullnameParts = currentFullname.componentsSeparatedByString(" ")
var newFullname = ""
if parentCellIndexPath?.row == 13 {
if fullnameParts.count == 2 {
newFullname = "\(newText) \(fullnameParts[1])"
}
else {
newFullname = newText
}
}
else {
newFullname = "\(fullnameParts[0]) \(newText)"
}
cellDescriptors[0][12].setValue(newFullname, forKey: "primaryTitle")
tblExpandable.reloadData()
}
What most likely is happening is that when you're calling tblExpandable.reloadData(), cellForRowAtIndexPath is getting called, and it's overwriting your changes that you're making here. You either need to do some cell caching or possibly consider using a UIScrollView instead of a UITableView.

Detect a Null value in NSDictionary

I have an NSDictionary that's populated from a JSON response from an API server. Sometimes the values for a key in this dictionary are Null
I am trying to take the given value and drop it into the detail text of a table cell for display.
The problem is that when I try to coerce the value into an NSString I get a crash, which I think is because I'm trying to coerce Null into a string.
What's the right way to do this?
What I want to do is something like this:
cell.detailTextLabel.text = sensor.objectForKey( "latestValue" ) as NSString
Here's an example of the Dictionary:
Printing description of sensor:
{
"created_at" = "2012-10-10T22:19:50.501-07:00";
desc = "<null>";
id = 2;
"latest_value" = "<null>";
name = "AC Vent Temp";
"sensor_type" = temp;
slug = "ac-vent-temp";
"updated_at" = "2013-11-17T15:34:27.495-07:00";
}
If I just need to wrap all of this in a conditional, that's fine. I just haven't been able to figure out what that conditional is. Back in the Objective-C world I would compare against [NSNull null] but that doesn't seem to be working in Swift.
You can use the as? operator, which returns an optional value (nil if the downcast fails)
if let latestValue = sensor["latestValue"] as? String {
cell.detailTextLabel.text = latestValue
}
I tested this example in a swift application
let x: AnyObject = NSNull()
if let y = x as? String {
println("I should never be printed: \(y)")
} else {
println("Yay")
}
and it correctly prints "Yay", whereas
let x: AnyObject = "hello!"
if let y = x as? String {
println(y)
} else {
println("I should never be printed")
}
prints "hello!" as expected.
You could also use is to check for the presence of a null:
if sensor["latestValue"] is NSNull {
// do something with null JSON value here
}
I'm using this combination and it also checks if object is not "null".
func isNotNull(object: AnyObject?) -> Bool {
guard let object = object else { return false }
return isNotNSNull(object) && isNotStringNull(object)
}
func isNotNSNull(object: AnyObject) -> Bool {
object.classForCoder != NSNull.classForCoder()
}
func isNotStringNull(object: AnyObject) -> Bool {
guard let object = object as? String where object.uppercaseString == "NULL" else {
return true
}
return false
}
It's not that pretty as extension but work as charm :)
NSNull is a class like any other. Thus you can use is or as to test an AnyObject reference against it.
Thus, here in one of my apps I have an NSArray where every entry is either a Card or NSNull (because you can't put nil in an NSArray). I fetch the NSArray as an Array and cycle through it, switching on which kind of object I get:
for card:AnyObject in arr {
switch card { // how to test for different possible types
case let card as NSNull:
// do one thing
case let card as Card:
// do a different thing
default:
fatalError("unexpected object in card array") // should never happen!
}
}
That is not identical to your scenario, but it is from a working app converted to Swift, and illustrates the full general technique.
my solution for now:
func isNull(someObject: AnyObject?) -> Bool {
guard let someObject = someObject else {
return true
}
return (someObject is NSNull)
}
tests look good so far...
I had a very similar problem and solved it with casting to the correct type of the original NSDictionary value. If your service returns a mixed type JSON object like this
{"id":2, "name":"AC Vent Temp", ...}
you'll have to fetch it's values like that.
var id:int = sensor.valueForKey("id") as Int;
var name:String? = sensor.valueForKey("name") as String;
This did solve my problem. See BAD_INSTRUCTION within swift closure

Resources