I have my function
func synch(cID: Int) -> AnyPublisher<Void, Error> {
summariesCache
.lastUpdateTimestamp(cID: cID)
.prefix(1)
.flatMap { self.sendRequest(withTimestamp: $0, cID: cID) }
.map { self.cache.uCache(with: $0.data) }
.eraseToAnyPublisher()
}
which accepts one argument cID: Int. I want to refactor it so that this function accepts argument cID: [Int], which is an array, and performs all operations inside on every array's element still returning AnyPublisher<Void, Error>, just like the use of DispatchGroup inside for loop for asynchronous operations and completion on end of all.
Assuming your methods (sendRequest, uCache etc) all do things asynchronously, you can get the publisher of the cIDs, and flat map that to what you currently have.
func synch(cIDs: [Int]) -> AnyPublisher<Void, Error> {
cIDs.publisher.flatMap { cID in
summariesCache
.lastUpdateTimestamp(cID: cID)
.prefix(1)
.flatMap { self.sendRequest(withTimestamp: $0, cID: cID) }
.map { self.cache.uCache(with: $0.data) }
.eraseToAnyPublisher()
}.eraseToAnyPublisher()
}
If you are sending requests to a web server, sending many requests in a short time could cause a heavy load on the server. I would recommend that you send all the cIDs in a single request, if that is possible.
Related
I'm familiar with JavaScript promises, but I'm new to swift and Firebase, and I don't have anyone to ask on my team. I've tried researching different ways of handling async operations without callback hell, but I can't understand how to make it work with firebase functions. Right now I'm using a really complicated mess of DispatchGroups and callbacks to make the code somewhat work, but I really want to make it cleaner and more maintainable.
My code looks something like this (error handling removed for conciseness):
var array = []
let dispatch = DispatchGroup()
db.collection("documentA").getDocuments() { (querySnapshot, err) in
for document in querySnapshot.documents
dispatch.enter()
let dataA = document.data()["dataA"]
...
db.collection("documentB").documents(dataA).getDocuments() { (document, error) in
let dataB = document.data()["dataB"]
...
db.collection("documentC").documents(dataB).getDocuments() { (document, error) in
let dataC = document.data()["dataC"]
let newObject = NewObject(dataA,dataB,dataC)
self.array.append(newObject)
dispatch.leave()
}
}
}
//Use dispatch group to notify main queue to update tableView using contents of this array
Does anyone have any recommended learning resources or advice on how I can tackle this problem?
I recommend you consider bringing in the RxFirebase library. Rx is a great way to clean up nested closures (callback hell).
In looking over your sample code, the first thing you have to understand is that only so much can be done. The problem itself has a lot of essential complexity. Also, there's a lot going on in this code that can be broken out. Once you do that, you can boil down the problem to the following:
import Curry // the Curry library is in Cocoapods
func example(db: Firestore) -> Observable<[NewObject]> {
let getObjects = curry(getData(db:collectionId:documentId:))(db)
let xs = getObjects("documentA")("dataA")
let xys = xs.flatMap { parentsAndChildren(fn: getObjects("documentB"), parent: { $0 }, xs: $0) }
let xyzs = xys.flatMap { parentsAndChildren(fn: getObjects("documentC"), parent: { $0.1 }, xs: $0) }
return xyzs.mapT { NewObject(dataA: $0.0.0, dataB: $0.0.1, dataC: $0.1) }
}
Note that this is making extensive use of higher order functions, so understanding those will help a lot. If you don't want to use higher order functions, you could use classes instead, but the amount of code you would have to write would at least double and you would likely have problems with memory cycles.
To make the above so simple requires some support code:
func getData(db: Firestore, collectionId: String, documentId: String) -> Observable<[String]> {
return db.collection(collectionId).rx.getDocuments()
.map { getData(documentId: documentId, snapshot: $0) }
}
func parentsAndChildren<X>(fn: (String) -> Observable<[String]>, parent: (X) -> String, xs: [X]) -> Observable<[(X, String)]> {
Observable.combineLatest(xs.map { x in
fn(parent(x)).map { apply(x: x, ys: $0) }
})
.map { $0.flatMap { $0 } }
}
extension ObservableType {
func mapT<T, U>(_ transform: #escaping (T) -> U) -> Observable<[U]> where Element == [T] {
map { $0.map(transform) }
}
}
The getData(db:collectionId:documentId:) function asks for the strings in the collection associated with the document.
The parentsAndChildren(fn:parent:xs:) function is probably the most complex. It will extract the appropriate parent object from the generic X type, get the children from the server and roll them up into a single dimensional array of parents and children. For example if the parents are ["a", "b"], the children of "a" are ["w", "x"] and the children of "b" are ["y", "z"], then the function will output [("a", "w"), ("a", "x"), ("b", "y"), ("b", "z")] (contained in an Observable.)
The Observable.mapT(_:) function allows us to map through an Observable Array of objects and do something to them. Of course, you could just do xyzs.map { $0.map { NewObject(dataA: $0.0.0, dataB: $0.0.1, dataC: $0.1) } }, but I feel this is cleaner.
Here is the support code for the above functions:
extension Reactive where Base: CollectionReference {
func getDocuments() -> Observable<QuerySnapshot> {
Observable.create { [base] observer in
base.getDocuments { snapshot, error in
if let snapshot = snapshot {
observer.onNext(snapshot)
observer.onCompleted()
}
else {
observer.onError(error ?? RxError.unknown)
}
}
return Disposables.create()
}
}
}
func getData(documentId: String, snapshot: QuerySnapshot) -> [String] {
snapshot.documents.compactMap { $0.data()[documentId] as? String }
}
func apply<X>(x: X, ys: [String]) -> [(X, String)] {
ys.map { (x, $0) }
}
The Reactive.getDocuments() function actually makes the firebase request. Its job is to turn the callback closure into an object so that you can deal with it easier. This is the piece that RxFirebase should give you, but as you can see, it's pretty easy to write it on your own.
The getData(documentId:snapshot:) function just extracts the appropriate data out of the snapshot.
The app(x:ys:) function is what keeps the whole thing in a single dimensional array by copying the X for each child.
Lastly, notice that most of the functions above are easily and independently unit testable and the ones that aren't are exceptionally simple...
I'm exploring Combine Swift with this project https://github.com/sgl0v/TMDB
and I'm trying to replace its imageLoader with something that supports Combine: https://github.com/JanGorman/MapleBacon
The project has a function that returns the type AnyPublisher<UIImage?, Never>.
But the imageLoader MapleBacon library returns the type AnyPublisher<UIImage, Error>.
So I'm trying to convert types with this function:
func convert(_ loader: AnyPublisher<UIImage, Error>) -> AnyPublisher<UIImage?, Never> {
// here.
}
I actually found a question that is kinda similar to mine, but the answers weren't helpful:
https://stackoverflow.com/a/58234908/3231194
What I've tried to so far (Matt's answer to the linked question).
The sample project has this function:
func loadImage(for movie: Movie, size: ImageSize) -> AnyPublisher<UIImage?, Never> {
return Deferred { return Just(movie.poster) }
.flatMap({ poster -> AnyPublisher<UIImage?, Never> in
guard let poster = movie.poster else { return .just(nil) }
let url = size.url.appendingPathComponent(poster)
let a = MapleBacon.shared.image(with: url)
.replaceError(with: UIImage(named: "")!) // <----
})
.subscribe(on: Scheduler.backgroundWorkScheduler)
.receive(on: Scheduler.mainScheduler)
.share()
.eraseToAnyPublisher()
}
if I do replaceError,
I get the type Publishers.ReplaceError<AnyPublisher<UIImage, Error>>
BUT, I was able to solve this one, by extending the library.
extension MapleBacon {
public func image(with url: URL, imageTransformer: ImageTransforming? = nil) -> AnyPublisher<UIImage?, Never> {
Future { resolve in
self.image(with: url, imageTransformer: imageTransformer) { result in
switch result {
case .success(let image):
resolve(.success(image))
case .failure:
resolve(.success(UIImage(named: "")))
}
}
}
.eraseToAnyPublisher()
}
}
First, you need to map a UIImage to a UIImage?. The sensible way to do this is of course to wrap each element in an optional.
Then, you try to turn a publisher that sometimes produces errors to a publisher that Never produces errors. You replaceError(with:) an element of your choice. What element should you replace errors with? The natural answer, since your publisher now publishes optional images, is nil! Of course, assertNoFailure works syntactically too, but you might be downloading an image here, so errors are very likely to happen...
Finally, we need to turn this into an AnyPublisher by doing eraseToAnyPublisher
MapleBacon.shared.image(with: url)
.map(Optional.some)
.replaceError(with: nil)
.eraseToAnyPublisher()
In the following code, which is a simplified version of a more elaborate pipeline, "Done processing" is never called for 2.
Why is that?
I suspect this is a problem due to the demand, but I cannot figure out the cause.
Note that if I remove the combineLatest() or the compactMap(), the value 2 is properly processed (but I need these combineLatest and compactMap for correctness, in my real example they are more involved).
var cancellables = Set<AnyCancellable>([])
func process<T>(_ value: T) -> AnyPublisher<T, Never> {
return Future<T, Never> { promise in
print("Starting processing of \(value)")
DispatchQueue.global(qos: .userInitiated).asyncAfter(deadline: .now() + 0.1) {
promise(.success(value))
}
}.eraseToAnyPublisher()
}
let s = PassthroughSubject<Int?, Never>()
s
.print("Combine->Subject")
.combineLatest(Just(true))
.print("Compact->Combine")
.compactMap { value, _ in value }
.print("Sink->Compact")
.flatMap(maxPublishers: .max(1)) { process($0) }
.sink {
print("Done processing \($0)")
}
.store(in: &cancellables)
s.send(nil)
// Give time for flatMap to finish
Thread.sleep(forTimeInterval: 1)
s.send(2)
It sounds like a bug of combineLatest. When a downstream request additional demand "synchronously" (as per-print publisher output), that demand doesn't flow upstream.
One way to overcome this is to wrap the downstream of combineLatest in a flatMap:
s
.combineLatest(Just(true))
.flatMap(maxPublishers: .max(1)) {
Just($0)
.compactMap { value, _ in value }
.flatMap { process($0) }
}
.sink {
print("Done processing \($0)")
}
.store(in: &cancellables)
The outer flatMap now creates the back pressure, and the inner flatMap doesn't need it anymore.
I am working with a web API which delivers results up to a given limit (pageSize parameter of the request). If the number of results surpasses this limit, the response message is pre-populated with an URL to which the follow-up request can be made to fetch more results. If there are even more results, this is again indicated in the same manner.
My intend is to fetch all results at once.
Currently I have something like the following request and response structures:
// Request structure
struct TvShowsSearchRequest {
let q: String
let pageSize: Int?
}
// Response structure
struct TvShowsSearchResponse: Decodable {
let next: String?
let total : Int
let searchTerm : String
let searchResultListShow: [SearchResult]?
}
When resolving the problem 'old style' using completion handlers, I had to write a handler, which is triggering a 'handle more' request with the URL of the response:
func handleResponse(request: TvShowsSearchRequest, result: Result<TvShowsSearchResponse, Error>) -> Void {
switch result {
case .failure(let error):
fatalError(error.localizedDescription)
case .success(let value):
print("> Total number of shows matching the query: \(value.total)")
print("> Number of shows fetched: \(value.searchResultListShow?.count ?? 0)")
if let moreUrl = value.next {
print("> URL to fetch more entries \(moreUrl)")
// start recursion here: a new request, calling the same completion handler...
dataProvider.handleMore(request, nextUrl: moreUrl, completion: handleResponse)
}
}
}
let request = TvShowsSearchRequest(query: "A", pageSize: 50)
dataProvider.send(request, completion: handleResponse)
Internally the send and handleMore functions are both calling the same internalSend which is taking the request and the url, to call afterwards URLSession.dataTask(...), check for HTTP errors, decode the response and call the completion block.
Now I want to use the Combine framework and use a Publisher which is providing the paged responses automatically, without the need to call for another Publisher.
I have therefore written a requestPublisher function which takes request and the (initial) url and returns a URLSession.dataTaskPublisher which checks for HTTP errors (using tryMap), decode the response.
Now I have to ensure that the Publisher automatically "renews" itself whenever the last emitted value had a valid next URL and the completion event occurs.
I've found that there is a Publisher.append method which would exactly do this, but the problem I had so far: I want to append only under a certain condition (=valid next).
The following pseudo-code illustrates it:
func requestPublisher(for request: TvShowsSearchRequest, with url: URL) -> AnyPublisher<TvShowsSearchResponse, Error> {
// ... build urlRequest, skipped here ...
let apiCall = self.session.dataTaskPublisher(for: urlRequest)
.tryMap { data, response -> Data in
guard let httpResponse = response as? HTTPURLResponse else {
throw APIError.server(message: "No HTTP response received")
}
if !(200...299).contains(httpResponse.statusCode) {
throw APIError.server(message: "Server respondend with status: \(httpResponse.statusCode)")
}
return data
}
.decode(type: TvShowsSearchResponse.self, decoder: JSONDecoder())
.eraseToAnyPublisher()
return apiCall
}
// Here I'm trying to use the Combine approach
var moreURL : String?
dataProvider.requestPublisher(request)
.handleEvents(receiveOutput: {
moreURL = $0.next // remember the "next" to fetch more data
})
.append(dataProvider.requestPublisher(request, next: moreURL)) // this does not work, because moreUrl was not prepared at the time of creation!!
.sink(receiveCompletion: { print($0) }, receiveValue: { print($0) })
.store(in: &cancellableSet)
I suppose there are people out there who have already resolved this problem in a reactive way. Whenever I find a doable solution, it involves again recursion. I don't think this is how a proper solution should look like.
I'm looking for a Publisher which is sending the responses, without me providing a callback function. Probably there must be a solution using Publisher of Publishers, but I'm not yet understanding it.
After the comment of #kishanvekariya I've tried to build everything with multiple publishers:
The mainRequest publisher which is getting the response to the "main" request.
A new urlPublisher which is receiving all the next URLs of the "main" or any follow-up requests.
A new moreRequest publisher which is fetching for each value of urlPublisher a new request, sending all next URLs back to the urlPublisher.
Then I tried to attach the moreRequest publisher to the mainRequest with append.
var urlPublisher = PassthroughSubject<String, Error>()
var moreRequest = urlPublisher
.flatMap {
return dataProvider.requestPublisher(request, next: $0)
.handleEvents(receiveOutput: {
if let moreURL = $0.next {
urlPublisher.send(moreURL)
}
})
}
var mainRequest = dataProvider.requestPublisher(request)
.handleEvents(receiveOutput: {
if let moreURL = $0.next {
urlPublisher.send(moreURL)
}
})
.append(moreRequest)
.sink(receiveCompletion: { print($0) }, receiveValue: { print($0) })
.store(in: &cancellableSet)
But this still does not work... I always get the result of the "main" request. All follow up requests are missing.
It seems that I've found the solution myself.
The idea is, that I have an urlPublisher which is initialized with the first URL which is then executed and may feed a next URL to the urlPublisher and by doing so causing a follow-up request.
let url = endpoint(for: request) // initial URL
let urlPublisher = CurrentValueSubject<URL, Error>(url)
urlPublisher
.flatMap {
return dataProvider.requestPublisher(for: request, with: $0)
.handleEvents(receiveOutput: {
if let next = $0.next, let moreURL = URL(string: self.transformNextUrl(nextUrl: next)) {
urlPublisher.send(moreURL)
} else {
urlPublisher.send(completion: .finished)
}
})
}
.sink(receiveCompletion: { print($0) }, receiveValue: { print($0) })
.store(in: &cancellableSet)
So in the end, I used composition of two publishers and flatMap instead of the non-functional append. Probably this is also the solution one would target from the start...
Before I dive into the response, just wanted to say that requesting all pages at once might not be the best idea:
it adds stress on the server, likely the paged API is there for a reason, to avoid costly operations on the backend
there is always a discussion on what to do when a page request fails: you report error, you report the partial results, you retry the request?
keep in mind that once you launch your product and have many clients requests for the whole data set of TV shows, the backend server might become overloaded, and generating even more failures
Now, back to the business, assuming your requestPublisher is properly working, what you can do is to write a publisher that chains those calls, and doesn't report values until the last page was received.
The code might look like this:
func allPages(for request: TvShowsSearchRequest, with url: URL) -> AnyPublisher<TvShowsSearchResponse, Error> {
// helper function to chain requests for all pages
func doRequest(with pageURL: URL, accumulator: TvShowsSearchResponse) -> AnyPublisher<TvShowsSearchResponse, Error> {
requestPublisher(for: request, with: pageURL)
.flatMap { (r: TvShowsSearchResponse) -> AnyPublisher<TvShowsSearchResponse, Error> in
if let next = r.next, let nextURL = URL(string: next) {
// we have a `next` url, append the received page,
// and make the next request
return doRequest(with: nextURL, accumulator: accumulator.accumulating(from: r))
} else {
// no more pages, we have the response already build up
// just report it
return Just(accumulator).setFailureType(to: Error.self).eraseToAnyPublisher()
}
}
.eraseToAnyPublisher()
}
return doRequest(with: url, accumulator: TvShowsSearchResponse())
}
You basically use a TvShowsSearchResponse as an accumulator for the results of the chained request.
The above code also needs the following extension:
extension TvShowsSearchResponse {
init() {
self.init(next: nil, total: 0, searchTerm: "", searchResultListShow: nil)
}
func accumulating(from other: TvShowsSearchResponse) -> TvShowsSearchResponse {
TvShowsSearchResponse(
next: nil,
total: other.total,
searchTerm: other.searchTerm,
searchResultListShow: (searchResultListShow ?? []) + (other.searchResultListShow ?? []))
}
}
, as for clarity the code that accumulates the values of searchResultListShow was placed into a dedicated extension.
I have a function that fires an API request to the server. I want to loop over it until it returns false (no more data).
func getData(id: Int) -> Observable<Bool> {
return Observable.create { observer in
// Alamofire request
// parse data
// if can decode,
// return true and increment page's property
// otherwise false
// error, if there's a problem
}
}
First try: I've tried using takeWhile, like : getData(id).takeWhile {$0}. It only iterate over my function 1x only.
Second try: using a range. The problem here is that even if my getData function errors out, instead of stopping, the loop continues !
Observable.range(start: 1, count: 100)
.enumerated()
.flatMapLatest({ _ in
self.getData(someID)
})
.subscribe(onNext: { _ in
// save to DB
observer.onNext(true)
observer.onCompleted()
}, onError: { error in
observer.onError(error)
})
.disposed(by: self.disposeBag)
Is there a way to do it, rx style ?
Something like this?
let callApiTrigger = BehaviorRelay<Bool>(value: true)
let callApiEnoughTimes = callApiTrigger.asObservable()
.takeWhile { $0 }
.flatMap { _ in
return getData(someId)
}
.do(onNext: { (apiResult: Bool) in
callApiTrigger.accept(apiResult)
})
The reason why takeWhile and take(X) do not work, is because they do not resubscribe to the Observable. A network request observable typically emits one value at the most.
What you are looking for requires some form of recursion / resubscribing. If you want to it hardcore Rx I suggest you reverse engineer the retry operator to suit your needs. Although I consider myself experienced with RxSwift, that seems like a bridge too far.
Fortunately, I whipped up a recursive approach that works just fine too :)
class PageService {
typealias Page = (pageNumber: Int, items: [String]?)
private func getPage(_ pageNumber: Int) -> Observable<Page> {
let pageToReturn = Page(pageNumber: pageNumber, items: (pageNumber < 3) ? ["bert, henk"] : nil)
return Observable<Page>
.just(pageToReturn)
.delay(0.5, scheduler: MainScheduler.instance)
}
func allPagesFollowing(pageNumber: Int) -> Observable<Page> {
let objectToReturnInCaseOfError = Page(pageNumber: pageNumber + 1, items: nil)
return getPage(pageNumber + 1)
// in order to error out gracefully, you could catch the network error and interpret it as 0 results
.catchErrorJustReturn(objectToReturnInCaseOfError)
.flatMap { page -> Observable<Page> in
// if this page has no items, do not continue the recursion
guard page.items != nil else { return .empty() }
// glue this single page together with all the following pages
return Observable<Page>.just(page)
.concat(self.allPagesFollowing(pageNumber: page.pageNumber))
}
}
}
_ = PageService().allPagesFollowing(pageNumber: 0)
.debug("get page")
.subscribe()
This will print:
2018-03-30 11:56:24.707: get page -> subscribed
2018-03-30 11:56:25.215: get page -> Event next((pageNumber: 1, data: Optional(["bert, henk"])))
2018-03-30 11:56:25.718: get page -> Event next((pageNumber: 2, data: Optional(["bert, henk"])))
2018-03-30 11:56:26.223: get page -> Event completed
2018-03-30 11:56:26.223: get page -> isDisposed