Using the new Combine framework in iOS 13.
Suppose I have an upstream publisher sending values at a highly irregular rate - sometimes seconds or minutes may go by without any values, and then a stream of values may come through all at once. I'd like to create a custom publisher that subscribes to the upstream values, buffers them and emits them at a regular, known cadence when they come in, but publishes nothing if they've all been exhausted.
For a concrete example:
t = 0 to 5000ms: no upstream values published
t = 5001ms: upstream publishes "a"
t = 5002ms: upstream publishes "b"
t = 5003ms: upstream publishes "c"
t = 5004ms to 10000ms: no upstream values published
t = 10001ms: upstream publishes "d"
My publisher subscribed to the upstream would produce values every 1 second:
t = 0 to 5000ms: no values published
t = 5001ms: publishes "a"
t = 6001ms: publishes "b"
t = 7001ms: publishes "c"
t = 7001ms to 10001ms: no values published
t = 10001ms: publishes "d"
None of the existing publishers or operators in Combine seem to quite do what I want here.
throttle and debounce would simply sample the upstream values at a certain cadence and drop ones that are missing (e.g. would only publish "a" if the cadence was 1000ms)
delay would add the same delay to every value, but not space them out (e.g. if my delay was 1000ms, it would publish "a" at 6001ms, "b" at 6002ms, "c" at 6003ms)
buffer seems promising, but I can't quite figure out how to use it - how to force it to publish a value from the buffer on demand. When I hooked up a sink to buffer it seemed to just instantly publish all the values, not buffering at all.
I thought about using some sort of combining operator like zip or merge or combineLatest and combining it with a Timer publisher, and that's probably the right approach, but I can't figure out exactly how to configure it to give the behavior I want.
Edit
Here's a marble diagram that hopefully illustrates what I'm going for:
Upstream Publisher:
-A-B-C-------------------D-E-F--------|>
My Custom Operator:
-A----B----C-------------D----E----F--|>
Edit 2: Unit Test
Here's a unit test that should pass if modulatedPublisher (my desired buffered publisher) works as desired. It's not perfect, but it stores events (including the time received) as they're received and then compares the time intervals between events, ensuring they are no smaller than the desired interval.
func testCustomPublisher() {
let expectation = XCTestExpectation(description: "async")
var events = [Event]()
let passthroughSubject = PassthroughSubject<Int, Never>()
let cancellable = passthroughSubject
.modulatedPublisher(interval: 1.0)
.sink { value in
events.append(Event(value: value, date: Date()))
print("value received: \(value) at \(self.dateFormatter.string(from:Date()))")
}
// WHEN I send 3 events, wait 6 seconds, and send 3 more events
passthroughSubject.send(1)
passthroughSubject.send(2)
passthroughSubject.send(3)
DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(6000)) {
passthroughSubject.send(4)
passthroughSubject.send(5)
passthroughSubject.send(6)
DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(4000)) {
// THEN I expect the stored events to be no closer together in time than the interval of 1.0s
for i in 1 ..< events.count {
let interval = events[i].date.timeIntervalSince(events[i-1].date)
print("Interval: \(interval)")
// There's some small error in the interval but it should be about 1 second since I'm using a 1s modulated publisher.
XCTAssertTrue(interval > 0.99)
}
expectation.fulfill()
}
}
wait(for: [expectation], timeout: 15)
}
The closest I've gotten is using zip, like so:
public extension Publisher where Self.Failure == Never {
func modulatedPublisher(interval: TimeInterval) -> AnyPublisher<Output, Never> {
let timerBuffer = Timer
.publish(every: interval, on: .main, in: .common)
.autoconnect()
return timerBuffer
.zip(self, { $1 }) // should emit one input element ($1) every timer tick
.eraseToAnyPublisher()
}
}
This properly attunes the first three events (1, 2, and 3), but not the second three (4, 5, and 6). The output:
value received: 1 at 3:54:07.0007
value received: 2 at 3:54:08.0008
value received: 3 at 3:54:09.0009
value received: 4 at 3:54:12.0012
value received: 5 at 3:54:12.0012
value received: 6 at 3:54:12.0012
I believe this is happening because zip has some internal buffering capacity. The first three upstream events are buffered and emitted on the Timer's cadence, but during the 6 second wait, the Timer's events are buffered - and when the second set ups upstream events are fired, there are already Timer events waiting in the queue, so they're paired up and fired off immediately.
This is an interesting problem. I played with various combinations of Timer.publish, buffer, zip, and throttle, but I couldn't get any combination to work quite the way you want. So let's write a custom subscriber.
What we'd really like is an API where, when we get an input from upstream, we also get the ability to control when the upstream delivers the next input. Something like this:
extension Publisher {
/// Subscribe to me with a stepping function.
/// - parameter stepper: A function I'll call with each of my inputs, and with my completion.
/// Each time I call this function with an input, I also give it a promise function.
/// I won't deliver the next input until the promise is called with a `.more` argument.
/// - returns: An object you can use to cancel the subscription asynchronously.
func step(with stepper: #escaping (StepEvent<Output, Failure>) -> ()) -> AnyCancellable {
???
}
}
enum StepEvent<Input, Failure: Error> {
/// Handle the Input. Call `StepPromise` when you're ready for the next Input,
/// or to cancel the subscription.
case input(Input, StepPromise)
/// Upstream completed the subscription.
case completion(Subscribers.Completion<Failure>)
}
/// The type of callback given to the stepper function to allow it to continue
/// or cancel the stream.
typealias StepPromise = (StepPromiseRequest) -> ()
enum StepPromiseRequest {
// Pass this to the promise to request the next item from upstream.
case more
// Pass this to the promise to cancel the subscription.
case cancel
}
With this step API, we can write a pace operator that does what you want:
extension Publisher {
func pace<Context: Scheduler, MySubject: Subject>(
_ pace: Context.SchedulerTimeType.Stride, scheduler: Context, subject: MySubject)
-> AnyCancellable
where MySubject.Output == Output, MySubject.Failure == Failure
{
return step {
switch $0 {
case .input(let input, let promise):
// Send the input from upstream now.
subject.send(input)
// Wait for the pace interval to elapse before requesting the
// next input from upstream.
scheduler.schedule(after: scheduler.now.advanced(by: pace)) {
promise(.more)
}
case .completion(let completion):
subject.send(completion: completion)
}
}
}
}
This pace operator takes pace (the required interval between outputs), a scheduler on which to schedule events, and a subject on which to republish the inputs from upstream. It handles each input by sending it through subject, and then using the scheduler to wait for the pace interval before asking for the next input from upstream.
Now we just have to implement the step operator. Combine doesn't give us too much help here. It does have a feature called “backpressure”, which means a publisher cannot send an input downstream until the downstream has asked for it by sending a Subscribers.Demand upstream. Usually you see downstreams send an .unlimited demand upstream, but we're not going to. Instead, we're going to take advantage of backpressure. We won't send any demand upstream until the stepper completes a promise, and then we'll only send a demand of .max(1), so we make the upstream operate in lock-step with the stepper. (We also have to send an initial demand of .max(1) to start the whole process.)
Okay, so need to implement a type that takes a stepper function and conforms to Subscriber. It's a good idea to review the Reactive Streams JVM Specification, because Combine is based on that specification.
What makes the implementation difficult is that several things can call into our subscriber asynchronously:
The upstream can call into the subscriber from any thread (but is required to serialize its calls).
After we've given promise functions to the stepper, the stepper can call those promises on any thread.
We want the subscription to be cancellable, and that cancellation can happen on any thread.
All this asynchronicity means we have to protect our internal state with a lock.
We have to be careful not to call out while holding that lock, to avoid deadlock.
We'll also protect the subscriber from shenanigans involving calling a promise repeatedly, or calling outdated promises, by giving each promise a unique id.
Se here's our basic subscriber definition:
import Combine
import Foundation
public class SteppingSubscriber<Input, Failure: Error> {
public init(stepper: #escaping Stepper) {
l_state = .subscribing(stepper)
}
public typealias Stepper = (Event) -> ()
public enum Event {
case input(Input, Promise)
case completion(Completion)
}
public typealias Promise = (Request) -> ()
public enum Request {
case more
case cancel
}
public typealias Completion = Subscribers.Completion<Failure>
private let lock = NSLock()
// The l_ prefix means it must only be accessed while holding the lock.
private var l_state: State
private var l_nextPromiseId: PromiseId = 1
private typealias PromiseId = Int
private var noPromiseId: PromiseId { 0 }
}
Notice that I moved the auxiliary types from earlier (StepEvent, StepPromise, and StepPromiseRequest) into SteppingSubscriber and shortened their names.
Now let's consider l_state's mysterious type, State. What are all the different states our subscriber could be in?
We could be waiting to receive the Subscription object from upstream.
We could have received the Subscription from upstream and be waiting for a signal (an input or completion from upstream, or the completion of a promise from the stepper).
We could be calling out to the stepper, which we want to be careful in case it completes a promise while we're calling out to it.
We could have been cancelled or have received completion from upstream.
So here is our definition of State:
extension SteppingSubscriber {
private enum State {
// Completed or cancelled.
case dead
// Waiting for Subscription from upstream.
case subscribing(Stepper)
// Waiting for a signal from upstream or for the latest promise to be completed.
case subscribed(Subscribed)
// Calling out to the stopper.
case stepping(Stepping)
var subscription: Subscription? {
switch self {
case .dead: return nil
case .subscribing(_): return nil
case .subscribed(let subscribed): return subscribed.subscription
case .stepping(let stepping): return stepping.subscribed.subscription
}
}
struct Subscribed {
var stepper: Stepper
var subscription: Subscription
var validPromiseId: PromiseId
}
struct Stepping {
var subscribed: Subscribed
// If the stepper completes the current promise synchronously with .more,
// I set this to true.
var shouldRequestMore: Bool
}
}
}
Since we're using NSLock (for simplicity), let's define an extension to ensure we always match locking with unlocking:
fileprivate extension NSLock {
#inline(__always)
func sync<Answer>(_ body: () -> Answer) -> Answer {
lock()
defer { unlock() }
return body()
}
}
Now we're ready to handle some events. The easiest event to handle is asynchronous cancellation, which is the Cancellable protocol's only requirement. If we're in any state except .dead, we want to become .dead and, if there's an upstream subscription, cancel it.
extension SteppingSubscriber: Cancellable {
public func cancel() {
let sub: Subscription? = lock.sync {
defer { l_state = .dead }
return l_state.subscription
}
sub?.cancel()
}
}
Notice here that I don't want to call out to the upstream subscription's cancel function while lock is locked, because lock isn't a recursive lock and I don't want to risk deadlock. All use of lock.sync follows the pattern of deferring any call-outs until after the lock is unlocked.
Now let's implement the Subscriber protocol requirements. First, let's handle receiving the Subscription from upstream. The only time this should happen is when we're in the .subscribing state, but .dead is also possible in which case we want to just cancel the upstream subscription.
extension SteppingSubscriber: Subscriber {
public func receive(subscription: Subscription) {
let action: () -> () = lock.sync {
guard case .subscribing(let stepper) = l_state else {
return { subscription.cancel() }
}
l_state = .subscribed(.init(stepper: stepper, subscription: subscription, validPromiseId: noPromiseId))
return { subscription.request(.max(1)) }
}
action()
}
Notice that in this use of lock.sync (and in all later uses), I return an “action” closure so I can perform arbitrary call-outs after the lock has been unlocked.
The next Subscriber protocol requirement we'll tackle is receiving a completion:
public func receive(completion: Subscribers.Completion<Failure>) {
let action: (() -> ())? = lock.sync {
// The only state in which I have to handle this call is .subscribed:
// - If I'm .dead, either upstream already completed (and shouldn't call this again),
// or I've been cancelled.
// - If I'm .subscribing, upstream must send me a Subscription before sending me a completion.
// - If I'm .stepping, upstream is currently signalling me and isn't allowed to signal
// me again concurrently.
guard case .subscribed(let subscribed) = l_state else {
return nil
}
l_state = .dead
return { [stepper = subscribed.stepper] in
stepper(.completion(completion))
}
}
action?()
}
The most complex Subscriber protocol requirement for us is receiving an Input:
We have to create a promise.
We have to pass the promise to the stepper.
The stepper could complete the promise before returning.
After the stepper returns, we have to check whether it completed the promise with .more and, if so, return the appropriate demand upstream.
Since we have to call out to the stepper in the middle of this work, we have some ugly nesting of lock.sync calls.
public func receive(_ input: Input) -> Subscribers.Demand {
let action: (() -> Subscribers.Demand)? = lock.sync {
// The only state in which I have to handle this call is .subscribed:
// - If I'm .dead, either upstream completed and shouldn't call this,
// or I've been cancelled.
// - If I'm .subscribing, upstream must send me a Subscription before sending me Input.
// - If I'm .stepping, upstream is currently signalling me and isn't allowed to
// signal me again concurrently.
guard case .subscribed(var subscribed) = l_state else {
return nil
}
let promiseId = l_nextPromiseId
l_nextPromiseId += 1
let promise: Promise = { request in
self.completePromise(id: promiseId, request: request)
}
subscribed.validPromiseId = promiseId
l_state = .stepping(.init(subscribed: subscribed, shouldRequestMore: false))
return { [stepper = subscribed.stepper] in
stepper(.input(input, promise))
let demand: Subscribers.Demand = self.lock.sync {
// The only possible states now are .stepping and .dead.
guard case .stepping(let stepping) = self.l_state else {
return .none
}
self.l_state = .subscribed(stepping.subscribed)
return stepping.shouldRequestMore ? .max(1) : .none
}
return demand
}
}
return action?() ?? .none
}
} // end of extension SteppingSubscriber: Publisher
The last thing our subscriber needs to handle is the completion of a promise. This is complicated for several reasons:
We want to protect against a promise being completed multiple times.
We want to protect against an older promise being completed.
We can be in any state when a promise is completed.
Thus:
extension SteppingSubscriber {
private func completePromise(id: PromiseId, request: Request) {
let action: (() -> ())? = lock.sync {
switch l_state {
case .dead, .subscribing(_): return nil
case .subscribed(var subscribed) where subscribed.validPromiseId == id && request == .more:
subscribed.validPromiseId = noPromiseId
l_state = .subscribed(subscribed)
return { [sub = subscribed.subscription] in
sub.request(.max(1))
}
case .subscribed(let subscribed) where subscribed.validPromiseId == id && request == .cancel:
l_state = .dead
return { [sub = subscribed.subscription] in
sub.cancel()
}
case .subscribed(_):
// Multiple completion or stale promise.
return nil
case .stepping(var stepping) where stepping.subscribed.validPromiseId == id && request == .more:
stepping.subscribed.validPromiseId = noPromiseId
stepping.shouldRequestMore = true
l_state = .stepping(stepping)
return nil
case .stepping(let stepping) where stepping.subscribed.validPromiseId == id && request == .cancel:
l_state = .dead
return { [sub = stepping.subscribed.subscription] in
sub.cancel()
}
case .stepping(_):
// Multiple completion or stale promise.
return nil
}
}
action?()
}
}
Whew!
With all that done, we can write the real step operator:
extension Publisher {
func step(with stepper: #escaping (SteppingSubscriber<Output, Failure>.Event) -> ()) -> AnyCancellable {
let subscriber = SteppingSubscriber<Output, Failure>(stepper: stepper)
self.subscribe(subscriber)
return .init(subscriber)
}
}
And then we can try out that pace operator from above. Since we don't do any buffering in SteppingSubscriber, and the upstream in general isn't buffered, we'll stick a buffer in between the upstream and our pace operator.
var cans: [AnyCancellable] = []
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
let erratic = Just("A").delay(for: 0.0, tolerance: 0.001, scheduler: DispatchQueue.main).eraseToAnyPublisher()
.merge(with: Just("B").delay(for: 0.3, tolerance: 0.001, scheduler: DispatchQueue.main).eraseToAnyPublisher())
.merge(with: Just("C").delay(for: 0.6, tolerance: 0.001, scheduler: DispatchQueue.main).eraseToAnyPublisher())
.merge(with: Just("D").delay(for: 5.0, tolerance: 0.001, scheduler: DispatchQueue.main).eraseToAnyPublisher())
.merge(with: Just("E").delay(for: 5.3, tolerance: 0.001, scheduler: DispatchQueue.main).eraseToAnyPublisher())
.merge(with: Just("F").delay(for: 5.6, tolerance: 0.001, scheduler: DispatchQueue.main).eraseToAnyPublisher())
.handleEvents(
receiveOutput: { print("erratic: \(Double(DispatchTime.now().rawValue) / 1_000_000_000) \($0)") },
receiveCompletion: { print("erratic: \(Double(DispatchTime.now().rawValue) / 1_000_000_000) \($0)") }
)
.makeConnectable()
let subject = PassthroughSubject<String, Never>()
cans += [erratic
.buffer(size: 1000, prefetch: .byRequest, whenFull: .dropOldest)
.pace(.seconds(1), scheduler: DispatchQueue.main, subject: subject)]
cans += [subject.sink(
receiveCompletion: { print("paced: \(Double(DispatchTime.now().rawValue) / 1_000_000_000) \($0)") },
receiveValue: { print("paced: \(Double(DispatchTime.now().rawValue) / 1_000_000_000) \($0)") }
)]
let c = erratic.connect()
cans += [AnyCancellable { c.cancel() }]
return true
}
And here, at long last, is the output:
erratic: 223394.17115897 A
paced: 223394.171495405 A
erratic: 223394.408086369 B
erratic: 223394.739186984 C
paced: 223395.171615624 B
paced: 223396.27056174 C
erratic: 223399.536717127 D
paced: 223399.536782847 D
erratic: 223399.536834495 E
erratic: 223400.236808469 F
erratic: 223400.236886323 finished
paced: 223400.620542561 E
paced: 223401.703613078 F
paced: 223402.703828512 finished
Timestamps are in units of seconds.
The erratic publisher's timings are, indeed, erratic and sometimes close in time.
The paced timings are always at least one second apart even when the erratic events occur less than one second apart.
When an erratic event occurs more than one second after the prior event, the paced event is sent immediately following the erratic event without further delay.
The paced completion occurs one second after the last paced event, even though the erratic completion occurs immediately after the last erratic event. The buffer doesn't send the completion until it receives another demand after it sends the last event, and that demand is delayed by the pacing timer.
I've put the the entire implementation of the step operator in this gist for easy copy/paste.
EDIT
There's an even simpler approach to the original one outlined below, which doesn't require a pacer, but instead uses back-pressure created by flatMap(maxPublishers: .max(1)).
flatMap sends a demand of 1, until its returned publisher, which we could delay, completes. We'd need a Buffer publisher upstream to buffer the values.
// for demo purposes, this subject sends a Date:
let subject = PassthroughSubject<Date, Never>()
let interval = 1.0
let pub = subject
.buffer(size: .max, prefetch: .byRequest, whenFull: .dropNewest)
.flatMap(maxPublishers: .max(1)) {
Just($0)
.delay(for: .seconds(interval), scheduler: DispatchQueue.main)
}
ORIGINAL
I know this is an old question, but I think there's a much simpler way to implement this, so I thought I'd share.
The idea is similar to a .zip with a Timer, except instead of a Timer, you would .zip with a time-delayed "tick" from a previously sent value, which can be achieved with a CurrentValueSubject. CurrentValueSubject is needed instead of a PassthroughSubject in order to seed the first ever "tick".
// for demo purposes, this subject sends a Date:
let subject = PassthroughSubject<Date, Never>()
let pacer = CurrentValueSubject<Void, Never>(())
let interval = 1.0
let pub = subject.zip(pacer)
.flatMap { v in
Just(v.0) // extract the original value
.delay(for: .seconds(interval), scheduler: DispatchQueue.main)
.handleEvents(receiveOutput: { _ in
pacer.send() // send the pacer "tick" after the interval
})
}
What happens is that the .zip gates on the pacer, which only arrives after a delay from a previously sent value.
If the next value comes earlier than the allowed interval, it waits for the pacer.
If, however, the next value comes later, then the pacer already has a new value to provide instantly, so there would be no delay.
If you used it like in your test case:
let c = pub.sink { print("\($0): \(Date())") }
subject.send(Date())
subject.send(Date())
subject.send(Date())
DispatchQueue.main.asyncAfter(deadline: .now() + 1.0) {
subject.send(Date())
subject.send(Date())
}
DispatchQueue.main.asyncAfter(deadline: .now() + 10.0) {
subject.send(Date())
subject.send(Date())
}
the result would be something like this:
2020-06-23 19:15:21 +0000: 2020-06-23 19:15:21 +0000
2020-06-23 19:15:21 +0000: 2020-06-23 19:15:22 +0000
2020-06-23 19:15:21 +0000: 2020-06-23 19:15:23 +0000
2020-06-23 19:15:22 +0000: 2020-06-23 19:15:24 +0000
2020-06-23 19:15:22 +0000: 2020-06-23 19:15:25 +0000
2020-06-23 19:15:32 +0000: 2020-06-23 19:15:32 +0000
2020-06-23 19:15:32 +0000: 2020-06-23 19:15:33 +0000
Could Publishers.CollectByTime be useful here somewhere?
Publishers.CollectByTime(upstream: upstreamPublisher.share(), strategy: Publishers.TimeGroupingStrategy.byTime(RunLoop.main, .seconds(1)), options: nil)
Just wanted to mention that I adapted Rob's answer from earlier and converted it to a custom Publisher, in order to allow for a single unbroken pipeline (see comments below his solution). My adaptation is below, but all the credit still goes to him. It also still makes use of Rob's step operator and SteppingSubscriber, as this custom Publisher uses those internally.
Edit: updated with buffer as part of the modulated operator, otherwise that would be required to be attached to buffer the upstream events.
public extension Publisher {
func modulated<Context: Scheduler>(_ pace: Context.SchedulerTimeType.Stride, scheduler: Context) -> AnyPublisher<Output, Failure> {
let upstream = buffer(size: 1000, prefetch: .byRequest, whenFull: .dropNewest).eraseToAnyPublisher()
return PacePublisher<Context, AnyPublisher>(pace: pace, scheduler: scheduler, source: upstream).eraseToAnyPublisher()
}
}
final class PacePublisher<Context: Scheduler, Source: Publisher>: Publisher {
typealias Output = Source.Output
typealias Failure = Source.Failure
let subject: PassthroughSubject<Output, Failure>
let scheduler: Context
let pace: Context.SchedulerTimeType.Stride
lazy var internalSubscriber: SteppingSubscriber<Output, Failure> = SteppingSubscriber<Output, Failure>(stepper: stepper)
lazy var stepper: ((SteppingSubscriber<Output, Failure>.Event) -> ()) = {
switch $0 {
case .input(let input, let promise):
// Send the input from upstream now.
self.subject.send(input)
// Wait for the pace interval to elapse before requesting the
// next input from upstream.
self.scheduler.schedule(after: self.scheduler.now.advanced(by: self.pace)) {
promise(.more)
}
case .completion(let completion):
self.subject.send(completion: completion)
}
}
init(pace: Context.SchedulerTimeType.Stride, scheduler: Context, source: Source) {
self.scheduler = scheduler
self.pace = pace
self.subject = PassthroughSubject<Source.Output, Source.Failure>()
source.subscribe(internalSubscriber)
}
public func receive<S>(subscriber: S) where S : Subscriber, Failure == S.Failure, Output == S.Input {
subject.subscribe(subscriber)
subject.send(subscription: PaceSubscription(subscriber: subscriber))
}
}
public class PaceSubscription<S: Subscriber>: Subscription {
private var subscriber: S?
init(subscriber: S) {
self.subscriber = subscriber
}
public func request(_ demand: Subscribers.Demand) {
}
public func cancel() {
subscriber = nil
}
}
I have a few asynchronous, network tasks that I need to perform on my app. Let's say I have 3 resources that I need to fetch from a server, call them A, B, and C. Let's say I have to finish fetching resource A first before fetching either B or C. Sometimes, I'd want to fetch B first, other times C first.
Right now, I just have a long-chained closure like so:
func fetchA() {
AFNetworking.get(completionHandler: {
self.fetchB()
self.fetchC()
})
}
This works for now, but the obvious limitation is I've hard-coded the order of execution into the completion handler of fetchA. Now, say I want to only fetchC after fetchB has finished in that completion handler, I'd have to go change my implementation for fetchB...
Essentially, I'd like to know if there's some magic way to do something like:
let orderedAsync = [fetchA, fetchB, fetchC]
orderedAsync.executeInOrder()
where fetchA, fetchB, and fetchC are all async functions, but fetchB won't execute until fetchA has finished and so on. Thanks!
You can use a serial DispatchQueue mixed with a DispatchGroup which will ensure that only one execution block will run at a time.
let serialQueue = DispatchQueue(label: "serialQueue")
let group = DispatchGroup()
group.enter()
serialQueue.async{ //call this whenever you need to add a new work item to your queue
fetchA{
//in the completion handler call
group.leave()
}
}
serialQueue.async{
group.wait()
group.enter()
fetchB{
//in the completion handler call
group.leave()
}
}
serialQueue.async{
group.wait()
group.enter()
fetchC{
group.leave()
}
}
Or if you are allowed to use a 3rd party library, use PromiseKit, it makes handling and especially chaining async methods way easier than anything GCD provides. See the official GitHub page for more info.
You can wrap an async method with a completion handler in a Promise and chain them together like this:
Promise.wrap(fetchA(completion:$0)).then{ valueA->Promise<typeOfValueB> in
return Promise.wrap(fetchB(completion:$0)
}.then{ valueB in
}.catch{ error in
//handle error
}
Also, all errors are propagated through your promises.
You could use combination of dispatchGroup and dispatchSemaphore to perform the asynchronous code blocks in sequence.
DispatchGroup will maintain the enter and leave to notify when all the task are completed.
DispatchSemaphore with value 1 will make sure only one block of task is executed
Sample
code where fetchA, fetchB, fetchC are functions with closure (completion handler)
// Create DispatchQueue
private let dispatchQueue = DispatchQueue(label: "taskQueue", qos: .background)
//value 1 indicate only one task will be performed at once.
private let semaphore = DispatchSemaphore(value: 1)
func sync() -> Void {
let group = DispatchGroup()
group.enter()
self.dispatchQueue.async {
self.semaphore.wait()
fetchA() { (modelResult) in
// success or failure handler
// semaphore signal to remove wait and execute next task
self.semaphore.signal()
group.leave()
}
}
group.enter()
self.dispatchQueue.async {
self.semaphore.wait()
fetchB() { (modelResult) in
// success or failure handler
// semaphore signal to remove wait and execute next task
self.semaphore.signal()
group.leave()
}
}
group.enter()
self.dispatchQueue.async {
self.semaphore.wait()
fetchC() { (modelResult) in
// success or failure handler
// semaphore signal to remove wait and execute next task
self.semaphore.signal()
group.leave()
}
}
group.notify(queue: .main) {
// Perform any task once all the intermediate tasks (fetchA(), fetchB(), fetchC()) are completed.
// This block of code will be called once all the enter and leave statement counts are matched.
}
}
Not sure why other answers are adding unnecessary code, what you are describing is already the default behavior for a serial queue:
let fetchA = { print("a starting"); sleep(1); print("a done")}
let fetchB = { print("b starting"); sleep(1); print("b done")}
let fetchC = { print("c starting"); sleep(1); print("c done")}
let orderedAsync = [fetchA, fetchB, fetchC]
let queue = DispatchQueue(label: "fetchQueue")
for task in orderedAsync{
queue.async(execute: task) //notice "async" here
}
print("all enqueued")
sleep(5)
"all enqueued" will print immediately, and each task will wait for the previous one to finish before it starts.
FYI, if you added attributes: .concurrent to your DispatchQueue initialization, then they wouldn't be guaranteed to execute in order. But even then you can use the .barrier flag when you want things to execute in order.
In other words, this would also fulfill your requirements:
let queue = DispatchQueue(label: "fetchQueue", attributes: .concurrent)
for task in orderedAsync{
queue.async(flags: .barrier, execute: task)
}
I am using Swift 3 GCD in order to perform some operations in my code. But I'm getting _dispatch_call_block_and_release error often. I suppose the reason behind this error is because different threads modify same variable, but I'm not sure how to fix problem. Here is my code and explanations:
I have one variable which is accessed and modified in different threads:
var queueMsgSent: Dictionary<Date,BTCommand>? = nil
func lock(obj: AnyObject, blk:() -> ()) {
objc_sync_enter(obj)
blk()
objc_sync_exit(obj)
}
func addMsgSentToQueue(msg: BTCommands) {
if queueMsgSent == nil {
queueMsgSent = Dictionary.init()
}
let currentDate = Date()
lock(obj: queueMsgSent as AnyObject) {
queueMsgSent?.updateValue(msg, forKey: currentDate)
}
}
func deleteMsgSentWithId(id: Int) {
if queueMsgSent == nil { return }
for (date, msg) in queueMsgSent! {
if msg.isAck() == false && msg.getId()! == id {
lock(obj: queueMsgSent as AnyObject) {
queueMsgSent?.removeValue(forKey: date)
}
}
}
}
func runSent() -> Void {
while(true) {
if queueMsgSent == nil { continue }
for (date, msg) in queueMsgSent! {
if msg.isSent() == false {
mainSearchView?.btCom?.write(str: msg.getCommand()!)
msg.setSent(val: true)
lastMsgSent = Date()
continue
}
if msg.isAck() == true {
lock(obj: queueMsgSent as AnyObject) {
queueMsgSent?.removeValue(forKey: date)
}
continue
}
}
}
}
I start runSent method as:
DispatchQueue.global().async(execute: runSent)
I need that runSent continuously check some conditions withinn queueMsgSent, and other functions addMsgSentToQueueue and deleteMsgSentWithId are called in main thread id necessary. I am using some locking mechanism but its not working properly
I strongly suggest you to use the DispatchQueue(s) provided by Grand Central Dispatch, they makes multithreading management much easier.
Command
Let's start with your command class
class Command {
let id: String
var isAck = false
var isSent = false
init(id:String) {
self.id = id
}
}
Queue
Now we can build our Queue class, it will provide the following functionalities
This is our class should not be confused with the concept of DispatchQueue!
push a Command into the queue
delete a Command from the queue
start the processing of all the elements into the queue
And now the code:
class Queue {
typealias Element = (date:Date, command:Command)
private var storage: [Element] = []
private let serialQueue = DispatchQueue(label: "serialQueue")
func push(command:Command) {
serialQueue.async {
let newElement = (Date(), command)
self.storage.append(newElement)
}
}
func delete(by id: String) {
serialQueue.async {
guard let index = self.storage.index(where: { $0.command.id == id }) else { return }
self.storage.remove(at: index)
}
}
func startProcessing() {
Timer.scheduledTimer(withTimeInterval: 10, repeats: true) { timer in
self.processElements()
}
}
private func processElements() {
serialQueue.async {
// send messages where isSent == false
let shouldBeSent = self.storage.filter { !$0.command.isSent }
for elm in shouldBeSent {
// TODO: add here code to send message
elm.command.isSent = true
}
// remove from storage message where isAck == true
self.storage = self.storage.filter { !$0.command.isAck }
}
}
}
How does it work?
As you can see the storage property is an array holding a list of tuples, each tuple has 2 components: Date and Command.
Since the storage array is accesses by multiple threads we need to make sure it is accessed in a thread safe way.
So each time we access storage we wrap our code into this
serialQueue.async {
// access self.storage safely
}
Each code we write into the closure 👆👆👆 shown above is added to our Serial Dispatch Queue.
The Serial Queue does process 1 closure at the time. That's why our storage property is accessed in a thread safe way!
Final consideration
The following block of code is evil
while true {
...
}
It does use all the available CPU time, it does freeze the UI (when executed on the main thread) and discharge the battery.
As you can see I replaced it with
Timer.scheduledTimer(withTimeInterval: 10, repeats: true) { timer in
self.processElements()
}
which calls self.processElements() every 10 seconds leaving plenty of time to the CPU to process other threads.
Of course it's up to you changing the number of seconds to better fit your scenario.
If you're uncomfortable with the objc mechanisms, you might take a look here. Using that, you create a PThreadMutex for the specific synchronizations you want to coordinate, then use mutex.fastsync{ *your code* } to segregate accesses. It's a simple, very lightweight mechanism using OS-level calls, but you'll have to watch out for creating deadlocks.
The example you provide depends on the object always being the same physical entity, because the objc lock uses the address as the ID of what's being synchronized. Because you seem to have to check everywhere for the existence of queueMsgSent, I'm wondering what the update value routine is doing - if it ever deletes the dictionary, expecting it to be created later, you'll have a potential race as different threads can be looking at different synchronizers.
Separately, your loop in runSent is a spin loop - if there's nothing to do, it's just going to burn CPU rather than waiting for work. Perhaps you could consider revising this to use semaphores or some more appropriate mechanism that would allow the workers to block when there's nothing to do?
I am designing a chat application and I have set up the following mechanism for users to upload messages. Basically, I push the messages onto a queue and upload them one after the other. When the queue is empty, I call finishedUploading which runs every second and reruns the task if there is anything in the queue.
var uploadQueue:[UploadMessage]?
let session = NSURLSession.sharedSession()
let lockQueue = dispatch_queue_create("com.dsdevelop.lockQueue", nil)
// RETURNS AMOUNT OF ITEMS STILL IN QUEUE
func getRemainingActiveUploads() -> Int {
return (self.uploadQueue != nil) ? self.uploadQueue!.count : 0
}
//REMOVES MESSAGE FROM QUEUE ONCE UPLOADED
func removeMessageFromUploadQueue(messageToBeRemoved : UploadMessage) {
if (uploadQueue != nil) {
dispatch_sync(lockQueue) {
self.uploadQueue = self.uploadQueue?.filter({$0.date!.compare(messageToBeRemoved.date!) == NSComparisonResult.OrderedSame})
}
}
}
var uploadTimer : NSTimer?
// CALLED ONLY WHEN UPLOADQUEUE IS EMPTY, RERUNS THE UPLOAD FUNCTION AFTER 1 SECOND
func finishedUploading() {
uploadTimer = NSTimer.scheduledTimerWithTimeInterval(1, target: self, selector: #selector(uploadAllLinks), userInfo: nil, repeats: false)
if (needToRefetch) {
needToRefetch = false
newMessageReceived()
}
}
func uploadAllLinks()
{
uploadTimer?.invalidate()
uploadTimer = nil
// suspending queue so they don't all finish before we can show it
session.delegateQueue.suspended = true
session.delegateQueue.maxConcurrentOperationCount = 1
let myUrl = NSURL(string: "http://****")
// create tasks
if (uploadQueue != nil) {
if (uploadQueue?.count > 0) {
for message in uploadQueue!
{
let request = NSMutableURLRequest(URL:myUrl!)
request.HTTPMethod = "POST"
request.timeoutInterval = 10
request.HTTPShouldHandleCookies=false
var postString = "sender=" + message.sender!
request.HTTPBody = postString.dataUsingEncoding(NSUTF8StringEncoding);
let dltask = session.dataTaskWithRequest(request, completionHandler: { (data, response, error) in
if data != nil
{
do {
let jsonArray = try NSJSONSerialization.JSONObjectWithData(data_fixed!, options:[])
dispatch_async(dispatch_get_main_queue(), {
if let errorToken = jsonArray["error"] as! Bool? {
if !errorToken {
self.uploadQueue = self.uploadQueue!.filter({$0.date!.compare(message.date!) != NSComparisonResult.OrderedSame})
let remaining = self.getRemainingActiveUploads()
print("Downloaded. Remaining: \(remaining)")
if (remaining == 0) {
self.finishedUploading()
}
}
else {
let remaining = self.getRemainingActiveUploads()
print("Downloaded. Remaining: \(remaining)")
if (remaining == 0) {
self.finishedUploading()
}
}
}
else {
let remaining = self.getRemainingActiveUploads()
print("Downloaded. Remaining: \(remaining)")
if (remaining == 0) {
self.finishedUploading()
}
}
})
}
catch {
print("Error: \(error)")
}
}
})
print("Queuing task \(dltask)")
dltask.resume()
}
session.delegateQueue.suspended = false
}
else {
finishedUploading()
}
// resuming queue so all tasks run
}
}
Now this works fine in the following two cases :
Queue is empty -> finishedUploading gets called and uploadAllLinks is run every second to check for items in uploadQueue
Queue has one item -> the one item gets posted, remaining == 0 hence finishedUploading is called
However, whenever the queue has more than one item, the first one gets uploaded, if remaining == 0 fails, and then nothing happens. I don't understand why the for loop is not run for the other items in the queue at this point.
I suspect that the problem is your 10-second timeout interval. That begins ticking as soon as the data task is created and terminates the task if it remains idle (without receiving new data) for more than ten seconds.
If you have multiple tasks and the OS is only allowed to upload one or two of them at a time, then any task that is queued up waiting to start will never complete. I don't think the documentation mentions that.
In practice, this design makes NSURLSession's queueing less than ideal, and as a result, most folks seem to write their own queues and handle the concurrency limiting on their own, ensuring that each task is created right before it should start running. I would suggest doing something similar:
Create a method that starts the next upload in the queue or calls the "everything complete" method if the queue is empty—basically the body of your loop.
Instead of the loop itself, call that method to start the first upload.
In your completion handler (inside that method), call that method semi-recursively to start the next upload.
Also, 10 seconds is way too short for the timeout interval unless your device is mounted to a wall and is on Wi-Fi with a guaranteed solid signal. Flaky Wi-Fi and weak cellular signals can result in serious latency, so IIRC, the default is 120 seconds, though I've read 60 in various places. Either way, you do not want to use 10 seconds. Such a short timeout would pretty much guarantee that your app will be hopelessly unreliable.
I am writing an application that depends on data from various sites/service, and involves performing calculations based on data from these different sources to produce an end product.
I have written an example class with two functions below that gathers data from the two sources. I have chosen to make the functions different, because sometimes we apply different authentication methods depending on the source, but in this example I have just stripped them down to their simplest form. Both of the functions use Alamofire to fire off and handle the requests.
I then have an initialisation function, which says if we have successfully gathered data from both sources, then load another nib file, otherwise wait up to for seconds, if no response has been returned, then load a server error nib file.
I've tried to make this example as simple as possible. Essentially. This is the kind of logic I would like to follow. Unfortunately it appears this does not currently work in its current implementation.
import Foundation
class GrabData{
var data_source_1:String?
var data_source_2:String?
init(){
// get data from source 1
get_data_1{ data_source_1 in
println("\(data_source_1)")
}
// get data from source 2
get_data_2{ data_source_1 in
println("\(data_source_1)")
}
var timer = 0;
while(timer<5){
if((data_source_1 == nil) && (data_source_2 == nil)){
// do nothing unless 4 seconds has elapsed
if (timer == 4){
// load server error nib
}
}else{
// load another nib, and start manipulating data
}
// sleep for 1 second
sleep(1)
timer = timer+1
}
}
func get_data_1(completionHandler: (String) -> ()) -> () {
if let datasource1 = self.data_source_1{
completionHandler(datasource1)
}else{
var url = "http://somewebsite.com"
Manager.sharedInstance.request(.GET, url).responseString {
(request, response, returnedstring, error) in
println("getting data from source 1")
let datasource1 = returnedstring
self.data_source_1 = datasource1
completionHandler(datasource1!)
}
}
}
func get_data_2(completionHandler: (String) -> ()) -> () {
if let datasource2 = self.data_source_2{
completionHandler(datasource2)
}else{
var url = "http://anotherwebsite.com"
Manager.sharedInstance.request(.GET, url).responseString {
(request, response, returnedstring, error) in
println("getting data from source 2")
let datasource2 = returnedstring
self.data_source_2 = datasource2
completionHandler(datasource2!)
}
}
}
}
I know that i could put the second closure within the first inside the init function, however, I don't think this would be best practice and I am actually pulling from more than 2 sources, so the closure would be n closures deep.
Any help to figuring out the best way to checking if multiple data sources gave a valid response, and handling that appropriately would be much appreciated.
Better than that looping process, which would block the thread, you could use dispatch group to keep track of when the requests were done. So "enter" the group before issuing each of the requests, "leave" the group when the request is done, and set up a "notify" block/closure that will be called when all of the group's tasks are done.
For example, in Swift 3:
let group = DispatchGroup()
group.enter()
retrieveDataFromURL(url1, parameters: firstParameters) {
group.leave()
}
group.enter()
retrieveDataFromURL(url2, parameters: secondParameters) {
group.leave()
}
group.notify(queue: .main) {
print("both requests done")
}
Or, in Swift 2:
let group = dispatch_group_create()
dispatch_group_enter(group)
retrieveDataFromURL(url1, parameters: firstParameters) {
dispatch_group_leave(group)
}
dispatch_group_enter(group)
retrieveDataFromURL(url2, parameters: secondParameters) {
dispatch_group_leave(group)
}
dispatch_group_notify(group, dispatch_get_main_queue()) {
print("both requests done")
}
The other approach is to wrap these requests within an asynchronous NSOperation subclass (making them cancelable, giving you control over constraining the degree of concurrency, etc.), but that's more complicated, so you might want to start with dispatch groups as shown above.