I am trying to understand the use of Future.cancel() with asyncio. The Python documentation is very light on this. I've had no success with existing questions on here or search engines. I just want to understand happens when a task is awaiting a future which is cancelled.
Here is my code:
import asyncio
async def foo(future):
await asyncio.sleep(3)
future.cancel()
async def bar(future):
await future
print("hi")
async def baz(future):
await bar(future)
print("ho")
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.create_task(baz(future))
loop.create_task(foo(future))
loop.run_forever()
"hi" is not seen printed. So I initially guessed bar was returning at the line await future in the case of a cancel.
However, "ho" is not printed either. So it seems logical that cancelling a future never yields back to tasks awaiting it? But then these tasks are sitting in the event loop forever? This seems undesirable, where have I misunderstood?
In this case the answer lies in the documentation, but you have to look for it a bit. First, a reminder of what it means to await a future:
# the expression:
x = await future
# is equivalent to:
... magically suspend the coroutine until the future.done() becomes true ...
x = future.result()
In other words, once the execution of the coroutine that contains await resumes, the value of the await statement will be the result() of the awaited future.
The question is: when you cancel a future, what is its result? The documentation says:
If the Future has been cancelled, this method raises a CancelledError exception.
So when someone cancel a future you awaited, the await future expression will raise an exception! This neatly explains why bar doesn't print hi (because await future has raised), and why baz doesn't print ho (because await bar(...) has raised).
A traceback is never printed because loop.create_task spawns the coroutine in the "background" (of sorts) - if no one inspects the return value, the exception will be lost. And since you threw away the task object returned by create_task and used run_forever to have the loop running forever, the loop just continues running, waiting (forever) for new tasks to somehow arrive.
If you changed the code to actually collect the result of bar, you would easily observe the CancelledError:
if __name__ == '__main__':
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.create_task(foo(future))
loop.run_until_complete(baz(future))
Output:
Traceback (most recent call last):
File "xxx.py", line 19, in <module>
loop.run_until_complete(baz(future))
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 266, in result
raise CancelledError
concurrent.futures._base.CancelledError
Related
I need to do a lot of work, but luckily it's easy to decouple into different tasks for asynchronous execution. Some of those depend on each other, and it's perfectly clear to me how on task can await multiple others to get their results. However, I don't know how I can have multiple different tasks await the same coroutine, and both get the result. The Documentation also doesn't mention this case as far as I can find.
Consider the following minimal example:
from asyncio import create_task, gather
async def TaskA():
... # This is clear
return result
async def TaskB(task_a):
task_a_result = await task_a
... # So is this
return result
async def TaskC(task_a):
task_a_result = await task_a
... # But can I even do this?
return result
async def main():
task_a = create_task(TaskA())
task_b = create_task(TaskB(task_a))
task_c = create_task(TaskC(task_a))
gather(task_b, task_c) # Can I include task_a here to signal the intent of "wait for all tasks"?
For the actual script, all tasks do some database operations, some of which involve foreign keys, and therefore depend on other tables already being filled. Some depend on the same table. I definitely need:
All tasks run once, and only once
Some tasks are dependent on others being done before starting.
In brief, the question is, does this work? Can I await the same instantiated coroutine multiple times, and get the result every time? Or do I need to put awaits in main(), and pass the result? (which is the current setup, and I don't like it.)
You can await the same task multiple times:
from asyncio import create_task, gather, run
async def coro_a():
print("executing coro a")
return 'a'
async def coro_b(task_a):
task_a_result = await task_a
print("from coro_b: ", task_a_result)
return 'b'
async def coro_c(task_a):
task_a_result = await task_a
print("from coro_a: ", task_a_result)
return 'c'
async def main():
task_a = create_task(coro_a())
print(await gather(coro_b(task_a), coro_c(task_a)))
if __name__ == "__main__":
run(main())
Will output:
executing coro a
from coro_b: a
from coro_a: a
['b', 'c']
What you can not do is to await the same coroutine multiples times:
...
async def main():
task_a = coro_a()
print(await gather(coro_b(task_a), coro_c(task_a)))
...
Will raise RuntimeError: cannot reuse already awaited coroutine.
As long as you schedule your coroutine coro_a using create_task your code will work.
import 'dart:async';
void main() async {
Future.microtask(() => print(1));
Future.value(2).then(print);
Future.sync(() => print(3));
Future.sync(() => 4).then(print);
}
Output I observe in dartpad:
3
1
2
4
Why didn't microtask get executed first?And what is different in the two Future.sync functions that printed them in different orders.
An important aspect with the .then() method, is the following you can find in the documentation:
When this future completes with a value, the onValue callback will be called with that value. If this future is already completed, the callback will not be called immediately, but will be scheduled in a later microtask.
https://api.dart.dev/stable/2.15.1/dart-async/Future/then.html
So what happens is that:
You schedule a microtask to call print(1).
Future.value(2) is done in sync and the following .then will be done in a microtask. The queue of microtasks is now: print(1), print(2).
The third line is sync and runs immediately in full so we run print(3). This gives the first line in your input.
We schedule a new microtask because of the .then(). The queue of microtasks is now: print(1), print(2), print(4).
After main() is done we run the microtasks in order they come in which explains the rest of your output.
An important note is that you are never awaiting on any of the returned Future from the .then() methods so any async stuff will first be executed when main() is done.
The question should be simple enough, but I couldn't find anything about it.
I have an async Python program that contains a rather long-running task that I want to be able to suspend and restart at arbitrary points (arbitrary of course meaning everywhere where there's an await keyword).
I was hoping there was something along the lines of task.suspend() and task.resume() but it seems there isn't.
Is there an API for this on task- or event-loop-level or would I need to do this myself somehow? I don't want to place an event.wait() before every await...
What you're asking for is possible, but not trivial. First, note that you can never have suspends on every await, but only on those that result in suspension of the coroutine, such as asyncio.sleep(), or a stream.read() that doesn't have data ready to return. Awaiting a coroutine immediately starts executing it, and if the coroutine can return immediately, it does so without dropping to the event loop. await only suspends to the event loop if the awaitee (or its awaitee, etc.) requests it. More details in these questions: [1], [2], [3], [4].
With that in mind, you can use the technique from this answer to intercept each resumption of the coroutine with additional code that checks whether the task is paused and, if so, waits for the resume event before proceeding.
import asyncio
class Suspendable:
def __init__(self, target):
self._target = target
self._can_run = asyncio.Event()
self._can_run.set()
self._task = asyncio.ensure_future(self)
def __await__(self):
target_iter = self._target.__await__()
iter_send, iter_throw = target_iter.send, target_iter.throw
send, message = iter_send, None
# This "while" emulates yield from.
while True:
# wait for can_run before resuming execution of self._target
try:
while not self._can_run.is_set():
yield from self._can_run.wait().__await__()
except BaseException as err:
send, message = iter_throw, err
# continue with our regular program
try:
signal = send(message)
except StopIteration as err:
return err.value
else:
send = iter_send
try:
message = yield signal
except BaseException as err:
send, message = iter_throw, err
def suspend(self):
self._can_run.clear()
def is_suspended(self):
return not self._can_run.is_set()
def resume(self):
self._can_run.set()
def get_task(self):
return self._task
Test:
import time
async def heartbeat():
while True:
print(time.time())
await asyncio.sleep(.2)
async def main():
task = Suspendable(heartbeat())
for i in range(5):
print('suspending')
task.suspend()
await asyncio.sleep(1)
print('resuming')
task.resume()
await asyncio.sleep(1)
asyncio.run(main())
I'm exploring Futures in Dart, and I'm confused about these two methods that Future offers, .then() and .whenCompleted(). What's the main difference between them?
Lets say I want to read a .txt using .readAsString(), I would do it like this:
void main(){
File file = new File('text.txt');
Future content = file.readAsString();
content.then((data) {
print(content);
});
}
So .then() is like a callback that fires a function once the Future is completed.
But I see there is also .whenComplete() that can also fire a function once Future completes. Something like this :
void main(){
File file = new File('text.txt');
Future content = file.readAsString();
content.whenComplete(() {
print("Completed");
});
}
The difference I see here is that .then() has access to data that was returned!
What is .whenCompleted() used for? When should we choose one over the other?
.whenComplete will fire a function either when the Future completes with an error or not, instead .then will fire a function after the Future completes without an error.
Quote from the .whenComplete API DOC
This is the asynchronous equivalent of a "finally" block.
then runs if the future completes successfully.
catchError runs if the future fails.
whenComplete runs regardless of the future completed with a value or with an error.
Here's the basic flow:
someFuture().then((value) {
print('Future finished successfully i.e. without error');
}).catchError((error) {
print('Future finished with error');
}).whenComplete(() {
print('Either of then or catchError has run at this point');
});
.whenComplete = The function inside .whenComplete is called when this future completes, whether it does so with a value or with an error.
.then = Returns a new Future which is completed with the result of the call to onValue (if this future completes with a value) or to onError (if this future completes with an error)
Read detail on API DOC
whenComplete then
I know for sure my brain isn't totally clear on async/await so I need some clarity. I see a lot of examples where await is used on some lines and not others inside a function marked async. For example, I rarely if ever have seen an example like this (I am using the print statement as an example of something simple/basic.):
myFunction() async {
await functionA();
await print("This really long thing that's going to print out.");
await functionB();
await MyExtraClass.lookupSomethingQuickly();
...
}
So the examples I see it's usually something like this:
myFunction() async {
await functionA();
print("This really long thing that's going to print out.");
await functionB();
MyExtraClass.lookupSomethingQuickly();
...
}
So I am wondering if there's just an assumption that simple things will complete in order or if theoretically, putting await in front of each line is what I should be doing in cases where I absolutely need line 1 to follow line 2 to follow line 3, etc... Like what if I absolutely need that print to finish before functionB() goes off?
Essentially I find myself making a judgment call on every line every time I am writing a function with async/await and I never know if my code is working because of good timing and luck or if there would ever be cases that would throw the execution off.
async / await is to make asynchronuos code easier to write, to read and to reason about. Synchronuos code doesn't need such support.
For async programming in Dart see also https://www.dartlang.org/docs/tutorials/futures/
If you take this code example
import 'dart:async' show Future;
void main() {
doSomethingAsync().then((_) => print('afterwards'));
print('at last');
}
Future doSomethingAsync() {
return new Future.delayed(const Duration(seconds: 1), () {
print('done something');
});
}
Try it in DartPad
which prints
at last
done something
afterwards
If you're not familiar with async execution this might be surprising
This is because code passed to Future.delayed() is executed with a delay of 1 second. The Future instance returned by doSomethingAsync() "completes" when the code in Future.delayed() has been executed.
In this line
doSomethingAsync().then((_) => print('afterwards'));
we call .then(...) on the Future returned by doSomethingAsync() and pass a closure (inline function) to .then(...). ((_) => print('afterwards')).
A feature of Future is that it calls the code passed to then(...) after it was completed (in our case when done something was printed after 1 sec delay).
So the execution goes something like
call doSomethingAsync() which schedules a call to print('done something) for later execution and returns a Future
call print('at last'); which just prints at last
after 1 second delay print('done something') is called
the Future returned from doSomethingAsync() is completed
the Future calls `(_) => print('afterwards')
main() ends.
When we use async / await the code looks like
import 'dart:async' show Future;
Future main() async {
await doSomethingAsync();
print('afterwards');
print('at last');
}
Future doSomethingAsync() {
return new Future.delayed(const Duration(seconds: 1), () {
print('done something');
});
}
Try it in DartPad
When run, the output is
done something
afterwards
at last
we also could use async / await in doSomethingAsync() but now we only focus on main()
Now the execution looks like
call doSomething() and wait for the returned Future to complete
print('done something') is executed and the Future completed
the execution of code after await continues
print('afterwards');
print('at last');
This is probably the behavior you expected.
To your original question. await is only necessary when a call returns a Future and you want the following code only be executed when the Future was completed. If the call doesn't return a Future there is nothing to wait for.
await print('xxx') is still valid code. This is to support functions that sometimes do some async work and return a Future but sometimes the don't have async work to do and execute the code immediately and just return afterwards. In this case there is nothing to wait for.
someTimesAsync() {
if(new DateTime.now().weekday == 1) {
return new Future.delayed(const Duration(seconds: 1), () {
print('done something');
});
} else {
print('done something');
}
}
await someTimesAsync();
works in both cases. If it wouldn't this would be cumbersome.
For more details about async / await see also https://www.dartlang.org/articles/await-async/