I need to do a lot of work, but luckily it's easy to decouple into different tasks for asynchronous execution. Some of those depend on each other, and it's perfectly clear to me how on task can await multiple others to get their results. However, I don't know how I can have multiple different tasks await the same coroutine, and both get the result. The Documentation also doesn't mention this case as far as I can find.
Consider the following minimal example:
from asyncio import create_task, gather
async def TaskA():
... # This is clear
return result
async def TaskB(task_a):
task_a_result = await task_a
... # So is this
return result
async def TaskC(task_a):
task_a_result = await task_a
... # But can I even do this?
return result
async def main():
task_a = create_task(TaskA())
task_b = create_task(TaskB(task_a))
task_c = create_task(TaskC(task_a))
gather(task_b, task_c) # Can I include task_a here to signal the intent of "wait for all tasks"?
For the actual script, all tasks do some database operations, some of which involve foreign keys, and therefore depend on other tables already being filled. Some depend on the same table. I definitely need:
All tasks run once, and only once
Some tasks are dependent on others being done before starting.
In brief, the question is, does this work? Can I await the same instantiated coroutine multiple times, and get the result every time? Or do I need to put awaits in main(), and pass the result? (which is the current setup, and I don't like it.)
You can await the same task multiple times:
from asyncio import create_task, gather, run
async def coro_a():
print("executing coro a")
return 'a'
async def coro_b(task_a):
task_a_result = await task_a
print("from coro_b: ", task_a_result)
return 'b'
async def coro_c(task_a):
task_a_result = await task_a
print("from coro_a: ", task_a_result)
return 'c'
async def main():
task_a = create_task(coro_a())
print(await gather(coro_b(task_a), coro_c(task_a)))
if __name__ == "__main__":
run(main())
Will output:
executing coro a
from coro_b: a
from coro_a: a
['b', 'c']
What you can not do is to await the same coroutine multiples times:
...
async def main():
task_a = coro_a()
print(await gather(coro_b(task_a), coro_c(task_a)))
...
Will raise RuntimeError: cannot reuse already awaited coroutine.
As long as you schedule your coroutine coro_a using create_task your code will work.
Related
The question should be simple enough, but I couldn't find anything about it.
I have an async Python program that contains a rather long-running task that I want to be able to suspend and restart at arbitrary points (arbitrary of course meaning everywhere where there's an await keyword).
I was hoping there was something along the lines of task.suspend() and task.resume() but it seems there isn't.
Is there an API for this on task- or event-loop-level or would I need to do this myself somehow? I don't want to place an event.wait() before every await...
What you're asking for is possible, but not trivial. First, note that you can never have suspends on every await, but only on those that result in suspension of the coroutine, such as asyncio.sleep(), or a stream.read() that doesn't have data ready to return. Awaiting a coroutine immediately starts executing it, and if the coroutine can return immediately, it does so without dropping to the event loop. await only suspends to the event loop if the awaitee (or its awaitee, etc.) requests it. More details in these questions: [1], [2], [3], [4].
With that in mind, you can use the technique from this answer to intercept each resumption of the coroutine with additional code that checks whether the task is paused and, if so, waits for the resume event before proceeding.
import asyncio
class Suspendable:
def __init__(self, target):
self._target = target
self._can_run = asyncio.Event()
self._can_run.set()
self._task = asyncio.ensure_future(self)
def __await__(self):
target_iter = self._target.__await__()
iter_send, iter_throw = target_iter.send, target_iter.throw
send, message = iter_send, None
# This "while" emulates yield from.
while True:
# wait for can_run before resuming execution of self._target
try:
while not self._can_run.is_set():
yield from self._can_run.wait().__await__()
except BaseException as err:
send, message = iter_throw, err
# continue with our regular program
try:
signal = send(message)
except StopIteration as err:
return err.value
else:
send = iter_send
try:
message = yield signal
except BaseException as err:
send, message = iter_throw, err
def suspend(self):
self._can_run.clear()
def is_suspended(self):
return not self._can_run.is_set()
def resume(self):
self._can_run.set()
def get_task(self):
return self._task
Test:
import time
async def heartbeat():
while True:
print(time.time())
await asyncio.sleep(.2)
async def main():
task = Suspendable(heartbeat())
for i in range(5):
print('suspending')
task.suspend()
await asyncio.sleep(1)
print('resuming')
task.resume()
await asyncio.sleep(1)
asyncio.run(main())
I want to set up a queue of functions in Darts. Queuing should be asynchronous, allowing multiple functions to run concurrently. However, a maximum of three functions should be executed simultaneously. How can I achieve this?
I already tied working off a list but i am struggeling at adding a limit on same time running functions
List<String> queue = new List();
main(){
queue.add("...");
queue.add("...");
queue.add("...");
for(String q in queue){
await crawl(q);
}
}
crawl(String) async{
...
}
I would use a queue:
import "dart:collection";
final queue = Queue<String>();
main() {
queue
..add("...")
..add("...")
..add("...");
while (queue.isNotEmpty) {
await crawl(queue.removeFirst());
}
}
crawl(String x) async {
.... queue.add(...) ...
}
This should work. It will not do concurrent crawling because await each operation. If you want concurrent crawling, I recommend being a little more clever. Look for worker pools or similar structures to ensure that you only have a certain number of operations running at the same time.
I am trying to understand the use of Future.cancel() with asyncio. The Python documentation is very light on this. I've had no success with existing questions on here or search engines. I just want to understand happens when a task is awaiting a future which is cancelled.
Here is my code:
import asyncio
async def foo(future):
await asyncio.sleep(3)
future.cancel()
async def bar(future):
await future
print("hi")
async def baz(future):
await bar(future)
print("ho")
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.create_task(baz(future))
loop.create_task(foo(future))
loop.run_forever()
"hi" is not seen printed. So I initially guessed bar was returning at the line await future in the case of a cancel.
However, "ho" is not printed either. So it seems logical that cancelling a future never yields back to tasks awaiting it? But then these tasks are sitting in the event loop forever? This seems undesirable, where have I misunderstood?
In this case the answer lies in the documentation, but you have to look for it a bit. First, a reminder of what it means to await a future:
# the expression:
x = await future
# is equivalent to:
... magically suspend the coroutine until the future.done() becomes true ...
x = future.result()
In other words, once the execution of the coroutine that contains await resumes, the value of the await statement will be the result() of the awaited future.
The question is: when you cancel a future, what is its result? The documentation says:
If the Future has been cancelled, this method raises a CancelledError exception.
So when someone cancel a future you awaited, the await future expression will raise an exception! This neatly explains why bar doesn't print hi (because await future has raised), and why baz doesn't print ho (because await bar(...) has raised).
A traceback is never printed because loop.create_task spawns the coroutine in the "background" (of sorts) - if no one inspects the return value, the exception will be lost. And since you threw away the task object returned by create_task and used run_forever to have the loop running forever, the loop just continues running, waiting (forever) for new tasks to somehow arrive.
If you changed the code to actually collect the result of bar, you would easily observe the CancelledError:
if __name__ == '__main__':
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.create_task(foo(future))
loop.run_until_complete(baz(future))
Output:
Traceback (most recent call last):
File "xxx.py", line 19, in <module>
loop.run_until_complete(baz(future))
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 266, in result
raise CancelledError
concurrent.futures._base.CancelledError
I know for sure my brain isn't totally clear on async/await so I need some clarity. I see a lot of examples where await is used on some lines and not others inside a function marked async. For example, I rarely if ever have seen an example like this (I am using the print statement as an example of something simple/basic.):
myFunction() async {
await functionA();
await print("This really long thing that's going to print out.");
await functionB();
await MyExtraClass.lookupSomethingQuickly();
...
}
So the examples I see it's usually something like this:
myFunction() async {
await functionA();
print("This really long thing that's going to print out.");
await functionB();
MyExtraClass.lookupSomethingQuickly();
...
}
So I am wondering if there's just an assumption that simple things will complete in order or if theoretically, putting await in front of each line is what I should be doing in cases where I absolutely need line 1 to follow line 2 to follow line 3, etc... Like what if I absolutely need that print to finish before functionB() goes off?
Essentially I find myself making a judgment call on every line every time I am writing a function with async/await and I never know if my code is working because of good timing and luck or if there would ever be cases that would throw the execution off.
async / await is to make asynchronuos code easier to write, to read and to reason about. Synchronuos code doesn't need such support.
For async programming in Dart see also https://www.dartlang.org/docs/tutorials/futures/
If you take this code example
import 'dart:async' show Future;
void main() {
doSomethingAsync().then((_) => print('afterwards'));
print('at last');
}
Future doSomethingAsync() {
return new Future.delayed(const Duration(seconds: 1), () {
print('done something');
});
}
Try it in DartPad
which prints
at last
done something
afterwards
If you're not familiar with async execution this might be surprising
This is because code passed to Future.delayed() is executed with a delay of 1 second. The Future instance returned by doSomethingAsync() "completes" when the code in Future.delayed() has been executed.
In this line
doSomethingAsync().then((_) => print('afterwards'));
we call .then(...) on the Future returned by doSomethingAsync() and pass a closure (inline function) to .then(...). ((_) => print('afterwards')).
A feature of Future is that it calls the code passed to then(...) after it was completed (in our case when done something was printed after 1 sec delay).
So the execution goes something like
call doSomethingAsync() which schedules a call to print('done something) for later execution and returns a Future
call print('at last'); which just prints at last
after 1 second delay print('done something') is called
the Future returned from doSomethingAsync() is completed
the Future calls `(_) => print('afterwards')
main() ends.
When we use async / await the code looks like
import 'dart:async' show Future;
Future main() async {
await doSomethingAsync();
print('afterwards');
print('at last');
}
Future doSomethingAsync() {
return new Future.delayed(const Duration(seconds: 1), () {
print('done something');
});
}
Try it in DartPad
When run, the output is
done something
afterwards
at last
we also could use async / await in doSomethingAsync() but now we only focus on main()
Now the execution looks like
call doSomething() and wait for the returned Future to complete
print('done something') is executed and the Future completed
the execution of code after await continues
print('afterwards');
print('at last');
This is probably the behavior you expected.
To your original question. await is only necessary when a call returns a Future and you want the following code only be executed when the Future was completed. If the call doesn't return a Future there is nothing to wait for.
await print('xxx') is still valid code. This is to support functions that sometimes do some async work and return a Future but sometimes the don't have async work to do and execute the code immediately and just return afterwards. In this case there is nothing to wait for.
someTimesAsync() {
if(new DateTime.now().weekday == 1) {
return new Future.delayed(const Duration(seconds: 1), () {
print('done something');
});
} else {
print('done something');
}
}
await someTimesAsync();
works in both cases. If it wouldn't this would be cumbersome.
For more details about async / await see also https://www.dartlang.org/articles/await-async/
I am doing F# programming, I have some special requirements.
I have 3 class instances; each class instance has to run for one hour every day, from 9:00AM to 10:00AM. I want to control them from main program, starting them at the same time, and stop them also at the same time. The following is my code to start them at the same time, but I don’t know how to stop them at the same time.
#light
module Program
open ClassA
open ClassB
open ClassC
let A = new CalssA.A("A")
let B = new ClassB.B("B")
let C = new ClassC.C("C")
let task = [ async { return A.jobA("A")};
async { return B.jobB("B")};
async { return C.jobC("C")} ]
task |> Async.Parallel |> Async.RunSynchronously |> ignore
Anyone knows hows to stop all 3 class instances at 10:00AM, please show me your code.
Someone told me that I can use async with cancellation tokens, but since I am calling instance of classes in different modules, it is difficult for me to find suitable code samples.
Thanks,
The jobs themselves need to be stoppable, either by having a Stop() API of some sort, or cooperatively being cancellable via CancellationTokens or whatnot, unless you're just talking about some job that spins in a loop and you'll just thread-abort it eventually? Need more info about what "stop" means in this context.
As Brian said, the jobs themselves need to support cancellation. The programming model for cancellation that works the best with F# is based on CancellationToken, because F# keeps CancellationToken automatically in asynchronous workflows.
To implement the cancellation, your JobA methods will need to take additional argument:
type A() =
member x.Foo(str, cancellationToken:CancellationToken) =
for i in 0 .. 10 do
cancellationToken.ThrowIfCancellationRequested()
someOtherWork()
The idea is that you call ThrowIfCancellationRequested frequently during the execution of your job. If a cancellation is requested, the method thorws and the operation will stop. Once you do this, you can write asynchronous workflow that gets the current CancellationToken and passes it to JobA member when calling it:
let task =
[ async { let! tok = Async.CancellationToken
return A.JobA("A", tok) };
async { let! tok = Async.CancellationToken
return B.JobB("B") }; ]
Now you can create a new token using CancellationTokenSource and start the workflow. When you then cancel the token source, it will automatically stop any jobs running as part of the workflow:
let src = new CancellationTokenSource()
Async.Start(task, cancellationToken = src.Token)
// To cancel the job:
src.Cancel()
You asked this question on hubfs.net, and I'll repeat here my answer: try using Quartz.NET. You'd just implement IInteruptableJob in A,B,C, defining how they stop. Then another job at 10:00AM to stop the others.
Quartz.NET has a nice tutorial, FAQ, and lots of examples. It's pretty easy to use for simple cases like this, yet very powerful if you ever need more complex scheduling, monitoring jobs, logging, etc.