Id'like to develop a web services + web sockets server using dart but the problem is I can't ensure the server's high availability because of uncatched exceptions in isolates.
Of course, I have try-catched my main function, but this is not enough.
If an exception occurs in the then() part of a future, the server will crash.
Which means that ONE flawd request can put the server down.
I realize that this is an open issue but is there any way to acknoledge any crash WITHOUT crashing the VM so that the server can continue serving other requests ?
Thank you.
What I've done in the past is use the main isolate to launch a child isolate which hosts the actual web server. When you launch an isolate, you can pass in an "uncaught exception" handler to the child isolate (I also think you should be able to register one at the top-level as well, to prevent this particular issue, as referenced by the issue in the original question).
Example:
import 'dart:isolate';
void main() {
// Spawn a child isolate
spawnFunction(isolateMain, uncaughtExceptionHandler);
}
void isolateMain() {
// this is the "real" entry point of your app
// setup http servers and listen etc...
}
bool uncaughtExceptionHandler(ex) {
// TODO: add logging!
// respawn a new child isolate.
spawnFunction(isolateMain, uncaughtException);
return true; // we've handled the uncaught exception
}
Chris Buckett gave you a good way to restart your server when it fails. However, you still don't want your server to go down.
The try-catch only works for synchronous code.
doSomething() {
try {
someSynchronousFunc();
someAsyncFunc().then(() => print('foo'));
} catch (e) {
// ...
}
}
When your async method completes or fails, it happens "long" after the program is done with the doSomething method.
When you write asynchronous code, it's generally a good idea to start a method by returning a future:
Future doSomething() {
return new Future(() {
// your code here.
var a = b + 5; // throws and is caught.
return someAsyncCall(); // Errors are forwarded if you return the Future directly.
});
}
This ensures that if you have code that throws, it catches them and the caller can then catchError() them.
If you write this way, you have much less crashes, assuming that you have some error handling at the top level at least.
Whenever you are calling a method that returns a Future, either return it directly (like shown above) or catchError() for it so that you are handling the possible errors locally.
There's a great lengthy article on the homepage that you should read.
Related
I tried to show a progress in angulardart, and thought that a Future would be good for this. But then i realized that a Future must be recursive to show a progress, since the Future returns immediately and the lengthy operation is executed afterwards.
If i create a Future that calls itself until the end condition is met it works with the progressbar. But i think this could not be a very good practice sind these calls will raise the memory on the stack with every recursion. Just consider a loop going through 1 billion datasets that could run a few hours and every loop calls a new Future within the current Future.
Is there a better way to create a loop that needs a certain amount of time to do work on every element (including calling a website that must be done asynchronous and evaluating the return value)? During the loop the user should see a progress that shows him "x/1000000 done".
I think it must be done with a Future since the UI needs to reload after initiating the loop, but a recursive Future seems like a bad idea to me.
You need the future to return back to you right away on the web because it is a single threaded platform. If an async action didn't return until it was complete then you would hang the browser and it wouldn't be a great experience to the user.
Instead you have a couple of options:
Dart has the ability to make the future look like it is synchronous with the await keyword. So you can do something like:
void performAction() async {
showProgress = true;
await expensiveRpc();
showProgress = false;
}
This would require the progress to be intermediate, as you aren't actually updating the progress bar as it goes along. That said if you don't really get progress events from your RPC this is probably the better solution.
Now if your RPC or action gives you some kind of feedback as it goes you can do something a bit nicer with a stream.
void performAction() {
showProgress = true;
expensiveRpc().listen((progress) {
if (progress.done) {
showProgress = false;
} else {
percentComplete = progress.value;
});
}
Really it depends more on the RPC or service you are interacting with on how you can update the progress nicely more than the progress itself.
Meanwhile i recognized that a Future-method returns immediately without executing anything in the method-body. So the solution is pretty easy:
Just declare the rpc with a Future, do whatever you need to do in the method and when calling it, use then(...) to do what you need to do after collecting the data.
int progress = 0;
int progressMax = 100;
bool progressCanceled = false;
Future rpc(var data)
async{
for(progress=0; progress<progressMax, progress++)
{
// do whatever you need to do with data
if(progressCanceled)
return;
}
}
rpc(data).then(
{
if(progressCanceled)
return;
// do whatever is needed after having received that data
});
rpc is executed and the calling process can continue while rpc does what rpc has to do. The main program can handle button clicks to set progressCanceled to true and the rpc-method will ask for the state and stop processing if it is set.
In short
We have a mobile app that streams fairly high volumes of data to and from a server through various bidirectional streams. The streams need to be closed on occasion (for example when the app is backgrounded). They are then reopened as needed. Sometimes when this happens, something goes wrong:
From what I can tell, the stream is up and running on the device's side (the status of both the GRPCProtocall and the GRXWriter involved is either started or paused)
The device sends data on the stream fine (the server receives the data)
The server seems to send data back to the device fine (the server's Stream.Send calls return as successful)
On the device, the result handler for data received on the stream is never called
More detail
Our code is heavily simplified below, but this should hopefully provide enough detail to indicate what we're doing. A bidirection stream is managed by a Switch class:
class Switch {
/** The protocall over which we send and receive data */
var protocall: GRPCProtoCall?
/** The writer object that writes data to the protocall. */
var writer: GRXBufferedPipe?
/** A static GRPCProtoService as per the .proto */
static let service = APPDataService(host: Settings.grpcHost)
/** A response handler. APPData is the datatype defined by the .proto. */
func rpcResponse(done: Bool, response: APPData?, error: Error?) {
NSLog("Response received")
// Handle response...
}
func start() {
// Create a (new) instance of the writer
// (A writer cannot be used on multiple protocalls)
self.writer = GRXBufferedPipe()
// Setup the protocall
self.protocall = Switch.service.rpcToStream(withRequestWriter: self.writer!, eventHandler: self.rpcRespose(done:response:error:))
// Start the stream
self.protocall.start()
}
func stop() {
// Stop the writer if it is started.
if self.writer.state == .started || self.writer.state == .paused {
self.writer.finishWithError(nil)
}
// Stop the proto call if it is started
if self.protocall?.state == .started || self.protocall?.state == .paused {
protocall?.cancel()
}
self.protocall = nil
}
private var needsRestart: Bool {
if let protocall = self.protocall {
if protocall.state == .notStarted || protocall.state == .finished {
// protocall exists, but isn't running.
return true
} else if writer.state == .notStarted || writer.state == .finished {
// writer isn't running
return true
} else {
// protocall and writer are running
return false
}
} else {
// protocall doesn't exist.
return true
}
}
func restartIfNeeded() {
guard self.needsRestart else { return }
self.stop()
self.start()
}
func write(data: APPData) {
self.writer.writeValue(data)
}
}
Like I said, heavily simplified, but it shows how we start, stop, and restart streams, and how we check whether a stream is healthy.
When the app is backgrounded, we call stop(). When it is foregrounded and we need the stream again, we call start(). And we periodically call restartIfNeeded(), eg. when screens that use the stream come into view.
As I mentioned above, what happens occasionally is that our response handler (rpcResponse) stops getting called when server writes data to the stream. The stream appears to be healthy (server receives the data we write to it, and protocall.state is neither .notStarted nor .finished). But not even the log on the first line of the response handler is executed.
First question: Are we managing the streams correctly, or is our way of stopping and restarting streams prone to errors? If so, what is the correct way of doing something like this?
Second question: How do we debug this? Everything we could think of that we can query for a status tells us that the stream is up and running, but it feels like the objc gRPC library keeps a lot of its mechanics hidden from us. Is there a way to see whether responses from server may do reach us, but fail to trigger our response handler?
Third question: As per the code above, we use the GRXBufferedPipe provided by the library. Its documentation advises against using it in production because it doesn't have a push-back mechanism. To our understanding, the writer is only used to feed data to the gRPC core in a synchronised, one-at-a-time fashion, and since server receives data from us fine, we don't think this is an issue. Are we wrong though? Is the writer also involved in feeding data received from server to our response handler? I.e. if the writer broke due to overload, could that manifest as a problem reading data from the stream, rather than writing to it?
UPDATE: Over a year after asking this, we have finally found a deadlock bug in our server-side code that was causing this behaviour on client-side. The streams appeared to hang because no communication sent by the client was handled by server, and vice-versa, but the streams were actually alive and well. The accepted answer provides good advice for how to manage these bi-directional streams, which I believe is still valuable (it helped us a lot!). But the issue was actually due to a programming error.
Also, for anyone running into this type of issue, it might be worth investigating whether you're experiencing this known issue where a channel gets silently dropped when iOS changes its network. This readme provides instructions for using Apple's CFStream API rather than TCP sockets as a possible fix for that issue.
First question: Are we managing the streams correctly, or is our way of stopping and restarting streams prone to errors? If so, what is the correct way of doing something like this?
From what I can tell by looking at your code, the start() function seems to be right. In the stop() function, you do not need to call cancel() of self.protocall; the call will be finished with the previous self.writer.finishWithError(nil).
needsrestart() is where it gets a bit messy. First, you are not supposed to poll/set the state of protocall yourself. That state is altered by itself. Second, setting those state does not close your stream. It only pause a writer, and if app is in background, pausing a writer is like a no-op. If you want to close a stream, you should use finishWithError to terminate this call, and maybe start a new call later when needed.
Second question: How do we debug this?
One way is to turn on gRPC log (GRPC_TRACE and GRPC_VERBOSITY). Another way is to set breakpoint at here where gRPC objc library receives a gRPC message from the server.
Third question: Is the writer also involved in feeding data received from server to our response handler?
No. If you create a buffered pipe and feed that as request of your call, it only feed data to be sent to server. The receiving path is handled by another writer (which is in fact your protocall object).
I don't see where the usage of GRXBufferedPipe in production is discouraged. The known drawback about this utility is that if you pause the writer but keep writing data to it with writeWithValue, you end up buffering a lot of data without being able to flush them, which may cause memory issue.
I have two functions
callee() async {
// do something that takes some time
}
caller () async {
await callee();
}
In this scenario, caller() waits till callee() finishes. I don't want that. I want caller() to complete right after invoking callee(). callee() can complete whenever in the future, I don't care. I just want to start it just like a thread and then forget about it.
Is this possible?
When you call the callee function, it returns a Future. The await then waits for that future to complete. If you don't await the future, it will eventually complete anyway, but your caller function won't be blocked on waiting for that. So, you can just do:
caller() {
callee(); // Ignore returned Future (at your own peril).
}
If you do that, you should be aware of what happens if callee fails with an error. That would make the returned future complete with that error, and if you don't listen on the future, that error is considered "uncaught". Uncaught errors are handled by the current Zone and the default behavior is to act like a top-level uncaught error which may kill your isolate.
So, remember to handle the error.
If callee can't fail, great, you're done (unless it fails anyway, then you'll have fun debugging that).
Actually, because of the risk of just forgetting to await a future, the highly reocmmended unawaited_futures lint requires that you don't just ignore a returned future, and instead wants you to do unawaited(callee()); to signal that it's deliberate. (The unawaited function can be imported from package:meta and will be available from the dart:async library in SDK version 2.14).
The unawaited function doesn't handle errors though, so if you can have errors, you should do something more.
You can handle the error locally:
caller() {
callee().catchError((e, s) {
logErrorSomehow(e, s);
});
}
(Since null safety, this code only works if the callee() future has a nullable value type. From Dart 2.14, you'll be able to use callee().ignore() instead, until then you can do callee().then((_) => null, onError: (e, s) => logErrorSomehow(e, s)); instead.)
or you can install an error handling zone and run your code in that:
runZoned(() {
myProgram();
}, onError: logErrorSomehow);
See the runZoned function and it's onError parameter.
Sure, just omit await. This way callee() is called immediately and when an async operation is called the call will be scheduled in the event queue for later execution and caller() is continued immediately afterwards.
This isn't like a thread though. As mentioned processing is enqueued to the event queue which means it won't be executed until the current task and all previously enqueued tasks are completed.
If you want real parallel execution you need to utilize isolates.
See also
https://www.dartlang.org/articles/event-loop/
https://api.dartlang.org/stable/1.16.1/dart-isolate/dart-isolate-library.html
https://pub.dartlang.org/packages/isolate
Universal App with MVVMLight.
So I started wondering why all the SDK examples were done from code behind rather than using a solid Wrapper class.
So I wanted to write a reusable wrapper class. No luck. Even tried adding that wrapper to a ViewModel, still no luck.
Works fine from MainView.xaml.cs
IBandInfo[] pairedBands = BandClientManager.Instance.GetBandsAsync().Result;
if (pairedBands.Length > 0)
{
using (IBandClient bandClient = await BandClientManager.Instance.ConnectAsync(pairedBands[0]))
{
}
}
The moment I move to any kind of OOP or View Model, ConnectAsync will never return or throw exception. I have tried this 20 different ways, is the SDK broken? What Is happening? No message, no throw, just never returns.
If I throw in Code behind, wallah it works just fine and returns the client in 1/2 second.
I have spend 5-6 hours so far on this. I wanted to create a solid wrapper class for the SDK so I could call easy calls from Model and do things like StartListener(MicrosoftBandSensor sensorToActivate).
Any suggestions?
-- For Phil's comment
I was trying to create backing variables for both client and bandinfo which would be held in a class that the VM uses. I wrote my class as IDisposable so I could dispose of both when I was done with my wrapper. I may be using this wrong to be honest.
MicrosoftBand.MicrosoftBandClient = BandClientManager.Instance.ConnectAsync(pairedBands[0]).Result;
Is what I wanted to call making it a sync call since I wanted to make the calls to bandinfo and client in the constructor then hold both until the class was destroyed and just recall the vars when needed.
My VM has :
public BandInformation MicrosoftBand
{
get { return _microsoftBand; }
set { Set(() => MicrosoftBand, ref _microsoftBand, value); }
}
If they didn't pass the bandclient in the constructor I would use:
private async Task InitBand(IBandInfo bandInfo)
{
if (bandInfo == null)
{
var allBands = await BandClientManager.Instance.GetBandsAsync();
if (allBands.Length > 0)
{
bandInfo = allBands[0];
}
}
var bandClient = await BandClientManager.Instance.ConnectAsync(bandInfo);
MicrosoftBandInfo = bandInfo;
MicrosoftBandClient = bandClient;
if (MicrosoftBandClient == null)
{
AddErrorMessage("This sample app requires a Microsoft Band paired to your device.Also make sure that you have the latest firmware installed on your Band, as provided by the latest Microsoft Health app.");
}
}
This seems fine working with BandInfo. I get back a solid seeming to work object For the client I get "thread exited" and nothing else.
Note: I had it in a try catch throwaway version at one point and nothing threw n exception either.
I assume you can do this like you would any other IDisposable where you handle the disposing yourself.
I can reinstantiate the BandClient each time, just figured I needed to detach the events at some point, meaning I had to keep ahold of the bandclient. I could keep it until done and would add and remove events as I needed each time.
It's likely your blocking call to .Result within your VM constructor is what was causing the hang. IBandClientManager.ConnectAsync() may implicitly display UI (a Windows Runtime dialog asking the user to confirm that she wants to use that specific Bluetooth device). If you've blocked the UI thread when it attempts to display UI, you've now gotten yourself into a deadlock.
Calling Task.Result is almost never a good idea, much less doing so within a constructor where you have little idea on which thread the constructor is executing. If you're working with an async API (such as the Band SDK) then your best bet is to keep that interaction async as well. Instead, defer calling ConnectAsync() until you actually need to, and do so from an async method in your VM. (Deferring the connection is a good idea anyway because you want to minimize the time connected to the Band to preserve battery life.) Then call Dispose() as early as possible to close the Bluetooth connection.
So I went and looked at a bunch of examples. Finally I landed on the GravityHeroUAP demo on the MSDN site. https://msdn.microsoft.com/en-us/magazine/mt573717.aspx?f=255&MSPPError=-2147217396
I looked at his code and the source: https://github.com/kevinash/GravityHeroUWP
He was essentially doing what I wanted to do.
However, I noticed something Bizarre. In his viewmodel everything was static!
public static IBandInfo SelectedBand
{
get { return BandModel._selectedBand; }
set { BandModel._selectedBand = value; }
}
private static IBandClient _bandClient;
public static IBandClient BandClient
{
get { return _bandClient; }
set
{
_bandClient = value;
}
}
I ended up copying this pattern (though had to throw away my favorite MVVM lib in the process, though I am sure I can get it back).
My common pattern in my VM's:
public string ExceptionOnStart {
get { return _exceptionOnStart; }
set { Set(() => ExceptionOnStart, ref _exceptionOnStart, value); }
}
It seems to be working now!
That and I got data way too fast for the
await Windows.Storage.FileIO.AppendLinesAsync(dataFile, new List<string> { toWrite });
Thank you for the help Phil, it got me looking in the right direction!
Thank you very, very much. Spent WAY to long on this. Mark
I'm wondering if anyone has figured out a way to properly handle timeouts in the JavaFX 8 (jdk 1.8.0_31) WebView. The problem is the following:
Consider you have an instance of WebView and you tell it to load a specific URL. Furthermore, you want to process the document once it's loaded, so you attach a listener to the stateProperty of the LoadWorker of the WebEngine powering the web view. However, a certain website times out during loading, which causes the stateProperty to transition into Worker.State.RUNNING and remain stuck there.
The web engine is then completely stuck. I want to implement a system that detects a timeout and cancels the load. To that end, I was thinking of adding a listener to the progressProperty and using some form of Timer. The idea is the following:
We start a load request on the web view. A timeout timer starts running immediately. On every progress update, the timer is reset. If the progress reaches 100%, the timer is invalidated and stopped. However, if the timer finishes (because there are no progress updates in a certain time frame we assume a time out), the load request is cancelled and an error is thrown.
Does anyone know the best way to implement this?
Kind regards
UPDATE
I've produced a code snippet with behavior described in the question. The only thing still troubling me is that I can't cancel the LoadWorker: calling LoadWorker#cancel hangs (the function never returns).
public class TimeOutWebEngine implements Runnable{
private final WebEngine engine = new WebEngine();
private ScheduledExecutorService exec;
private ScheduledFuture<?> future;
private long timeOutPeriod;
private TimeUnit timeOutTimeUnit;
public TimeOutWebEngine() {
engine.getLoadWorker().progressProperty().addListener((ObservableValue<? extends Number> observable, Number oldValue, Number newValue) -> {
if (future != null) future.cancel(false);
if (newValue.doubleValue() < 1.0) scheduleTimer();
else cleanUp();
});
}
public void load(String s, long timeOutPeriod, TimeUnit timeOutTimeUnit){
this.timeOutPeriod = timeOutPeriod;
this.timeOutTimeUnit = timeOutTimeUnit;
exec = Executors.newSingleThreadScheduledExecutor();
engine.load(s);
}
private void scheduleTimer(){
future = exec.schedule(TimeOutWebEngine.this, timeOutPeriod, timeOutTimeUnit);
}
private void cleanUp(){
future = null;
exec.shutdownNow();
}
#Override
public void run() {
System.err.println("TIMED OUT");
// This function call stalls...
// engine.getLoadWorker().cancel();
cleanUp();
}
}
I don't think that you can handle timeouts properly now. Looks at this method. As you can see it has hardcoded value for setReadTimeout method. Is it mean that SocketTimeoutException exception will be raised after one hour of loading site. And state will be changed to FAILED only after that event.
So, you have only one way now: try to hack this problem use Timers as you described above.
P.S.
Try to create issue in JavaFX issue tracker. May be anyone fixed it after 5 years...
I have the same problem and used a simple PauseTransition. Same behavior, not so complicated. =)