go error handling of 'dial tcp i/o timeout' from docker - docker

We have an application hosted in a docker container calling a REST API, and we'd like to identify network timeouts and handle them in a specific way
Right now to detect timeouts we're using the following code snippet:
if netErr, ok := err.(net.Error); ok && netErr.Timeout() {
// do something...
}
While this did work in capturing some cases, when I setup an integration test where firewall/security groups was not configured correctly and the host isn't reachable, it resulted in the following error:
Post \"https://notreal.com/endpoint\": dial tcp 10.130.140.150:8443: i/o timeout
In this case it doesn't seem to fall under net.Error, and ideas on how else to identify this?
Edit:
Looks like this is an issue with our own libraries. The error was originally one that implemented net.Error but was swallowed, returning only the string and no longer implementing Timeout() or Temporary().

You have to implement it by yourself like you can see in this SO answer. However another way to this is to use an external package that will do that for you like retryablehttp.
resp, err := retryablehttp.Get("/foo")
if err != nil {
panic(err)
}
The returned response object is an *http.Response, the same thing you would usually get from net/http. Had the request failed one or more times, the above call would block and retry with exponential backoff.
If you need to distinguish transient errors from permanent errors (like endpoint that does not exist) you may try something like this (not tested by myself)?
if netErr, ok := err.(net.Error); ok && netErr.Temporary() {
// retry
}
if err != nil {
// unrecoverable error
log.Fatal(err)
}
Catching specific net errors can be tedious since there is a bunch of errors AddrError, DNSError and so on.

Related

Fine-grained control over what kind of failures trip a dapr circuit breaker

Question
Is it possible to further specifiy what kind of failures trip a dapr circuit breaker targeting a component (AWS SQS)?
Use case
I am sending emails via AWS SES. If you reach your sending limits, the AWS sdk throws an error (code 454). In this case, I want a circuit breaker to stop the processing of the queue and retry sending the emails later.
However, when you have another error, e.g. invalid email address, I don't want this to trip the circuit breaker as it is not transient. I would like to send the message to the DLQ though, as to manually examine these messages later (-> that's why I am still throwing here and not failing siltently).
Simplified setup
I have a circuit breaker defined that trips when my snssqs-pubsub component has more than 3 consecutive failures:
circuitBreakers:
pubsubCB:
# Available variables: requests, totalSuccesses, totalFailures, consecutiveSuccesses, consecutiveFailures
trip: consecutiveFailures > 3
# ...
targets:
components:
snssqs-pubsub-emails:
inbound:
circuitBreaker: pubsubCB
In my application, I want to retry sending emails that failed because the AWS SES sending limit was hit:
try {
await this.sendMail(options);
} catch (error) {
if (error.responseCode === '454') {
// This error should trip the cicuit breaker
throw new Error({ status: 429, 'Rate limited. Should be retried.' })
} else {
// This error should not trip the circuit breaker.
// Because status is 404, dapr directly puts the message into DLQ and skips retry
throw new NotFoundError({ status: 404 })
}
}
You may not have a problem to worry about if you have business case that does not violate AWS Terms of Service. You can put a support ticket and get SES Service Limit's raised.
It does not appear that dapr retry policies don't support the customization you need but .NET does.
If you don't want to process the message, then don't delete it. You can then set visibility timeout of the message in the SQS so they stay hidden to avoid processing again too quickly. Any exception thrown regardless will end up in the DLQ.

How to handle network connectivity issues (Firebase FireStore)?

For testing, I've turned off networking. I've got caching disabled:
settings.isPersistenceEnabled = false // https://firebase.google.com/docs/firestore/manage-data/enable-offline -- done to help demonstrate async handling
I'm noticing that error is not set even though I see this in my logs:
2018-08-04 10:26:26.441529-0400 Behavior-based Notifications
prototype[52077:2250670] 5.4.1 - [Firebase/Firestore][I-FST000001]
Could not reach Cloud Firestore backend. Connection failed 2 times.
Most recent error: Error Domain=FIRFirestoreErrorDomain Code=2
"Network error (such as timeout, interrupted connection or unreachable
host) has occurred." UserInfo={NSLocalizedDescription=Network error
(such as timeout, interrupted connection or unreachable host) has
occurred.} This typically indicates that your device does not have a
healthy Internet connection at the moment. The client will operate in
offline mode until it is able to successfully connect to the backend.
Here's relevant code snippet:
DBIFirebase.db.collection(URI).getDocuments() {
(snapshot, error) in
if tx < 1 {
print("returned from Firebase TX when tx count is exhausted, dropping")
return
}
if let error = error {
// in either case, there was a previous error
print("Firestore error loading data for app \(app) - \(error.localizedDescription)")
callback(samples, Errors.firebaseError)
tx = 0
return
}
error is always nil. documents are always empty.
edit:
Very near after that if let error, I use this to process the documents:
guard let documents = snapshot?.documents else {
print("no documents for app \(app)")
callback(samples, nil)
tx = 0
return
}
for document in documents {
I set a breakpoint. error is seen to be nil. stepping into this block skips over it (the guard block doesn't trigger, I'm talking about stepping into from a breakpoint at the for loop). If I turn networking on, it goes into it (not a swifty caching loops thing).
How would I distinguish between an empty but successful return and a network connectivity issue?

Swift Siesta Framework: do something before sending requests

I am trying the Siesta framework and I want to call a function before sending every API calls.
I saw that decorateRequests(with:) is best suited for what I am looking to do, but as the return value must be a Request, there's an error on the following code:
service.decorateRequests(with: { (res, req) -> Request in
if (res.url == self.tests.url) {
// do things..., then call req.repeated()
} else {
req.repeated()
}
})
However, I have this error:
Missing return in a closure expected to return 'Request'
Any idea how I can make it work? Thanks
The basic Swift syntax error here is that you need to use the return keyword if a value-returning closure contains more than one statement.
If what you need to do is something that either:
can synchronously block the main thread because it is brief (e.g. log the request, flip a test expectation), or
needs to be started when the request is sent, but the request can go ahead and start immediately without waiting for it to finish
…then it needn’t be complicated. Do your brief task while everyone waits, then return the request:
service.decorateRequests { req, res in
if res.url == self.tests.url {
doThatOtherThing() // blocks the main thread, which is fine if it’s brief
}
return req
}
If on the other hand you need to do something that will take an indefinite amount of time while the main thread continues, and then at a later time initiate that request, then Siesta currently doesn’t support that very well. You can do it by writing a custom implementation of the Request protocol, but that's laborious and error prone. A better approach is coming in a future version of Siesta.

Request not sent

I'm having a weird problem when i consume my API from my app. Sometimes, for no reason, the request is just not sent, and it fails at the end of the time-out with the following error:
Error Domain=NSURLErrorDomain Code=-1001 "The request timed out."
I have tried many API such as NSURLConnection delegates, NSURLSession and NSURLConnection.sendSynchronousRequest without success.
Here is a sample project i have made to highlight the issue. ConnectionBugApp
Here are the steps to reproduce:
Run the app in Xcode and stop debug just so the app is on your phone
Open the app, click Test Connection (it succeeds, loading wheel stops spinning right after)
Go to other apps like facebook/twitter/network games (somes that are a bit heavy) and switch to airplane mode a few times
Go back to my app and click Test Connection (loading wheel never stops)
A few details that might help:
If I use my server IP instead of my domain name, it succeeds
Issue only appears when on the LTE/4G network
Any ideas or workaround would be greatly appreciated ! Feel free to ask for more details.
Thanks
EDIT
I've edited the description a lot since i first posted it (hoping to make it cleaner and clearer), i'm sorry if some answers or comment don't really make sense anymore.
I have come across this issue when using an asynchronous request. It seems as though iOS limits the number of open connections to a single domain, such that all subsequent connections fail in the manner you have described.
If connections typically complete quickly, this possibly won't be an issue.
The solution is to limit the number of open connections to the same domain to prevent this from happening.
The answer posted by karlos works because the synchronisity of the connection blocks others from being opened.
Like mentioned in comments, I had DNSSEC (cache poisoning protection) enabled on my hosting service.
Disabling it, fixed the issue, even though that might not be a really nice solution. After a few weeks of searching, that'll be good enough.
I'll give the bounty to someone that can explain it, or who can provide a better solution.
In your code your request take default timeout is 60s, but you can change Request time out in your code as below.
in your NetworkItem class change time out.
init(request:NSMutableURLRequest){
self.request = request
self.request.timeoutInterval = 120
super.init()
}
Try the following code for the connection.This would help you.
let urlData = NSURLConnection.sendSynchronousRequest(request, returningResponse: &response, error: &error)
if error != nil || urlData!.length == 0
{
println("Error happend timeout======\(error?.code)!")
continue
}
else //else1
{if let httpResponse = response as? NSHTTPURLResponse
{
println("Status Code for successful---\(httpResponse.statusCode)")
// For example 502 is for Bad Gateway
if (httpResponse.statusCode == 502)
{
// Check the response code from your server and break
break
}
else
{
//Your code
}
}
}
You can get the list of HTTPS status code in the following link
StatusCode

Connection problems from a netty client in Android

I am using a Netty server in linux with one netty client in Android. (Netty 4.0.23).
It works perfect most of the time. However, sometimes, my client is not able to connect to server.
The ChannelFuture received when trying to connect tells me that connection is done but not success (with cause not null) so it is completed with failure.
About the failure I received it is a java.net.socketException - EHOSTUNREACH error (No route to host).
I am quite sure that the problem is not in server.
Important:
I always run the server and the client in the same lan wifi network
with local ip addresses.
I have tested it in two different wifi
networks. In one of them, the reported error occurs rarely and in the
other occurs quite often.
CONNECT_TIMEOUT_MILLIS is configured with enough time in both contexts
I attach the way I do the connection from client:
.............
final ChannelFuture futureConnection = bootstrap.connect(HOST, PORT);
futureConnection.awaitUninterruptibly();
if (futureConnection.isDone()){
if (futureConnection.isCancelled()) {
// log error
}
else if (futureConnection.isDone() && futureConnection.isSuccess()){
// do something
}
else if (futureConnection.isDone() && (futureConnection.cause() != null)) {
// log error THIS IS THE ERROR REPORTED
}
else {
// log error
}
}
............

Resources