I have a camel route in grails. The exception goes away when I comment out the unmarshall line in the processor as if it was taking too long for the TCP response to be sent back in time. If my assessment is correct, if there a setting in netty to send the response back without waiting?
Thanks.
from("netty4:tcp://172.25.200.66:12345?
clientMode=true&disconnectOnNoReply=false&connectTimeout=100000
&reconnect=true&allowDefaultCodec=false&decoder=#jsonDecoderFactory")
.to("log:dump?showAll=true")
.bean("messageProcessorService", "processMessage")
def processMessage (String msg) {
log.debug("MessageProcessorService processMessage")
def object = unmarshall(msg)
}
2017-01-13 13:20:58,186 | WARN | camel.component.netty4.NettyConsumer |
Caused by: [org.apache.camel.CamelExchangeException - Cannot write response to /172.25.200.66:12345.
Exchange[ID-wulg2-chij-udev-local-41278-1484313632001-0-20][Message: []].
Caused by: [java.lang.UnsupportedOperationException - unsupported message
type: Collections$UnmodifiableList (expected: ByteBuf, FileRegion)]]
org.apache.camel.CamelExchangeException: Cannot write response to
/172.25.200.66:12345.
Exchange[ID-wulg2-chij-udev-local-41278-1484313632001-0-20][Message: []].
Caused by: [java.lang.UnsupportedOperationException - unsupported message
It turns out that option "sync=false" sets the endpoint to one-way. Thus not requiring to write responses back.
Related
In the documentation it is mentioned that producer retries to send the message to the queue based on max_retries.
So I shutdown the Kafka and then tried my producer. I get this error
Fetching cluster metadata from kafka://localhost:9092
[topic_metadata] Opening connection to localhost:9092 with client id MYCLIENTID
ERROR -- : [topic_metadata] Failed to connect to localhost:9092: Connection refused
DEBUG -- : Closing socket to localhost:9092
ERROR -- : Failed to fetch metadata from kafka://localhost:9092
Completed 500 Internal Server Error in 486ms (ActiveRecord: 33.9ms)
which it make sense, however the retries never happens after that. I have read the doc inside-out and I can't figure it out how this retries actually going to trigger?
Here is my code:
def self.deliver_message(kafka, message, topic, transactional_id)
producer = kafka.producer(idempotent: true,
transactional_id: transactional_id,
required_acks: :all,
max_retries: 5,
retry_backoff: 5)
producer.produce(message, topic: topic)
producer.deliver_messages
end
link to doc:
https://www.rubydoc.info/gems/ruby-kafka/Kafka/Producer#initialize-instance_method
Thank you in advance.
The retries are based on the type of Exception thrown by the producer callback. According to the Callback Docs there are the following Exception possible happening during callback:
The exception thrown during processing of this record. Null if no error occurred. Possible thrown exceptions include:
Non-Retriable exceptions (fatal, the message will never be sent):
InvalidTopicException
OffsetMetadataTooLargeException
RecordBatchTooLargeException
RecordTooLargeException
UnknownServerException
Retriable exceptions (transient, may be covered by increasing #.retries):
CorruptRecordException
InchvalidMetadataException
NotEnoughReplicasAfterAppendException
NotEnoughReplicasException
OffsetOutOfRangeException
TimeoutException
UnknownTopicOrPartitionException
Shutting down Kafka completely rather looks like a non-retriable Exception.
I am trying to establish the max number of retries from my app to rabbit broker.
I have the retry interceptor,
#Bean
public RetryOperationsInterceptor retryOperationsInterceptor() {
return RetryInterceptorBuilder.stateless()
.maxAttempts(CommonConstants.MAX_AMQP_RETRIES)
.backOffOptions(500, 2.0, 3000)
.build();
}
and this is used inside listener container,
container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});
However, after a couple of retries, the consumer attempts connection all over again in an endless loop,
2017-02-21 15:03:12.229 WARN 9292 --- [nsumerThread_92] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
2017-02-21 15:03:12.229 INFO 9292 --- [nsumerThread_92] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer: tags=[{}], channel=null, acknowledgeMode=AUTO local queue size=0
2017-02-21 15:03:13.245 WARN 9292 --- [nsumerThread_93] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
2017-02-21 15:03:13.245 INFO 9292 --- [nsumerThread_93] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer: tags=[{}], channel=null, acknowledgeMode=AUTO local queue size=0
2017-02-21 15:03:13.261 ERROR 9292 --- [nsumerThread_83] o.s.a.r.l.SimpleMessageListenerContainer : Failed to check/redeclare auto-delete queue(s).
I want the app to fail and error out because of lack of connectivity to the broker after a MAX_RETRY # limit.
thanks for the help
EDIT
As suggested by #artem-bilan, I ended up using a Component
public class BrokerFailureEventListener implements ApplicationListener<ListenerContainerConsumerFailedEvent>
In this class the onApplicationEvent I counted the number of failures and then take appropriate action.
In case of producer-side, it's a little different scenario. As explained by #artem-bilan, the application would need to take care of any issues. I explored using netflix-hystrix and added a fallback method for the production method and will go with that route. thanks much again.
Well, you misunderstood a bit container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});. It is for the business errors during messages processing:
Business exception handling, as opposed to protocol errors and dropped connections, might need more thought and some custom configuration, especially if transactions and/or container acks are in use. Prior to 2.8.x, RabbitMQ had no definition of dead letter behaviour, so by default a message that is rejected or rolled back because of a business exception can be redelivered ad infinitum. To put a limit in the client on the number of re-deliveries, one choice is a StatefulRetryOperationsInterceptor in the advice chain of the listener. The interceptor can have a recovery callback that implements a custom dead letter action: whatever is appropriate for your particular environment.
In contradiction to the:
In fact it loops endlessly trying to restart the consumer, and only if the consumer is very badly behaved indeed will it give up. One side effect is that if the broker is down when the container starts, it will just keep trying until a connection can be established.
What you need is ListenerContainerConsumerFailedEvent, which is emitted as:
private void logConsumerException(Throwable t) {
if (logger.isDebugEnabled()
|| !(t instanceof AmqpConnectException || t instanceof ConsumerCancelledException)) {
logger.warn(
"Consumer raised exception, processing can restart if the connection factory supports it",
t);
}
else {
logger.warn("Consumer raised exception, processing can restart if the connection factory supports it. "
+ "Exception summary: " + t);
}
publishConsumerFailedEvent("Consumer raised exception, attempting restart", false, t);
}
So, you can listen for those events and stop your application when some condition is reached.
When I send a comments thread. List request to the YouTube API
I get the following exception "But not for all videos":
Google.GoogleApiException: Google.Apis.Requests.RequestError
The API server failed to successfully process the request.
While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the commentThread resource in the request body to ensure that it is valid. [400].
And for those videos I double checked the inputs sent with the request and I
make the request with the same data directly from the YouTube API requests trial section but everything goes right!
I want to know why this request becomes valid on some video ID's but invalid on the others ?
I'll appreciate any help.
Here is the full written log: System.AggregateException: One or more errors occurred. ---> Google.GoogleApiException: Google.Apis.Requests.RequestError
The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the commentThread resource in the request body to ensure that it is valid. [400]
Errors [
Message[The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the commentThread resource in the request body to ensure that it is valid.] Location[body - other] Reason[processingFailure] Domain[youtube.commentThread]
]
at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Google.Apis.Requests.ClientServiceRequest`1.d__0.MoveNext() in c:\ApiaryDotnet\default\Src\GoogleApis\Apis\Requests\ClientServiceRequest.cs:line 0
--- End of inner exception stack trace ---
CommentThreads.list API doesn't need a request body as indicated in the API reference
Request body
Do not provide a request body when calling this method.
This may have caused the RequestError on your call. Try to remove any objects passed when calling this API, hopefully this would fix the issue.
I get lots of Errors in my Umbraco 7.2.6 log file. And sometimes the CMS is broken, needs to restart IIS or browser. All errors are somethings with the load or save XML Lock or Thread aborted. I use the default Umbraco config on a simple shared hosting environment.
But It often works well. (so no file permission or somethings I expect)
What is wrong?
2015-07-07 09:45:02,833 [10] INFO umbraco.BusinessLogic.Log - [T15/D3] Log scrubbed. Removed all items older than 2015-05-08 09:45:02
2015-07-07 09:45:40,726 [10] ERROR umbraco.content - [T8/D3] Failed to load Xml from file.
System.Threading.ThreadAbortException: Thread was being aborted.
at System.Threading.WaitHandle.WaitOneNative(SafeHandle waitableSafeHandle, UInt32 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
at System.Threading.WaitHandle.InternalWaitOne(SafeHandle waitableSafeHandle, Int64 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout, Boolean exitContext)
at System.Threading.WaitHandle.WaitOne(Int32 millisecondsTimeout)
at Umbraco.Core.AsyncLock.Lock(Int32 millisecondsTimeout)
at umbraco.content.EnsureFileLock()
at umbraco.content.LoadXmlFromFile()
2015-07-07 09:45:40,741 [10] ERROR Umbraco.Web.WebServices.ScheduledPublishController - [T55/D3] Error executing scheduled task
System.Threading.ThreadAbortException: Thread was being aborted.
at umbraco.content.LoadXmlFromFile()
at umbraco.content.LoadXmlLocked(SafeXmlReaderWriter safeXml, Boolean& registerXmlChange)
at umbraco.content..ctor()
at umbraco.content.<.cctor>b__17()
at System.Lazy`1.CreateValue()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Lazy`1.LazyInitValue()
at System.Lazy`1.get_Value()
at Umbraco.Web.WebServices.ScheduledPublishController.Index()
2015-07-07 09:45:40,757 [10] ERROR Umbraco.Web.Scheduling.ScheduledPublishing - [T7/D3] An error occurred with the scheduled publishing. The base url used in the request was: http://acc.xxxxxxx.nl:80/umbraco/, see http://our.umbraco.org/documentation/Using-Umbraco/Config-files/umbracoSettings/#ScheduledTasks documentation for details on setting a baseUrl if this is in error
System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request)
at System.Net.WebClient.UploadString(Uri address, String method, String data)
at System.Net.WebClient.UploadString(String address, String data)
at Umbraco.Web.Scheduling.ScheduledPublishing.PerformRun()
2015-07-07 09:46:14,905 [10] INFO Umbraco.Core.PluginManager - [T19/D3] Starting resolution types of Umbraco.Core.PropertyEditors.IParameterEditor
2015-07-07 12:55:10,801 [17] ERROR umbraco.content - [T18/D11] Failed to save Xml to file.
System.TimeoutException: Failed to enter the lock within timeout.
at Umbraco.Core.AsyncLock.Lock(Int32 millisecondsTimeout)
at umbraco.content.EnsureFileLock()
at umbraco.content.<SaveXmlToFileAsync>d__d.MoveNext()
2015-07-07 14:03:09,702 [9] ERROR umbraco.content - [T7/D3] Failed to load Xml from file.
System.TimeoutException: Failed to enter the lock within timeout.
at Umbraco.Core.AsyncLock.Lock(Int32 millisecondsTimeout)
at umbraco.content.EnsureFileLock()
at umbraco.content.LoadXmlFromFile()
This issue was related to http://issues.umbraco.org/issue/U4-6802
it is fixed in the Umbraco v7.2.8 release.
I keep getting the following error on a seemingly random basis from a WildFly 8.1.0.Final install running under NetBeans:
08:51:09,742 ERROR [io.undertow.request] (default task-40) Blocking request failed HttpServerExchange{ GET /web/faces/javax.faces.resource/dynamiccontent.properties}: java.lang.RuntimeException: java.io.IOException: Broken pipe
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:527)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_20]
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [rt.jar:1.8.0_20]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) [rt.jar:1.8.0_20]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [rt.jar:1.8.0_20]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) [rt.jar:1.8.0_20]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) [rt.jar:1.8.0_20]
at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:150) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:531)
at io.undertow.conduits.ChunkedStreamSinkConduit.flush(ChunkedStreamSinkConduit.java:256)
at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
at org.xnio.channels.Channels.flushBlocking(Channels.java:63) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:625)
at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:451)
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:525)
... 9 more
The requested pages appear to load without a problem, so other than the exceptions in the log, I haven't noticed any breaks. Any ideas?
I've face a similar problem and thanks to the idea of this response, I advanced a little it. I'm going to expose my case.
I was creating a REST API using Java (Java 7) (javax.ws.rs) and deploying it on a JBoss server (8.x).
My Api responds to these paths:
/myapi/a
/myapi/a?filer=myfilter
So I code it this way:
private static final String FILTER = "filter";
#GET
#Path("/a")
#Produces(MediaType.APPLICATION_JSON)
public Object
foo(#Context UriInfo requestInfo) {
LOG.info("Http request: GET /myapi/a");
if (requestParameters.getQueryParameters().containsKey(FILTER)) {
return foo(requestInfo.getQueryParameters().get(FILTER));
}
// no params
return ...
}
public Object foo(List<String> filter) {
LOG.info(" > Requested filter");
return ...;
}
But I was getting sometimes this exception from the server (not my code)
UT005023: Exception handling request to ... sessionState: org.jboss.resteasy.spi.UnhandledException: Response is committed, can't handle exception caused by java.io.IOException: Broken pipe
Investigating it I come across something really interesting: it was only able to reproduce it from Safari browser, not Chrome. So what? The point is that Safari has a functionality Chrome doesn't: When Safari auto-completes the request, it sends the request. Chrome doesn't send the request until the enter button is pressed. And this is important because the bug appears only if:
request with Safari's autocomplete to /a?filter=f
request (hitting enter) to /a
At this point, I don't know the reason (it's something related to the http header) => as stephen-c, the problem is that you are trying to do stuff that would require a change to the HTTP response headers ... after the headers have been sent
[EDITED]
I'm almost sure (99%) that we could not handled that exception. basically it's saying that you have lose one request and, as a warning, the server is telling you that you're not going to handle it.
There is another way to recreate the exception: try to put your finger at F5 or CMD-R. Your are going to create hundred of requests... but you'll lose some of them (related to pool thread, workers, etc) and you'll see the exception for those lose requests.
I've decided not to worry about this anymore.
I had the same warnings, but only with Firefox. Daniel.lichtenberger's post explains well the issue and how to solve it.
Summarized, Firefox's RCWN makes two simultaneous requests and cancels the slowest, resulting in the broken pipe warning. To disable RCWN type about:config in Firefox and disable network.http.rcwn.enable
If you are sending multipart/form-data Request In IE,
you must append hidden type to form, like this
<form>
...
<!-- for IE -->
<input type='hidden' name='_4ie' value='for IE'>
</form>