I have one SqsListener in my spring boot app as below:
#SqsListener(value = "QUEUE-FQN", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
private void receiveNotifications(String payload, MessageHeaders headers, Acknowledgment acknowledgment)
throws IOException, ParseException, InterruptedException {
try {
// process message here
}
}
And the following two beans defined:
#Bean
public AmazonSQSAsync amazonSQSAsync(AWSCredentialsProvider awsCredentialsProvider) {
return AmazonSQSAsyncClientBuilder
.standard()
.withCredentials(awsCredentialsProvider)
.withRegion(Regions.US_EAST_1.getName())
.build();
}
#Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory(AmazonSQSAsync amazonSqs) {
SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
factory.setAmazonSqs(amazonSqs);
factory.setMaxNumberOfMessages(10);
factory.setAutoStartup(false);
return factory;
}
After the simpleMessageListenerContainer.start(QUEUE_NAME), I see below exceptions constantly and the listener is never able to poll for new messages (never makes any progress).
Anything that I am doing wrong here? How to get past this error:
WARN 1297 --- [enerContainer-2] i.a.c.m.l.SimpleMessageListenerContainer : An Exception occurred while polling queue 'https://sqs:us-east-1:amazonaws:com/ACCOUNTID/QUEUE_NAME'. The failing operation will be retried in 10000 milliseconds
org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#3cee2db[Running, pool size = 11, active threads = 11, queued tasks = 0, completed tasks = 1188]] did not accept task: io.awspring.cloud.messaging.listener.SimpleMessageListenerContainer$SignalExecutingRunnable#681b4433
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:363)
at io.awspring.cloud.messaging.listener.SimpleMessageListenerContainer$AsynchronousMessageListener.run(SimpleMessageListenerContainer.java:343)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.RejectedExecutionException: Task io.awspring.cloud.messaging.listener.SimpleMessageListenerContainer$SignalExecutingRunnable#681b4433 rejected from java.util.concurrent.ThreadPoolExecutor#3cee2db[Running, pool size = 11, active threads = 11, queued tasks = 0, completed tasks = 1188]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:360)
... 6 more
Related
I have a Grails 4 application providing a REST API. One of the endpoints sometimes fail with the following exception:
io.micronaut.http.client.exceptions.ReadTimeoutException: Read Timeout
at io.micronaut.http.client.exceptions.ReadTimeoutException.<clinit>(ReadTimeoutException.java:26)
at io.micronaut.http.client.DefaultHttpClient$10.exceptionCaught(DefaultHttpClient.java:1917)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:426)
at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)
at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:147)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:233)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268)
at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:98)
at io.netty.handler.timeout.ReadTimeoutHandler.channelIdle(ReadTimeoutHandler.java:90)
at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:505)
at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:477)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
The endpoint uses micronaut http client to call other systems. The remote system takes a very long time to respond, causing the ReadTimeOutException.
Here is the code calling the remote Service:
class RemoteTaskService implements GrailsConfigurationAware {
String taskStepperUrl
// initializes fields from configuration
void setConfiguration(Config config) {
taskStepperUrl = config.getProperty('services.stepper')
}
private BlockingHttpClient getTaskClient() {
HttpClient.create(taskStepperUrl.toURL()).toBlocking()
}
List<Map> loadTasksByProject(long projectId) {
try {
retrieveRemoteList("/api/tasks?projectId=${projectId}")
} catch(HttpClientResponseException e) {
log.error("Loading tasks of project failed with status: ${e.status.code}: ${e.message}")
throw new NotFoundException("No tasks found for project ${projectId}")
}
}
private List<Map> retrieveRemoteList(String path) {
HttpRequest request = HttpRequest.GET(path)
HttpResponse<List> response = taskClient.exchange(request, List) as HttpResponse<List>
response.body()
}
}
I've tried resolving it using the following configuration in my application.yml:
micronaut:
server:
read-timeout: 30
and
micronaut.http.client.read-timeout: 30
...with no success. Despite my configuration, the timeout still occurs around 10s after calling the endpoint.
How can I change the read timeout duration for the http rest client?
micronaut.http.client.read-timeout takes a duration, so you should add a measuring unit to the value, like 30s, 30m or 30h.
It seems that the configuration values are not injected in the manually created http clients.
A solution is to configure the HttpClient at creation, setting the readTimeout duration:
private BlockingHttpClient getTaskClient() {
HttpClientConfiguration configuration = new DefaultHttpClientConfiguration()
configuration.readTimeout = Duration.ofSeconds(30)
new DefaultHttpClient(taskStepperUrl.toURL(), configuration).toBlocking()
}
In my case I was streaming a file from a client as
#Get(value = "${service-path}", processes = APPLICATION_OCTET_STREAM)
Flowable<byte[]> fullImportStream();
so when I got this my first impulse was to increase the read-timeout value. Though, for streaming scenarios the property that applies is read-idle-timeout as stated in the docs https://docs.micronaut.io/latest/guide/configurationreference.html#io.micronaut.http.client.DefaultHttpClientConfiguration
I'm putting together a Solace Point-to-point solution in C#.
In my subscriber/listener, I am using ClientAck mode to ensure messages are successfully processed before being removed from the queue.
My question (probably due to my limited experience in messaging) is regarding failed messages, e.g. if I cannot process the message and hence not send the Ack, how is the message replayed?
An example of what I have is as follows:
using (ISession session = context.CreateSession(sessionProperties, null, null))
{
ReturnCode returnCode = session.Connect();
if (returnCode == ReturnCode.SOLCLIENT_OK)
{
var endpointProps = new EndpointProperties()
{
Permission = EndpointProperties.EndpointPermission.Consume,
AccessType = EndpointProperties.EndpointAccessType.Exclusive
};
using (IQueue queue = ContextFactory.Instance.CreateQueue(queueName))
{
session.Provision(queue, endpointProps,
ProvisionFlag.IgnoreErrorIfEndpointAlreadyExists | ProvisionFlag.WaitForConfirm, null);
_flow = session.CreateFlow(new FlowProperties { AckMode = MessageAckMode.ClientAck }, queue, null, HandleMessageEvent, HandleFlowEvent);
_flow.Start();
do { WaitEventWaitHandle.WaitOne(); } while (!cancellationToken.IsCancellationRequested);
};
return Task.CompletedTask;
}
else
{
throw new Exception($"Connection failed, return code: {returnCode}");
}
}
and then handling incoming messages
void HandleMessageEvent(object sender, MessageEventArgs args)
{
using (IMessage message = args.Message)
{
try
{
_handler(message.ApplicationMessageType, message.BinaryAttachment);
_flow.Ack(message.ADMessageId);
}
finally
{
WaitEventWaitHandle.Set();
}
}
}
So, if I don't Ack, message remains on queue as expected (and required), however, how (best-practice) can I re-process it without manual intervention?
After a message has been delivered to a consumer from a Solace PubSub+ queue, the message will only be resent if the client unbinds before sending an acknowledgement back. The exception to this is specific to JMS clients with the session.recover() action.
If a message needs to be re-delivered to a C# application after it has already been sent but not acknowledged, the client will need to unbind and rebind to the queue. Note that if there are other clients also bound to the queue, the message may be re-sent to those clients before your client rebinds.
I'm using for a web project JSF2 with Oracle Glassfish Server Open Source Edition 4.0 and Oracle Database 11g (Version 11.2.0-1.0).
The server and database are running on the same windows machine.
A connection pool managed the connections to the database.
Does anybody know why I sometimes get the following exception:
java.sql.SQLException: Connection closed
at com.sun.gjc.spi.base.ConnectionHolder.checkValidity(ConnectionHolder.java:766)
at com.sun.gjc.spi.base.ConnectionHolder.commit(ConnectionHolder.java:243)
at de.mydomain.myproject.Hl7MessageHandler.run(Hl7MessageHandler.java:123)
...
Or sometimes this one:
java.sql.SQLRecoverableException: Closed connection
at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:5675)
at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:5735)
at com.sun.gjc.spi.base.ConnectionHolder.commit(ConnectionHolder.java:244)
at de.mydomain.myproject.Hl7MessageHandler.run(Hl7MessageHandler.java:123)
...
The Database Class:
public static Connection getConnection() throws NamingException, SQLException {
Context initContext = new InitialContext();
DataSource datasSource = (DataSource)initContext.lookup("jdbc/Oracle");
Connection connnection = datasSource.getConnection();
return connnection;
}
Request handling in the servlet:
public IResponseSendable<String> run(String hl7MsgString, boolean publishErrorToDB) {
// ... do something
try {
con = Database.getConnection();
} catch (NamingException | SQLException conExc) {
return generateAck(true, conExc.getMessage(), hl7MsgString);
}
try {
con.setAutoCommit(false);
process();
con.commit();
} catch (HL7Exception | SQLException pe) {
logger.error(...);
// Exceptionhandling...
try {
con.rollback();
} catch (SQLException rollbackExc) {
logger.error(...);
}
return generateAck(true, pe.getMessage(),hl7MsgString, _log);
}
finally {
try {
con.setAutoCommit(true);
con.close();
} catch (SQLException e) {
logger.error(...);
}
}
return generateAck(false, "", hl7MsgString);
}
The process-Methode:
private void process() throws HL7Exception, SQLException {
// Do something...
String sql = "BEGIN save_patient_data(?,?,?,?,?,?,?); END;";
CallableStatement stmt = (CallableStatement) con.prepareCall(sql);
stmt.setString(1, ...);
// ...
stmt.registerOutParameter(6, java.sql.Types.VARCHAR);
stmt.registerOutParameter(7, java.sql.Types.NUMERIC);
stmt.execute();
// ...
stmt.close();
// More databse stored procedure can be called ...
}
Connection Pool Settingts:
Initial and Minimum Pool Size: 10 Connections
Maximum Pool Size: 60 Connections
Pool Resize Quantity: 2 Connections
Idle Timeout: 600 Seconds
Max Wait Time: 0 Milliseconds
Validate At Most Once: 0 Seconds
Connection Leak Timeout: 10 Seconds
Connection Leak Reclaim: enabled
Statement Leak Timeout: 6 Seconds
Statement Leak Reclaim: enabled
Creation Retry Attempts: 0
Retry Interval: 10 Seconds
Connection Validation: Required
Validation Method: meta-data
The database IDLE-Timeout setting is "UNLIMITED".
Notcie:
The exception occurred either when to call "con.prepareCall(sql);" (must not be at the first time) or when I try to commit the connection or later when to try to turn autocommit on.
Does any body know the reason or what is the best way to debug the application to find it out?
Thank you.
Edit:
Maybe it's important:
I can find in the server log many warnings about connection leaks:
2014-07-28T14:49:17.961+0200|Warnung: A potential connection leak detected for connection pool OraclePool. The stack trace of the thread is provided below :
com.sun.enterprise.resource.pool.ConnectionPool.setResourceStateToBusy(ConnectionPool.java:324)
com.sun.enterprise.resource.pool.ConnectionPool.getResourceFromPool(ConnectionPool.java:758)
com.sun.enterprise.resource.pool.ConnectionPool.getUnenlistedResource(ConnectionPool.java:632)
com.sun.enterprise.resource.pool.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:200)
com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:526)
com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:360)
com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:307)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:196)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:171)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:166)
com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:114)
de.mydomain.myproject.utilities.Database.getConnection(Database.java:17)
...
You have connection leak reclaim enabled and the connection leak timeout is 10 seconds. This means that if you hold onto a logical connection for longer than 10 seconds, it is forcibly revoked and closed by the connection pool manager (and the physical connection is returned to the connection pool). Subsequent attempts to use the logical connection will result in a SQLException as the connection is closed.
Find out which operation takes longer than 10 seconds and try to reduce the time it takes or configure a longer connection leak timeout (10 seconds is IMHO a bit short for connection leak detection). The same BTW applies to your statement leak detection (6 seconds is also pretty short).
i use executor service to launch multiple thread to sent request to api and get data back. sometimes i see some threads haven't finished their job yet, the service kill that thread already, how can i force the service to wait until the thread finish their job?
here is my code:
ExecutorService pool = Executors.newFixedThreadPool(10);
List<Future<List<Book>>> futures = Lists.newArrayList();
final ObjectMapper mapper1 = new ObjectMapper();
for (final Author a : authors) {
futures.add(pool.submit(new Callable<List<Book>>() {
#Override
public List<Book> call() throws Exception {
String urlStr = "http://localhost/api/book?limit=5000&authorId=" + a.getId();
List<JsonBook> Jsbooks = mapper1.readValue(
new URL(urlStr), BOOK_LIST_TYPE_REFERENCE);
List<Book> books = Lists.newArrayList();
for (JsonBook jsonBook : Jsbooks) {
books.add(jsonBook.toAvro());
}
return books;
}
}));
}
pool.shutdown();
pool.awaitTermination(3, TimeUnit.MINUTES);
List<Book> bookList = Lists.newArrayList();
for (Future<List<Book>> future : futures) {
if (!future.isDone()) {
LogUtil.info("future " + future.toString()); <-- future not finished yet
throw new RuntimeException("Future to retrieve books: " + future + " did not complete");
}
bookList.addAll(future.get());
}
and i saw some excepitons at the (!future.isDone()) block. how can i make sure every future is done when executor service shutdown?
I like to use the countdown latch.
Set the latch to the size that you're iterating and pass that latch into your callables, then in your run / call method have a try/finally block that decrements the countdown latch.
After everything has been enqueued to your executor service, just call your latch's await method, which will block until it's all done. At that time all your callables will be finished, and you can properly shut down your executor service.
This link has an example of how to set it up.
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CountDownLatch.html
I am developing an grails application(server) to track the mobile device which are in the Wi-Fi network. The users will send a request to the webservice which is running on grails applicion(server) along with Mobileid and Wi-Fi IP address.
In my grails application i am staring multiple external java threads, each thread will be pinging the Wi-Fi IP address of each mobile device(one thread per one device to track). If any device IP is not reachable then i will update mobile status as "Disconnected" in the database from the external thread. Here only i am facing the issue, if more than one device is in not reachable then multiple threads are going to update the status of each device in the same table using domain.withTransaction method while i am getting the following exception
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is java.lang.NullPointerException
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:596)
at org.codehaus.groovy.grails.orm.hibernate.GrailsHibernateTransactionManager.super$3$doBegin(GrailsHibernateTransactionManager.groovy)
at sun.reflect.GeneratedMethodAccessor492.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:127)
My Code:
Pinging device in thread
try {
final InetAddress inet = InetAddress.getByName(ipAddress);
boolean status = inet.isReachable(5000);
if (status) {
pool.run(MobileDeviceTracker.deviceMap.get(mobileId));
} else {
// Calling service to update the status of device as disconnected
getUserMobileService().deviceDisconnected(mobileId, ipAddress);
}
} catch (Exception e) { }
Updating Status in Database
class DisconnectionService implements UserMobileServiceInt{
static transactional = true
def void deviceDisconnected(String mobileId, String wifiIp){
try{
def mobile = Mobile.findByMobileId(mobileId)
def userMobile = UserMobile.findByMobileAndWifiIp(mobile, wifiIp)
userMobile.withTransaction {tx ->
userMobile.action = Constants.MOBILE_STATUS_DISCONNECTED
userMobile.alarmStatus = Constants.ALARM_STATUS_TURNED_ON
userMobile.modifiedDate = new Date()
userMobile.save(flush: true)
}
}catch(Exception e){
e.printStackTrace()
}
I am trying last 4 days but i am not able solve this.
Move the reads into the transaction, otherwise they'll be in a disconnected session and not the one that the transaction creates. Also, it's best to call static methods on the class, not an instance (in both Groovy and Java):
void deviceDisconnected(String mobileId, String wifiIp){
try {
UserMobile.withTransaction { tx ->
def mobile = Mobile.findByMobileId(mobileId)
def userMobile = UserMobile.findByMobileAndWifiIp(mobile, wifiIp)
userMobile.action = Constants.MOBILE_STATUS_DISCONNECTED
userMobile.alarmStatus = Constants.ALARM_STATUS_TURNED_ON
userMobile.modifiedDate = new Date()
userMobile.save(flush: true)
}
}
catch(e) {
e.printStackTrace()
}
}
Rather than using the verbose binding code suggested by Tiggerizzy. It is better to use the built in withNewSession method on domain classes:
Mobile.withNewSession {
// your code here
}
No need for me to spread mis-information and bad ways of doing things. Both the answers from Burt and Graeme will work. I just wrote a quick test app to prove this.