We are seeing a strange behavior in our prod environment only.
The application runs fine for several hours and after that we start seeing
java.sql.SQLException: Closed Connection
This is happening with our spring + ibatis code base( we migrated from Hibernate to iBatis).
we are using weblogic 10.0.3, spring 3.0.2 and iBatis 2.3.5
The behavior of the exceptions are:
Couple of database exceptions while update ( single update not transaction) using iBatis.
Then java.sql.SQLException: ORA-01013: user requested cancel of current operation
Then Cause: java.sql.SQLException: No more data to read from socket
Then Cause: java.sql.SQLException: OALL8 is in an inconsistent state.
Then Cause: java.sql.SQLException: Io exception: Broken pipe.
and then all subsequent database queries getting
Cause: java.sql.SQLException: Closed Connection and after then we have to restart the servers to bring the connections alive.
Related
Here I encountered an error on my database connection.
I've read some references but still can't find the answer to the cause of this error.
Error detail =https://pastebin.com/rqTSh7sG
Connection Database
Program
I have a mule application which exposes a rest webservice.
<inbound-endpoint exchange-pattern="request-response" address="${webservice.url}" doc:name="Generic"></inbound-endpoint>
I am using jersey resources for rest component.
The code works fine in QA but in UAT it throws the following error.
ERROR org.mule.exception.DefaultSystemExceptionStrategy - Caught exception in Exception Strategy: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[?:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_111]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_111]
at java.io.BufferedInputStream.read(BufferedInputStream.java:265) ~[?:1.8.0_111]
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) ~[commons-httpclient-3.1.jar:?]
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) ~[commons-httpclient-3.1.jar:?]
at org.mule.transport.http.HttpServerConnection.readLine(HttpServerConnection.java:245) ~[mule-transport-http-3.8.1.jar:3.8.1]
at org.mule.transport.http.HttpServerConnection.getRequestLine(HttpServerConnection.java:557) ~[mule-transport-http-3.8.1.jar:3.8.1]
at org.mule.transport.http.HttpRequestDispatcherWork.run(HttpRequestDispatcherWork.java:67) ~[mule-transport-http-3.8.1.jar:3.8.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Telnet to the port is working. cron jobs are working fine in the server except the rest methods.
Connection request means that the other side closed the connection. Given that the application works in one environment I assume that the problem is the environment. You have to find out what is different between the environments to cause that problem. Is there anything between the client and the Mule application? A load balancer, proxy, etc? Difference in timeouts? Performing a network traffic capture from both sides could be helpful to isolate the problem.
I have probably a very stupid error here, but I cannot get a simple Grails 4.0.4 project to connect to a Neo4j Desktop instance (1.3.8).
The neo4j service is running and I have been able to verify that I have connectivity to it from other tools, like DBeaver and Python.
The neo4j database settings in application.yml looks like this:
grails:
neo4j:
url: bolt://localhost:7687
username: "neo4j"
password: "neo4j"
The project I generated from start.grails.org running this:
curl -O start.grails.org/myapp.zip -d version=4.0.4 -d features=events,geb2,neo4j
The problem is when I try to launch a Grails project, the app fails to start with the message:
Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'neo4jDatastore': Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.grails.datastore.gorm.neo4j.Neo4jDatastore]: Constructor threw exception; nested exception is org.neo4j.driver.v1.exceptions.ServiceUnavailableException: Connection to the database terminated. This can happen due to network instabilities, or due to restarts of the database
Would be grateful for some hints that could help me get started.
Thanks!
As per http://gorm.grails.org/snapshot/neo4j/manual/#releaseNotes, there doesn't seem to be a version that yet supports Neo4j 4.x.
The latest supported version is Neo4j 3.x with GORM 7.0.x
apparently with no specific reason, and with nothing on neo4j logs, our application is getting this:
2019-01-30 14:15:08,715 WARN com.calenco.core.content3.ContentHandler:177 - Unable to acquire connection from the pool within configured maximum time of 60000ms
org.neo4j.driver.v1.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:192)
at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:89)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.neo4j.driver.internal.util.Futures.lambda$asCompletionStage$0(Futures.java:78)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:745)
The neo4j server is still running, and answering requests to either its web browser console, or the cypher-shell CLI. Also, restarting our application re-acquires the connection to neo4j with no issue.
Our application is connecting to neo4j once when it's started and then keeps that connection open for as lunch as it's running, opening and closing sessions against that connection as needed to fulfill the received requests.
It's the 2nd time in less than a month that we see the above exception thrown.
Any ideas?
Thanks in advance
I'm trying to get most basic example for Spring Cloud Dataflow running on CloudFoundry.
I've followed the steps here: http://docs.spring.io/spring-cloud-dataflow-admin-cloudfoundry/docs/current-SNAPSHOT/reference/htmlsingle/#getting-started to make the admin app available in my org/space.
Then I tried to create the most basic example from http://cloud.spring.io/spring-cloud-dataflow/, namely to create the "ticktock" stream:
dataflow:>stream create ticktock --definition "time | log" --deploy
I can see that both apps ticktock-time and ticktock-log are created in the space, the needed service "redis" is bound to these apps and they try to start. Unfortunately they don't start completely, because they have problem to access "redis" service. In the log we find:
Exception encountered during context initialization - cancelling
refresh attempt:
org.springframework.context.ApplicationContextException: Failed to
start bean 'outputBindingLifecycle'; nested exception is
org.springframework.context.ApplicationContextException: Failed to
start bean 'inputBindingLifecycle'; nested exception is
org.springframework.data.redis.RedisConnectionFailureException: Cannot
get Jedis connection; nested exception is
redis.clients.jedis.exceptions.JedisConnectionException: Could not get
a resource from the pool
which eventually is caused by
Caused by: redis.clients.jedis.exceptions.JedisConnectionException:
java.net.ConnectException: Connection refused
Am I missing some configuration step in between?
Alexander
There seems to be an issue with our deployer using the master branch of the Java buildpack. Try these settings for the Dataflow Server:
cf set-env s-c-dataflow-server CLOUDFOUNDRY_BUILDPACK https://github.com/cloudfoundry/java-buildpack.git#v3.6
cf restage s-c-dataflow-server
Also, be aware that we currently launch apps using "streamname-module" as part of the URL so unless you use unique stream names you might collide with other users and get a "400 Bad Request" error.