What is wrong with these Tomcat database connection pooling configuration - connection-pooling

From last two years on tomcat I am using this configuration for database connection pooling in tomcat
<Resource auth="Container"
driverClassName="com.mysql.jdbc.Driver"
logAbandoned="true"
maxActive="100"
maxIdle="30"
maxWait="10000"
name="jdbc/maindb"
password="xxxxx"
removeAbandoned="true"
removeAbandonedTimeout="60"
type="javax.sql.DataSource"
url="jdbc:mysql://localhost:3306/maindb?zeroDateTimeBehavior=convertToNull"
connectionProperties="useEncoding=true;"
username="sqladmin" validationQuery="select 1"/>
On production server, from last one month, with this configuration - suddenly tomcat stop responding to any requests. And results in timeout. There is no errors in logs, but as soon as I shutdown tomcat a huge amount of error log comes, which seems to show some kind of deadlock in database connections.
To rectify it, I used database connection pooling configuration from http://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html . After using this configuration, now I face two problems, on production, either a table lock occurs even after using INNODB engine, or some queries start returning empty result set even when query is perfectly fine.
<Resource name="jdbc/maindb"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
testWhileIdle="true"
testOnBorrow="true"
testOnReturn="false"
validationQuery="SELECT 1"
validationInterval="30000"
timeBetweenEvictionRunsMillis="30000"
maxActive="100"
minIdle="10"
maxWait="10000"
initialSize="10"
removeAbandonedTimeout="60"
removeAbandoned="true"
logAbandoned="true"
minEvictableIdleTimeMillis="30000"
jmxEnabled="true"
jdbcInterceptors="org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;
org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer"
username="sqladmin"
password="xxxxx"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/maindb"/>
In case of first configuration, after shutting down tomcat following error logs start coming
04-Feb-2015 20:44:46.048 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
04-Feb-2015 20:44:46.049 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-apr-8080"]
04-Feb-2015 20:44:46.100 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-apr-8009"]
04-Feb-2015 20:44:46.151 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
04-Feb-2015 20:44:46.157 INFO [localhost-startStop-2] org.apache.catalina.core.StandardWrapper.unload Waiting for 81 instance(s) to be deallocated for Servlet [dispatcher]
04-Feb-2015 20:44:47.158 INFO [localhost-startStop-2] org.apache.catalina.core.StandardWrapper.unload Waiting for 81 instance(s) to be deallocated for Servlet [dispatcher]
04-Feb-2015 20:44:48.160 INFO [localhost-startStop-2] org.apache.catalina.core.StandardWrapper.unload Waiting for 81 instance(s) to be deallocated for Servlet [dispatcher]
04-Feb-2015 20:44:48.260 INFO [localhost-startStop-2] org.springframework.context.support.AbstractApplicationContext.doClose Closing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue Feb 03 18:26:26 UTC 2015]; parent: Root WebApplicationContext
04-Feb-2015 20:44:48.307 INFO [localhost-startStop-2] org.springframework.context.support.AbstractApplicationContext.doClose Closing Root WebApplicationContext: startup date [Tue Feb 03 18:26:24 UTC 2015]; root of context hierarchy
04-Feb-2015 20:44:48.310 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ExecutorConfigurationSupport.shutdown Shutting down ExecutorService 'taskExecutor'
04-Feb-2015 20:44:48.329 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:582)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
com.myproj.dao.ConnectionPool.getConnection(ConnectionPool.java:41)
And in second case following error logs comes, while doing certain operation
22-Jan-2015 16:36:04.077 SEVERE [http-apr-8080-exec-2] com.myproj.dao.cart.impl.VisitorCartDaoImpl.addCartItem Lock wait timeout exceeded; try restarting transaction
java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2077)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2062)
at com.myproj.dao.cart.impl.VisitorCartDaoImpl.addCartItem(VisitorCartDaoImpl.java:96)
Is there anything wrong with the configuration or with the database? I am using MySQL 5.6 as the database and on production, mySQL is running on amazon RDS. And I am using tomcat 8.0.15

Related

Jenkins instance appears offline on Centos

I Installed Docker, Docker Compose and then Jenkins on CentOS 8. Seems Jenkins is installed correctly. However I could see the message Jenkins appears to be offline and get an exception as mentioned below. I changed the URL https://updates.jenkins.io/update-center.json in hudson.model.UpdateCenter.xml from https to http. Still the exception reappears. Plugins are not getting upgraded.
Any help on this is appreciated. Thanks.
Exception:
$: docker logs -f jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
2020-05-16 07:08:57.939+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized #4453ms to org.eclipse.jetty.util.log.JavaUtilLog
2020-05-16 07:09:02.052+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2020-05-16 07:09:04.197+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2020-05-16 07:09:04.990+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 1.8.0_242-b08
2020-05-16 07:09:12.128+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2020-05-16 07:09:12.439+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2020-05-16 07:09:12.439+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2020-05-16 07:09:12.476+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2020-05-16 07:09:14.143+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2020-05-16 07:09:14.794+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.#2235eaab{Jenkins v2.237,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
2020-05-16 07:09:14.871+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector#5315b42e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2020-05-16 07:09:14.872+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started #21390ms
2020-05-16 07:09:14.881+0000 [id=20] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2020-05-16 07:09:18.067+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2020-05-16 07:09:18.316+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2020-05-16 07:09:22.668+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2020-05-16 07:09:22.694+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2020-05-16 07:09:22.881+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2020-05-16 07:09:24.037+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2020-05-16 07:09:24.037+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2020-05-16 07:09:24.037+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2020-05-16 07:09:24.038+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2020-05-16 07:09:24.194+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
2020-05-16 07:09:24.255+0000 [id=39] INFO hudson.util.Retrier#start: Attempt #1 to do the action check updates server
2020-05-16 07:09:27.052+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext#6069d6b7: display name [Root WebApplicationContext]; startup date [Sat May 16 07:09:27 UTC 2020]; root of context hierarchy
2020-05-16 07:09:27.053+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext#6069d6b7]: org.springframework.beans.factory.support.DefaultListableBeanFactory#6481ce76
2020-05-16 07:09:27.084+0000 [id=25] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory#6481ce76: defining beans [authenticationManager]; root of factory hierarchy
2020-05-16 07:09:27.739+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext#3b4860bb: display name [Root WebApplicationContext]; startup date [Sat May 16 07:09:27 UTC 2020]; root of context hierarchy
2020-05-16 07:09:27.739+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext#3b4860bb]: org.springframework.beans.factory.support.DefaultListableBeanFactory#5c405df3
2020-05-16 07:09:27.747+0000 [id=25] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory#5c405df3: defining beans [filter,legacy]; root of factory hierarchy
2020-05-16 07:09:27.955+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-05-16 07:09:28.492+0000 [id=19] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
***2020-05-16 07:09:44.531+0000 [id=39] INFO hudson.util.Retrier#start: The attempt #1 to do the action check updates server failed with an allowed exception:
java.net.UnknownHostException: updates.jenkins.io***
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at hudson.model.DownloadService.loadJSON(DownloadService.java:114)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:212)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:207)
at hudson.PluginManager.checkUpdatesServer(PluginManager.java:1767)
at hudson.util.Retrier.start(Retrier.java:63)
at hudson.PluginManager.doCheckUpdatesServer(PluginManager.java:1738)
at jenkins.DailyCheck.execute(DailyCheck.java:93)
at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
at java.lang.Thread.run(Thread.java:748)
2020-05-16 07:09:44.536+0000 [id=39] INFO hudson.util.Retrier#start: Calling the listener of the allowed exception 'updates.jenkins.io' at the attempt #1 to do the action check updates server
2020-05-16 07:09:44.544+0000 [id=39] INFO hudson.util.Retrier#start: Attempted the action check updates server for 1 time(s) with no success
2020-05-16 07:09:44.547+0000 [id=39] SEVERE hudson.PluginManager#doCheckUpdatesServer: Error checking update sites for 1 attempt(s). Last exception was: UnknownHostException: updates.jenkins.io
2020-05-16 07:09:44.566+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 20,358 ms
2020-05-16 07:10:31.488+0000 [id=56] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Periodic background build discarder
2020-05-16 07:10:31.492+0000 [id=56] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Periodic background build discarder. 2 ms
I got the same exception with Kubernetes environment.
My Setup:
-Docker CE
-Kubernetes
Deployment of Jenkins:
deployment.apps/jenkins created
persistentvolume/jenkins created
persistentvolumeclaim/jenkins-claim created
serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created
service/jenkins created
In the pod's log, I could saw this
2020-06-19 05:04:12.590+0000 [id=39] INFO hudson.util.Retrier#start: The attempt #1 to do the action check updates server failed with an allowed exception:
java.net.UnknownHostException: updates.jenkins.io
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
at hudson.model.DownloadService.loadJSON(DownloadService.java:114)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:212)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:207)
at hudson.PluginManager.checkUpdatesServer(PluginManager.java:1767)
at hudson.util.Retrier.start(Retrier.java:63)
at hudson.PluginManager.doCheckUpdatesServer(PluginManager.java:1738)
at jenkins.DailyCheck.execute(DailyCheck.java:93)
at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
at java.lang.Thread.run(Thread.java:748)
<b>2020-06-19 05:04:12.591+0000 [id=39] INFO hudson.util.Retrier#start: Calling the listener of the allowed exception 'updates.jenkins.io' at the attempt #1 to do the action check updates server</b>
2020-06-19 05:04:12.593+0000 [id=39] INFO hudson.util.Retrier#start: Attempted the action check updates server for 1 time(s) with no success
2020-06-19 05:04:12.594+0000 [id=39] SEVERE hudson.PluginManager#doCheckUpdatesServer: Error checking update sites for 1 attempt(s). Last exception was: UnknownHostException: updates.jenkins.io
2020-06-19 05:04:12.597+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 20,180 ms
After some time if logs are fetched, you'd see the token for first time Jenkins setup.
get the token and in my case, I had to do a port-forward to 8080.
Once you access Jenkins on the browser, fill in the token and see if Jenkins is offline.
if it is, goto manage Jenkins then advanced tab, scroll down and locate the update.jenkins.io URL at last.
Here you just need to click on submit without tempering anything and apply.
Now check for the updates in the plugins
If this does not work then you can choose HTTP over HTTPS for updates.jenkins.io URL in the same advanced section and submit, apply the changes. Again check for updates in plugins.
If above 2 don't work then it is very much possible you'd have to configure the proxy settings of Jenkins so that Jenkins instance can reach to internet.
The exception is simply saying that Jenkins instance could not fetch the available plugins/updates from updates.jenkins.io site since it can not resolve the URL.
The main idea here is to resolve the URL.
There are a couple of issues related to this URL exception on git issue trackers as well.
Guess you've just hit the common issue.
In my case I had to consider https://kubernetes.io/docs/tasks/debug-application-cluster/dns-debugging-resolution/ for proper debugging of dns.
I guess you can check if you have the network interface with nameserver 8.8.8.8 or the /etc/resolve.conf has the same.
After trying different things, these commands worked for me:
Disable and stop the firewall on my Centos Host
sudo systemctl disable firewalld
sudo systemctl stop firewalld
Then, restart docker
sudo service docker restart

Unable to start neo4j with systemctl: 'Failed to load from plugin jar'

I've been trying to restart neo4j after adding new data on an EC2 instance. I stopped the neo4j instance, then I called systemctl start neo4j, but when I call cypher-shell it says Connection refused, and connection to the browser port doesn't work anymore.
In the beginning I assumed it was a heap space problem, since looking at the debug.log it said there was a memory issue. I adjusted the heap space and cache settings in neo4j.conf as recommended by neo4j-admin memrec, but still neo4j won't start.
Then I assumed it was because my APOC package was outdated. My neo4j version is 3.5.6, but APOC is 3.5.0.3. I download the latest 3.5.0.4 version, but still neo4j won't start.
At last I tried chmod 777 on every file in the data/database and plugin directories and the directories themselves, but still neo4j won't start.
What's strange is when I try neo4j console for all of these attempts, both cypher-shell and the neo4j browser port works just fine. However, obviously I would prefer to be able to launch neo4j with systemctl.
Right now the only hint of error I can find in debug.log is the following:
2019-06-19 21:19:55.508+0000 INFO [o.n.i.d.DiagnosticsManager] Storage summary:
2019-06-19 21:19:55.508+0000 INFO [o.n.i.d.DiagnosticsManager] Total size of store: 3.07 GB
2019-06-19 21:19:55.509+0000 INFO [o.n.i.d.DiagnosticsManager] Total size of mapped files: 3.07 GB
2019-06-19 21:19:55.509+0000 INFO [o.n.i.d.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:StoreFiles
END ---
2019-06-19 21:19:55.509+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Fulfilling of requirement 'Database available' mak
es database available.
2019-06-19 21:19:55.509+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database is ready.
2019-06-19 21:19:55.568+0000 INFO [o.n.k.i.DatabaseHealth] Database health set to OK
2019-06-19 21:19:56.198+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.s3.S3URLConnection` from plugin jar `
/var/lib/neo4j/plugins/apoc-3.5.0.4-all.jar`: com/amazonaws/ClientConfiguration
2019-06-19 21:19:56.199+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.s3.S3Aws` from plugin jar `/var/lib/n
eo4j/plugins/apoc-3.5.0.4-all.jar`: com/amazonaws/auth/AWSCredentials
2019-06-19 21:19:56.200+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.s3.S3Aws$1` from plugin jar `/var/lib
/neo4j/plugins/apoc-3.5.0.4-all.jar`: com/amazonaws/services/s3/model/S3ObjectInputStream
2019-06-19 21:19:56.207+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.hdfs.HDFSUtils$1` from plugin jar `/v
ar/lib/neo4j/plugins/apoc-3.5.0.4-all.jar`: org/apache/hadoop/fs/FSDataInputStream
2019-06-19 21:19:56.208+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.hdfs.HDFSUtils` from plugin jar `/var
/lib/neo4j/plugins/apoc-3.5.0.4-all.jar`: org/apache/hadoop/fs/FSDataOutputStream
...
...
...
2019-06-19 21:20:00.678+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutting down database.
2019-06-19 21:20:00.679+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2019-06-19 21:20:00.679+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database is unavailable.
2019-06-19 21:20:00.684+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" # txId: 1
checkpoint started...
2019-06-19 21:20:00.704+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" # txId: 1
checkpoint completed in 20ms
2019-06-19 21:20:00.705+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] No log version pruned, last checkpoint was made in vers
ion 0
2019-06-19 21:20:00.725+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics START ---
2019-06-19 21:20:00.725+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics END ---
2019-06-19 21:20:00.725+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2019-06-19 21:20:05.875+0000 INFO [o.n.g.f.m.e.CommunityEditionModule] No locking implementation specified, defaulting
to 'community'
2019-06-19 21:20:06.080+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Creating database.
2019-06-19 21:20:06.154+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Requirement `Database available` makes database unavailable.
2019-06-19 21:20:06.156+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database is unavailable.
2019-06-19 21:20:06.183+0000 INFO [o.n.i.d.DiagnosticsManager] --- INITIALIZED diagnostics START ---
I think the warning isn't an issue, since it's just a warning and not an error or exception. Also it seems that the database just shuts down automatically, and then restarts, creating an infinite loop. This loop does not happen when I call neo4j console (all the warnings still exist in the logs). All my ports are default.
Any clue why this is happening? I've never encountered this error when I previously launched neo4j on this instance.
If it works with neo4j console but not with systemctl, you should check the rights of the Neo4j folder.
I'm pretty sure you have a problem on it, and that the systemctl doesn't run Neo4j with the same user as you

Log4j2 Async logger

This is my log4j2.xml file (abbreviated, there are 4 more appenders and loggers, but all of them with exactly the same format):
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="trace" name="ventusproxy" packages="">
<Appenders>
<RollingFile name="ACCESS_LOG"
filePattern="${sys:log.dir}vproxy_access.%d{yyyy-MM-dd}"
append="true"
immediateFlush="false">
<PatternLayout>
<Pattern>%m%d{yyyy-MM-dd HH:mm:ss}%n</Pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Root level="off" additivity="false" />
<Logger name="LOGGER_ACCESS" level="info" includeLocation="true" additivity="false">
<AppenderRef ref="ACCESS_LOG"/>
</Logger>
</Loggers>
</Configuration>
We start tomcat with the option:
-DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
So we enable full async logging. That's why we are simply using Logger and Root instead of AsyncLogger and AsyncRoot.
When tomcat starts, we can see:
2018-08-07 21:03:56,535 localhost-startStop-1 TRACE Using default SystemClock for timestamps.
2018-08-07 21:03:56,536 localhost-startStop-1 TRACE Using DummyNanoClock for nanosecond timestamps.
2018-08-07 21:03:56,536 localhost-startStop-1 DEBUG Reconfiguration complete for context[name=AsyncContext#4d8d549e] at URI /home/ventusproxy/app/ROOT/WEB-INF/log4j.xml (org.apache.logging.log4j.core.async.AsyncLoggerContext#2c9ae56e) with optional ClassLoader: null
2018-08-07 21:03:56,536 localhost-startStop-1 DEBUG Shutdown hook enabled. Registering a new one.
2018-08-07 21:03:56,537 localhost-startStop-1 DEBUG LoggerContext[name=AsyncContext#4d8d549e, org.apache.logging.log4j.core.async.AsyncLoggerContext#2c9ae56e] started OK.
2018-08-07 21:03:56,538 localhost-startStop-1 TRACE Using default SystemClock for timestamps.
2018-08-07 21:03:56,539 localhost-startStop-1 DEBUG AsyncLogger.ThreadNameStrategy=CACHED
2018-08-07 21:03:56,556 Log4j2-TF-1-AsyncLogger[AsyncContext#4d8d549e]-1 DEBUG Now writing to /home/ventusproxy/logs/vproxy_system.2018-08-07 at 2018-08-07T21:03:56.556-0400
2018-08-07 21:03:57,081 localhost-startStop-1 DEBUG PluginManager 'Converter' found 42 plugins
2018-08-07 21:03:57,082 localhost-startStop-1 DEBUG Starting OutputStreamManager SYSTEM_OUT.false.false-2
2018-08-07 21:03:57,083 localhost-startStop-1 TRACE [DefaultAsyncContext#localhost-startStop-1] AsyncLoggerDisruptor creating new disruptor for this context.
2018-08-07 21:03:57,083 localhost-startStop-1 TRACE property AsyncLogger.WaitStrategy=TIMEOUT
2018-08-07 21:03:57,092 localhost-startStop-1 DEBUG [DefaultAsyncContext#localhost-startStop-1] Starting AsyncLogger disruptor for this context with ringbufferSize=262144, waitStrategy=TimeoutBlockingWaitStrategy, exceptionHandler=org.apache.logging.log4j.core.async.AsyncLoggerDefaultExceptionHandler#71fd86cb...
2018-08-07 21:03:57,093 localhost-startStop-1 TRACE [DefaultAsyncContext#localhost-startStop-1] AsyncLoggers use a threadlocal translator
2018-08-07 21:03:57,093 localhost-startStop-1 DEBUG Starting LoggerContext[name=DefaultAsyncContext#localhost-startStop-1, org.apache.logging.log4j.core.async.AsyncLoggerContext#77ae266d]...
2018-08-07 21:03:57,094 localhost-startStop-1 DEBUG Reconfiguration started for context[name=DefaultAsyncContext#localhost-startStop-1] at URI null (org.apache.logging.log4j.core.async.AsyncLoggerContext#77ae266d) with optional ClassLoader: null
2018-08-07 21:03:57,094 localhost-startStop-1 DEBUG Using configurationFactory org.apache.logging.log4j.core.config.ConfigurationFactory$Factory#4b6e6fdf
2018-08-07 21:03:57,133 localhost-startStop-1 DEBUG Initializing configuration XmlConfiguration[location=/home/ventusproxy/app/ROOT/WEB-INF/log4j.xml]
And finally (last debug line):
2018-08-07 21:03:57,200 localhost-startStop-1 DEBUG LoggerContext[name=DefaultAsyncContext#localhost-startStop-1, org.apache.logging.log4j.core.async.AsyncLoggerContext#77ae266d] started OK.
But when we perform a thread dump with JStack, instead of seeing one thread we see 2 threads:
"Log4j2-TF-7-AsyncLogger[DefaultAsyncContext#localhost-startStop-1]-2" #23 daemon prio=5 os_prio=0 tid=0x00007f1ee8ad9800 nid=0x625d runnable [0x00007f1f05817000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000925d3ad8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:124)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Log4j2-TF-1-AsyncLogger[AsyncContext#4d8d549e]-1" #21 daemon prio=5 os_prio=0 tid=0x00007f1ee88d8800 nid=0x625c runnable [0x00007f1f06073000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000093099540> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:124)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
And when we stop tomcat, all these errors are logged:
07-Aug-2018 21:07:29.991 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Log4j2-TF-7-AsyncLogger[DefaultAsyncContext#localhost-startStop-1]-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:124)
java.lang.Thread.run(Thread.java:748)
07-Aug-2018 21:07:29.993 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#494a6bc6]) and a value of type [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator] (value [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator#46d89786]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
07-Aug-2018 21:07:29.993 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#494a6bc6]) and a value of type [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator] (value [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator#7fca5b63]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
07-Aug-2018 21:07:29.994 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#7d2c87bb]) and a value of type [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator] (value [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator#316c8d06]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
07-Aug-2018 21:07:29.994 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#74afa5d5]) and a value of type [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator] (value [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator#5913f960]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
07-Aug-2018 21:07:29.994 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#401baca6]) and a value of type [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator] (value [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator#154a0d1e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
07-Aug-2018 21:07:29.995 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#1cc2936f]) and a value of type [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator] (value [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator#5a5ca16e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
07-Aug-2018 21:07:29.995 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#473f581f]) and a value of type [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator] (value [org.apache.logging.log4j.core.async.RingBufferLogEventTranslator#2d89f1b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
So just 3 questions because now I'm not sure if I'm using asynchronous logging here:
With these traces can I confirm I'm using full asynchronous logging?
I've read that async logging starts just one thread, but here I see 2 (like when we mix sync and async logging). Is this true?
How can I remove these errors from the catalina.out? The application stops correctly so these messages are not really a problem, but I'd prefer to remove them if it's possible.

Apache nifi is not starting up

I am trying to start Apache nifi version 1.2.0 on window 8 machine. It used to start properly. After I restarted the system the nifi is not starting at all. I had check status Its keep getting "Apacha Nifi not running".
Below are logs from nifi.bootstrap.log file:-
2017-07-05 15:41:57,105 WARN [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Failed to set permissions so that only the
owner can read pid file E:\softwares\nifi-1.2.0\bin\..\run\nifi.pid; this
may allows others to have access to the key needed to communicate with NiFi.
Permissions should be changed so that only the owner can read this file
2017-07-05 15:41:57,142 WARN [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Failed to set permissions so that only the
owner can read status file E:\softwares\nifi-1.2.0\bin\..\run\nifi.status;
this may allows others to have access to the key needed to communicate with
NiFi. Permissions should be changed so that only the owner can read this
file
2017-07-05 15:41:57,168 INFO [NiFi Bootstrap Command Listener]
org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for
Bootstrap requests on port 50765
2017-07-05 15:43:12,077 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Failed to start web server: Unable to start Flow Controller.
2017-07-05 15:43:12,078 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Shutting down...
2017-07-05 15:43:14,501 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi
never started. Will not restart NiFi
Stack trace from nifi.app.log: -
2017-07-05 15:43:12,077 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:876)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:839)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1480)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1442)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:799)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:540)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
at org.eclipse.jetty.server.Server.start(Server.java:452)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:419)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:695)
at org.apache.nifi.NiFi.<init>(NiFi.java:160)
at org.apache.nifi.NiFi.main(NiFi.java:267)
Caused by: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead
at org.apache.nifi.repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:65)
at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:115)
at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:109)
at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:46)
at org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1096)
at org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
at org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:381)
at org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:712)
at org.apache.nifi.controller.StandardFlowService.initializeController(StandardFlowService.java:953)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:534)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
... 28 common frames omitted
Thanks in advance
After Googling on this error "Caused by: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead" I found that this error indicates a partial write to the repos.
Here are a couple of things you can check/try to bring your Dataflow back online ;
check if your dsks are not full
Did you launch nifi with the same user ? Did you run it with administrator privileges ?
You can backup/move your repositories and try to start Nifi with empty repositories, you will still have your dataflows there but any file that was processing when you shutdown will be gone.
Could you please try that ?
I think the issue is with incompatible java version, use JAVA 8 version.
If you haven't set JAVA_HOME then set in environment variables with path Like "C:/program files/jdk1.8"
Jira addressing when NiFi run with java 9 version and the issue not resolved yet
https://issues.apache.org/jira/browse/NIFI-4419

How to grant permission to docker user write on /var/?

I'm trying to run confluence image on a docker container. This behavior is Intermittent - most of the time I can't run it, but at sometimes I can.
After downloading the image I run these commands in order put confluence online:
sudo chown -R daemon /data/confluence
sudo chown -R daemon /var/atlassian/application-data/confluence
docker run -v /data/your-confluence-home:/var/atlassian/application-data/confluence --name="confluence" -d -p 8090:8090 -p 8091:8091 atlassian/confluence-server
But then, I get this error on docker logs confluence
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
2017-05-23 15:33:41,519 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
23-May-2017 15:33:42.013 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
23-May-2017 15:33:42.026 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
23-May-2017 15:33:42.028 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 381 ms
23-May-2017 15:33:42.033 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
23-May-2017 15:33:42.033 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.41
SLF4J: A number (1) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
23-May-2017 15:33:43.517 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer#335dedfb]
23-May-2017 15:33:43.814 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
23-May-2017 15:33:43.814 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
23-May-2017 15:33:43.819 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue May 23 15:33:43 UTC 2017]; root of context hierarchy
23-May-2017 15:33:43.864 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
2017-05-23 15:33:44,007 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.2.0 [build 7111 based on commit hash 583f3f4010922d699723a091097524472954c767] - synchrony version 0.3.1-release-confluence_6.1-e20fb944
23-May-2017 15:33:44.371 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
23-May-2017 15:33:44.670 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/sockjs/v1/**] onto handler of type [class org.springframework.web.socket.sockjs.support.SockJsHttpRequestHandler]
23-May-2017 15:33:44.854 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
23-May-2017 15:33:44.855 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
23-May-2017 15:33:44.873 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
23-May-2017 15:33:44.939 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for #ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue May 23 15:33:43 UTC 2017]; root of context hierarchy
23-May-2017 15:33:45.056 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
23-May-2017 15:33:45.094 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1279 ms
2017-05-23 15:33:46,688 WARN [localhost-startStop-1] [com.atlassian.config.ApplicationConfig] setApplicationHome Application home does not exist. Creating directory: /var/atlassian/application-data/confluence/shared-home
2017-05-23 15:33:46,783 ERROR [localhost-startStop-1] [atlassian.config.bootstrap.DefaultAtlassianBootstrapManager] init Home is not configured properly:
com.atlassian.config.ConfigurationException: Couldn't save confluence.cfg.xml to /var/atlassian/application-data/confluence directory.
at com.atlassian.config.xml.AbstractDom4jXmlConfigurationPersister.saveDocument(AbstractDom4jXmlConfigurationPersister.java:86)
at com.atlassian.config.xml.DefaultDom4jXmlConfigurationPersister.save(DefaultDom4jXmlConfigurationPersister.java:24)
at com.atlassian.config.ApplicationConfig.save(ApplicationConfig.java:330)
at com.atlassian.confluence.setup.DefaultBootstrapManager.populateSynchronyConfiguration(DefaultBootstrapManager.java:298)
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:509)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:71)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:36)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:127)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:60)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at com.atlassian.config.xml.AbstractDom4jXmlConfigurationPersister.saveDocumentAtomically(AbstractDom4jXmlConfigurationPersister.java:92)
at com.atlassian.config.xml.AbstractDom4jXmlConfigurationPersister.saveDocument(AbstractDom4jXmlConfigurationPersister.java:82)
... 17 more
What am I doing wrong?
Have you tried via named volume (if using a docker version >= 1.9)?
$ sudo docker volume create --name confluenceVolume
$ sudo docker run -v confluenceVolume:/var/atlassian/application-data/confluence --name="confluence" -d -p 8090:8090 -p 8091:8091 atlassian/confluence-server
Add ENTRYPOINT script and use it. Example - https://github.com/AfsmNGhr/dockemacs/blob/master/Dockerfile#L34. See also - http://www.johnzaccone.io/entrypoint-vs-cmd-back-to-basics/.

Resources