When using neo4j-admin memrec there is a recommendation to activate the following in the neo4j.conf file:
dbms.jvm.additional=-XX:+ExitOnOutOfMemoryError
However if I add the line in, I am unable to restart the neo4j service at all. If I check the debug log the service does not even begin to startup. Is there an error in that particular suggested config line that prevents the service from starting before it has even checked anything else?
:~$ sudo service neo4j restart
:~$ tail -f /var/log/neo4j/debug.log
2022-01-24 14:49:26.937+0000 INFO [o.n.d.d.DefaultDatabaseManager] Stopped 'DatabaseId{21dafb04[neo4j]}' successfully.
2022-01-24 14:49:26.937+0000 INFO [o.n.d.d.DefaultDatabaseManager] Stopping 'DatabaseId{00000000[system]}'.
2022-01-24 14:49:26.938+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] [system/00000000] Requirement `Database unavailable` makes database system unavailable.
2022-01-24 14:49:26.938+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] [system/00000000] DatabaseId{00000000[system]} is unavailable.
2022-01-24 14:49:26.939+0000 INFO [o.n.k.d.Database] [system/00000000] Waiting for closing transactions.
2022-01-24 14:49:26.940+0000 INFO [o.n.k.d.Database] [system/00000000] All transactions are closed.
2022-01-24 14:49:26.940+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [system/00000000] Checkpoint triggered by "Database shutdown" # txId: 71 checkpoint started. ..
2022-01-24 14:49:26.956+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [system/00000000] Checkpoint triggered by "Database shutdown" # txId: 71 checkpoint completed in 15ms
2022-01-24 14:49:26.956+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] [system/00000000] No log version pruned. The strategy used was '1 days'.
2022-01-24 14:49:26.974+0000 INFO [o.n.d.d.DefaultDatabaseManager] Stopped 'DatabaseId{00000000[system]}' successfully.
Related
I am trying to setup Sonarqube in my Macbook but I am getting following error when I try to start it with sh sonar.sh console
sudo sh sonar.sh console
Password:
/usr/bin/java
Running SonarQube...
Removed stale pid file: ./SonarQube.pid
INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /Applications/sonarqube/temp
INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:60506]
INFO app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/Applications/sonarqube/elasticsearch]: /Applications/sonarqube/elasticsearch/bin/elasticsearch
INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
Exception in thread "main" java.lang.UnsupportedOperationException: The Security Manager is deprecated and will be removed in a future release
at java.base/java.lang.System.setSecurityManager(System.java:416)
at org.elasticsearch.bootstrap.Security.setSecurityManager(Security.java:99)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
2022.08.24 16:24:52 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 1
2022.08.24 16:24:52 INFO app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2022.08.24 16:24:52 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
After some research on internet I have installed Java 11 but it is not helping me.
I Installed Docker, Docker Compose and then Jenkins on CentOS 8. Seems Jenkins is installed correctly. However I could see the message Jenkins appears to be offline and get an exception as mentioned below. I changed the URL https://updates.jenkins.io/update-center.json in hudson.model.UpdateCenter.xml from https to http. Still the exception reappears. Plugins are not getting upgraded.
Any help on this is appreciated. Thanks.
Exception:
$: docker logs -f jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
2020-05-16 07:08:57.939+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized #4453ms to org.eclipse.jetty.util.log.JavaUtilLog
2020-05-16 07:09:02.052+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2020-05-16 07:09:04.197+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2020-05-16 07:09:04.990+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 1.8.0_242-b08
2020-05-16 07:09:12.128+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2020-05-16 07:09:12.439+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2020-05-16 07:09:12.439+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2020-05-16 07:09:12.476+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2020-05-16 07:09:14.143+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2020-05-16 07:09:14.794+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.#2235eaab{Jenkins v2.237,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
2020-05-16 07:09:14.871+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector#5315b42e{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2020-05-16 07:09:14.872+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started #21390ms
2020-05-16 07:09:14.881+0000 [id=20] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2020-05-16 07:09:18.067+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2020-05-16 07:09:18.316+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2020-05-16 07:09:22.668+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2020-05-16 07:09:22.694+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2020-05-16 07:09:22.881+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2020-05-16 07:09:24.037+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2020-05-16 07:09:24.037+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2020-05-16 07:09:24.037+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2020-05-16 07:09:24.038+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2020-05-16 07:09:24.194+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
2020-05-16 07:09:24.255+0000 [id=39] INFO hudson.util.Retrier#start: Attempt #1 to do the action check updates server
2020-05-16 07:09:27.052+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext#6069d6b7: display name [Root WebApplicationContext]; startup date [Sat May 16 07:09:27 UTC 2020]; root of context hierarchy
2020-05-16 07:09:27.053+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext#6069d6b7]: org.springframework.beans.factory.support.DefaultListableBeanFactory#6481ce76
2020-05-16 07:09:27.084+0000 [id=25] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory#6481ce76: defining beans [authenticationManager]; root of factory hierarchy
2020-05-16 07:09:27.739+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext#3b4860bb: display name [Root WebApplicationContext]; startup date [Sat May 16 07:09:27 UTC 2020]; root of context hierarchy
2020-05-16 07:09:27.739+0000 [id=25] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext#3b4860bb]: org.springframework.beans.factory.support.DefaultListableBeanFactory#5c405df3
2020-05-16 07:09:27.747+0000 [id=25] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory#5c405df3: defining beans [filter,legacy]; root of factory hierarchy
2020-05-16 07:09:27.955+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-05-16 07:09:28.492+0000 [id=19] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
***2020-05-16 07:09:44.531+0000 [id=39] INFO hudson.util.Retrier#start: The attempt #1 to do the action check updates server failed with an allowed exception:
java.net.UnknownHostException: updates.jenkins.io***
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at hudson.model.DownloadService.loadJSON(DownloadService.java:114)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:212)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:207)
at hudson.PluginManager.checkUpdatesServer(PluginManager.java:1767)
at hudson.util.Retrier.start(Retrier.java:63)
at hudson.PluginManager.doCheckUpdatesServer(PluginManager.java:1738)
at jenkins.DailyCheck.execute(DailyCheck.java:93)
at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
at java.lang.Thread.run(Thread.java:748)
2020-05-16 07:09:44.536+0000 [id=39] INFO hudson.util.Retrier#start: Calling the listener of the allowed exception 'updates.jenkins.io' at the attempt #1 to do the action check updates server
2020-05-16 07:09:44.544+0000 [id=39] INFO hudson.util.Retrier#start: Attempted the action check updates server for 1 time(s) with no success
2020-05-16 07:09:44.547+0000 [id=39] SEVERE hudson.PluginManager#doCheckUpdatesServer: Error checking update sites for 1 attempt(s). Last exception was: UnknownHostException: updates.jenkins.io
2020-05-16 07:09:44.566+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 20,358 ms
2020-05-16 07:10:31.488+0000 [id=56] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Periodic background build discarder
2020-05-16 07:10:31.492+0000 [id=56] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Periodic background build discarder. 2 ms
I got the same exception with Kubernetes environment.
My Setup:
-Docker CE
-Kubernetes
Deployment of Jenkins:
deployment.apps/jenkins created
persistentvolume/jenkins created
persistentvolumeclaim/jenkins-claim created
serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created
service/jenkins created
In the pod's log, I could saw this
2020-06-19 05:04:12.590+0000 [id=39] INFO hudson.util.Retrier#start: The attempt #1 to do the action check updates server failed with an allowed exception:
java.net.UnknownHostException: updates.jenkins.io
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
at hudson.model.DownloadService.loadJSON(DownloadService.java:114)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:212)
at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:207)
at hudson.PluginManager.checkUpdatesServer(PluginManager.java:1767)
at hudson.util.Retrier.start(Retrier.java:63)
at hudson.PluginManager.doCheckUpdatesServer(PluginManager.java:1738)
at jenkins.DailyCheck.execute(DailyCheck.java:93)
at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
at java.lang.Thread.run(Thread.java:748)
<b>2020-06-19 05:04:12.591+0000 [id=39] INFO hudson.util.Retrier#start: Calling the listener of the allowed exception 'updates.jenkins.io' at the attempt #1 to do the action check updates server</b>
2020-06-19 05:04:12.593+0000 [id=39] INFO hudson.util.Retrier#start: Attempted the action check updates server for 1 time(s) with no success
2020-06-19 05:04:12.594+0000 [id=39] SEVERE hudson.PluginManager#doCheckUpdatesServer: Error checking update sites for 1 attempt(s). Last exception was: UnknownHostException: updates.jenkins.io
2020-06-19 05:04:12.597+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 20,180 ms
After some time if logs are fetched, you'd see the token for first time Jenkins setup.
get the token and in my case, I had to do a port-forward to 8080.
Once you access Jenkins on the browser, fill in the token and see if Jenkins is offline.
if it is, goto manage Jenkins then advanced tab, scroll down and locate the update.jenkins.io URL at last.
Here you just need to click on submit without tempering anything and apply.
Now check for the updates in the plugins
If this does not work then you can choose HTTP over HTTPS for updates.jenkins.io URL in the same advanced section and submit, apply the changes. Again check for updates in plugins.
If above 2 don't work then it is very much possible you'd have to configure the proxy settings of Jenkins so that Jenkins instance can reach to internet.
The exception is simply saying that Jenkins instance could not fetch the available plugins/updates from updates.jenkins.io site since it can not resolve the URL.
The main idea here is to resolve the URL.
There are a couple of issues related to this URL exception on git issue trackers as well.
Guess you've just hit the common issue.
In my case I had to consider https://kubernetes.io/docs/tasks/debug-application-cluster/dns-debugging-resolution/ for proper debugging of dns.
I guess you can check if you have the network interface with nameserver 8.8.8.8 or the /etc/resolve.conf has the same.
After trying different things, these commands worked for me:
Disable and stop the firewall on my Centos Host
sudo systemctl disable firewalld
sudo systemctl stop firewalld
Then, restart docker
sudo service docker restart
I've been trying to restart neo4j after adding new data on an EC2 instance. I stopped the neo4j instance, then I called systemctl start neo4j, but when I call cypher-shell it says Connection refused, and connection to the browser port doesn't work anymore.
In the beginning I assumed it was a heap space problem, since looking at the debug.log it said there was a memory issue. I adjusted the heap space and cache settings in neo4j.conf as recommended by neo4j-admin memrec, but still neo4j won't start.
Then I assumed it was because my APOC package was outdated. My neo4j version is 3.5.6, but APOC is 3.5.0.3. I download the latest 3.5.0.4 version, but still neo4j won't start.
At last I tried chmod 777 on every file in the data/database and plugin directories and the directories themselves, but still neo4j won't start.
What's strange is when I try neo4j console for all of these attempts, both cypher-shell and the neo4j browser port works just fine. However, obviously I would prefer to be able to launch neo4j with systemctl.
Right now the only hint of error I can find in debug.log is the following:
2019-06-19 21:19:55.508+0000 INFO [o.n.i.d.DiagnosticsManager] Storage summary:
2019-06-19 21:19:55.508+0000 INFO [o.n.i.d.DiagnosticsManager] Total size of store: 3.07 GB
2019-06-19 21:19:55.509+0000 INFO [o.n.i.d.DiagnosticsManager] Total size of mapped files: 3.07 GB
2019-06-19 21:19:55.509+0000 INFO [o.n.i.d.DiagnosticsManager] --- STARTED diagnostics for KernelDiagnostics:StoreFiles
END ---
2019-06-19 21:19:55.509+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Fulfilling of requirement 'Database available' mak
es database available.
2019-06-19 21:19:55.509+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database is ready.
2019-06-19 21:19:55.568+0000 INFO [o.n.k.i.DatabaseHealth] Database health set to OK
2019-06-19 21:19:56.198+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.s3.S3URLConnection` from plugin jar `
/var/lib/neo4j/plugins/apoc-3.5.0.4-all.jar`: com/amazonaws/ClientConfiguration
2019-06-19 21:19:56.199+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.s3.S3Aws` from plugin jar `/var/lib/n
eo4j/plugins/apoc-3.5.0.4-all.jar`: com/amazonaws/auth/AWSCredentials
2019-06-19 21:19:56.200+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.s3.S3Aws$1` from plugin jar `/var/lib
/neo4j/plugins/apoc-3.5.0.4-all.jar`: com/amazonaws/services/s3/model/S3ObjectInputStream
2019-06-19 21:19:56.207+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.hdfs.HDFSUtils$1` from plugin jar `/v
ar/lib/neo4j/plugins/apoc-3.5.0.4-all.jar`: org/apache/hadoop/fs/FSDataInputStream
2019-06-19 21:19:56.208+0000 WARN [o.n.k.i.p.Procedures] Failed to load `apoc.util.hdfs.HDFSUtils` from plugin jar `/var
/lib/neo4j/plugins/apoc-3.5.0.4-all.jar`: org/apache/hadoop/fs/FSDataOutputStream
...
...
...
2019-06-19 21:20:00.678+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutting down database.
2019-06-19 21:20:00.679+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2019-06-19 21:20:00.679+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database is unavailable.
2019-06-19 21:20:00.684+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" # txId: 1
checkpoint started...
2019-06-19 21:20:00.704+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" # txId: 1
checkpoint completed in 20ms
2019-06-19 21:20:00.705+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] No log version pruned, last checkpoint was made in vers
ion 0
2019-06-19 21:20:00.725+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics START ---
2019-06-19 21:20:00.725+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics END ---
2019-06-19 21:20:00.725+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2019-06-19 21:20:05.875+0000 INFO [o.n.g.f.m.e.CommunityEditionModule] No locking implementation specified, defaulting
to 'community'
2019-06-19 21:20:06.080+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Creating database.
2019-06-19 21:20:06.154+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Requirement `Database available` makes database unavailable.
2019-06-19 21:20:06.156+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database is unavailable.
2019-06-19 21:20:06.183+0000 INFO [o.n.i.d.DiagnosticsManager] --- INITIALIZED diagnostics START ---
I think the warning isn't an issue, since it's just a warning and not an error or exception. Also it seems that the database just shuts down automatically, and then restarts, creating an infinite loop. This loop does not happen when I call neo4j console (all the warnings still exist in the logs). All my ports are default.
Any clue why this is happening? I've never encountered this error when I previously launched neo4j on this instance.
If it works with neo4j console but not with systemctl, you should check the rights of the Neo4j folder.
I'm pretty sure you have a problem on it, and that the systemctl doesn't run Neo4j with the same user as you
I get this error when start Alwayson sql, tried many ways but the results still same. any ideas why?
Im using 1 cluster, 1 analytics+search center, 2 ubuntu 16.04 nodes.
INFO [ALWAYSON-SQL] 2019-02-14 11:36:01,348 ALWAYSON-SQL AlwaysOnSqlRunner.scala:304 - Shutting down AlwaysOn SQL.
INFO [ALWAYSON-SQL] 2019-02-14 11:36:01,617 ALWAYSON-SQL AlwaysOnSqlRunner.scala:328 - Set status to stopped
INFO [ALWAYSON-SQL] 2019-02-14 11:36:01,620 ALWAYSON-SQL AlwaysOnSqlRunner.scala:382 - Reserve port for AlwaysOn SQL
INFO [ALWAYSON-SQL] 2019-02-14 11:36:04,621 ALWAYSON-SQL AlwaysOnSqlRunner.scala:375 - Release reserved port
INFO [ALWAYSON-SQL] 2019-02-14 11:36:04,622 ALWAYSON-SQL AlwaysOnSqlRunner.scala:805 - Set InCluster token to DseFs client
INFO [ForkJoinPool-1-worker-1] 2019-02-14 11:36:04,650 AlwaysOnSqlRunner.scala:740 - dsefs server heartbeat response: pong
INFO [ForkJoinPool-1-worker-3] 2019-02-14 11:36:04,757 AlwaysOnSqlRunner.scala:704 - Create DseFs directory /var/log/spark/alwayson_sql
INFO [ForkJoinPool-1-worker-3] 2019-02-14 11:36:04,758 AlwaysOnSqlRunner.scala:805 - Set InCluster token to DseFs client
ERROR [ForkJoinPool-1-worker-3] 2019-02-14 11:36:04,788 AlwaysOnSqlRunner.scala:722 - Failed to check dsefs directory alwayson_sql
com.datastax.bdp.fs.model.AccessDeniedException: Insufficient permissions to path /
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:258)
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:232)
at spray.json.JsValue.convertTo(JsValue.scala:31)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:48)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:44)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
INFO [ALWAYSON-SQL] 2019-02-14 11:36:04,788 ALWAYSON-SQL AlwaysOnSqlRunner.scala:247 - ALWAYSON-SQL caused an exception in state RUNNING : com.datastax.bdp.fs.model.AccessDeniedException: Insufficient permissions to path /
com.datastax.bdp.fs.model.AccessDeniedException: Insufficient permissions to path /
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:258)
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:232)
at spray.json.JsValue.convertTo(JsValue.scala:31)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:48)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:44)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
I have seen this problem too! It was a permissions problem in dsefs! To fix, login with the root Cassandra user, and change permissions of the your alwayson log directory to the alwayson user.
I have installed 3 instances of neo4j version 1.9.4 on a linux machine, in 3 different directories: Neo4j01, neo4j02, neo4j03.
I have updated the configuration files neo4j.properties and neo4j-server.properties as mentioned in the link (http://docs.neo4j.org/chunked/milestone/ha-setup-tutorial.html).
When I start all the neo4j instances one after the other, they are successfully installing, but after some time 2 of the 3 neo4j process/instances are automatically disappearing. I noticed it via ps -aef | grep neo4j.
When I checked console logs then i found below errors:
2013-11-12 16:37:32.512+0000 INFO [Cluster] Checking store consistency with master
2013-11-12 16:37:33.174+0000 INFO [Cluster] Store is consistent
2013-11-12 16:37:33.176+0000 INFO [Cluster] Catching up with master
2013-11-12 16:37:33.276+0000 INFO [Cluster] Now consistent with master
2013-11-12 16:37:34.442+0000 INFO [Cluster] ServerId 2, successfully moved to slave for master ha://localhost.localdomain:6363?serverId=1
2013-11-12 16:37:34.689+0000 INFO [Cluster] Instance 1 is available as backup at backup://localhost.localdomain:6366
2013-11-12 16:37:34.798+0000 INFO [Cluster] Instance 2 (this server) is available as slave at ha://localhost.localdomain:6364?serverId=2
2013-11-12 16:37:35.036+0000 INFO [Cluster] Database available for write transactions
2013-11-12 16:37:35.360+0000 INFO [API] Successfully started database
2013-11-12 16:37:36.079+0000 INFO [API] Starting HTTP on port :7474 with 10 threads available
2013-11-12 16:37:40.596+0000 INFO [Cluster] Instance 3 has failed
2013-11-12 16:37:43.654+0000 INFO [API] Enabling HTTPS on port :7473
2013-11-12 16:38:01.081+0000 INFO [API] Mounted REST API at: /db/manage/
2013-11-12 16:38:01.158+0000 INFO [API] Mounted discovery module at [/]
2013-11-12 16:38:02.375+0000 INFO [API] Loaded server plugin "CypherPlugin"
2013-11-12 16:38:02.449+0000 INFO [API] Loaded server plugin "GremlinPlugin"
2013-11-12 16:38:02.462+0000 INFO [API] Mounted REST API at [/db/data/]
2013-11-12 16:38:02.534+0000 INFO [API] Mounted management API at [/db/manage/]
2013-11-12 16:38:03.568+0000 INFO [API] Mounted webadmin at [/webadmin]
2013-11-12 16:38:06.189+0000 INFO [API] Mounting static content at [/webadmin] from [webadmin-html]
2013-11-12 16:38:30.844+0000 DEBUG [API] Failed to start Neo Server on port [7474], reason [org.mortbay.util.MultiException[java.net.BindException: Address already in use, java.net.BindException: Address already in use]]
2013-11-12 16:38:30.880+0000 DEBUG [API] org.neo4j.server.ServerStartupException: Starting Neo4j Server failed: org.mortbay.util.MultiException[java.net.BindException: Address already in use, java.net.BindException: Address already in use]
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:211) ~[neo4j-server-1.9.4.jar:1.9.4]
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:86) [neo4j-server-1.9.4.jar:1.9.4]
at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:49) [neo4j-server-1.9.4.jar:1.9.4]
Caused by: java.lang.RuntimeException: org.mortbay.util.MultiException[java.net.BindException: Address already in use, java.net.BindException: Address already in use]
at org.neo4j.server.web.Jetty6WebServer.startJetty(Jetty6WebServer.java:334) ~[neo4j-server-1.9.4.jar:1.9.4]
at org.neo4j.server.web.Jetty6WebServer.start(Jetty6WebServer.java:154) ~[neo4j-server-1.9.4.jar:1.9.4]
at org.neo4j.server.AbstractNeoServer.startWebServer(AbstractNeoServer.java:344) ~[neo4j-server-1.9.4.jar:1.9.4]
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187) ~[neo4j-server-1.9.4.jar:1.9.4]
... 2 common frames omitted
Caused by: org.mortbay.util.MultiException: Multiple exceptions
at org.mortbay.jetty.Server.doStart(Server.java:188) ~[jetty-6.1.25.jar:6.1.25]
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) ~[jetty-util-6.1.25.jar:6.1.25]
at org.neo4j.server.web.Jetty6WebServer.startJetty(Jetty6WebServer.java:330) ~[neo4j-server-1.9.4.jar:1.9.4]
... 5 common frames omitted
2013-11-12 16:38:30.894+0000 DEBUG [API] Failed to start Neo Server on port [7474]
Now, only neo4j01 process is running and neo4j02 and neo4j03 processes are disappeared. But even though neo4j01 process is up and running I am unable to access the webadmin page at http://htname:7474/webadmin/#/info/org.neo4j/High%20Availability/.
Please, can someone shed some light on this?
You might want to take a look at https://github.com/neo-technology/neo4j-enterprise-local-qa. This contains a rakefile that automates a local setup of 3 instances. Clone the repo locally, and use
rake setup_cluster start_cluster
to bring a locally running cluster online. Shutdown can be done via
rake stop_cluster
Find the configs in machine[ABC]/conf/.