After Remote System forced shutdown. Unable to start Neo4j Server. (neo4j-community-2.2.3)
data/log/console.log now says
2015-10-30 06:42:21.689+0000 INFO [API] Setting startup timeout to: 120000ms based on 120000
2015-10-30 06:42:22.188+0000 INFO [API] Successfully shutdown Neo4j Server.
Updated Link to messages.log
On searching the internet, possible solutions were
Changing the startup time
and the permission related issue
which I don't think is the solution since the server was running fine before.
How to start the server now?
Related
I was running a large delete query and got an out of memory error, so the DB shutdown automatically. I restarted it, but it is still showing as 'offline' in Neo4j desktop.
Here are the log entries from the restart:
2021-08-01 23:47:03.506+0000 INFO Starting...
2021-08-01 23:47:06.804+0000 INFO ======== Neo4j 4.2.1 ========
Exception in thread "neo4j.Scheduler-1" java.lang.OutOfMemoryError: Java heap space
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2021-08-01 23:47:22.505+0000 INFO Sending metrics to CSV file at /Users/my_user/Library/Application Support/Neo4j Desktop/Application/relate-data/dbmss/dbms-########-####-####-####-##########/metrics
2021-08-01 23:47:22.524+0000 INFO Bolt enabled on localhost:7687.
2021-08-01 23:47:23.836+0000 INFO Remote interface available at http://localhost:7474/
2021-08-01 23:47:23.837+0000 INFO Started.
Similarly, when I attempt to connect from a browser it tells me that the Neo4j database is unavailable.
In the log I can see that there is a Java out of memory error. Why would this appear? Does Neo4j queue/cache incomplete queries? And how do I go about clearing it if I can't access the server?
The data is only test data, so I don't need to save it. I do need to understand if I can fix it, and how, since I am putting the product through its paces for a new project.
apparently with no specific reason, and with nothing on neo4j logs, our application is getting this:
2019-01-30 14:15:08,715 WARN com.calenco.core.content3.ContentHandler:177 - Unable to acquire connection from the pool within configured maximum time of 60000ms
org.neo4j.driver.v1.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:192)
at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:89)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.neo4j.driver.internal.util.Futures.lambda$asCompletionStage$0(Futures.java:78)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:745)
The neo4j server is still running, and answering requests to either its web browser console, or the cypher-shell CLI. Also, restarting our application re-acquires the connection to neo4j with no issue.
Our application is connecting to neo4j once when it's started and then keeps that connection open for as lunch as it's running, opening and closing sessions against that connection as needed to fulfill the received requests.
It's the 2nd time in less than a month that we see the above exception thrown.
Any ideas?
Thanks in advance
I am trying to switch from Java embbeded Neo4j database to standalone server but cannot find a way to import my database.
Whatever I do, I am getting the error: `Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2190bb62' was successfully initialized, but failed to start. Please see attached cause exception.'
The problem is that the log file doesn't show anything useful:
2017-04-12 12:25:34.919+0000 INFO [o.n.s.CommunityNeoServer] ======== Neo4j .1.3 ========
2017-04-12 12:25:35.071+0000 INFO [o.n.s.d.LifecycleManagingDatabase] Starting...
2017-04-12 12:25:35.608+0000 INFO [o.n.b.v.r.WorkerFactory] Bolt enabled on localhost:7687.
2017-04-12 12:25:35.722+0000 INFO [o.n.k.i.s.p.StoreMigrator] Starting upgrade of database
I have activated the dbms.directories.import=import option to import an existing database that was created in 2.x
What should I do? is there any way to have more verbose logs?
I tried using Neo4j version 2.x, but I still get the same error when trying to start the server.
I started the influxdb. The meta server is getting started at 8088 and I am seeing a series of [wal] logs. When I try to connect with the server using influx command it throws
Failed to connect to http://localhost:8086
Please check your connection settings and ensure 'influxd' is running.
The server is running in the background. What could be the reason ? I have been writing continuously and then I restarted my server. After restarting I am not able to connect to the server. I also tried connecting after an hour of restarting to make sure it was not due to some startup tasks.
What could be the reason for this ?
The db had huge number of series and it took more than 2 hours for the meta server to be up fully. Later, the http listener was up after the initial start tasks.
We are running v1.9.1, (latest stable release), of Neo4j in Embedded mode. We have had a couple of situations where the process has shutdown unexpectedly and the neo4j.shutdown() has not been called.
Note: when this has occurred we know there is no outstanding updates or changes occurring to the neoDB. Also this is on a linux OS.
When the application is started up again and it starts the connection to neo4j it begins the recovery process but is hanging forever. The messages.log file shows:
2013-07-17 21:05:09.143+0000 INFO [o.n.k.i.t.x.XaLogicalLog]: XaResourceManager[nioneo_logical.log] recovery completed.
2013-07-17 21:05:09.143+0000 INFO [o.n.k.i.t.x.XaLogicalLog]: Recovery on log [/opt/pricing/data/database/app/nioneo_logical.log.1] completed.
2013-07-17 21:05:09.156+0000 INFO [o.n.k.i.t.TxManager]: TM opening log: /opt/pricing/data/database/app/tm_tx_log.2
2013-07-17 21:05:09.245+0000 INFO [o.n.b.BackupServer]: BackupServer communication server started and bound to /0.0.0.0:6362
2013-07-17 21:05:09.271+0000 INFO [o.n.k.i.t.x.XaLogicalLog]: Non clean shutdown detected on log [/opt/pricing/data/database/app/index/lucene.log.2]. Recovery started ...
2013-07-17 21:05:09.271+0000 INFO [o.n.k.i.t.x.XaLogicalLog]: [/opt/pricing/data/database/app/index/lucene.log.2] logVersion=3 with committed tx=317
What's most interesting, we copied the DB over to a desktop and created a little program that just starts the DB then shuts it down and ran it against the DB. It recovered no problems and in only a couple of seconds, (this may be because the hang process had partially recovered the DB, but we don't think so because the application does recover the DB if we kill it and try running it again)
We repeated this on the linux machine with the same successful results.
We are obviously working on trying ensure shutdown will always be called on an unexpected termination of the application, but the real problem is why is the recovery process hanging when starting up?
We did find the following https://groups.google.com/forum/#!msg/neo4j/CBvuMybTRFw/NMIOpBjrIYIJ but that talks about running the DB as a server and just increasing the timeout. Although the point in the messages.log is exactly the same location as mine.
As a temporary solution if the recovery hangs we can run the little 'dummy' program to see if the DB will get fixed, but would rather get to the root cause.
Does anybody have any advice?
What does the CPU/memory/disk do when you say it hangs? Is everything quiet?
Also if you do a jstack or profile or similar of the JVM doing the recovery what does the "main" thread do?
Providing answers to these questions would help a great deal.