unable to start neo4j after exhausting storage in the machine - neo4j

I have running the neo4j community version from last 5 months and it was working fine. But today all the storage space on my machine where neo4j is running got exhausted. So I deleted few files not related to neo4j. But still, neo4j was not responding to the queries. So I restarted neo4j. But it does not start anymore. Can anyone point me in the right direction?
Here are the logs
ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#652d6ca4' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#652d6ca4' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#652d6ca4' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:215)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:90)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:67)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#652d6ca4' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /var/lib/neo4j/data/databases/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:40)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:55)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery#39ef543' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:416)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:62)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:98)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:511)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:99)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
... 10 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -1 for version -1. At position LogPosition{logVersion=79, byteOffset=85258244} and entry version V2_1
at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:141)
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:84)
at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:78)
at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:80)
at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:80)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:406)
... 19 more

Related

Open Externally Locked Neo4j Store

I am currently unable to start a neo4j instance on a server where it was previously working. Neo4j produces an error saying one of the files is externally locked. From reading answers to similar questions I know this usually means another instance is currently using the store.
I have used lsof to look for any open files with 'neo' in their name but there are no open files.
I have used ps to look for any process with 'neo' or 'java' it it's name but no luck with that either.
I am not sure where to go from here to get neo4j working again. How do I figure out why this file is locked?
[site#x neo4j]$ neo4jstart-234.sh
[site#x neo4j]$ WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Starting Neo4j Server console-mode...
2017-07-20 10:37:51.073-0400 INFO Successfully shutdown Neo4j Server
2017-07-20 10:37:51.074-0400 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:235)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:195)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /scratch/neo4j/neo4j-community-2.3.4/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource#14c7f72f' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: org.neo4j.kernel.impl.store.UnderlyingStorageException: Unable to open store file: /scratch/neo4j/neo4j-community-2.3.4/data/graph.db/neostore.nodestore.db.labels
at org.neo4j.kernel.impl.store.CommonAbstractStore.checkStorage(CommonAbstractStore.java:161)
at org.neo4j.kernel.impl.store.CommonAbstractStore.initialise(CommonAbstractStore.java:102)
at org.neo4j.kernel.impl.store.NeoStores.initialize(NeoStores.java:429)
at org.neo4j.kernel.impl.store.NeoStores$StoreType$1.open(NeoStores.java:85)
at org.neo4j.kernel.impl.store.NeoStores$StoreType$1.open(NeoStores.java:78)
at org.neo4j.kernel.impl.store.NeoStores.openStore(NeoStores.java:422)
at org.neo4j.kernel.impl.store.NeoStores.getOrCreateStore(NeoStores.java:467)
at org.neo4j.kernel.impl.store.NeoStores.<init>(NeoStores.java:365)
at org.neo4j.kernel.impl.store.StoreFactory.openNeoStores(StoreFactory.java:174)
at org.neo4j.kernel.impl.store.StoreFactory.openAllNeoStores(StoreFactory.java:138)
at org.neo4j.kernel.NeoStoreDataSource.buildNeoStore(NeoStoreDataSource.java:661)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:535)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 15 more
Caused by: org.neo4j.io.pagecache.impl.FileLockException: Externally locked: /scratch/neo4j/neo4j-community-2.3.4/data/graph.db/neostore.nodestore.db.labels
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.acquireLock(SingleFilePageSwapper.java:209)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.<init>(SingleFilePageSwapper.java:160)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:64)
at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.<init>(MuninnPagedFile.java:95)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:357)
at org.neo4j.kernel.impl.store.CommonAbstractStore.checkStorage(CommonAbstractStore.java:140)
... 27 more

neo4j starts and crashes instantly after update to 3.1.3

today I´ve updated my Neo4j Server (running on Ubuntu) from version 3.1.0 to 3.1.3! Everything works so far, so good.
After restarting my server, it crashes instantly. This happens everytime when I start/restart the server...
This is the part from the error log neo4.log
2017-05-01 15:18:30.896+0000 INFO Starting...
2017-05-01 15:18:31.950+0000 INFO Bolt enabled on localhost:7687.
2017-05-01 15:18:35.181+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:227)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:68)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:193)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.proc.Procedures#78b283fb' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
... 9 more
Caused by: org.neo4j.kernel.api.exceptions.ProcedureException: Unable to register procedure, because the name `apoc.algo.betweenness` is already in use.
at org.neo4j.kernel.impl.proc.ProcedureRegistry.register(ProcedureRegistry.java:81)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:103)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:76)
at org.neo4j.kernel.impl.proc.Procedures.start(Procedures.java:209)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 11 more
nohup: ignoring input
2017-05-01 15:19:58.689+0000 INFO Starting...
2017-05-01 15:19:59.725+0000 INFO Bolt enabled on localhost:7687.
2017-05-01 15:20:01.504+0000 ERROR Cannot register procedure AssertSchemaProcedure org.neo4j.kernel.api.proc.ProcedureSignature$Builder.mode(Lorg/neo4j/procedure/Mode;)Lorg/neo4j/kernel/api/proc/ProcedureSignature$Builder;
java.lang.NoSuchMethodError: org.neo4j.kernel.api.proc.ProcedureSignature$Builder.mode(Lorg/neo4j/procedure/Mode;)Lorg/neo4j/kernel/api/proc/ProcedureSignature$Builder;
at apoc.schema.AssertSchemaProcedure.signature(AssertSchemaProcedure.java:154)
at org.neo4j.kernel.impl.proc.ProcedureRegistry.register(ProcedureRegistry.java:53)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:103)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:76)
at apoc.ApocKernelExtensionFactory$ApocLifecycle.registerCustomProcedures(ApocKernelExtensionFactory.java:79)
at apoc.ApocKernelExtensionFactory$ApocLifecycle.start(ApocKernelExtensionFactory.java:67)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.extension.KernelExtensions.start(KernelExtensions.java:86)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
Does anyone had this problem after updating Neo4j or can help me with "understanding" that error log?
Thanks in advance ;)
UPDATE
When I start Neo4j I´ll get this message:
Starting Neo4j.
Started neo4j (pid 28724). By default, it is available at http://localhost:7474/
/usr/share/neo4j/bin/neo4j: line 176: /usr/share/neo4j/logs /var/log/neo4j/neo4j.log: No such file or directory
There may be a short delay until the server is ready.
See /usr/share/neo4j/logs /var/log/neo4j/neo4j.log for current status.
I´ve highlighted the 'suspect' part, which doesn´t appear before updating...
This is the ultimate cause in your call stack:
Caused by: org.neo4j.kernel.api.exceptions.ProcedureException: Unable to register procedure, because the name `apoc.algo.betweenness` is already in use.
This type of exception, where an APOC procedure name is already in use, can be seen when you install a new APOC plugin .jar without removing an existing APOC .jar.
If this applies to you, delete the old APOC .jar and try starting neo4j again.

NEO4J Server startup issue java.lang.NegativeArraySizeException

The neo4j server is not able to start and i am getting an error java.lang.NegativeArraySizeException
I have added the complete trace below:
2017-04-17 06:31:12.832+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#40e621fd' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#40e621fd' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#40e621fd' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:227)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:68)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#40e621fd' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:193)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery#26a34b49' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:415)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:62)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:98)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:517)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
... 9 more
Caused by: java.lang.NegativeArraySizeException
at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV3_0_2.readDynamicRecord(PhysicalLogCommandReaderV3_0_2.java:492)
at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV3_0_2.readDynamicRecords(PhysicalLogCommandReaderV3_0_2.java:506)
at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV3_0_2.readPropertyRecord(PhysicalLogCommandReaderV3_0_2.java:564)
at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV3_0_2.visitPropertyCommand(PhysicalLogCommandReaderV3_0_2.java:189)
at org.neo4j.kernel.impl.transaction.command.PhysicalLogCommandReaderV3_0_2.read(PhysicalLogCommandReaderV3_0_2.java:76)
at org.neo4j.kernel.impl.transaction.command.BaseCommandReader.read(BaseCommandReader.java:49)
at org.neo4j.kernel.impl.transaction.command.BaseCommandReader.read(BaseCommandReader.java:34)
at org.neo4j.kernel.impl.transaction.log.entry.LogEntryParsersV2_3$3.parse(LogEntryParsersV2_3.java:94)
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:98)
at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:78)
at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:80)
at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:80)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:405)

Neo4j service not starting

I have been using Neo4j for more than 3 months on this server. Not sure what changed, but today morning I am unable to start Neo4j server.
Till today everything was fine and there were not changes that I know of were made to Neo4j or anything else.
I am currently using neo4j 3.1.0.
The following is the error log.
xxx#dataserver:~$ sudo neo4j console
Starting Neo4j.
WARNING: Max 1024 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2017-05-01 03:09:17.948+0000 INFO Starting...
2017-05-01 03:09:18.616+0000 INFO Bolt enabled on 0.0.0.0:7687.
2017-05-01 03:09:20.766+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:227)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:68)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/default.graphdb
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:193)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery#4b1ec7fa' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:415)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:62)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:98)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:517)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
... 9 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -1 for version -1. At position LogPosition{logVersion=94, byteOffset=157773910} and entry version V2_1
at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:141)
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:84)
at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:78)
at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:80)
at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:80)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:405)
... 18 more
Stop service
Change dbms.allow_upgrade to true in neo4j config file
Start service again
I had the same problem, and upgrading the db fixed it. I would also suggest changing the password if you are still using default password. Once you start the service it should upgrade the database, once upgraded, you can change the dbms.allow_upgrade to false again if you dont want neo4j to update automatically.

Neo4j 3.0.3 HA cluster doesnot start

Neo4j 3.0.3 HA cluster fails to start with the following error message from the log:
ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#1b38563a' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#1b38563a' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#1b38563a' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:217)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:87)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:66)
at org.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#1b38563a' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:189)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory, /var/lib/neo4j/data/databases/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
at org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory.newFacade(HighlyAvailableFacadeFactory.java:42)
at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.(HighlyAvailableGraphDatabase.java:41)
at org.neo4j.server.enterprise.EnterpriseNeoServer.lambda$static$0(EnterpriseNeoServer.java:80)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.cluster.client.ClusterJoin#27564996' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
... 10 more
Caused by: java.util.concurrent.TimeoutException
at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:300)
at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:143)
at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:82)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 14 more
Since there's a timeout, it's probably an error in the network configuration of the cluster: wrong IP addresses, TCP port filtered by a firewall, etc.
I have the same problem on 3.0.1 in an 2 node ha cluster with an arbiter. However, my arbiter fails first, and if I restart it two more times while the nodes are still trying to start, I'm able to start entire cluster. This is a ridiculous procedure, but this is the only on I've found so far to start the cluster. And not, it is not a networking issue, because after that elaborate restart, everything is working.

Resources