Repair corrupted neo4j database (2.3.2 community) - neo4j

My neo4j-2.3.2-community database ran out of disk space (at 80G). I did a clean shutdown of it, then added more disk, and tried to start it back up, only to get cryptic error messages and have it refuse to start.
I shut it down this morning when I realized the disk was full and it wasn't accepting any more inserts. I didn't get back to trying to bring it back up until this evening. I apologize for the apparent gap in the logs between the shutdown and restart. The database is still in a prototype/proof-of-concept phase. I've been running jobs to load data in it for the past week or so. I really, really don't want to have to start over with that data loading process. It is way too slow as it is.
I'm hoping someone will tell me there is a "neo4j --force_repair" sort of command line option that will do the trick. My data loader is smart enough to check the last records it successfully inserted in the database before trying to load new records, so if we lose a few it should be ok.
Here is what I see in the console.log.
2016-01-31 16:25:56.648+0000 INFO Successfully shutdown Neo4j Server
2016-01-31 16:25:56.810+0000 INFO Successfully stopped database
2016-01-31 16:25:56.811+0000 INFO Successfully shutdown database
2016-02-01 01:16:10.591+0000 INFO Successfully shutdown Neo4j Server
2016-02-01 01:16:10.593+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /home/neo4j/neo4j-community-2.3.2/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery#4ec23a22' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -1 for version -1. At position LogPosition{logVersion=252, byteOffset=100663356} and entry version V2_1
at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:207)
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:92)
at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:77)
at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:135)
at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:72)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
... 19 more

This looks like the transaction logs have only been partially written. If that's true the datastore itself is unlikely to be corrupt.
WARNING: before doing any manual change in the data/graph.db folder be sure to have a backup copy in place.
Before proceeding, read the above paragraph. Backup taken? If yes, try to remove the transaction log files in data/graph.db by rm neostore.transaction.db.*. Try to start Neo4j. If it still fails to start, paste a snippet of data/graph.db/messages.log containing the latest startup sequence - this should give more insight.

Related

Open Externally Locked Neo4j Store

I am currently unable to start a neo4j instance on a server where it was previously working. Neo4j produces an error saying one of the files is externally locked. From reading answers to similar questions I know this usually means another instance is currently using the store.
I have used lsof to look for any open files with 'neo' in their name but there are no open files.
I have used ps to look for any process with 'neo' or 'java' it it's name but no luck with that either.
I am not sure where to go from here to get neo4j working again. How do I figure out why this file is locked?
[site#x neo4j]$ neo4jstart-234.sh
[site#x neo4j]$ WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Starting Neo4j Server console-mode...
2017-07-20 10:37:51.073-0400 INFO Successfully shutdown Neo4j Server
2017-07-20 10:37:51.074-0400 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:235)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:195)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /scratch/neo4j/neo4j-community-2.3.4/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource#14c7f72f' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: org.neo4j.kernel.impl.store.UnderlyingStorageException: Unable to open store file: /scratch/neo4j/neo4j-community-2.3.4/data/graph.db/neostore.nodestore.db.labels
at org.neo4j.kernel.impl.store.CommonAbstractStore.checkStorage(CommonAbstractStore.java:161)
at org.neo4j.kernel.impl.store.CommonAbstractStore.initialise(CommonAbstractStore.java:102)
at org.neo4j.kernel.impl.store.NeoStores.initialize(NeoStores.java:429)
at org.neo4j.kernel.impl.store.NeoStores$StoreType$1.open(NeoStores.java:85)
at org.neo4j.kernel.impl.store.NeoStores$StoreType$1.open(NeoStores.java:78)
at org.neo4j.kernel.impl.store.NeoStores.openStore(NeoStores.java:422)
at org.neo4j.kernel.impl.store.NeoStores.getOrCreateStore(NeoStores.java:467)
at org.neo4j.kernel.impl.store.NeoStores.<init>(NeoStores.java:365)
at org.neo4j.kernel.impl.store.StoreFactory.openNeoStores(StoreFactory.java:174)
at org.neo4j.kernel.impl.store.StoreFactory.openAllNeoStores(StoreFactory.java:138)
at org.neo4j.kernel.NeoStoreDataSource.buildNeoStore(NeoStoreDataSource.java:661)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:535)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 15 more
Caused by: org.neo4j.io.pagecache.impl.FileLockException: Externally locked: /scratch/neo4j/neo4j-community-2.3.4/data/graph.db/neostore.nodestore.db.labels
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.acquireLock(SingleFilePageSwapper.java:209)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.<init>(SingleFilePageSwapper.java:160)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:64)
at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.<init>(MuninnPagedFile.java:95)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:357)
at org.neo4j.kernel.impl.store.CommonAbstractStore.checkStorage(CommonAbstractStore.java:140)
... 27 more

neo4j starts and crashes instantly after update to 3.1.3

today I´ve updated my Neo4j Server (running on Ubuntu) from version 3.1.0 to 3.1.3! Everything works so far, so good.
After restarting my server, it crashes instantly. This happens everytime when I start/restart the server...
This is the part from the error log neo4.log
2017-05-01 15:18:30.896+0000 INFO Starting...
2017-05-01 15:18:31.950+0000 INFO Bolt enabled on localhost:7687.
2017-05-01 15:18:35.181+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:227)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:68)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#486c3088' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:193)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.proc.Procedures#78b283fb' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
... 9 more
Caused by: org.neo4j.kernel.api.exceptions.ProcedureException: Unable to register procedure, because the name `apoc.algo.betweenness` is already in use.
at org.neo4j.kernel.impl.proc.ProcedureRegistry.register(ProcedureRegistry.java:81)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:103)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:76)
at org.neo4j.kernel.impl.proc.Procedures.start(Procedures.java:209)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 11 more
nohup: ignoring input
2017-05-01 15:19:58.689+0000 INFO Starting...
2017-05-01 15:19:59.725+0000 INFO Bolt enabled on localhost:7687.
2017-05-01 15:20:01.504+0000 ERROR Cannot register procedure AssertSchemaProcedure org.neo4j.kernel.api.proc.ProcedureSignature$Builder.mode(Lorg/neo4j/procedure/Mode;)Lorg/neo4j/kernel/api/proc/ProcedureSignature$Builder;
java.lang.NoSuchMethodError: org.neo4j.kernel.api.proc.ProcedureSignature$Builder.mode(Lorg/neo4j/procedure/Mode;)Lorg/neo4j/kernel/api/proc/ProcedureSignature$Builder;
at apoc.schema.AssertSchemaProcedure.signature(AssertSchemaProcedure.java:154)
at org.neo4j.kernel.impl.proc.ProcedureRegistry.register(ProcedureRegistry.java:53)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:103)
at org.neo4j.kernel.impl.proc.Procedures.register(Procedures.java:76)
at apoc.ApocKernelExtensionFactory$ApocLifecycle.registerCustomProcedures(ApocKernelExtensionFactory.java:79)
at apoc.ApocKernelExtensionFactory$ApocLifecycle.start(ApocKernelExtensionFactory.java:67)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.extension.KernelExtensions.start(KernelExtensions.java:86)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
Does anyone had this problem after updating Neo4j or can help me with "understanding" that error log?
Thanks in advance ;)
UPDATE
When I start Neo4j I´ll get this message:
Starting Neo4j.
Started neo4j (pid 28724). By default, it is available at http://localhost:7474/
/usr/share/neo4j/bin/neo4j: line 176: /usr/share/neo4j/logs /var/log/neo4j/neo4j.log: No such file or directory
There may be a short delay until the server is ready.
See /usr/share/neo4j/logs /var/log/neo4j/neo4j.log for current status.
I´ve highlighted the 'suspect' part, which doesn´t appear before updating...
This is the ultimate cause in your call stack:
Caused by: org.neo4j.kernel.api.exceptions.ProcedureException: Unable to register procedure, because the name `apoc.algo.betweenness` is already in use.
This type of exception, where an APOC procedure name is already in use, can be seen when you install a new APOC plugin .jar without removing an existing APOC .jar.
If this applies to you, delete the old APOC .jar and try starting neo4j again.

Neo4j service not starting

I have been using Neo4j for more than 3 months on this server. Not sure what changed, but today morning I am unable to start Neo4j server.
Till today everything was fine and there were not changes that I know of were made to Neo4j or anything else.
I am currently using neo4j 3.1.0.
The following is the error log.
xxx#dataserver:~$ sudo neo4j console
Starting Neo4j.
WARNING: Max 1024 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2017-05-01 03:09:17.948+0000 INFO Starting...
2017-05-01 03:09:18.616+0000 INFO Bolt enabled on 0.0.0.0:7687.
2017-05-01 03:09:20.766+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:227)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:68)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:28)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2241b683' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:443)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:199)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory, /var/lib/neo4j/data/databases/default.graphdb
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:193)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:124)
at org.neo4j.server.CommunityNeoServer.lambda$static$0(CommunityNeoServer.java:57)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery#4b1ec7fa' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:415)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:62)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:98)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:517)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:433)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:189)
... 9 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -1 for version -1. At position LogPosition{logVersion=94, byteOffset=157773910} and entry version V2_1
at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:141)
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:84)
at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:78)
at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:80)
at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:80)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:405)
... 18 more
Stop service
Change dbms.allow_upgrade to true in neo4j config file
Start service again
I had the same problem, and upgrading the db fixed it. I would also suggest changing the password if you are still using default password. Once you start the service it should upgrade the database, once upgraded, you can change the dbms.allow_upgrade to false again if you dont want neo4j to update automatically.

Docker container with Neo4j 2.3 enterprise edition exits without error message when mounting graph.db volume

I have a snapshot of a graph.db that is over 40 gb in size. That snapshot came from a server running Neo4j 2.2.8. Now I'm trying to run this database locally to explore the existing graph without wasting resources and potentially crashing the server.
To do so, I'm using Neo4j via Docker and mounting that snapshot. At least that's what I'm trying to do via docker run -p 7474:7474 --ulimit=nofile=40000:40000 --env=NEO4J_CACHE_MEMORY=8G --env=NEO4J_ALLOW_STORE_UPGRADE=true --env=NEO4J_AUTH=none --volume=$HOME/graph.db:/data/graph.db neo4j:enterprise
This docker image uses the enterprise edition of Neo4j 2.3.2 and therefore needs to set allow_store_upgrade to true.
The only output I'm getting is this:
Starting Neo4j Server console-mode...
2016-03-02 22:43:44.277+0000 INFO No SSL certificate found, generating a self-signed certificate..
2016-03-02 22:43:45.718+0000 INFO Initiating metrics..
Then the container stops and I get control back of the command line.
docker ps -l shows that the container exits with status code 137.
My question here is: How can I troubleshoot this (non-)error and run this dataset on my local machine in a safe environment?
Now when I start Neo4j 2.3.2 community edition on localhost with $HOME/graph.db as the database path, the store gets updated and after a short while, Neo4j is accessible.
With the upgraded store in place, I stopped Neo4j on localhost and tried to re-run docker on the upgraded database.
This is the (logged) output:
Starting Neo4j Server console-mode...
2016-03-07 17:14:17.835+0000 INFO No SSL certificate found, generating a self-signed certificate..
2016-03-07 17:14:19.214+0000 INFO Initiating metrics..
2016-03-07 17:15:05.068+0000 INFO Successfully shutdown Neo4j Server
2016-03-07 17:15:05.070+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.enterprise.EnterpriseBootstrapper.main(EnterpriseBootstrapper.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.enterprise.EnterpriseFacadeFactory, /var/lib/neo4j/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.enterprise.EnterpriseFacadeFactory.newFacade(EnterpriseFacadeFactory.java:40)
at org.neo4j.graphdb.EnterpriseGraphDatabase.<init>(EnterpriseGraphDatabase.java:57)
at org.neo4j.server.enterprise.EnterpriseNeoServer$2.newGraphDatabase(EnterpriseNeoServer.java:67)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.api.index.IndexingService#7d237704' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:184)
at org.apache.lucene.index.TermInfosReaderIndex.<init>(TermInfosReaderIndex.java:77)
at org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:116)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:256)
at org.neo4j.kernel.api.impl.index.LuceneIndexWriter.isOnline(LuceneIndexWriter.java:74)
at org.neo4j.kernel.api.impl.index.LuceneSchemaIndexProvider.getInitialState(LuceneSchemaIndexProvider.java:119)
at org.neo4j.kernel.impl.api.index.IndexingService.init(IndexingService.java:225)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
at org.neo4j.kernel.impl.enterprise.EnterpriseFacadeFactory.newFacade(EnterpriseFacadeFactory.java:40)
at org.neo4j.graphdb.EnterpriseGraphDatabase.<init>(EnterpriseGraphDatabase.java:57)
at org.neo4j.server.enterprise.EnterpriseNeoServer$2.newGraphDatabase(EnterpriseNeoServer.java:67)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)

neo4j - UnexpectedUpgradingStoreVersionException. File is version 'NodeStore v0.A.4'

Our neo4j server just crashed, and when I tried to call bin/neo4j start, I'm getting this error:
2015-04-21 07:50:49.529 ERROR Failed to start Neo Server on port [unknown port] Starting Neo4j Server failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2c651755' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j Server failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2c651755' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:244)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:125)
at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:68)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2c651755' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:499)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:108)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:192)
... 2 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, /var/lib/neo4j/hangwith.db
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:314)
at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:59)
at org.neo4j.server.database.LifecycleManagingDatabase$1.newGraphDatabase(LifecycleManagingDatabase.java:43)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:103)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:493)
... 4 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.state.DataSourceManager#10c9f7b4' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:499)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:108)
at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:309)
... 8 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource#138924b9' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:499)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:108)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:117)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:493)
... 10 more
Caused by: org.neo4j.kernel.impl.storemigration.StoreUpgrader$UnexpectedUpgradingStoreVersionException: 'neostore.nodestore.db' has a store version number that we cannot upgrade from. Expected 'v0.A.3' but file is version 'NodeStore v0.A.4'.
at org.neo4j.kernel.impl.storemigration.UpgradableDatabase.checkUpgradeable(UpgradableDatabase.java:88)
at org.neo4j.kernel.impl.storemigration.StoreMigrator.needsMigration(StoreMigrator.java:158)
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.getParticipantsEagerToMigrate(StoreUpgrader.java:259)
at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:134)
at org.neo4j.kernel.NeoStoreDataSource.upgradeStore(NeoStoreDataSource.java:532)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:493)
... 13 more
The server lib were magically updated to 2.3.0-M01.
How do I fix this problem? Would downgrade to the older binary help me? If so, which binary supports NodeStore v0.A.4?
Seems like you have upgraded your Neo4j instance but it will not start since the database store files have not been upgraded. These require an explicit upgrade. What was the version of Neo4j you were running previously?
To upgrade your graph database explicitly you need to set
allow_store_upgrade=true
in \conf\neo4j.properties. Note that this line should be present in the file already but commented out with a #.
Bear in mind that 2.3.0-M1 is a milestone release and may not be stable. The latest stable version is 2.2.1.

Resources