I have a Neo4j database dump created with a particular version of Neo4j. I am trying to restore the database but I am getting an error (show below) that I suspect is because the dump was created with a version different from my current neo4j version. Is there a way to get the version with which the dump was created using the dump itself ?
Error when loading database dump from debug.log
org.neo4j.kernel.impl.store.UnderlyingStorageException: java.io.FileNotFoundException: /var/lib/neo4j/data/databases/graph.db/neostore (Permission denied)
at org.neo4j.kernel.impl.store.NeoStores.verifyRecordFormat(NeoStores.java:217)
at org.neo4j.kernel.impl.store.NeoStores.<init>(NeoStores.java:144)
at org.neo4j.kernel.impl.store.StoreFactory.openNeoStores(StoreFactory.java:129)
at org.neo4j.kernel.impl.store.StoreFactory.openAllNeoStores(StoreFactory.java:93)
at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.<init>(RecordStorageEngine.java:187)
at org.neo4j.kernel.NeoStoreDataSource.buildStorageEngine(NeoStoreDataSource.java:514)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:367)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:116)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:225)
at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:146)
at org.neo4j.server.database.CommunityGraphFactory.newGraphDatabase(CommunityGraphFactory.java:41)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:90)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:32)
Caused by: java.io.FileNotFoundException: /var/lib/neo4j/data/databases/graph.db/neostore (Permission denied)
at java.base/java.io.RandomAccessFile.open0(Native Method)
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:347)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:261)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:65)
at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:50)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.<init>(SingleFilePageSwapper.java:170)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:66)
at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.<init>(MuninnPagedFile.java:149)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:412)
at org.neo4j.kernel.impl.store.MetaDataStore.getRecord(MetaDataStore.java:285)
at org.neo4j.kernel.impl.store.NeoStores.verifyRecordFormat(NeoStores.java:198
You have an issue with file permissions:
Caused by: java.io.FileNotFoundException: /var/lib/neo4j/data/databases/graph.db/neostore (Permission denied)
Check that the /var/lib/neo4j/ directory and all files inside (recursively) are accessible to the user you run Neo4j database under (typically neo4j).
This situation usually happens when you copy the data under different user or root. You can fix this by running:
chown -R neo4j:neo4j /var/lib/neo4j/
Related
I am getting the following error while trying to build an application.
It worked properly from long time, recently it stopped the build process and throws the error. The configuration of the build uses slave node, In google many suggested that the issue may related to the Java version issues in slave node and the selected version in jenkings configuration section.
FATAL: Unable to produce a script file
hudson.util.IOException2: Failed to create a temp file on
/scratch/jenkins/workspace/
at hudson.FilePath.createTextTempFile(FilePath.java:1223)
at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:115)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:75)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
at hudson.model.Run.execute(Run.java:1593)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:247)
Caused by: hudson.util.IOException2: remote file operation failed:
/scratch/jenkins/workspace/ERT_PC_CF7 at
hudson.remoting.Channel#1f9efe8:build- linux-1
at hudson.FilePath.act(FilePath.java:901)
at hudson.FilePath.act(FilePath.java:878)
at hudson.FilePath.createTextTempFile(FilePath.java:1200)
... 12 more
Caused by: hudson.util.IOException2: Failed to create a temporary directory in /tmp
at hudson.FilePath$14.invoke(FilePath.java:1212)
at hudson.FilePath$14.invoke(FilePath.java:1200)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2393)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: Read-only file system
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.checkAndCreate(File.java:1717)
at java.io.File.createTempFile0(File.java:1738)
at java.io.File.createTempFile(File.java:1815)
at hudson.FilePath$14.invoke(FilePath.java:1210)
This is not a permission issue. The issue has been resolved by rebooting the slave nodes as the whole file system in slave node went into read mode, even the root user gets exception with touch command on the directory /scratch/jenkins/workspace/ .
Here is my setup. I have created a basic solr container (version 5.5.5 from official solr docker hub) using docker. In this container I have created some 30000 documents. What I am now trying to do is create a new docker solr image that has these documents in it by default.
I thought I could copy the index out of my existing container and copy it into my new image through the dockerfile. I tried this by copying it out.
docker cp my_solr_container:/opt/solr/server/solr/core_name/data .
and then in the Dockerfile for my new solr image copying this data folder into the corresponding folder
COPY --chown=solr:solr data /opt/solr/server/solr/core_name/data/
but when I build the image and try to run it I get the following error:
3035 ERROR (coreLoadExecutor-6-thread-1) [ x:Development] o.a.s.c.CoreContainer Error creating core [Development]: Error opening new searcher
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:820)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:658)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:820)
at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:90)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:473)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:464)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1696)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1807)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:914)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
... 10 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=MMapIndexInput(path="/opt/solr/server/solr/dev/data/index/_33.fdt"))
at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:418)
at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:403)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:180)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121)
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:117)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:99)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:435)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:100)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:40)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1660)
... 13 more
3046 ERROR (coreContainerWorkExecutor-2-thread-1) [ ] o.a.s.c.CoreContainer Error waiting for SolrCore to be created
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [Development]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.core.CoreContainer$2.run(CoreContainer.java:502)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Unable to create core [Development]
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:833)
at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:90)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:473)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:464)
... 5 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:820)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:658)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:820)
... 8 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1696)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1807)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:914)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
... 10 more
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=MMapIndexInput(path="/opt/solr/server/solr/dev/data/index/_33.fdt"))
at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:418)
at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:403)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:180)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:121)
at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsReader(Lucene50StoredFieldsFormat.java:173)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:117)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:197)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:99)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:435)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:100)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:79)
at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:40)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1660)
... 13 more
I also tried to copy the index in another fashion by hitting the url
http://host:8080/solr/replication?command=backup&location=/home/backup
on my existing container, then copying down the backup index from the container and copying into my new docker image in the same way but ran into the same error.
What would be a possible way to copy data from my existing container such that I can build a new image with this data already set.
Here is my Dockerfile for reference as well:
FROM solr:5.5.5
ARG core_name="Development"
ARG core_path="/opt/solr/server/solr/dev"
COPY --chown=solr:solr schema.xml $core_path/conf/
COPY --chown=solr:solr solrconfig.xml $core_path/conf/
COPY --chown=solr:solr stopwords.txt $core_path/conf/
COPY --chown=solr:solr data $core_path/data/
USER solr
RUN echo "name=$core_name" > $core_path/core.properties
you could use [docker commit][1] to just commit the container instance where you have created the documents as a new image.
I am currently unable to start a neo4j instance on a server where it was previously working. Neo4j produces an error saying one of the files is externally locked. From reading answers to similar questions I know this usually means another instance is currently using the store.
I have used lsof to look for any open files with 'neo' in their name but there are no open files.
I have used ps to look for any process with 'neo' or 'java' it it's name but no luck with that either.
I am not sure where to go from here to get neo4j working again. How do I figure out why this file is locked?
[site#x neo4j]$ neo4jstart-234.sh
[site#x neo4j]$ WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Starting Neo4j Server console-mode...
2017-07-20 10:37:51.073-0400 INFO Successfully shutdown Neo4j Server
2017-07-20 10:37:51.074-0400 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:235)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#79b877d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:195)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /scratch/neo4j/neo4j-community-2.3.4/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource#14c7f72f' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: org.neo4j.kernel.impl.store.UnderlyingStorageException: Unable to open store file: /scratch/neo4j/neo4j-community-2.3.4/data/graph.db/neostore.nodestore.db.labels
at org.neo4j.kernel.impl.store.CommonAbstractStore.checkStorage(CommonAbstractStore.java:161)
at org.neo4j.kernel.impl.store.CommonAbstractStore.initialise(CommonAbstractStore.java:102)
at org.neo4j.kernel.impl.store.NeoStores.initialize(NeoStores.java:429)
at org.neo4j.kernel.impl.store.NeoStores$StoreType$1.open(NeoStores.java:85)
at org.neo4j.kernel.impl.store.NeoStores$StoreType$1.open(NeoStores.java:78)
at org.neo4j.kernel.impl.store.NeoStores.openStore(NeoStores.java:422)
at org.neo4j.kernel.impl.store.NeoStores.getOrCreateStore(NeoStores.java:467)
at org.neo4j.kernel.impl.store.NeoStores.<init>(NeoStores.java:365)
at org.neo4j.kernel.impl.store.StoreFactory.openNeoStores(StoreFactory.java:174)
at org.neo4j.kernel.impl.store.StoreFactory.openAllNeoStores(StoreFactory.java:138)
at org.neo4j.kernel.NeoStoreDataSource.buildNeoStore(NeoStoreDataSource.java:661)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:535)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 15 more
Caused by: org.neo4j.io.pagecache.impl.FileLockException: Externally locked: /scratch/neo4j/neo4j-community-2.3.4/data/graph.db/neostore.nodestore.db.labels
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.acquireLock(SingleFilePageSwapper.java:209)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.<init>(SingleFilePageSwapper.java:160)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:64)
at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.<init>(MuninnPagedFile.java:95)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:357)
at org.neo4j.kernel.impl.store.CommonAbstractStore.checkStorage(CommonAbstractStore.java:140)
... 27 more
I have a snapshot of a graph.db that is over 40 gb in size. That snapshot came from a server running Neo4j 2.2.8. Now I'm trying to run this database locally to explore the existing graph without wasting resources and potentially crashing the server.
To do so, I'm using Neo4j via Docker and mounting that snapshot. At least that's what I'm trying to do via docker run -p 7474:7474 --ulimit=nofile=40000:40000 --env=NEO4J_CACHE_MEMORY=8G --env=NEO4J_ALLOW_STORE_UPGRADE=true --env=NEO4J_AUTH=none --volume=$HOME/graph.db:/data/graph.db neo4j:enterprise
This docker image uses the enterprise edition of Neo4j 2.3.2 and therefore needs to set allow_store_upgrade to true.
The only output I'm getting is this:
Starting Neo4j Server console-mode...
2016-03-02 22:43:44.277+0000 INFO No SSL certificate found, generating a self-signed certificate..
2016-03-02 22:43:45.718+0000 INFO Initiating metrics..
Then the container stops and I get control back of the command line.
docker ps -l shows that the container exits with status code 137.
My question here is: How can I troubleshoot this (non-)error and run this dataset on my local machine in a safe environment?
Now when I start Neo4j 2.3.2 community edition on localhost with $HOME/graph.db as the database path, the store gets updated and after a short while, Neo4j is accessible.
With the upgraded store in place, I stopped Neo4j on localhost and tried to re-run docker on the upgraded database.
This is the (logged) output:
Starting Neo4j Server console-mode...
2016-03-07 17:14:17.835+0000 INFO No SSL certificate found, generating a self-signed certificate..
2016-03-07 17:14:19.214+0000 INFO Initiating metrics..
2016-03-07 17:15:05.068+0000 INFO Successfully shutdown Neo4j Server
2016-03-07 17:15:05.070+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.enterprise.EnterpriseBootstrapper.main(EnterpriseBootstrapper.java:32)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#13ae26d2' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.enterprise.EnterpriseFacadeFactory, /var/lib/neo4j/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.enterprise.EnterpriseFacadeFactory.newFacade(EnterpriseFacadeFactory.java:40)
at org.neo4j.graphdb.EnterpriseGraphDatabase.<init>(EnterpriseGraphDatabase.java:57)
at org.neo4j.server.enterprise.EnterpriseNeoServer$2.newGraphDatabase(EnterpriseNeoServer.java:67)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.api.index.IndexingService#7d237704' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:184)
at org.apache.lucene.index.TermInfosReaderIndex.<init>(TermInfosReaderIndex.java:77)
at org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:116)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:256)
at org.neo4j.kernel.api.impl.index.LuceneIndexWriter.isOnline(LuceneIndexWriter.java:74)
at org.neo4j.kernel.api.impl.index.LuceneSchemaIndexProvider.getInitialState(LuceneSchemaIndexProvider.java:119)
at org.neo4j.kernel.impl.api.index.IndexingService.init(IndexingService.java:225)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
at org.neo4j.kernel.impl.enterprise.EnterpriseFacadeFactory.newFacade(EnterpriseFacadeFactory.java:40)
at org.neo4j.graphdb.EnterpriseGraphDatabase.<init>(EnterpriseGraphDatabase.java:57)
at org.neo4j.server.enterprise.EnterpriseNeoServer$2.newGraphDatabase(EnterpriseNeoServer.java:67)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
My neo4j-2.3.2-community database ran out of disk space (at 80G). I did a clean shutdown of it, then added more disk, and tried to start it back up, only to get cryptic error messages and have it refuse to start.
I shut it down this morning when I realized the disk was full and it wasn't accepting any more inserts. I didn't get back to trying to bring it back up until this evening. I apologize for the apparent gap in the logs between the shutdown and restart. The database is still in a prototype/proof-of-concept phase. I've been running jobs to load data in it for the past week or so. I really, really don't want to have to start over with that data loading process. It is way too slow as it is.
I'm hoping someone will tell me there is a "neo4j --force_repair" sort of command line option that will do the trick. My data loader is smart enough to check the last records it successfully inserted in the database before trying to load new records, so if we lose a few it should be ok.
Here is what I see in the console.log.
2016-01-31 16:25:56.648+0000 INFO Successfully shutdown Neo4j Server
2016-01-31 16:25:56.810+0000 INFO Successfully stopped database
2016-01-31 16:25:56.811+0000 INFO Successfully shutdown database
2016-02-01 01:16:10.591+0000 INFO Successfully shutdown Neo4j Server
2016-02-01 01:16:10.593+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase#2d9e9010' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.impl.factory.CommunityFacadeFactory, /home/neo4j/neo4j-community-2.3.2/data/graph.db
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:143)
at org.neo4j.kernel.impl.factory.CommunityFacadeFactory.newFacade(CommunityFacadeFactory.java:43)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:108)
at org.neo4j.server.CommunityNeoServer$1.newGraphDatabase(CommunityNeoServer.java:66)
at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:95)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery#4ec23a22' failed to initialize. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:434)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:66)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:102)
at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:600)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:112)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:139)
... 10 more
Caused by: java.lang.IllegalArgumentException: Unknown entry type -1 for version -1. At position LogPosition{logVersion=252, byteOffset=100663356} and entry version V2_1
at org.neo4j.kernel.impl.transaction.log.entry.LogEntryVersion.entryParser(LogEntryVersion.java:207)
at org.neo4j.kernel.impl.transaction.log.entry.VersionAwareLogEntryReader.readLogEntry(VersionAwareLogEntryReader.java:92)
at org.neo4j.kernel.impl.transaction.log.LogEntryCursor.next(LogEntryCursor.java:54)
at org.neo4j.kernel.recovery.LatestCheckPointFinder.find(LatestCheckPointFinder.java:77)
at org.neo4j.kernel.recovery.PositionToRecoverFrom.apply(PositionToRecoverFrom.java:53)
at org.neo4j.kernel.recovery.DefaultRecoverySPI.getPositionToRecoverFrom(DefaultRecoverySPI.java:135)
at org.neo4j.kernel.recovery.Recovery.init(Recovery.java:72)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
... 19 more
This looks like the transaction logs have only been partially written. If that's true the datastore itself is unlikely to be corrupt.
WARNING: before doing any manual change in the data/graph.db folder be sure to have a backup copy in place.
Before proceeding, read the above paragraph. Backup taken? If yes, try to remove the transaction log files in data/graph.db by rm neostore.transaction.db.*. Try to start Neo4j. If it still fails to start, paste a snippet of data/graph.db/messages.log containing the latest startup sequence - this should give more insight.