My local Neo4j has a lot of transaction logs in data/graph.db:
251M 3 Sep 16:44 neostore.transaction.db.0
255M 3 Sep 20:01 neostore.transaction.db.1
255M 3 Sep 23:20 neostore.transaction.db.2
251M 4 Sep 19:34 neostore.transaction.db.3
250M 4 Sep 22:16 neostore.transaction.db.4
134M 5 Sep 05:02 neostore.transaction.db.5
16B 5 Sep 09:57 neostore.transaction.db.6
16B 7 Sep 16:44 neostore.transaction.db.7
I'm backing the graph.db folder up (I have stopped the neo4j instance) in order to reload in another offsite instance, so it would be nice to reduce the folder size.
What methods are there to control these logs?
How do I check if a given neostore.transaction.db.X file has been successfully processed?
Is it safe to remove older processed files?
Logical logs are referred to in the docs, which I believe are the same files:
http://neo4j.com/docs/stable/configuration-logical-logs.html
In conf/neo4j.properties I've changed the option keep_logical_logs to 100M size:
# Keep logical logs, helps debugging but uses more disk space, enabled for
# legacy reasons To limit space needed to store historical logs use values such
# as: "7 days" or "100M size" instead of "true".
keep_logical_logs=100M size
and restarted neo4j, but it hasn't removed any of the old log files.
Can I do this manually when neo4j has stopped? Or are all of these files required?
I stopped neo4j, made a backup of the graph.db directory, removed all bar neostore.transaction.db.7 and started neo4j again. It appears to be happy but...
Thanks!
If you database in good condition you can delete all neostore.transaction.db.x files, but I recommend to you backup them.
Stop Neo4j
Delete neostore.transaction.db.x files
Start Neo4j
Updating:
In neo4j 3.x, the config has changed to:
dbms.tx_log.rotation.retention_policy=<true/false>
dbms.tx_log.rotation.retention_policy=<amount> <type>
see https://neo4j.com/docs/operations-manual/current/configuration/transaction-logs/ for more details
The properties file has a parameter for rotating TX logs:
# Will keep logical logs which contains any transaction committed within 2 days
keep_logical_logs=2 days
Check the docs for more detailed information on the different options that are available.
Related
I'm trying to remotely back up my Neo4J database for already 2 days and nothing works.
I run
sudo neo4j-admin backup --backup-dir=backup --name=graph.db-backup --from=1.1.1.1:1111 --timeout=50m
The files start to get saved and then simply erased after a while and there's no backup.
I tried setting up --pagecache=16M and HEAP_SIZE but it has no effect. Sometime it just stalls sometimes I get an error like this:
unexpected error: java.io.IOException: org.neo4j.com.ComException: Channel has been closed
The DB I'm backing up is Enterprise 3.3.3 and the one I'm backing up with is 3.5.14
Thank you for any help.
Is this the normal Neo4J behavior?
The following points are taken from the Up gradation Link
1. Cleanly shut down the older version of Neo4j, if it is running.
2. Install Neo4j 2.1.5, and set it up to use the same database store directory (typically data/graph.db).
3. Set the Neo4j configuration parameter allow_store_upgrade=true in your conf/neo4j.properties file. Neo4j will fail to start without this configuration set.
4. Start up Neo4j.
5. The database store upgrade will occur during startup.
6. The allow_store_upgrade configuration parameter should be removed, set to false or commented out.
7. Information about the upgrade and a progress indicator are logged into the messages.log file, inside the database store directory.
The sixth point, After starting the higher version of neo4j, should we again set allow_store_upgrade=false.
I have followed the same steps, but the older database which was 10gb after upgrading to new version, database size is only 1.7gb.
I am worried, all my relations that are imported in older version can be seen in newer version?
Any suggestions.
You're mentioning "from 2.1 to 2.3". Please note that as of today there's just a milestone build from 2.3 M01 available. Upgrading a database from a milestone version is not guaranteed, see release notes.
Aside from your graph data itself the graph.db directory contains (depending on your config) transaction logs. These might be a large fraction of the complete directory.
To be sure that your datastore is consistent, use the consistency checker, see http://www.markhneedham.com/blog/2014/01/22/neo4j-backup-store-copy-and-consistency-check/.
Yes, you should complete step #6 after you have successfully started up the new version once (which is what triggers the upgrade). Perhaps a slightly more detailed process could be written as:
4. Start up Neo4j.
5. The database store upgrade will occur during startup.
6. Review information about the upgrade contained in the messages.log file, inside the database store directory.
7. After a successful upgrade:
7a. Shut down Neo4j
7b. Remove, comment out or set to false the allow_store_upgrade configuration parameter.
7c. Start up Neo4j.
This is my problem. I have copied graph.db for a backup.
Neo4j was started when I did that.
After that I deleted graph.db and neo4j started a new one. Neo4j is working fine, no problems.
But when wanted to copy again graph.db (the backup one) I get:
Starting Neo4j Server...WARNING: not changing user
waiting for server to be ready... Failed to start within 120 seconds.
I have uninstalled and installed again neo4j and it does not work.
I think is a problem of permissions... If yes how to solve it? I have tried to change permissions with chmod but it does not work... Someone can help me with the code for changing the permissions or configure neo4j with good permissions.
On the log: this appears multiple times:
2014-08-14 15:27:58.102+0000 INFO [API] Setting startup timeout to: 120000ms based on -1
Detected incorrectly shut down database, performing recovery..
Any other suggestion?
Thank you for your help.
Try to use
bin/neo4j start-no-wait
instead of
bin/neo4j start
As commented before the data was corrupted. It was impossible to recover it.
The correct way to do the backup is: link
Never copy the files while running the database.
We need to zip the graph.db file and then copy it in another location as a backup.
For restore stop the database just delete recursively graph.db and unzip the backup.
Sadly the first data could not be restored...
I just installed log4c and log4c-devel on CentOS 6.x by yum. All works fine except the log time, it's slow for 8 hours
Following is the output of my machine:
[rafael#localhost ~]$ date
Tue Jan 14 15:17:34 CST 2014
Following is the log produced by log4c:
20140114 07:17:34.453 DEBUG GFXLog- CTimerManager::killTimer: time id-0x0000000198c4e0.
you see, slow for 8 hours. I have setup my machine with correct time and timezone, and also keep ntpd running. I think maybe i should setup log4c timezone, but I don't know how? log4crc or code? cannot find any document about that by google.
I had exactly the same problem, but in Ubuntu 14.04 dated_local is not available because log4c is an older version. I wrote an update based on the latest log4c code which does not require modifying the supplied shared library.
See https://bugs.launchpad.net/ubuntu/+source/log4c/+bug/1314576
I am receiving a "not cleanly shutdown" log message when attempting to open a 2 year old Neo4j database posted on github at https://github.com/IGS/disease-ontology/tree/master/database/neo4j. How do I open/update this database?
Here's how I got it initialized: Downloaded the graph_db.tgz to my local win7-64 machine (neo4j-community-1.9.RC2), extracted the graph.db file to the data subfolder, renamed it graph_ontology.db, and edited the server config file to point to it as follows:
org.neo4j.server.database.location=data/graph_ontology.db
When starting Neo4j, the last entry in the log file is:
2013-05-21 17:23:33.418+0000 INFO [o.n.k.EmbeddedGraphDatabase]: Startup failed: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager#1f91d41b' was successfully initialized, but failed to start. Please see attached cause exception.: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource#b86527e' was successfully initialized, but failed to start. Please see attached cause exception.: Mismatching store version found (Uknown while expecting v0.A.0) and the store is not cleanly shutdown. Recover the database with the previous database version and then attempt to upgrade
Thanks,
Jeff
Look at the messages.log of the database to determine which neo4j version it was started with last.
In this case it is 1.4
You can get it from here: http://dist.neo4j.org/neo4j-community-1.4.2-windows.zip
You can easily just point the neo4j-shell to it:
create an upgrade.properties that contains: allow_store_upgrade=true
and then use: bin/neo4j-shell -path graph_ontology.db -config upgrade.properties
Here is the converted db for 1.9.GA: https://dl.dropboxusercontent.com/u/14493611/disease_ontology_graphdb-1.9.tgz