Neo4j 2.1.2 incremental backup fails but full backup succeeds - neo4j

We recently upgraded our database from 2.0.1 to 2.1.2 (Enterprise) using the explicit upgrade procedure.
When trying to take a backup post-upgrade, full backups succeed, but incremental backups fail.
When running this command the first time, it succeeds:
~/neo4j-enterprise-2.1.2/bin/neo4j-backup -from single://127.0.0.1 -to /mnt/backups/neo4j-test-backup
Running it a second time gives the following error:
Performing backup from '127.0.0.1'
00:18:44.907 [main] INFO o.n.k.InternalAbstractGraphDatabase - No locking implementation specified, defaulting to 'forseti'
Transactions applied
Exception in thread "main" org.neo4j.consistency.ConsistencyCheckingError: Inconsistencies in transaction:
Start[3,xid=GlobalId[NEOKERNL|2772027681176372421|40044|-1], BranchId[ 52 49 52 49 52 49 ],master=-1,me=-1,time=2014-06-23 23:56:53.637+0000/1403567813637,lastCommittedTxWhenTransactionStarted=752027]
1PC[3, txId=752028, 2014-06-23 23:56:53.647+0000/1403567813647]
ConsistencySummaryStatistics{
Number of errors: 2
Number of warnings: 0
Number of inconsistent RELATIONSHIP records: 2
}
at org.neo4j.consistency.checking.incremental.intercept.CheckingTransactionInterceptor.complete(CheckingTransactionInterceptor.java:181)
at org.neo4j.kernel.impl.transaction.xaframework.LogEntryVisitorAdapter.apply(LogEntryVisitorAdapter.java:62)
at org.neo4j.kernel.impl.transaction.xaframework.LogEntryVisitorAdapter.apply(LogEntryVisitorAdapter.java:28)
at org.neo4j.kernel.impl.nioneo.xa.command.LogFilter.endLog(LogFilter.java:87)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.applyTransaction(XaLogicalLog.java:1120)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.applyCommittedTransaction(XaResourceManager.java:856)
at org.neo4j.kernel.impl.transaction.xaframework.XaDataSource.applyCommittedTransaction(XaDataSource.java:246)
at org.neo4j.com.ServerUtil.applyReceivedTransactions(ServerUtil.java:461)
at org.neo4j.backup.BackupService.unpackResponse(BackupService.java:401)
at org.neo4j.backup.BackupService.incrementalWithContext(BackupService.java:315)
at org.neo4j.backup.BackupService.doIncrementalBackup(BackupService.java:257)
at org.neo4j.backup.BackupService.doIncrementalBackup(BackupService.java:210)
at org.neo4j.backup.BackupService.doIncrementalBackupOrFallbackToFull(BackupService.java:231)
at org.neo4j.backup.BackupTool.doBackup(BackupTool.java:240)
at org.neo4j.backup.BackupTool.run(BackupTool.java:168)
at org.neo4j.backup.BackupTool.main(BackupTool.java:71)
Any help/workarounds are appreciated.
Update: The same behavior persists after upgrading to 2.1.3

Could you please check again in the issue is resolved with 2.1.4? I darkly remember a resolved issue regarding incremental backups.

Related

Possible root cause for mix ecto.migrate resulting in recv failed timeout

I am new to Elixir/Erlang. As I need to setup an existing application, I am executing below step to generate migrations:
mix ecto.migrate
After migrating 9 tables, everytime it is throwing following error:
<b> 00:55:30.074 [error] Mariaex.Protocol (#PID<0.415.0>) disconnected: ** (Mariaex.Error) [tcp] `recv` failed with: :timeout </b>
I am really stuck here and do not know how to move ahead. Theare are around 175 migration files.
I tried to re-run the migrations command but that is not working.
Probably the migration whos generate this error performs a long operation and exceeds your MariaDB/MySQL timeout.
Try increase your timeouts for this operation:
https://hexdocs.pm/ecto/Ecto.Repo.html#module-shared-options

Errors in opscenterd.log after starting repair service

After upgrading from Opscenter 5.x > 6.0.8 > 6.1.0 > 6.1.1, I see lots of errors and warnings in the opscenterd.log, like the following. I'm using DSE 4.8.10. I've turned on the Repair Service, which seems to be working as expected. But I see WARNings in the log. Are these anything to be concerned about?
2017-06-27 01:00:00,356 [local] ERROR: The best practice rule 'Tombstone count' has failed. (MainThread)
2017-06-27 01:00:00,358 [local] ERROR: The best practice rule 'Wide partitions' has failed. (MainThread)
2017-06-27 01:00:00,451 [local] ERROR: The best practice rule 'Secondary indexes cardinality' has failed. (MainThread)
2017-06-27 13:10:11,672 [opscenterd] WARN: Unknown request 54688d7f-7c5f-4bcb-bc4d-07b7a0a79c3c (running {'started': 1498569009, 'details': u'Repair session f260e7b0-5b39-11e7-87cf-612516369059 for range (1042910172352712044,1065269862139026652] finished', 'details-type': None}) (MainThread)
2017-06-27 13:12:40,885 [opscenterd] WARN: Unknown request 341c9bc9-1c00-4771-aa64-27206ad4152a (running {'started': 1498569160, 'details': u'Repair session 4c3d5c50-5b3a-11e7-87cf-612516369059 for range (-1555782662812296764,-1538702344225528661] finished', 'details-type': None}) (MainThread)
OpsCenter is warning about repair sessions on the Cassandra nodes that it doesn't recognize. They could be sessions that are hung up; there are a number of tickets related to that in Cassandra 2.1.x and older:
https://issues.apache.org/jira/browse/CASSANDRA-10012
https://issues.apache.org/jira/browse/CASSANDRA-10992
https://issues.apache.org/jira/browse/CASSANDRA-10961
https://issues.apache.org/jira/browse/CASSANDRA-12901
You could try to clean up these sessions by invoking forceTerminateAllRepairSessions via JMX on the StorageSession mbean. Or, a rolling restart on the cluster should do the trick.
Regardless, OpsCenter's Repair Service will operate normally.

Prepared data set "Musicbrainz" doesn't work - Detected incorrectly shut down database, performing recovery

I struggling to get the "Musicbrainz" data set running on my local machine (MacBook Air, Mountain Lion).
What I did? (Basically what the video said)
Downloaded data set directly from http://www.neo4j.org/develop/example_data
Downloaded fresh Neo4j-Community-2.1.5
Extracted the data set, renamed it to graph.db and copied to /neo4j-community-2.1.5/data/graph.db
Run the server: ./bin/neo4j console, but neo does not start - not even an exception is thrown!!!!
Gerardos-MacBook-Air:neo4j-community-2.1.5 Gery$ ./bin/neo4j console
WARNING: Max 256 open files allowed, minimum of 40 000 recommended.
See the Neo4j manual. Starting Neo4j Server console-mode... Using
additional JVM arguments: -server -XX:+DisableExplicitGC
-Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled 2014-10-09 15:40:31.700+0000 INFO [API] Setting startup timeout to: 120000ms
based on -1 Detected incorrectly shut down database, performing
recovery..
I tried several settings in neo4j.properties, but nothing, see properties below ...
use_memory_mapped_buffers=true
neostore.nodestore.db.mapped_memory=300M
neostore.relationshipstore.db.mapped_memory=3G
neostore.propertystore.db.mapped_memory=500M
neostore.propertystore.db.strings.mapped_memory=500M
neostore.propertystore.db.arrays.mapped_memory=0M
neostore.propertystore.db.index.keys.mapped_memory=15M
neostore.propertystore.db.index.mapped_memory=15M
allow_store_upgrade=true
I was hoping that this would work out of the box. What am I missing ...?

CouchDB 1.3.1 on Centos 6.4

I compiled CouchDB and installed. It seems to work great except when I use views on the database, then it just spins the wheel and nothing happens and the cpu load spikes to 100% and slowly it eats away all memory and starts to swap a lot which in return increases the cpu load.
I have tried both with the js-1.70-12 that comes with centos 6.4, as well as build and install my own js-1.85-1. All erlang packages are installed from epel :
erlang-crypto-R14B-04.2.el6.x86_64
erlang-syntax_tools-R14B-04.2.el6.x86_64
erlang-mnesia-R14B-04.2.el6.x86_64
erlang-ssl-R14B-04.2.el6.x86_64
erlang-cosProperty-R14B-04.2.el6.x86_64
erlang-asn1-R14B-04.2.el6.x86_64
erlang-cosEventDomain-R14B-04.2.el6.x86_64
erlang-eunit-R14B-04.2.el6.x86_64
erlang-erl_docgen-R14B-04.2.el6.x86_64
erlang-toolbar-R14B-04.2.el6.x86_64
erlang-debugger-R14B-04.2.el6.x86_64
erlang-tools-R14B-04.2.el6.x86_64
erlang-typer-R14B-04.2.el6.x86_64
erlang-megaco-R14B-04.2.el6.x86_64
erlang-oauth-1.1.1-1.el6.x86_64
erlang-stdlib-R14B-04.2.el6.x86_64
erlang-hipe-R14B-04.2.el6.x86_64
erlang-kernel-R14B-04.2.el6.x86_64
erlang-runtime_tools-R14B-04.2.el6.x86_64
erlang-snmp-R14B-04.2.el6.x86_64
erlang-public_key-R14B-04.2.el6.x86_64
erlang-inets-R14B-04.2.el6.x86_64
erlang-ibrowse-2.2.0-4.el6.x86_64
erlang-cosEvent-R14B-04.2.el6.x86_64
erlang-cosNotification-R14B-04.2.el6.x86_64
erlang-edoc-R14B-04.2.el6.x86_64
erlang-otp_mibs-R14B-04.2.el6.x86_64
erlang-cosFileTransfer-R14B-04.2.el6.x86_64
erlang-cosTransactions-R14B-04.2.el6.x86_64
erlang-inviso-R14B-04.2.el6.x86_64
erlang-jinterface-R14B-04.2.el6.x86_64
erlang-erl_interface-R14B-04.2.el6.x86_64
erlang-diameter-R14B-04.2.el6.x86_64
erlang-gs-R14B-04.2.el6.x86_64
erlang-tv-R14B-04.2.el6.x86_64
erlang-appmon-R14B-04.2.el6.x86_64
erlang-odbc-R14B-04.2.el6.x86_64
erlang-wx-R14B-04.2.el6.x86_64
erlang-et-R14B-04.2.el6.x86_64
erlang-observer-R14B-04.2.el6.x86_64
erlang-sasl-R14B-04.2.el6.x86_64
erlang-dialyzer-R14B-04.2.el6.x86_64
erlang-common_test-R14B-04.2.el6.x86_64
erlang-os_mon-R14B-04.2.el6.x86_64
erlang-examples-R14B-04.2.el6.x86_64
erlang-compiler-R14B-04.2.el6.x86_64
erlang-erts-R14B-04.2.el6.x86_64
erlang-xmerl-R14B-04.2.el6.x86_64
erlang-orber-R14B-04.2.el6.x86_64
erlang-cosTime-R14B-04.2.el6.x86_64
erlang-ssh-R14B-04.2.el6.x86_64
erlang-docbuilder-R14B-04.2.el6.x86_64
erlang-percept-R14B-04.2.el6.x86_64
erlang-parsetools-R14B-04.2.el6.x86_64
erlang-ic-R14B-04.2.el6.x86_64
erlang-pman-R14B-04.2.el6.x86_64
erlang-webtool-R14B-04.2.el6.x86_64
erlang-test_server-R14B-04.2.el6.x86_64
erlang-reltool-R14B-04.2.el6.x86_64
erlang-R14B-04.2.el6.x86_64
erlang-mochiweb-1.4.1-5.el6.x86_64
Every thing configures and makes and installs as expected. You can dump data into the database, you can create documents and all that. But I can not run any view, temporary or not.
The only error I see in the logs is like this one, and it is a lot of these errors :
[Sun, 18 Aug 2013 23:10:38 GMT] [error] [<0.124.0>] {error_report,<0.30.0>,
{<0.124.0>,crash_report,
[[{initial_call,
{mochiweb_socket_server,init,['Argument__1']}},
{pid,<0.124.0>},
{registered_name,[]},
{error_info,
{exit,eaddrinuse,
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_secondary_services,couch_server_sup,
<0.31.0>]},
{messages,[]},
{links,[<0.93.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,459}],
[]]}}
But I have no idea what they mean.
Do I need to compile and install erlang as well ? All the above packages or just erlang ?
Your compilation and installation looks fine. At least your error (note: eaddrinuse in traceback) is about that there is some process that listens same address and port as your CouchDB try to. Check other listening processes with netstat -anp command or change CouchDB's listen port to different.

Neo4j server failed to start

Tried starting neo4j service and got a message like
WARNING: Detected a limit of 1024 for maximum open files, while a
minimum value of 40000 is recommended.
WARNING: Problems with the
operation of the server may occur. Please refer to the Neo4j manual
regarding lifting this limitation. Starting Neo4j Server...
WARNING:
not changing user process [17348]... waiting for server to be
ready... BAD. Neo4j Server may have failed to start, please check the
logs.
The log says :
Opened [/home/ub/graph_db/neo4j-community-1.7.M01/data/graph.db/nioneo_logical.log.1] clean empty log, version=224, lastTxId=654769
2013-03-14 11:26:28.111+0000: TM opening log: /home/ub/graph_db/neo4j-community-1.7.M01/data/graph.db/tm_tx_log.1
2013-03-14 11:26:28.159+0000: Failed to load index provider lucene Target file[lucene.log.v318] already exists
org.neo4j.graphdb.NotFoundException: Target file[lucene.log.v318] already exists
at org.neo4j.kernel.impl.util.FileUtils.renameFile(FileUtils.java:165)
at org.neo4j.kernel.DefaultFileSystemAbstraction.renameFile(DefaultFileSystemAbstraction.java:78)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.renameLogFileToRightVersion(XaLogicalLog.java:700)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.renameIfExists(XaLogicalLog.java:219)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:171)
at org.neo4j.kernel.impl.transaction.xaframework.XaContainer.openLogicalLog(XaContainer.java:64)
at org.neo4j.index.impl.lucene.LuceneDataSource.<init>(LuceneDataSource.java:229)
at org.neo4j.index.lucene.LuceneIndexProvider.load(LuceneIndexProvider.java:71)
at org.neo4j.kernel.AbstractGraphDatabase$DefaultKernelExtensionLoader.loadIndexImplementations(AbstractGraphDatabase.java:986)
at org.neo4j.kernel.AbstractGraphDatabase$DefaultKernelExtensionLoader.init(AbstractGraphDatabase.java:958)
at org.neo4j.kernel.LifeSupport$LifecycleInstance.init(LifeSupport.java:362)
at org.neo4j.kernel.LifeSupport.init(LifeSupport.java:76)
at org.neo4j.kernel.LifeSupport.start(LifeSupport.java:110)
at org.neo4j.kernel.AbstractGraphDatabase.run(AbstractGraphDatabase.java:178)
at org.neo4j.kernel.EmbeddedGraphDatabase.<init>(EmbeddedGraphDatabase.java:69)
at org.neo4j.server.NeoServerBootstrapper$1.createDatabase(NeoServerBootstrapper.java:65)
at org.neo4j.server.database.Database.createDatabase(Database.java:80)
at org.neo4j.server.database.Database.<init>(Database.java:63)
at org.neo4j.server.NeoServerWithEmbeddedWebServer.startDatabase(NeoServerWithEmbeddedWebServer.java:186)
at org.neo4j.server.NeoServerWithEmbeddedWebServer.start(NeoServerWithEmbeddedWebServer.java:97)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87)
at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:52)
2013-03-14 11:26:28.160+0000: TM shutting down
2013-03-14 11:26:28.382+0000: Closed log /home/biju/graph_db/neo4j-community-1.7.M01/data/graph.db/nioneo_logical.log
2013-03-14 11:26:28.945+0000: NeoStore closed
2013-03-14 11:26:28.946+0000: --- SHUTDOWN diagnostics START ---
2013-03-14 11:26:28.947+0000: --- SHUTDOWN diagnostics END ---
This started happening when I have installed ElasticSearch on my machine. There was one issue with starting Elastic search "JAVA_HOME issue", which is sorted.
I had such a problem when I was installing Neo4j the first time on my Linux laptop, I solved putting this couple of rows at the end of the /etc/security/limits.conf file:
user hard nofile 100000
user soft nofile 40000
where user is the login name of the user who starts Neo4j.
The 10000 and 40000 are somewhat arbirtrary, they were ok for me, in case you still get the error try to increase them.
If you've got a db with that problem, upgrading won't make it go away. 1.8.2 will prevent this from happening though. You're running community I see so keeping old logs around isn't all that necessary. Try deleting the existing lucene.log.v318 file, or move it away at least and see what happens the next startup.

Resources