Cassandra rpc timeout occurring during heavy writes and deletes - timeout

Question

I'm using cassandra 2.0, and I have created a column family that looks like this:
CREATE TABLE user_id_timestamp_index (
user_id int,
timestamp text,
PRIMARY KEY (user_id, timestamp)
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='NONE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'LeveledCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
I've written over 2 million rows to this table without any issues, and I perform numerous deletes as well.
The problem arises after about 10k deletes or so in rapid succession, and I start encountering numerous rpc_timeouts. A simple "delete from user_id_timestamp_index where user_id = 5 AND timestamp = '12345'" via cqlsh fails during this period.
Things I've noticed and tried:
During rpc timeouts, Load Average on 2 nodes (out of 5) shoot up to about 50.
Compaction is done almost every 5 minutes during these load-intensive writes and deletes.
During rpc_timeouts, tpstats shows pending mutation stages:
MutationStage 64 (active) 395 (pending) 48182373 (completed) 0 0
Timeouts tend to occur when memtable data size exceeds 3 mb for this CF.
After I perform a nodetool flush, pending mutations goes to zero and rpc time disappears, until memtable size creeps up again to past 3 mb.
My question is, there a configuration I can tune? For example, is the solution to simply force a memtable flush on this column family every 5 minutes? Lessen the write load on this table? A way to have faster writes and reduce pending stages? Or is there a better solution?

Answer 1

If you're experiencing GC pressure (which you can tell from the GCInspector lines in the log), you can reduce the amount of memory used by memtables by adjusting memtable_total_space_in_mb in cassandra.yaml. You may also need to reduce the key or row cache settings instead or as well.