I am using cassandra version 1.1.4 and for bulk loading data. I am creating SSTables and then trying to stream them to the cassandra cluster. I am trying this only on a single node (my own system) and thus have kept replication factor as 1. But streaming fails every time I try to, and the following error message occurs
progress: [/10.66.92.92 0/6 (0)] [total: 0 - 0MB/s (avg: 0MB/s)]Streaming session to /10.66.92.92 failed
Exception in thread "Streaming to /10.66.92.92:1" java.lang.RuntimeException: java.io.EOFException
at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(Unknown Source)
at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
progress: [/10.66.92.92 1/6 (16)] [total: 16 - 0MB/s (avg: 0MB/s)]Streaming to the following hosts failed:
[/10.66.92.92]
When I close the server and restart it, I can see my tables inserted, but I dont want this. I want streaming to happen successfully and avoid restarting the server again and again. Cassandra folks please help! I am Using a Windows 7 environment
Related
Looks like something got corrupted with our H2 database. Here is the output of server.log upon startup. Is there any way to recover the data, or are we better off starting fresh?
2018-02-26 13:24:03,867 INFO [main] server: Starting server
2018-02-26 13:24:04,602 INFO [main] server: Starting database in directory E:\ProgramData\perfino
2018-02-26 13:24:05,381 ERROR [main] server: Cannot connect to database or access schema version
org.h2.jdbc.JdbcSQLException: General error: "java.lang.RuntimeException: rowcount remaining=1 SYS" [50000-176]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.convert(DbException.java:294)
at org.h2.engine.Database.openDatabase(Database.java:291)
at org.h2.engine.Database.<init>(Database.java:254)
at org.h2.engine.Engine.openSession(Engine.java:57)
at org.h2.engine.Engine.openSession(Engine.java:164)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:142)
at org.h2.engine.Engine.createSession(Engine.java:125)
at org.h2.engine.Engine.createSession(Engine.java:27)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:331)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:107)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:91)
at org.h2.Driver.connect(Driver.java:74)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at com.perfino.c.a.a(ejt:207)
at com.perfino.c.a.a(ejt:138)
at com.perfino.server.ServerMain.main(ejt:248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:65)
at com.exe4j.runtime.WinLauncher$2.run(WinLauncher.java:96)
Caused by: java.lang.RuntimeException: rowcount remaining=1 SYS
at org.h2.message.DbException.throwInternalError(DbException.java:241)
at org.h2.table.RegularTable.addIndex(RegularTable.java:284)
at org.h2.engine.Database.open(Database.java:705)
at org.h2.engine.Database.openDatabase(Database.java:260)
... 21 more
There is no data recovery tool for H2. If you have a backup of the database, I would recommend switching to the backup.
To reset the database, go to the database directory and remove the files perfino.h2.db and perfino.trace.db while the server is stopped.
The configuration data is located in config.h2.db and will be preserved.
I am working with Elastic Search 5.5.0 on Spring boot 1.5.8.RELEASE using the Java Transport Client library.
The Elastic Search was deployed with docker in a container. It works well. The queries from my java application works well too.
The problem is that the Elasticsearch Health check failed almost every time when Spring boot started from my local machine as the following exception says. I didn't call any health check explicitly in my application.
How can I remove the health check every time on start up or are there any way of passing the health check?
Thanks.
20180201 09:04:12.499 [restartedMain] INFO c.k.a.Application - Log info On
20180201 09:04:12.499 [restartedMain] WARN c.k.a.Application - Log warn On
20180201 09:04:12.499 [restartedMain] ERROR c.k.a.Application - Log error On
20180201 09:04:15.628 [RMI TCP Connection(9)-10.10.20.187] WARN o.s.b.a.h.ElasticsearchHealthIndicator - Health check failed
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:71)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:58)
at org.springframework.boot.actuate.health.ElasticsearchHealthIndicator.doHealthCheck(ElasticsearchHealthIndicator.java:52)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43)
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:85)
at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:35)
at org.springframework.boot.actuate.endpoint.jmx.DataEndpointMBean.getData(DataEndpointMBean.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1252)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1246)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1085)
at org.springframework.jmx.export.SpringModelMBean.invoke(SpringModelMBean.java:90)
at javax.management.modelmbean.RequiredModelMBean.getAttribute(RequiredModelMBean.java:1562)
at org.springframework.jmx.export.SpringModelMBean.getAttribute(SpringModelMBean.java:109)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:232)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:69)
... 47 common frames omitted
You can disable the Elasticsearch health check by adding the following line to your application.xml file
management.health.elasticsearch.enabled: false
When I went deep into the same issue happening in my sprint boot app too, I found that elasticsearch health check use org.elasticsearch.client.RestClient but I was using org.elasticsearch.client.RestHighLevelClient and had created Bean for the same.
In case you want to keep the recurring elasticsearch's healthcheck,
Create a Bean for RestClient using the right host and port.
Or add following entry in app config yml/xml to remove healthcheck itself:
management.health.elasticsearch.enabled: false
disable the health indicator auto configuration
#SpringBootApplication(exclude = {
ElasticSearchRestHealthIndicatorAutoConfiguration.class
})
public class YourApplication
Disable elastic search health check. Mention this property in your yml/xml file
management.health.elasticsearch.enabled=false
I install "Usergrid 2.1" follow by link
https://usergrid.apache.org/docs/installation/ug2-deploy-to-tomcat.html
But when I start tomcat, it throw exception as
(JobSchedulerService.java:118)
at com.google.common.util.concurrent.AbstractScheduledService$1$1.run(Ab
stractScheduledService.java:174)
at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: InvalidRequestException(why:Keyspace 'Usergrid_Applications' does not
exist)
at org.apache.cassandra.thrift.Cassandra$set_keyspace_result.read(Cassan
dra.java:5540)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_set_keyspace(Cassan
dra.java:540)
at org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.j
ava:527)
at me.prettyprint.cassandra.connection.client.HThriftClient.getCassandra
(HThriftClient.java:110)
How to resolve my issue?
After struggling with this problem for very long, i realized that in my case elasticsearch 1.7.6 as well as 2.x does not start without an additional config in /etc/default/elasticsearch.
Just adding this line START_DAEMON=true in the file and restarting elasticsearch worked for me on Ubuntu 16.04.
Now able to connect to setup usergrid successfully. Had searched for the answer everywhere but could not find any reference till now. Hence thought of posting.
The stack I am using is
1. apache-usergrid-2.1.0
2. cassandra 3.7
3. tomcat8
4. elasticsearch 2.4.4
I had almost given up on using usergrid due to this issue, Hope this is one of the solution that might help anyone looking for this issue.
I wrote a program using Processing(which uses jogl and gluegen), and am using PApplet to run it in a java application in NetBeans.
I am unable to run my program (by running the jar) outside the IDE - neither on my local windows 64-bit machine using command line, nor on a external linux server (which is where I would like to run the program) using X11 forwarding
I previously was able to run the program via command line on my windows machine, but ran into trouble after testing on the linux server. I think there may have been some issues with paths or something after I downloaded the linux versions of the same jars
This is the error I receive from the linux machine:
Exception in thread "Animation Thread" java.lang.UnsatisfiedLinkError: jogamp.common.jvm.JVMUtil.initialize(Ljava/nio/ByteBuffer;)Z
at jogamp.common.jvm.JVMUtil.initialize(Native Method)
at jogamp.common.jvm.JVMUtil.<clinit>(JVMUtil.java:58)
at com.jogamp.common.os.Platform$1.run(Platform.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at com.jogamp.common.os.Platform.<clinit>(Platform.java:179)
at javax.media.opengl.GLProfile.<clinit>(GLProfile.java:82)
at processing.opengl.PJOGL.initSurface(PJOGL.java:250)
at processing.opengl.PGraphicsOpenGL.initPrimary(PGraphicsOpenGL.java:6310)
at processing.opengl.PGraphicsOpenGL.requestDraw(PGraphicsOpenGL.java:1653)
at processing.core.PApplet.run(PApplet.java:2256)
at java.lang.Thread.run(Thread.java:724)
this is the error I get from trying to run the jar on my machine using command line:
java.lang.NoSuchMethodError: com.jogamp.common.nio.Buffers.slice2Float(Ljava/nio
/Buffer;[FII)Ljava/nio/FloatBuffer;
at jogamp.opengl.ProjectFloat.<init>(ProjectFloat.java:181)
at jogamp.opengl.ProjectFloat.<init>(ProjectFloat.java:167)
at jogamp.opengl.ProjectFloat.<init>(ProjectFloat.java:163)
at javax.media.opengl.glu.GLU.<init>(GLU.java:164)
at processing.opengl.PJOGL.<init>(PJOGL.java:218)
at processing.opengl.PGraphicsOpenGL.createPGL(PGraphicsOpenGL.java:1744
)
at processing.opengl.PGraphicsOpenGL.<init>(PGraphicsOpenGL.java:518)
at processing.opengl.PGraphics3D.<init>(PGraphics3D.java:37)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou
rce)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at processing.core.PApplet.makeGraphics(PApplet.java:1919)
at processing.core.PApplet.size(PApplet.java:1771)
at processing.core.PApplet.size(PApplet.java:1742)
at noblis.farag.MySketch.setup(MySketch.java:132)
at processing.core.PApplet.handleDraw(PApplet.java:2361)
at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:240)
at processing.core.PApplet.run(PApplet.java:2256)
at java.lang.Thread.run(Unknown Source)
Exception in thread "Animation Thread" java.lang.RuntimeException: com.jogamp.co
mmon.nio.Buffers.slice2Float(Ljava/nio/Buffer;[FII)Ljava/nio/FloatBuffer;
at processing.core.PApplet.makeGraphics(PApplet.java:1944)
at processing.core.PApplet.size(PApplet.java:1771)
at processing.core.PApplet.size(PApplet.java:1742)
at noblis.farag.MySketch.setup(MySketch.java:132)
at processing.core.PApplet.handleDraw(PApplet.java:2361)
at processing.core.PGraphicsJava2D.requestDraw(PGraphicsJava2D.java:240)
at processing.core.PApplet.run(PApplet.java:2256)
at java.lang.Thread.run(Unknown Source)
Any idea about why I can run it in the IDE but not outside? or any way to package it so that it can run outside the IDE?
thanks
We are trying to add a new Solr node to our cluster:
DC Cassandra
Cassandra node 1
DC Solr
Solr node 1 <-- new node (actually, a replacement for an old node)
Solr node 2
Solr node 3
Solr node 4
Solr node 5
During the bootstrap process:
The stream from node 3 to node 1 failed with an exception:
ERROR [STREAM-OUT-/IP_OF_NODE1] 2014-04-01 01:14:40,887 CassandraDaemon.java (line 196) Exception in thread Thread[STREAM-OUT-/IP_OF_NODE1,5,main]
java.lang.NullPointerException
at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
at java.lang.Thread.run(Thread.java:744)
The stream from node 4 to node 1 never started. The last relevant line in node 4's system.log is:
Received streaming plan for Bootstrap.
It should have been followed by:
Prepare completed. Receiving 0 files(0 bytes), sending x files(y bytes)
It seems that the bootstrap process is now stalled because the data file sizes are not changing anymore. How can I force those streams to be retried?
EDIT:
I restarted all nodes today in an attempt to force new node to retry the bootstrap process. Unfortunately, it encountered some stream failures again. This time, the exception in node 1 is as follows:
WARN [STREAM-IN-/IP_OF_NODE3] 2014-04-06 20:48:17,963 StreamSession.java (line 532) [Stream #c84effb0-bda9-11e3-a07d-89325af2f6bf] Retrying for following error
java.lang.RuntimeException: java.io.FileNotFoundException: /home/cassandra/data/my_keyspace/my_table/my_keyspace-my_table-tmp-jb-1209-Data.db (Too many open files)
at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:75)
at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:71)
at org.apache.cassandra.io.compress.CompressedSequentialWriter.open(CompressedSequentialWriter.java:42)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:107)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:60)
at org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:111)
at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:65)
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:47)
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:37)
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:283)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.FileNotFoundException: /home/cassandra/data/my_keyspace/my_table/my_keyspace-my_table-tmp-jb-1209-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:71)
ERROR [STREAM-IN-/78.46.63.218] 2014-04-06 20:48:17,964 StreamSession.java (line 418) [Stream #c84effb0-bda9-11e3-a07d-89325af2f6bf] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:89)
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:283)
at java.lang.Thread.run(Thread.java:724)
There are tons of similar errors in the log. e.g.:
ERROR [CompactionExecutor:129] 2014-04-06 20:50:06,401 CassandraDaemon.java (line 196) Exception in thread Thread[CompactionExecutor:129,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /home/cassandra/data/my_keyspace/my_table/my_keyspace-my_table-jb-51-Data.db (Too many open files)
at org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:154)
at org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:137)
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:400)
at org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
at org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:833)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /home/cassandra/data/my_keyspace/my_table/my_keyspace-my_table-jb-51-Data.db (Too many open files)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47)
at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:57)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1550)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.service.pager.SliceQueryPager.queryNextPage(SliceQueryPager.java:77)
at org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:84)
at org.apache.cassandra.service.pager.SliceQueryPager.fetchPage(SliceQueryPager.java:33)
at org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:148)
... 10 more
Caused by: java.io.FileNotFoundException: /home/cassandra/data/my_keyspace/my_table/my_keyspace-my_table-jb-51-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
... 28 more
This appears to be very similar to a Cassandra bug/issue:
https://issues.apache.org/jira/browse/CASSANDRA-6965
I'll follow up on that.
Meanwhile, you could run rebuild/repair on that new node.
EDIT: Another Cassandra issue that appears to be related:
CASSANDRA-6984 - "NullPointerException in Streaming During Repair"
https://issues.apache.org/jira/browse/CASSANDRA-6984
That issue is labeled as a Blocker, so it should get some prompt attention. I've inquired as to whether there is a workaround.
Stay tuned.
(Too many open files)
Looks like you need to increase your ulimit.