I am currently sending json batches (mass creating nodes and relationships) to my neo4j instance at http://host:7474/db/data/batch. These batches have grown in size and now I am getting the error shown below.
Seemingly this error is caused when trying to create the response to the request, which is just too big and blowing up an Array?
Is there a way around this and still allow the use of the json batch endpoint? The json is being created by a downstream process and so moving to some other mechanism (e.g. csv loading) is not possible due to the necessary changes in other processes.
Is there any way around this? It is not possible to do multiple smaller batches because of the many relationships between the nodes being created. Later nodes and relationships being created refer to node IDs from earlier on the batch. If I break the batch up then I'll refer to node IDs which don't exist in the current transaction.
2015-05-27 13:15:16.425+0000 INFO [API] Remote interface ready and available at [http://localhost:7474/]
09:18:38.822 [qtp20290798-38] WARN o.e.jetty.servlet.ServletHandler -
javax.servlet.ServletException: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420) ~[jersey-servlet-1.18.1.jar:1.18.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540) ~[jersey-servlet-1.18.1.jar:1.18.1]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715) ~[jersey-servlet-1.18.1.jar:1.18.1]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800) ~[jetty-servlet-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) ~[jetty-servlet-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.neo4j.server.rest.dbms.AuthorizationFilter.doFilter(AuthorizationFilter.java:120) ~[neo4j-server-2.2.1.jar:2.2.1]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.4.v20141103.jar:9.2.4.v20141103]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.4.v20141103.jar:9.2.4.v20141103]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2367) ~[na:1.7.0_75]
.
.
.
Limit it to 10k nodes that you create and use -H X-Stream:true as header then it doesn't build up gigantic json response strings in memory.
Related
I'd like to use MQTT in tomcat8 in order to publish messages via MQTT. I use the Paho library version 3.1.1. When I try to instantiate the client using
String clientId = Mqtt.generateClientId();
MqttClient client = new MqttClient("tcp://localhost:1883", clientId);
I get an MqttException(0).
Below you'll find the details of the error message and the stacktrace:
reason 0
msg MqttException
loc MqttException
cause null
excep MqttException (0)
MqttException (0)
at org.eclipse.paho.client.mqttv3.persist.MqttDefaultFilePersistence.open(MqttDefaultFilePersistence.java:80)
at org.eclipse.paho.client.mqttv3.MqttAsyncClient.<init>(MqttAsyncClient.java:304)
at org.eclipse.paho.client.mqttv3.MqttAsyncClient.<init>(MqttAsyncClient.java:185)
at org.eclipse.paho.client.mqttv3.MqttClient.<init>(MqttClient.java:226)
at org.eclipse.paho.client.mqttv3.MqttClient.<init>(MqttClient.java:138)
at de.rz.homeautomation.service.HomeAutomationService.publishSensorData(HomeAutomationService.java:807)
at de.rz.homeautomation.service.HomeAutomationService.postSensorData(HomeAutomationService.java:857)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1081)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1580)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1537)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
The same code works fine in a command line program, so I guess I have to do some kind of configuration of MQTT and/or tomcat. I also don't know if I need additional libraries. I just copied the paho library in the lib-folder of WEB-INF.
Just right now I am only guessing what could be the problem.
Can somebody explain me how I have to configure tomcat or where I have to move the library?
Thanks a lot,
Rudi
The exception is being thrown when the MQTT Client tries to open the persistence file, I will assume it is probably a permisions error for the default location and the user running Tomcat.
You probably need to set the path by passing in a MQTTDefaultFilePersistence object to the MQTTClient() constructor after the client ID or pass in a MemoryPersistence to remove the need to store to disk at all (this will potentially loose messages if Tomcat crashes).
I realize the db is old and this is a long shot...
I've got an older project that has been running on Neo4j 2.1.6 with no issues. Yesterday I started getting [{Neo.DatabaseError.Statement.ExecutionFailure }] errors when I try to make updates to the graph using Cypher via the transaction endpoint.
I finally tracked down the problem to a couple of relationships attached to a couple of nodes that seem to be corrupted/invalid. Trying to delete them on the console just returns 'null' and trying to delete them via the transaction endpoint leads to:
{"code":"Neo.DatabaseError.Statement.ExecutionFailure","message":null,"stackTrace":"java.lang.NullPointerException
org.neo4j.kernel.impl.nioneo.xa.RelationshipDeleter.updateNodesForDeletedRelationship(RelationshipDeleter.java:187)
org.neo4j.kernel.impl.nioneo.xa.RelationshipDeleter.relDelete(RelationshipDeleter.java:67)
org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransactionContext.relationshipDelete(NeoStoreTransactionContext.java:85)
org.neo4j.kernel.impl.nioneo.xa.NeoStoreTransaction.relDelete(NeoStoreTransaction.java:818)
org.neo4j.kernel.impl.persistence.PersistenceManager.relDelete(PersistenceManager.java:140)
org.neo4j.kernel.impl.core.NodeManager.deleteRelationship(NodeManager.java:780)
org.neo4j.kernel.impl.api.state.OldTxStateBridgeImpl.deleteRelationship(OldTxStateBridgeImpl.java:164)
org.neo4j.kernel.impl.api.state.TxStateImpl.relationshipDoDelete(TxStateImpl.java:439)
org.neo4j.kernel.impl.api.StateHandlingStatementOperations$1.visit(StateHandlingStatementOperations.java:138)
org.neo4j.kernel.impl.api.store.CacheLayer.visit(CacheLayer.java:476)
org.neo4j.kernel.impl.api.StateHandlingStatementOperations.relationshipDelete(StateHandlingStatementOperations.java:133)
org.neo4j.kernel.impl.api.ConstraintEnforcingEntityOperations.relationshipDelete(ConstraintEnforcingEntityOperations.java:159)
org.neo4j.kernel.impl.api.LockingStatementOperations.relationshipDelete(LockingStatementOperations.java:231)
org.neo4j.kernel.impl.api.OperationsFacade.relationshipDelete(OperationsFacade.java:528)
org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$RelationshipOperations.delete(TransactionBoundQueryContext.scala:180)
org.neo4j.cypher.internal.spi.v2_1.TransactionBoundQueryContext$RelationshipOperations.delete(TransactionBoundQueryContext.scala:178)
org.neo4j.cypher.internal.compiler.v2_1.spi.DelegatingOperations.delete(DelegatingQueryContext.scala:110)
org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations.org$neo4j$cypher$internal$compiler$v2_1$spi$ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$super$delete(ExceptionTranslatingQueryContext.scala:118)
org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$delete$1.apply$mcV$sp(ExceptionTranslatingQueryContext.scala:118)
org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$delete$1.apply(ExceptionTranslatingQueryContext.scala:118)
org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations$$anonfun$delete$1.apply(ExceptionTranslatingQueryContext.scala:118)
org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext.org$neo4j$cypher$internal$compiler$v2_1$spi$ExceptionTranslatingQueryContext$$translateException(ExceptionTranslatingQueryContext.scala:152)
org.neo4j.cypher.internal.compiler.v2_1.spi.ExceptionTranslatingQueryContext$ExceptionTranslatingOperations.delete(ExceptionTranslatingQueryContext.scala:118)
org.neo4j.cypher.internal.compiler.v2_1.spi.UpdateCountingQueryContext$CountingOps.delete(UpdateCountingQueryContext.scala:120)
org.neo4j.cypher.internal.compiler.v2_1.mutation.DeleteEntityAction.org$neo4j$cypher$internal$compiler$v2_1$mutation$DeleteEntityAction$$delete(DeleteEntityAction.scala:51)
org.neo4j.cypher.internal.compiler.v2_1.mutation.DeleteEntityAction.exec(DeleteEntityAction.scala:37)
org.neo4j.cypher.internal.compiler.v2_1.pipes.ExecuteUpdateCommandsPipe.org$neo4j$cypher$internal$compiler$v2_1$pipes$ExecuteUpdateCommandsPipe$$exec(ExecuteUpdateCommandsPipe.scala:57)
org.neo4j.cypher.internal.compiler.v2_1.pipes.ExecuteUpdateCommandsPi$$$$1019fdff8b266d7d9d5647386930b3d8$$$$ands$1$$anonfun$apply$2.apply(ExecuteUpdateCommandsPipe.scala:46)
org.neo4j.cypher.internal.compiler.v2_1.pipes.ExecuteUpdateCommandsPi$$$$1019fdff8b266d7d9d5647386930b3d8$$$$ands$1$$anonfun$apply$2.apply(ExecuteUpdateCommandsPipe.scala:46)
scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
org.neo4j.cypher.internal.compiler.v2_1.pipes.EmptyResultPipe.internalCreateResults(EmptyResultPipe.scala:29)
org.neo4j.cypher.internal.compiler.v2_1.pipes.PipeWithSource.createResults(Pipe.scala:105)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1$$anonfun$apply$2.apply(ExecutionPlanBuilder.scala:120)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1$$anonfun$apply$2.apply(ExecutionPlanBuilder.scala:119)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionWorkflowBuilder.runWithQueryState(ExecutionPlanBuilder.scala:168)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1.apply(ExecutionPlanBuilder.scala:118)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anonfun$getExecutionPlanFunction$1.apply(ExecutionPlanBuilder.scala:103)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anon$1.execute(ExecutionPlanBuilder.scala:68)
org.neo4j.cypher.internal.compiler.v2_1.executionplan.ExecutionPlanBuilder$$anon$1.execute(ExecutionPlanBuilder.scala:67)
org.neo4j.cypher.internal.ExecutionPlanWrapperForV2_1.execute(CypherCompiler.scala:159)
org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:76)
org.neo4j.cypher.ExecutionEngine.execute(ExecutionEngine.scala:71)
org.neo4j.cypher.javacompat.ExecutionEngine.execute(ExecutionEngine.java:84)
org.neo4j.server.rest.transactional.TransactionHandle.executeStatements(TransactionHandle.java:277)
org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:139)
org.neo4j.server.rest.web.TransactionalService$2.write(TransactionalService.java:194)
com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1506)
org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
ch.qos.logback.access.servlet.TeeFilter.doFilter(TeeFilter.java:55)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
org.neo4j.server.guard.GuardingRequestFilter.doFilter(GuardingRequestFilter.java:68)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:211)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
org.eclipse.jetty.server.Server.handle(Server.java:445)
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
java.lang.Thread.run(Thread.java:745)\n"
Is there anyway to nuke these relationships? Looks like the first one was corrupted two days ago and I'd prefer not to lose two days of data if at all possible.
Just in case anybody else finds their way here, Michael Hunger wrote a tool that will copy a database and skip anything that is broken which is exactly what I needed. You can find it at https://github.com/jexp/store-utils/tree/21.
Turns out all of my issues were due to one relationship attached to a NULL node... Ran the tool, copied over the fixed up database and everything is working perfectly again.
Are there additional steps you need to take after joining new Riak nodes to a cluster to set up yokozuna/solr?
Solr was working on one node. When I joined two additional nodes solr seems to have stopped. Solr is not running any of the nodes, including on the original node that was working before the cluster was created.
Update: I had the original node leave the cluster and restarted it to test it on it's own. It is still throwing the same error.
Update: contents of log/riak/solr.log
2014-11-05 19:34:13,581 [WARN] #CoreAdminHandler.java:495 Creating a core with existing name is not allowed
2014-11-05 19:34:13,582 [ERROR] #SolrException.java:109 org.apache.solr.common.SolrException: Core with name 'normalized_people' already exists.
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:496)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:152)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:732)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:268)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
2014-11-05 19:34:13,582 [ERROR] #SolrException.java:120 null:org.apache.solr.common.SolrException: Core with name 'normalized_people' already exists.
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:496)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:152)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:732)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:268)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
No additional steps are necessary. Please check the log files, especially /var/log/riak/solr.log for clues as to what the error could be.
Based on your description, reproduction steps should be:
Start one Riak node with search enabled and create a search index (normalized_people)
Start other Riak nodes with search enabled and join to cluster
Check for error
I finished just to configure some buildjobs. But when i clicked apply / save (doesn't matter, because the same error appears) the following error shows up. Changes are saved somehow and the configuration page for this jobs loads frustrating long afterwards. What could be the reason and how to fix it?
ERROR Message
HTTP Status 500 - An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
type Exception report
message An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
description The server encountered an internal error that prevented it from fulfilling this request.
exception
org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
org.apache.coyote.http11.AbstractOutputBuffer.checkLengthBeforeWrite(AbstractOutputBuffer.java:546)
org.apache.coyote.http11.AbstractOutputBuffer.write(AbstractOutputBuffer.java:510)
org.apache.coyote.http11.AbstractOutputBuffer.write(AbstractOutputBuffer.java:427)
org.apache.coyote.http11.AbstractOutputBuffer.sendHeader(AbstractOutputBuffer.java:393)
org.apache.coyote.http11.AbstractHttp11Processor.prepareResponse(AbstractHttp11Processor.java:1495)
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:767)
org.apache.coyote.Response.action(Response.java:174)
org.apache.coyote.Response.sendHeaders(Response.java:365)
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:345)
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:303)
org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:108)
com.jcraft.jzlib.DeflaterOutputStream.close(DeflaterOutputStream.java:122)
org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36)
net.bull.javamelody.FilterServletOutputStream.close(FilterServletOutputStream.java:45)
java.io.FilterOutputStream.close(Unknown Source)
sun.nio.cs.StreamEncoder.implClose(Unknown Source)
sun.nio.cs.StreamEncoder.close(Unknown Source)
java.io.OutputStreamWriter.close(Unknown Source)
java.io.BufferedWriter.close(Unknown Source)
org.dom4j.io.XMLWriter.close(XMLWriter.java:286)
org.kohsuke.stapler.jelly.HTMLWriterOutput.close(HTMLWriterOutput.java:70)
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:56)
org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
org.kohsuke.stapler.ResponseImpl.forward(ResponseImpl.java:121)
hudson.model.AbstractModelObject.sendError(AbstractModelObject.java:70)
hudson.model.Job.doConfigSubmit(Job.java:1167)
hudson.model.AbstractProject.doConfigSubmit(AbstractProject.java:786)
sun.reflect.GeneratedMethodAccessor1531.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
org.kohsuke.stapler.S
please read the comments to my question above.
"Just updated all plugins -> it did not help. BUT I think i found the reason: i have uninstalled the Plugin „Publish over SSH“ and it helped for now. System update will be done soon too. Thank you for your help."
-> The deinstallation solved the problem not. The server went a little bit faster afterwards, but not as much as expected and hoped.
Now i have updated to the newest version of Jenkins and hope that it will fix my issues.
We have two DSE nodes (3.2.4) for development purpose. Replication factor for a Cassandra keyspace is set to 2. When performing Solr delete by query without specifying consistency level (i.e.: quorum), occasionally we got the following exception when deleting a lot of rows (about 500K or so). With consistency level set to one, the same exception also observed (when deleting about 1 million rows). I wonder if there is a work around for this. The exception is as follows:
WARN [http-8983-1] 2014-03-27 09:21:59,008 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 1 out of 4 with CL ONE...
WARN [http-8983-1] 2014-03-27 09:22:09,109 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 2 out of 4 with CL ONE...
WARN [http-8983-1] 2014-03-27 09:22:19,211 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 3 out of 4 with CL ONE...
WARN [http-8983-1] 2014-03-27 09:22:29,312 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 4 out of 4 with CL ONE...
ERROR [http-8983-1] 2014-03-27 09:22:29,415 SolrException.java (line 124) org.apache.solr.common.SolrException: java.lang.RuntimeException: Insert command failed after 4 attempts, source exception follows.
at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.deleteByQuery(CassandraDirectUpdateHandler.java:236)
at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:80)
at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:899)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:757)
at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1841)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:669)
at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.execute(CassandraDispatchFilter.java:165)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:137)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:194)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:94)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Insert command failed after 4 attempts, source exception follows.
at com.datastax.bdp.util.CassandraUtil.robustInsert(CassandraUtil.java:56)
at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.doDeletes(ThriftCassandraRowWriter.java:189)
at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.access$100(ThriftCassandraRowWriter.java:39)
at com.datastax.bdp.search.solr.ThriftCassandraRowWriter$DeletingHitCollector.collect(ThriftCassandraRowWriter.java:241)
at org.apache.lucene.search.Scorer.score(Scorer.java:65)
at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:245)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.deleteByQuery(ThriftCassandraRowWriter.java:168)
at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.deleteByQuery(CassandraDirectUpdateHandler.java:217)
... 38 more
Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:91)
at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:198)
at com.datastax.bdp.util.CassandraUtil.robustInsert(CassandraUtil.java:39)
... 47 more
Thanks for reporting this issue. We're taking a look at it and considering improvements.
For now, all I can recommend is to retry the operation if it fails, with a slight delay since the problem may simply be that the cluster is busy.
You might also want to consider smaller deletes, using narrower queries that select fewer documents/rows.
I'll check into whether you can reasonably tune the timeout to handle cluster latency issues.
EDIT: You can raise the timeout with write_request_timeout_in_ms, in cassandra.yaml