So I'm trying to deploy 2 instances of Janusgraph (v 0.3.1) to insert data in the same keyspace of ScyllaDb backend. To do that i deploy 2 janusgraph containers with docker. The first one is starting without errors and creates the keyspace in my ScyllaDb but the second one displays some errors.
So my Janusgraph containers works on a cluster while my backend works on another cluster. Actually, I tried with only one container to restart it when my keyspace Scylla is already created and I have the same problem.
I also found a script clean.groovy that allowed to force shutdown the graph instances opened. But nothing works...
5316 [main] INFO org.janusgraph.diskstorage.Backend - Initiated backend operations thread pool of size 8
5648 [main] WARN org.apache.tinkerpop.gremlin.server.GremlinServer - Graph [graph] configured at [/janusgraph-config/janusgraph.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.lambda$new$0(DefaultGraphManager.java:57)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:120)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:84)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:343)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 13 more
Caused by: org.janusgraph.core.JanusGraphException: A JanusGraph graph with the same instance id [0a0a01657-node11] is already open. Might required forced shutdown.
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:165)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)
... 18 more
5651 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
5800 [main] INFO org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Initialized GremlinExecutor and preparing GremlinScriptEngines instances.
7837 [gremlin-server-exec-1] ERROR org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager - Could not create GremlinScriptEngine for gremlin-groovy
java.lang.IllegalStateException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.lambda$createGremlinScriptEngine$16(DefaultGremlinScriptEngineManager.java:464)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.createGremlinScriptEngine(DefaultGremlinScriptEngineManager.java:450)
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.getEngineByName(DefaultGremlinScriptEngineManager.java:219)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.lambda$getEngineByName$0(CachedGremlinScriptEngineManager.java:57)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.getEngineByName(CachedGremlinScriptEngineManager.java:57)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:263)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:397)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager.lambda$createGremlinScriptEngine$16(DefaultGremlinScriptEngineManager.java:460)
... 24 more
Caused by: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:713)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:395)
... 26 more
Caused by: groovy.lang.MissingPropertyException: No such property: graph for class: Script1
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:66)
at org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:310)
at Script1.run(Script1.groovy:16)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:690)
... 27 more
7841 [main] WARN org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Could not initialize gremlin-groovy GremlinScriptEngine as init script could not be evaluated
java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: gremlin-groovy is not an available GremlinScriptEngine
at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.lambda$new$4(ServerGremlinExecutor.java:141)
at java.util.LinkedHashMap$LinkedKeySet.forEach(LinkedHashMap.java:559)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:136)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:120)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:84)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:343)
Caused by: java.lang.IllegalArgumentException: gremlin-groovy is not an available GremlinScriptEngine
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.registerLookUpInfo(CachedGremlinScriptEngineManager.java:95)
at org.apache.tinkerpop.gremlin.jsr223.CachedGremlinScriptEngineManager.getEngineByName(CachedGremlinScriptEngineManager.java:58)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:263)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I think all those errors are created by this :
Caused by: org.janusgraph.core.JanusGraphException: A JanusGraph graph with the same instance id [0a0a01657-node11] is already open. Might required forced shutdown.
And this is the script clean.groovy
graph = JanusGraphFactory.open('janusgraph.properties')
mgmt = graph.openManagement()
instances = mgmt.getOpenInstances()
instances.iterator().findAll {
!it.contains('current')
}.each {
mgmt.forceCloseInstance(it)
}
mgmt.commit()
So did anybody solved this problem of graph ?
Thanks in advance.
I finally got answer ! So you can use ConfiguredGraphFactory but it failed for me. So for the first container i started it normally but the second one i added another configuration in the janusgraph.properties file :
graph.replace-instance-if-exists=true`
And it works perfectly !
I am not sure, if this is the correct community to ask Jenkins questions - if not please feel free to move the question.
I am trying to add my local machine as a local slave to an existing job. My problem is that I cannot make the job run, as I get a remote call fails exception. As EnvInject tells me that two variables are missing just the line above I though there maybe is a connection there. But when I look into the environment variables the variables are set correctly and I have read and write access there. So what else could be the cause of the failed remote call? The Stacktrace doesn't tell me much:
C:\Temp\test_slave>exit 0
[EnvInject] - Script executed successfully.
[EnvInject] - Injecting contributions.
Building remotely on test_slave (win_test) in workspace C:\Temp\test_slave\workspace\<job>
[EnvInject] - Unset unresolved 'HOMESHARE' variable.
[EnvInject] - Unset unresolved 'pkidata' variable.
RTC : checkout...
FATAL: RTC : checkout failure: remote file operation failed: C:\Temp\test_slave\workspace\<job> at hudson.remoting.Channel#1453477:test_slave: java.io.IOException: Remote call on test_slave failed
java.io.IOException: remote file operation failed: C:\Temp\test_slave\workspace\<job> at hudson.remoting.Channel#1453477:test_slave: java.io.IOException: Remote call on test_slave failed
at hudson.FilePath.act(FilePath.java:986)
at hudson.FilePath.act(FilePath.java:968)
at com.ibm.team.build.internal.hjplugin.RTCScm.checkout(RTCScm.java:1606)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: java.io.IOException: Remote call on test_slave failed
at hudson.remoting.Channel.call(Channel.java:789)
at hudson.FilePath.act(FilePath.java:979)
... 11 more
Caused by: java.lang.NoClassDefFoundError: org/eclipse/core/runtime/OperationCanceledException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at com.ibm.team.build.internal.hjplugin.RTCFacadeFactory.newFacade(RTCFacadeFactory.java:240)
at com.ibm.team.build.internal.hjplugin.RTCFacadeFactory.getFacade(RTCFacadeFactory.java:71)
at com.ibm.team.build.internal.hjplugin.RTCBuildResultSetupTask.invoke(RTCBuildResultSetupTask.java:120)
at com.ibm.team.build.internal.hjplugin.RTCBuildResultSetupTask.invoke(RTCBuildResultSetupTask.java:38)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:62)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to test_slave(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
... 12 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.core.runtime.OperationCanceledException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at com.ibm.team.build.internal.hjplugin.RTCFacadeClassLoader.loadClass(RTCFacadeClassLoader.java:81)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at com.ibm.team.build.internal.hjplugin.RTCFacadeFactory.newFacade(RTCFacadeFactory.java:240)
at com.ibm.team.build.internal.hjplugin.RTCFacadeFactory.getFacade(RTCFacadeFactory.java:71)
at com.ibm.team.build.internal.hjplugin.RTCBuildResultSetupTask.invoke(RTCBuildResultSetupTask.java:120)
at com.ibm.team.build.internal.hjplugin.RTCBuildResultSetupTask.invoke(RTCBuildResultSetupTask.java:38)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:62)
at java.lang.Thread.run(Thread.java:745)
ERROR: RTC : checkout failure: remote file operation failed: C:\Temp\test_slave\workspace\GETRAG\GG-ECMot_AP_0160A_DEV_IFX~SlaveBuild at hudson.remoting.Channel#1453477:test_slave: java.io.IOException: Remote call on test_slave failed
Performing Post build task...
Could not match :Code generation phase : False
Logical operation result is FALSE
Additional information:
The variable EnvInject says are missing are set like the following:
HOMESHARE \\<server>\<user>$
[...]
pkidata \\<server>\<user>$
From the Slave side everything seems to be connected just fine. This is the commandline read-out.
Nov 23, 2016 10:49:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://<address>.com:8080/]
Nov 23, 2016 10:49:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Nov 23, 2016 10:49:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <address>.com:49599
Nov 23, 2016 10:49:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Nov 23, 2016 10:49:29 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
I have a hazelcast server and client. The client successfully connects to its server. But due to some problem when the server needs to restarts, the client is not able to connect to the server.
My configuration on client side is
<network>
<cluster-members>
<address>127.0.0.1</address>
</cluster-members>
<smart-routing>true</smart-routing>
<redo-operation>true</redo-operation>
<connection-timeout>100000</connection-timeout>
<connection-attempt-limit>10</connection-attempt-limit>
<connection-pool-size>30</connection-pool-size>
</network>
And the error I get in the client is
Hazelcast instance is not active!. Stacktrace follows:
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is no
t active!
at com.hazelcast.client.HazelcastClientProxy.getClient(HazelcastClientPr
oxy.java:245)
at com.hazelcast.client.HazelcastClientProxy.getMap(HazelcastClientProxy
.java:95)
at com.hazelcast.web.WebFilter.getClusterMap(WebFilter.java:302)
at com.hazelcast.web.WebFilter.loadHazelcastSession(WebFilter.java:252)
at com.hazelcast.web.WebFilter.createNewSession(WebFilter.java:246)
at com.hazelcast.web.WebFilter.access$1100(WebFilter.java:59)
at com.hazelcast.web.WebFilter$RequestWrapper.getSession(WebFilter.java:
403)
at com.hazelcast.web.WebFilter$RequestWrapper.getSession(WebFilter.java:
314)
at com.k12report.security.AuthController$_closure4.doCall(AuthController
.groovy:40)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(Pag
eFragmentCachingFilter.java:198)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter
.java:63)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:694)
at com.hazelcast.web.WebFilter$RequestWrapper$1.forward(WebFilter.java:3
42)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:704)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:745)
[2014-07-10 21:22:18,929] - errors.GrailsExceptionResolver Unable to render erro
rs view: Hazelcast instance is not active!
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is no
t active!
at com.hazelcast.client.HazelcastClientProxy.getClient(HazelcastClientPr
oxy.java:245)
at com.hazelcast.client.HazelcastClientProxy.getMap(HazelcastClientProxy
.java:95)
at com.hazelcast.web.WebFilter.getClusterMap(WebFilter.java:302)
at com.hazelcast.web.WebFilter.access$1000(WebFilter.java:59)
at com.hazelcast.web.WebFilter$HazelcastHttpSession.getAttribute(WebFilt
er.java:448)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(Pag
eFragmentCachingFilter.java:198)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter
.java:63)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:694)
at com.hazelcast.web.WebFilter$RequestWrapper$1.forward(WebFilter.java:3
42)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:704)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:745)
[2014-07-10 21:22:19,130] - [/frontend].[grails] Servlet.service() for servlet g
rails threw exception
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is no
t active!
at com.hazelcast.client.HazelcastClientProxy.getClient(HazelcastClientPr
oxy.java:245)
at com.hazelcast.client.HazelcastClientProxy.getMap(HazelcastClientProxy
.java:95)
at com.hazelcast.web.WebFilter.getClusterMap(WebFilter.java:302)
at com.hazelcast.web.WebFilter.access$1000(WebFilter.java:59)
at com.hazelcast.web.WebFilter$HazelcastHttpSession.getAttribute(WebFilt
er.java:448)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(Pag
eFragmentCachingFilter.java:198)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter
.java:63)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:694)
at com.hazelcast.web.WebFilter$RequestWrapper$1.forward(WebFilter.java:3
42)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:704)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:745)
[2014-07-10 21:22:19,371] - [/frontend].[default] Servlet.service() for servlet
[default] in context with path [/frontend] threw exception [org.springframework.
web.util.NestedServletException: Request processing failed; nested exception is
org.codehaus.groovy.grails.exceptions.GrailsRuntimeException: com.hazelcast.core
.HazelcastInstanceNotActiveException: Hazelcast instance is not active!] with ro
ot cause
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is no
t active!
at com.hazelcast.client.HazelcastClientProxy.getClient(HazelcastClientPr
oxy.java:245)
at com.hazelcast.client.HazelcastClientProxy.getMap(HazelcastClientProxy
.java:95)
at com.hazelcast.web.WebFilter.getClusterMap(WebFilter.java:302)
at com.hazelcast.web.WebFilter.access$1000(WebFilter.java:59)
at com.hazelcast.web.WebFilter$HazelcastHttpSession.getAttribute(WebFilt
er.java:448)
at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(Pag
eFragmentCachingFilter.java:198)
at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter
.java:63)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:694)
at com.hazelcast.web.WebFilter$RequestWrapper$1.forward(WebFilter.java:3
42)
at com.hazelcast.web.WebFilter.doFilter(WebFilter.java:704)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:745)
And there is no any error on the server side.
Thanks
I'm trying to use the flume connectors for storm with a flume instance which is getting syslog messages.
my storm config is the following
flumeConf.put("flume-agent.source.type", "AVRO");
flumeConf.put("flume-agent.source.bind", "localhost");
flumeConf.put("flume-agent.source.port", "3564");
flumeConf.put("flume-agent.channel.type", "MEMORY");
flumeConf.put("flume-agent.channel.capacity", "2000");
my flume config is the following
mainAgent.sinks = consoleSink
mainAgent.sources = syslog-source
mainAgent.channels = mainChannel
mainAgent.sinks = stormSink
mainAgent.sinks.consoleSink.channel = mainChannel
mainAgent.sources.syslog-source.channels = mainChannel
mainAgent.sinks.stormSink.channel = mainChannel
# the source
mainAgent.sources.syslog-source.type = syslogudp
mainAgent.sources.syslog-source.bind = localhost
mainAgent.sources.syslog-source.port = 3216
mainAgent.channels.mainChannel.type = memory
mainAgent.channels.mainChannel.capacity = 2000
mainAgent.channels.mainChannel.transactionCapacity = 200
mainAgent.sinks.consoleSink.type = logger
# avro sink properties
mainAgent.sources.stormSink.type = avro
mainAgent.sources.stormSink.hostname = #my-storm-server.localdomain
mainAgent.sources.stormSink.port = 3564
When I try to run that I'm getting this:
[WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:697)] Could not configure sink stormSink due to: Component has no type. Cannot configure. stormSink
org.apache.flume.conf.ConfigurationException: Component has no type. Cannot configure. stormSink
at org.apache.flume.conf.ComponentConfiguration.configure(ComponentConfiguration.java:76)
at org.apache.flume.conf.sink.SinkConfiguration.configure(SinkConfiguration.java:44)
at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:680)
at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:346)
at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:212)
at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:126)
at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:108)
at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:193)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:94)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
So I'm obviously missing something, I tried changing the lines of avro sink properties to
# avro sink properties
mainAgent.sinks.stormSink.type = avro
mainAgent.sinks.stormSink.hostname = #my-storm-server-ip
mainAgent.sinks.stormSink.port = 3564
and I'm getting
[INFO - org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:126)] Attempting to create Avro Rpc client.
2013-11-25 15:12:47,550 (Old I/O datagram worker ([id: 0x50b279f3, 0.0.0.0/0.0.0.0:3216])) [ERROR - org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:80)] Error writting to channel
org.apache.flume.ChannelException: Unable to put event on required channel: org.apache.flume.channel.MemoryChannel{name: mainChannel}
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:275)
at org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:76)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:95)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:54)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:70)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:267)
... 12 more
2013-11-25 15:12:50,554 (Old I/O datagram worker ([id: 0x50b279f3, 0.0.0.0/0.0.0.0:3216])) [ERROR - org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:80)] Error writting to channel
org.apache.flume.ChannelException: Unable to put event on required channel: org.apache.flume.channel.MemoryChannel{name: mainChannel}
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:275)
at org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:76)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:95)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:54)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:70)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:267)
... 12 more
2013-11-25 15:12:53,555 (Old I/O datagram worker ([id: 0x50b279f3, 0.0.0.0/0.0.0.0:3216])) [ERROR - org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:80)] Error writting to channel
org.apache.flume.ChannelException: Unable to put event on required channel: org.apache.flume.channel.MemoryChannel{name: mainChannel}
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:275)
at org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:76)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:95)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:54)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:70)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:267)
... 12 more
2013-11-25 15:12:56,557 (Old I/O datagram worker ([id: 0x50b279f3, 0.0.0.0/0.0.0.0:3216])) [ERROR - org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:80)] Error writting to channel
org.apache.flume.ChannelException: Unable to put event on required channel: org.apache.flume.channel.MemoryChannel{name: mainChannel}
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:275)
at org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:76)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:95)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:54)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:70)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:267)
... 12 more
^C2013-11-25 15:12:57,444 (agent-shutdown-hook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:79)] Stopping lifecycle supervisor 10
2013-11-25 15:12:59,559 (Old I/O datagram worker ([id: 0x50b279f3, 0.0.0.0/0.0.0.0:3216])) [ERROR - org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:80)] Error writting to channel
org.apache.flume.ChannelException: Unable to put event on required channel: org.apache.flume.channel.MemoryChannel{name: mainChannel}
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:275)
at org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:76)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:95)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:54)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:70)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:267)
... 12 more
2013-11-25 15:13:01,262 (lifecycleSupervisor-1-1) [WARN - org.apache.flume.sink.AbstractRpcSink.start(AbstractRpcSink.java:291)] Unable to create Rpc client using hostname: <ip>, port: 3564
org.apache.flume.FlumeException: NettyAvroRpcClient { host: <ip>, port: 3564 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:161)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:115)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:590)
at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88)
at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
at org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:209)
at org.apache.flume.sink.AbstractRpcSink.start(AbstractRpcSink.java:289)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Error connecting to /<ip>:3564
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:147)
... 16 more
2013-11-25 15:13:01,263 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.sink.AbstractRpcSink.start(AbstractRpcSink.java:300)] Rpc sink stormSink started.
2013-11-25 15:13:01,265 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:205)] Rpc sink stormSink: Building RpcClient with hostname: <ip>, port: 3564
2013-11-25 15:13:01,265 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:126)] Attempting to create Avro Rpc client.
2013-11-25 15:13:01,271 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: <ip>, port: 3564 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:161)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:115)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:590)
at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88)
at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
at org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:209)
at org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:269)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:339)
... 3 more
Caused by: java.io.IOException: Error connecting to /<ip>:3564
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:147)
... 10 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:677)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:134)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:97)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:55)
at org.jboss.netty.channel.Channels.connect(Channels.java:642)
at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:253)
... 13 more
Storm is running and using tcpdump I can't see any flume-related traffic on the storm-server
Could you provide some info on what am I missing?
I changed my storm config to:
flumeConf.put("flume-agent.source.type", "AVRO");
flumeConf.put("flume-agent.source.bind", "the ip of my storm server");
flumeConf.put("flume-agent.source.port", "3564");
flumeConf.put("flume-agent.channel.type", "MEMORY");
flumeConf.put("flume-agent.channel.capacity", "2000");
and the flume config to
# Name the components on this agent
mainAgent.sinks = consoleSink
mainAgent.sources = syslog-source
mainAgent.channels = mainChannel
mainAgent.sinks = stormSink
# Bind the source and sink to the channel
mainAgent.sinks.consoleSink.channel = mainChannel
mainAgent.sources.syslog-source.channels = mainChannel
mainAgent.sinks.stormSink.channel = mainChannel
# Describe/configure the source
mainAgent.sources.syslog-source.type = syslogudp
mainAgent.sources.syslog-source.bind = localhost
mainAgent.sources.syslog-source.port = 3216
# Use a channel which buffers events in memory
mainAgent.channels.mainChannel.type = memory
mainAgent.channels.mainChannel.capacity = 2000
mainAgent.channels.mainChannel.transactionCapacity = 200
# Describe the sink
mainAgent.sinks.consoleSink.type = logger
# avro sink properties
mainAgent.sinks.stormSink.type = avro
mainAgent.sinks.stormSink.hostname = the ip of my storm server
mainAgent.sinks.stormSink.port = 3564
and it works
Please Have a look at the following error:
Flume Code:
local1.channels.mem-ch-1.type = memory
local1.sources.exc-src-1.channels = mem-ch-1
local1.sources.exc-src-1.type = exec
local1.sources.exc-src-1.command =tail -F /u01/accts/a.ntimbadi/flumtest.txt
local1.sinks.avro-snk-1.channel = mem-ch-1
local1.sinks.avro-snk-1.type = avro
local1.sinks.avro-snk-1.hostname = localhost
local1.sinks.avro-snk-1.port = 4141
local1.sources = exc-src-1
local1.sinks = avro-snk-1
local1.channels = mem-ch-1
Error Log:
org.apache.flume.EventDeliveryException: Failed to send events
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 4141 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:145)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:101)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:564)
at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88)
at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
at org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:209)
at org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:269)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:339)
... 3 more
Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:4141
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:131)
... 10 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
Let me know if I am doing something wrong. I am actually trying to create an avro sink here and then redirect it with other agent(whose code is not here) to HDFS. I seem to have error connecting to the port at localhost. Please suggest some solution.
Your configuration is to read events from the exec source and send them on using Avro on localhost:4141.
However, the error log indicates that you do not have an Avro source running at localhost:4141.
You'll be needing an Avro source (maybe another agent) listening on that port.