Capturing networking traffic using Tshark and Flume - flume

Question

Hello everyone i'm trying to capture a network traffic using tshark and i'm using apache flume to send those results to spark .
Problem is when i use the exec source in flume's configuration flume doesn't work it stops instantly right after starting
My configuration :
agent.sources = tsharkSource
agent.channels = memoryChannel
agentasinks = avroSink
# Configuring the sources to pull the bashes from Tshark
agent.sources.tsharkSource.type = exec
agent.sources.tsharkSource.command = tshark -T json
agent.sources.tsharkSource.channels = memoryChannel
# Configuring the sink to push logs to spark change hostname to 116's ip adress
agent.sinks.avroSink.type = avro
agent.sinks.avroSink.channel = memoryChannel
agent.sinks.avroSink.hostname = 192.168.1.112
agent.sinks.avroSink.port= 6969
# Configuring the memory channel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 1000
agent.channels.memoryChannel.transactionCapacity = 100
the shell output :
flume-ng agent -f conf/flume-conf.properties -n agent
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hive libraries found via () for Hive access
+ exec /usr/lib/jum/jaua-1.11.0-openjdk-amd64//bin/jaua -Xmx20m -cp
1/home/oshiflume/flume/apache-flume-1.8.0-bin/lib/*:/libfle -
Djava.library.path= org.apache.flume.node.Application -f conf/flume-
conf.properties -n agent
log4j:WARN No appenders could be found for logger (org.apache.flume.node.Application).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.htmlDnoconfig for more info.

Answer 1

Your execution command is
flume-ng agent -f conf/flume-conf.properties -n agent
I see two mistakes here. First, you must specify the configuration directory with -c conf and generally flume configuration files are named as some-config.conf
The warning on the console is No configuration directory set! Use --conf, -c and --conf are the same thing.
You may want to rename your configuration file from flume-conf.properties to flume.conf
As a solution you can try this command:
flume-ng agent -c conf -f conf/flume.conf -n agent
If you want to display the logs after execution use this command
flume-ng agent -c conf -f conf/flume.conf -n agent -Dflume.root.logger=INFO,console
To display the logs log4j.properties must be in your conf directory as conf/log4j.properties.
My properties are below:
flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume.log
log4j.logger.org.apache.flume.lifecycle = INFO
log4j.logger.org.jboss = WARN
log4j.logger.org.mortbay = INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN
log4j.logger.org.apache.hadoop = INFO
log4j.logger.org.apache.hadoop.hive = ERROR
# Define the root logger to the system property "flume.root.logger".
log4j.rootLogger=${flume.root.logger}
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}
log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
log4j.appender.DAILY.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n