Flume agent - using tail -F - flume

Question

I am new to Apache Flume.
I have created my agent like:
agent.sources=exec-source
agent.sinks=hdfs-sink
agent.channels=ch1
agent.sources.exec-source.type=exec
agent.sources.exec-source.command=tail -F /var/log/apache2/access.log
agent.sinks.hdfs-sink.type=hdfs
agent.sinks.hdfs-sink.hdfs.path=hdfs://<Host-Name of name node>/
agent.sinks.hdfs-sink.hdfs.filePrefix=apacheaccess
agent.channels.ch1.type=memory
agent.channels.ch1.capacity=1000
agent.sources.exec-source.channels=ch1
agent.sinks.hdfs-sink.channel=ch1
And the ouput i am getting is :
13/01/22 17:31:48 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
13/01/22 17:31:48 INFO node.FlumeNode: Flume node starting - agent
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
13/01/22 17:31:48 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 9
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:conf/flume_exec.conf
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: agent
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent]
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: Creating channels
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: created channel ch1
13/01/22 17:31:48 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: hdfs
13/01/22 17:31:48 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{exec-source=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor#715d44 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} }
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: ch1, registered successfully.
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink hdfs-sink
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting Source exec-source
13/01/22 17:31:48 INFO source.ExecSource: Exec source starting with command:tail -F /var/log/apache2/access.log
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: hdfs-sink, registered successfully.
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink started
But it's not writing logs to HDFS.
When I run cat /var/log/apache2/access.log instead of tail –F /var/log/apache2/access.log it runs and my files are created on HDFS.
Please help me out.

Answer 1

"tail -F" by default prints only last 10 lines of file at start. It seems that 10 lines is not enough to fill HDFS block, so you don't see anything written by Flume.
You can:
Try "tail -n $X -F" to print last X lines at start (value of X will vary depending on size of block in your HDFS setup)
Wait until access.log will grow big enough while Flume is running (again, time to wait will depend on size of block and rate of access.log growing; in production mode it will be fast enough, I think)
Add follwing lines to your flume.conf. It will force Flume to roll new file every 10 seconds regardless of size of written data (assuming it is not zero):
agent.sinks.hdfs-sink.hdfs.rollInterval = 10
agent.sinks.hdfs_sink.hdfs.rollSize = 0