flume : find ip/hostname of event sender? - flume

Question

I am trying to setup data pipeline where applications servers send (using log4j logging) logevents to flume (using flume log4j appender) over network , to a avrosource that flume agent is using
I tried with below configration but It only appends IP of the host on which agent is running
A1.sources.logE.interceptors = hostint
A1.sources.logE.interceptors.hostint.type = org.apache.flume.interceptor.HostInterceptor$Builder
A1.sources.logE.interceptors.hostint.preserveExisting = true
A11.sources.logE.interceptors.hostint.useIP = false

Answer 1

This is actually quite easy if your using AvroSource, which is actually an Avro RPC implementation. As you are creating Flume Events on the client, just make sure that you add custom headers to you Events which contain the values you want to use, e.g. the sender IP. Do not use the interceptors, because those overwrite the host address in the headers with that of the flume agent executing the interceptor. You can access the values of your header in flume config by using the following syntax %{header_key}

Answer 2

For anyone who has similar problem , I ended up removing log4j-flume appender at application side , replaced it with log4j-syslog appender.
on flume side I configured syslogudp source.
so finally it looks something like,
at application side log4j-config
log4j.appender.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.syslog.Facility=LOCAL7
log4j.appender.syslog.FacilityPrinting=false
log4j.appender.syslog.Header=true
log4j.appender.syslog.SyslogHost=flume1.host.net:41473
log4j.appender.syslog.layout=org.apache.log4j.PatternLayout
log4j.appender.syslog.layout.ConversionPattern= DUMMY %p: (%F:%L) %x %m %n
at flume agent , syslogudp source
A1.sources.logMR.type = syslogudp
A1.sources.logMR.bind = 0.0.0.0
A1.sources.logMR.port = 41473
so now my flume event looks like,
2015-03-17 08:43:02,658 INFO org.apache.flume.sink.LoggerSink: Event: { headers:{timestamp=1426682579754, Severity=7, host=apphost1.host.net, Facility=23, log_type=maplogs} body: 54 4F 46 4C
55 4D 45 20 5B 6D 61 69 6E 2C 5D 20 TOFLUME [main,] }
and I can now finally acces host key in header with %{host}