apache flume log directory - flume

Just begin to learn Apache Flume. I follow the instructions on Flume official getting started website:
https://cwiki.apache.org/confluence/display/FLUME/Getting+Started
Almost everything is fine after follow the instructions on above link. But I could not find any log file afterwards. I suppose log file is under {flume.directory}/logs. Any idea to find flume log files?
Here comes my log4j.properties:
flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume.log
log4j.logger.org.apache.flume.lifecycle = INFO
log4j.logger.org.jboss = WARN
log4j.logger.org.mortbay = INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN
log4j.logger.org.apache.hadoop = INFO
log4j.logger.org.apache.hadoop.hive = ERROR
# Define the root logger to the system property "flume.root.logger".
log4j.rootLogger=${flume.root.logger}
# Stock log4j rolling file appender
# Default log rotation configuration
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
# Warning: If you enable the following appender it will fill up your disk if you don't have a cleanup job!
# This uses the updated rolling file appender from log4j-extras that supports a reliable time-based rolling policy.
# See http://logging.apache.org/log4j/companions/extras/apidocs/org/apache/log4j/rolling/TimeBasedRollingPolicy.html
# Add "DAILY" to flume.root.logger above if you want to use this
log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}
log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
log4j.appender.DAILY.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
# console
# Add "console" to flume.root.logger above if you want to use this
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n

Cause your configuration use a relative path :
flume.log.dir=./logs
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
Flume use your current directory to log.
(the directory where you are when you launch flume)
You can use an absolute path for flume.log.dir if you want to force flume to log in that path, no matter where you launch it.

Based on your log4j.properties, it would be in the same directory where you start your flume-agent, thats what ./ would do.
So, if you run from /home directory, it would be
/home/logs/flume.log
Also, you can use linux commands anytime to find that out,
find / -name flume.log ##search in whole disk
find . -name flume.log ##search in current directory

Related

Tomcat application server logs are not printed in a rolling file other than catalina.log using log4j2.properties

I have recently migrated application from log4j1.x to log4j2.17.1. Application logs are printed well in console as well as in Rolling file when start the tomcat server. I could see Tomcat server start up logs are all printing in catalina.log but the same server startup logs and any other server internal logs are not printed in rolling file. Rolling file prints only those logs are coming from class files. Could you anyone of you suggest what can be done to print all application server start up logs like some logs gets printed in console before server start or connection refused logs etc. also get printed into Rolling file? Same things was working well when I had log4j1.x version.It means when I had log4j1.x server start up logs are printed well in catalina as well as in a DailyRollingFileAppender but Now issue is only with log4j2, it means server start up logs are printed in catalain only not in a rolling file which I want. Please suggest.
Here is my log4j2.properties
rootLogger.level = info
property.filename =D:/log/TEST.log
appenders = R, console
appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d %5p [%t] (%F:%L) - %m%n
appender.R.type = RollingFile
appender.R.name = File
appender.R.fileName = ${filename}
appender.R.filePattern = ${filename}.%d{yyyy-MM-dd}
appender.R.layout.type = PatternLayout
appender.R.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%15.15t]: %c{1}:%L - %m%n
appender.R.policies.type = Policies
appender.R.policies.time.type = TimeBasedTriggeringPolicy
appender.R.policies.time.interval = 1
rootLogger.appenderRef.console.ref = STDOUT
rootLogger.appenderRef.R.ref = File

Can't add a rolling file appender with Log4j2.properties

I have a java application running in a docker container. I can't change the program but I can change the log4j2.properties file. I want to add a rolling file appender but when I do it I get this error:
ERROR An exception occurred processing Appender file_appender java.security.AccessControlException: access denied ("java.io.FilePermission" "shared_logs" "read")
Then I gave all permissions to my shared_logs folder
chmod 777 shared_logs
But the error is still there.
This is how I added the the appender:
appender.fa.type = RollingFile
appender.fa.name = file_appender
appender.fa.fileName = shared_logs/elastic.log
appender.fa.filePattern = shared_logs/elastic-%d{yyyy-dd-MM}-%i.log.gz
appender.fa.layout.type = PatternLayout
appender.fa.layout.pattern = [%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{1} - %msg%n
appender.fa.policies.type = Policies
appender.fa.policies.time.type = TimeBasedTriggeringPolicy
appender.fa.policies.size.type = SizeBasedTriggeringPolicy
appender.fa.policies.size.size = 500
appender.fa.strategy.type = DefaultRolloverStrategy
appender.fa.strategy.max = 5
"fa" stands for fileappender
Well, you obviously still have a permissions problem. Your fileName is a relative file so where it is being created will be based on the currently directory of your app. That would typically be either the directory your app is in or the root directory. If it is the root directory and you are expecting it to be in your app this would explain the problem. If you add -Dlog4j2.debug to the startup options you would see where Log4j is trying to create the file.
Writing to log files in Docker Containers is also not usually considered a best practice. If you haven't already, I would suggest taking a look at Logging in the Cloud on the Log4j web site.

Why does log4j work correctly on windows and Linux, the same configuration not working in docker?

I use tomcat as server and log4j can work on windows and Linux. When I deploy it with Docker, log4j can't work normally. And I find nothing is created in related folder. I don't find any error in catalina.log. I check the access rights for tmp folder and writing is allowed. Do you have any idea about source of the problem? Thanks.
log4j.rootLogger=INFO, DailyRolling, Terminal, CONSOLE
#log4j.rootLogger=DEBUG, A1, A2, DailyRolling, Terminal, CONSOLE
log4j.appender.DailyRolling=org.apache.log4j.RollingFileAppender
log4j.appender.DailyRolling.layout=org.apache.log4j.PatternLayout
log4j.appender.DailyRolling.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-7p (%c) %m%n
log4j.appender.DailyRolling.DatePattern='.'yyyy-MM-dd-a
#LINUX
log4j.appender.DailyRolling.File=/tmp/RolingLog.log
log4j.appender.DailyRolling.DatePattern=yyyy-MM-dd'.log'
log4j.appender.DailyRolling.Threshold=INFO
log4j.appender.DailyRolling.BufferedIO=true
log4j.appender.DailyRolling.BufferSize=8192
log4j.appender.DailyRolling.MaxFileSize=20MB
log4j.appender.DailyRolling.MaxBackupIndex=100
log4j.appender.DailyRolling.MaxBackupDay=10
log4j.appender.Terminal=org.apache.log4j.RollingFileAppender
log4j.appender.Terminal.MaxBackupDay=10
#LINUX
log4j.appender.Terminal.File=/tmp/LogConsole.log
log4j.appender.Terminal.layout=org.apache.log4j.PatternLayout
log4j.appender.Terminal.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-7p (%c) %m%n
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.encoding=UTF-8
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.conversionPattern=%d [%t] %-5p %c - %m%n
log4j.logger.httpclient.wire=INFO
log4j.logger.org.apache.commons=INFO
log4j.logger.org.hibernate=WARN
log4j.logger.org.oracle=INFO
log4j.logger.com.mchange.v2=INFO
log4j.logger.org.apache.commons.httpclient=WARN
log4j.logger.org.apache.xerces=ERROR
log4j.logger.org.springframework=INFO

Separate Stderr and stdout stream at CLI using log4j configuration

I have an ANT based application and I have used slf4j logger to log at the exception, info, fatal etc. It is working fine. I am using log4j configuration for logging details. I used slf4j-log4j12-1.5.2.jar for this. Below is the configuration for logs.
log4j.rootLogger=info, file, stdout, stderr
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=cmd.log
log4j.appender.file.MaxFileSize=1024KB
log4j.appender.file.MaxBackupIndex=4
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p (%13F:%L) %3x - %m%n
log4j.appender.errorFile=org.apache.log4j.RollingFileAppender
log4j.appender.errorFile.File=error_stream.log
log4j.appender.errorFile.MaxFileSize=1024KB
log4j.appender.errorFile.MaxBackupIndex=4
log4j.appender.errorFile.layout=org.apache.log4j.PatternLayout
log4j.appender.errorFile.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p (%13F:%L) %3x - %m%n
log4j.appender.errorFile.Threshold=ERROR
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss} %5p - %m%n
log4j.appender.stderr=org.apache.log4j.ConsoleAppender
log4j.appender.stderr.Threshold=WARN
log4j.appender.stderr.Target=System.err
log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
log4j.appender.stderr.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss} %5p - %m%n
I am calling my application through ant script as ant -f build.xml 1>out.txt 2>err.txt
Now it should stream all errors to err.txt. But its going into out.txt
Note : it is able to create error_stream.log and have errors. But not in err.txt
Do I am using wrong configuration properties?
Are you trying to define logging for your application launched by ANT? Or are you trying to define logging for ANT (the process encapsulating your application)?
If it's the latter, then ANT supports loggers and listeners, for an example:
Real time ant build analyzer
There is an issue with ant Task class. It handleErrorOutput as "warning" and further warning is stream into output stream.
You can modify the ant.jar with below changes. It will work.
Task_before_changes.java
protected void handleErrorOutput(String output) {
log(output, Project.MSG_WARN);
}
Task_after_changes.java
protected void handleErrorOutput(String output) {
log(output, Project.MSG_ERR);
}

diagnose why my Cobertura output from Ant is in debug mode

I've set up an Ant script to run unit tests with Cobertura. For some reason I'm getting the log4j DEBUG output from Cobertura. I've been staring at this for hours using all the diagnostic fu I can come up with, and I don't see why this is happening.
I used SysInternals ProcessMonitor to check to see if I manually loaded "log4j.properties" or "log4j.xml". None found.
I made Ant turn on verbose class loading, and I never saw it load either "log4j.properties" or "log4j.xml". This tells me this wasn't passed to forked processes, because I know the "cobertura.jar" has a "log4j.properties", which has the following contents:
# set default logging level and appender
log4j.rootCategory=WARN, console
#log4j.category.net.sourceforge.cobertura=WARN, console
#log4j.category.net.sourceforge.cobertura.check=DEBUG, console
#log4j.category.net.sourceforge.cobertura.coverage=DEBUG, file
#log4j.category.net.sourceforge.cobertura.coverage.CoverageDataFactory=INFO, file
#log4j.category.net.sourceforge.cobertura.coverage.SaveInstrumentationTimer=DEBUG, file
#log4j.category.net.sourceforge.cobertura.merge=DEGUG, console
#log4j.category.net.sourceforge.cobertura.reporting=DEBUG, file
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%-6p %M, %m%n
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=cobertura.log
log4j.appender.file.MaxFileSize=3000KB
log4j.appender.file.MaxBackupIndex=3
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d %-2p [%t] %c{3} - %m - (%F:%L) %n
The only irrelevant problem in here is the misspelled "DEGUG", but that's commented out.
I can't see how else this is happening. I've been asking one of the Cobertura maintainers, and he doesn't see why this might be happening.
I figured it out. I had forgotten about the very handy "log4j.debug" property, which when enabled tells you where it found a "log4j.xml" or "log4j.properties" file. I found the former in an unexpected jar file, which I've now patched (it was setting the default level to DEBUG), and now I'm not getting debug output by default.

Resources