Modify existing log4j2 FileAppender configuration during runtime - log4j2

I am migrating application (huge web application) from log4j1.x to log4j 2.11.0.
I need help/clarification to migrate following scenarios -
--------scenario1
log4j1.x:
<appender name="import_log_file" class="xxxx">
During runtime i want to change the file so i would just do appender.setFile(...new file...). And done.
log4j2:
how do I migrate above code?
Few ideas but not a straight answer:
Creating brand new appender via LoggerContext, Configuration might be a way but i want to update an existing appender's configuration and reload the log4j2 xml. How do I do that?
Another way could be redefining something like this
<appender name="import_log_file" class="xxxx">
And then setup "dynamic_name" property in threadcontext. But I am loosing original default file import.log
Any help is appreciated. Let me know if you have ideas.

During runtime i want to change the file
You can use the RoutingAppender together with a lookup to do this. See the log4j2 FAQ page for details.
Here is a very simple example of how to change the log file name at runtime:
package pkg;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;
public class Log4j2DiffFilePerCtxVarMain {
private static final Logger LOG = LogManager.getLogger();
public static void main(String[] args){
ThreadContext.put("myFileNameVar", "file1");
LOG.info("This should appear in file1.log");
ThreadContext.put("myFileNameVar", "file2");
LOG.info("This should appear in file2.log");
}
}
The configuration looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Routing name="myAppender">
<Routes pattern="$${ctx:myFileNameVar}">
<Route>
<File
fileName="logs/${ctx:myFileNameVar}.log"
name="myAppender-${ctx:myFileNameVar}">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</File>
</Route>
</Routes>
</Routing>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="myAppender" />
</Root>
</Loggers>
</Configuration>
The result of running the above code will be two files - file1.log an file2.log each with a single entry. The file1.log will contain the first log message and file2.log will contain the second message.
Hope this helps!

Related

Using RoutingAppender in commerical product programmatically

I have a commerical product, using log4j2 internally.
I can add java/groovy script as an internal api. Here I can also use log4j2 as logger
Each script is running within a thread.
The logging for each of the scripts, I want have a own logfile (and all subordinate loggers)
Before I start within the product, I was playing with Log4J2 locally, using the RoutingAppender examples.
RoutingAppender: it works as expected when I have a log4j2.xml during startup.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout
pattern="myCONSOLE - [%d] [%-5p] [%-7t] - %m%n" />
</Console>
<Routing name="Routing">
<Routes pattern="$${ctx:ROUTINGKEY}">
<Route>
<RollingFile name="Rolling-${ctx:ROUTINGKEY}"
fileName="logs/test-${ctx:ROUTINGKEY}.log"
filePattern="./logs/test-${ctx:ROUTINGKEY}-%d{yyyyMMdd}-%i.log.gz">
<PatternLayout>
<pattern>%d{ISO8601} [%t] %p %c{3} - %m%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy interval="6"
modulate="true" />
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
</RollingFile>
</Route>
</Routes>
</Routing>
</Appenders>
<Loggers>
<Root level="ALL">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="Routing" />
</Root>
</Loggers>
</Configuration>
In the code I use parts as:
In the script script (simulated), I do ThreadContext.put("ROUTINGKEY", "exampleA"); , in a second script I use "exampleB".
In main programm I set:
static {
System.setProperty("isThreadContextMapInheritable", "true"); // So all objects/loggers in same thread will inherit the ThreadC
}
Other classes/objects are created as usual with normal logger technique: private static Logger logger = LogManager.getLogger(Grassfrosch.class);; and using logger.info....
All logging was in correct logfile including the subordinate elements.
Back to the commerical product. Here I want to add the RoutingAppender (and the RollingFileAppender) dynamically with programming code in the groovy scripts.
Script "exampleA" should add the route for the own part by itself
Script "exampleB" should add the route for the own part by itself
I don't want to use the product config file itself due to product upgrades/patches, overwritten by the manufacturer.
So, I also implemented locally such logic "programmatically approach".
I create a FileAppender/RollingFileAppender
I create a Routes plus Route(s) object
I create a RoutingAppender, adding the routes etc.
Simplified code is:
private void fragment() {
Configuration cfg;
org.apache.logging.log4j.core.Logger rl = (org.apache.logging.log4j.core.Logger) LogManager.getRootLogger();
cfg = rl.getContext().getConfiguration();
String name = "exampleA";
PatternLayout layout = PatternLayout.newBuilder().withPattern("DYNAMIC[" + name + "] - %d [%p] - %m%n").build();
FileAppender fileAppender = FileAppender.newBuilder() //
.setName("FileAppender-" + name).withFileName("logs/FileAppender-" + name + ".log") //
.withAppend(true).setLayout(layout) //
// .setConfiguration(cfg)
.build();
fileAppender.start();
rl.addAppender(fileAppender);
Route route01 = Route.createRoute("FileAppender-" + name , name, null);
Routes routes = Routes.newBuilder().withPattern("$${ctx:ROUTINGKEY}").withRoutes(new Route[] { route01}).build();
RoutingAppender routingAppender = RoutingAppender.newBuilder().setName("EPI-Routing").withRoutes(routes) //
.setConfiguration(cfg) //
.build();
routingAppender.start();
rl.addAppender(routingAppender);
rl.getContext().updateLoggers();
}
With this code, both scripts "exampleA" and "exampleB", will written to both logfiles "FileAppender-exampleA" and "FileAppender-exampleB" and not as expected in the "log4j2.xml" example.
I guess, the problem is:
I have to add a FileAppender to existing configuration
due to I have to "reference" Route.createRoute.
Finally the FileAppender I have added to rootlogger (wrong).
I also try to add to script logger, but no success.
I stuck. Any hints?
Uwe

Translating prometheus LOG4j conf from XML to properties file

I have used LOG4j many times on quite common basics, and mainly through a properties file. I am though very unfamiliar with the XML form and with uncommon features (such as third party lib custom logging).
The prometheus log4j2 configuration is written for xml conf files, and I actually don't understand it. Although I could use it as such, I would like to understand it by translating in a form I am confortable with : as a properties file .
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="io.prometheus.client.log4j2">
<Appenders>
<Prometheus name="METRICS"/>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="METRICS"/>
</Root>
</Loggers>
</Configuration>
properties file :
name=PropertiesConfig
property.filename = /var/logs
appenders = console, METRICS?
appenders.METRICS?.
...?
rootLogger.appenderRefs = METRICS, console
...?
Can anyone help me on this one?
like this:
log4j.rootLogger=CONSOLE,METRICS
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.Threshold=DEBUG
log4j.appender.CONSOLE.Target=System.out
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=[%-5p][%d{yyyy-MM-dd HH\:mm\:ss,SSS}][%c] \:%m%n
log4j.appender.METRICS=io.prometheus.client.log4j.InstrumentedAppender
after see log4j_appender_total flag in prometheus
converstion refer: https://www.journaldev.com/10698/log4j-properties-file-example

Log4j2: Empty log file in case of parallel tests

I have in my test automation project problem with logging. I'm using log4j2 logger with FileAppender. The way I'm using it is:
Logger logger = (Logger) LogManager.getLogger(loggerName);
Appender appender = FileAppender.newBuilder()
.withAppend(false)
.withBufferedIo(true)
.withFileName(DIR_NAME + File.separator + loggerName + ".log")
.withIgnoreExceptions(false)
.withImmediateFlush(true)
.withLocking(false)
.withLayout(PatternLayout.newBuilder().withPattern("%d{HH:mm:ss.SSS} [%-5level] %msg%n").withCharset(Charset.forName("UTF-8")).build())
.withName(loggerName)
.build();
appender.start();
logger.addAppender(appender);
It works when I'm running single test. All data are visible in console, the file is created and test log is written in the file. Problem occurs in case of tests are running in parallel - in different threads.
In this case, two different loggers and file appenders are created. Log files from both file appenders are created too and logs from both tests are visible in console. Everything seems to be fine, but every time one of these log files is empty. The empty log belongs to test which started later.
I suspect problem with caching. The first file appender holds all cache for writing so the second one is not able to write. Am I right? What is the solution for this?
Thank you.
You should be able to achieve what you want without using programmatic configuration. There are many reasons not to configure log4j2 programmatically, but the best one, in my opinion, is that in doing so you would make your code dependent on aspects of log4j2 that are not part of the public API. This means that if the implementation of log4j2 changes your code has to change as well. This creates more work for you in the long run.
So, with that in mind I will provide a demo of how to set up log4j2 using an XML config file such that it will generate separate logs for each test. I am assuming, since it was not specified in your question, that your goal is to create a log for each method with a Test annotation and that each of these methods is executed in parallel.
First, here is my TestNG class:
package testpkg;
import java.lang.reflect.Method;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class NewTest {
private static final Logger log = LogManager.getLogger();
#BeforeMethod
public void setThreadName(Method method){
ThreadContext.put("threadName", method.getName());
}
#Test
public void test1() {
log.info("This is the first test!");
log.warn("Something may be wrong, better take a look.");
}
#Test
public void test2() {
log.info("Here's the second test!");
log.error("There's a problem, better fix it");
}
}
As you can see here I have two Test methods and a BeforeMethod called setThreadName. The setThreadName method is, obviously, executed before each of the Test methods. It places a key named threadName into the log4j2 ThreadContext using the name of the method that is about to be run. This will be used as part of the log file name in the log4j2 config file.
Here is the log4j2.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Routing name="MyRoutingAppender">
<Routes pattern="$${ctx:threadName}">
<Route>
<File
fileName="logs/${ctx:threadName}.log"
name="appender-${ctx:threadName}"
append="false">
<PatternLayout>
<Pattern>[%date{ISO8601}][%-5level][%t] %m%n</Pattern>
</PatternLayout>
</File>
</Route>
</Routes>
</Routing>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="[%date{ISO8601}][%-5level][%t] %m%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="testpkg" level="TRACE" additivity="false">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="MyRoutingAppender" />
</Logger>
<Root level="WARN">
<AppenderRef ref="STDOUT" />
</Root>
</Loggers>
</Configuration>
As you can see I've set up the config file to use a RoutingAppender to dynamically generate appenders at runtime based on the ThreadContext key threadName and that threadName is also used in the fileName attribute of the FileAppender.
Here is my testNG config file:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="My suite" parallel="methods" thread-count="5" verbose="1">
<test name="testpkg" >
<classes>
<class name="testpkg.NewTest" />
</classes>
</test>
</suite>
As you can see here I've set it up so that each Test method within my class is run in parallel.
When executed this results in the following console output:
[RemoteTestNG] detected TestNG version 6.14.3
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-2] Here's the second test!
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-1] This is the first test!
[2018-05-04T21:54:54,709][WARN ][TestNG-test=testpkg-1] Something may be wrong, better take a look.
[2018-05-04T21:54:54,709][ERROR][TestNG-test=testpkg-2] There's a problem, better fix it
===============================================
My suite
Total tests run: 2, Failures: 0, Skips: 0
===============================================
You can clearly see that the output of the two methods is interleaved, so we know that the methods are indeed running in parallel.
The execution of the test class also creates two log files as expected. They are named test1.log and test2.log
Here are their contents:
test1.log:
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-1] This is the first test!
[2018-05-04T21:54:54,709][WARN ][TestNG-test=testpkg-1] Something may be wrong, better take a look.
test2.log:
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-2] Here's the second test!
[2018-05-04T21:54:54,709][ERROR][TestNG-test=testpkg-2] There's a problem, better fix it
So we see here that as expected the logs from the first method went to test1.log and the logs from the second method went to test2.log
Enjoy!

how to change htmllayout in log4j2

When I use log4j, I can create a new class extends org.apache.log4j.HTMLLayout and override it to make a new format of the log. When using log4j2, I can only use the layout but not override it. In my log4j2.xml writing
<Appenders>
<File name="log" fileName="D:\log.html" append="false">
<HTMLLayout/>
</File>
</Appenders>
in log4j I may use layout class="log.FormatHTMLLayout" (log.FormatHTMLLayout is my new class which extends the HTMLLayout), but now I can only use HTMLLayout.
Is there any way to override the HTMLayout? I need to do a lot of things, like changing the output table, the title and so on.
Here is what i did,
I needed to add <img> HTML tags in logs generated by log4j2 and by default HTML elements like <, >, " are replaced by escape characters like lt; , gt; , quot;.
(Ref:https://issues.apache.org/jira/browse/LOG4J2-439)
So i just copied whole HTMLLayout class from source, which is available at
log4j-core / src / main / java / org / apache / logging / log4j / core / layout
and changed its name to "CustomHTMLLayout" and updated it wherever required (you can choose any name), now your custom layout class is as good as HTMLLayout class.
there is method called toSerializable which contains actual formatting of each record, so you can manipulate it as per your need.
once class is modified, you need to provide the new layout in your log4j2.html
as following:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.redknee.bssauto.helpers">
<Appenders>
<RollingFile name="Rolling-default" fileName="logs/bssauto.html" filePattern="logs/$${date:yyyy-MM}/bssauto-%d{MM-dd-yyyy}-%i.log.gz">
<CustomHTMLLayout charset="UTF-8" title="BSSAuto Logs" locationInfo="true" />
<Policies>
<TimeBasedTriggeringPolicy />
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="Rolling-default"/>
</Root>
</Loggers>
</Configuration>
Notice following
<Configuration packages="com.bssauto.helpers">
here packages should have all packages containing custom class for layouts. so here com.bssauto.helpers is package under which i have CustomHTMLLayout class.
<CustomHTMLLayout charset="UTF-8" title="BSSAuto Logs" locationInfo="true" />
and CustomHTMLLayout is custom layout class created by extending AbstractStringLayout
Make sure you are using latest log4j2 version, I used log4j 2.2

Log4j2 (beta9) How to configure custom layout for FileAppender?

Can someone tell me, How to configure custom layout for FileAppender?
Can anyone tell me, how to configure custom layout for FileAppender?
I'm created copy of HTMLLayout and made some changes there (it's cannot be extends because it's final class) and now I want use this layout, but I don't know how :(
This error is showed with bellow listed configuration:
ERROR File contains an invalid element or attribute "ibtrader.log4j2.MYHTMLLayout"
Here is my log4j2.xml configuration file
<?xml version="1.0" encoding="UTF-8"?>
<configuration strict="true" monitorInterval="30">
<appenders>
<appender name="Console" type="Console" target="SYSTEM_OUT">
<layout type="PatternLayout" pattern="%highlight{%d{ISO8601} [%t] %-5level %logger{36} - %msg%n}" />
</appender>
<appender name="DEBUG_FILE" type="File" fileName="logs/errors.txt" >
<layout type="PatternLayout" pattern="%d{ISO8601} [%t] %-5level %logger{36} - %msg%n}" />
</appender>
<appender name="HTMLAppender" type="File" fileName="logs/mainlog.html">
<layout type="ibtrader.log4j2.MYHTMLLayout" charset="UTF-8" title="IBTRader logs" locationInfo="true" />
</appender>
</appenders>
<loggers>
<root level="trace">
<appender-ref ref="Console"/>
<appender-ref ref="DEBUG_FILE" level="WARN" />
<appender-ref ref="HTMLAppender" />
</root>
</loggers>
</configuration>
Thanks for help!
The comments mention this is already fixed but just for completeness, a (simplified) configuration with a custom MYHTMLLayout plugin could look like this:
<?xml version="1.0" encoding="UTF-8"?>
<!-- the packages attribute contains a comma-separated list
of the packages that Log4J2 will scan for custom plugins. -->
<configuration packages="ibtrader.log4j2">
<appenders>
<appender name="HTMLAppender" type="File" fileName="logs/mainlog.html">
<!-- Each plugin has a *name*, declared with annotation on the
plugin implementation class.
The plugin name does not need to match the class name.
The plugin name needs to match the *type* attribute in the config.
For example:
package ibtrader.log4j2;
import ...;
#Plugin(name="MYHTMLLayout", category="Core",
elementType="layout", printObject=true)
public class MyHTMLLayoutImpl extends AbstractStringLayout {...
-->
<layout type="MYHTMLLayout" charset="UTF-8" title="IBTRader logs" locationInfo="true" />
</appender>
</appenders>
<loggers>
<root level="trace">
<appender-ref ref="HTMLAppender" />
</root>
</loggers>
</configuration>
You have to extend AbstractStringLayout, setting the #Plugin properties with category = "Core", elementType = "layout" and name="The name of the class itself that should be refered from the configuration file" (i.e.: "ExtHtmlLayout")
You could just copy the whole class HtmlLayout from the sources and change whatever you want, for example, the time in millis column to the time in hours.
Define the package property in "Configuration" tag to use the package to the extended class you have created.
Finally, just call the new layout from the configuration file:
<ExtHtmlLayout/>

Resources