log4j2: registering custom TriggeringPolicy - log4j2

I have written a custom TriggeringPolicy for log4j2 that is suppose to roll-over .log file at the end of every hour/day/your_interval following advices from this SO post.
Though I followed TimeBasedTriggeringPolicy conventions (naming, etc) I am not able to see my policy being instantiated and used.
Solution comprise of 3 java files + a maven file and is available at the github.
Here you can find main lines from the policy itself:
#Plugin(name = "FTimeBasedTriggeringPolicy", category = "Core", printObject = true)
public class FTimeBasedTriggeringPolicy implements TriggeringPolicy {
private final TimeBasedTriggeringPolicy timeBasedTriggeringPolicy;
private RollingFileManager manager;
private FTimeBasedTriggeringPolicy(final int interval, final boolean modulate) {
timeBasedTriggeringPolicy = TimeBasedTriggeringPolicy.createPolicy(String.valueOf(interval), String.valueOf(modulate));
LogRotateThread.registerPolicy(this);
}
public void checkRollover(final LogEvent event) {
this.manager.checkRollover(event);
}
#Override
protected void finalize() throws Throwable {
LogRotateThread.unregisterPolicy(this);
super.finalize();
}
#Override
public void initialize(final RollingFileManager manager) {
this.manager = manager;
timeBasedTriggeringPolicy.initialize(manager);
}
#Override
public boolean isTriggeringEvent(final LogEvent event) {
return timeBasedTriggeringPolicy.isTriggeringEvent(event);
}
#Override
public String toString() {
return "FTimeBasedTriggeringPolicy";
}
#PluginFactory
public static FTimeBasedTriggeringPolicy createPolicy(
#PluginAttribute("interval") final String interval,
#PluginAttribute("modulate") final String modulate) {
final int increment = Integers.parseInt(interval, 1);
final boolean mod = Boolean.parseBoolean(modulate);
return new FTimeBasedTriggeringPolicy(increment, mod);
}
}
The log4j2.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="debug" name="RoutingLoggingConfiguration" packages="org.log4j2plugin" verbose="true">
<Properties>
<Property name="routing_filename">${log.path}/table-$${sd:type}.log</Property>
</Properties>
<Appenders>
<Console name="STDOUT">
<PatternLayout pattern="%d{yyyyMMddHH}{GMT+0} %m%n"/>
</Console>
<Routing name="Routing">
<Routes pattern="$${sd:type}">
<Route>
<RollingFile name="RollingFile-${sd:type}"
fileName="${routing_filename}"
filePattern="${log.path}/%d{yyyyMMdd}{GMT+0}/%d{yyyyMMddHH}{GMT+0}-${sd:type}-${hostName}.%i.log.gz">
<PatternLayout>
<Pattern>%d{yyyyMMddHH}{GMT+0},'%d{yyyy-MM-dd HH:mm:ss}{GMT+0}',%K{v}%n</Pattern>
</PatternLayout>
<Policies>
<FTimeBasedTriggeringPolicy interval="1"/>
<SizeBasedTriggeringPolicy size="64 MB"/>
</Policies>
<DefaultRolloverStrategy max="999"/>
</RollingFile>
</Route>
</Routes>
</Routing>
</Appenders>
<Loggers>
<Logger name="EventLogger" level="debug" additivity="false">
<AppenderRef ref="Routing"/>
</Logger>
<Root level="warn">
<AppenderRef ref="STDOUT"/>
</Root>
</Loggers>
</Configuration>
EDIT:
During debugging I understood that Routes appenders (defined by the $${sd:type} in my case) are not known during log4j2.xml parsing. Thus - their creation/initialization is delayed in time to the moment when the first message for destination $${sd:type} arrives. My next plan is to:
add a StructuredDataFilter to the Routes appender
provide an empty message at the system start-up to all known $${sd:type}, which on one hand should initialize Route appender and cause FTimeBasedTriggeringPolicy to register itself in the LogRotateThread, but on other - should be discarded by the StructuredDataFilter
allow LogRotateThread to query registered FTimeBasedTriggeringPolicy and rotate logs if needed

While debugging, it appeared that Routes appenders (defined by the $${sd:type} in my case) are not known during log4j2.xml parsing. Thus, their creation/initialization is delayed in time to the moment when the first message for the destination $${sd:type} arrives.
Thus, the original subject of the post could be answered as "custom policy is registered, however not always instantly (or during the .xml parse time)".
The original problem, however, was to force .log files to roll-over at the end of the time period (an hour in my case). To address it, I have implemented following algorithm:
wrote a thin wrapper around TimeBasedTriggeringPolicy - FTimeBasedTriggeringPolicy that registers itself on instantiation at LogRotateThread
wrote a simple LogRotateThread that queries registered FTimeBasedTriggeringPolicy once every few minutes and makes them rotate .log if needed
added a StructuredDataFilter to the Routes appender, so that they discard specific messages (with id=SKIP in my case)
provide an empty message at the system start-up to all known $${sd:type}, which:
a. instantiate Route appender and cause FTimeBasedTriggeringPolicy to register itself in the LogRotateThread
b. are discarded by the StructuredDataFilter
Solution is published under Apache 2.0 license, and is available at github

Related

Using RoutingAppender in commerical product programmatically

I have a commerical product, using log4j2 internally.
I can add java/groovy script as an internal api. Here I can also use log4j2 as logger
Each script is running within a thread.
The logging for each of the scripts, I want have a own logfile (and all subordinate loggers)
Before I start within the product, I was playing with Log4J2 locally, using the RoutingAppender examples.
RoutingAppender: it works as expected when I have a log4j2.xml during startup.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout
pattern="myCONSOLE - [%d] [%-5p] [%-7t] - %m%n" />
</Console>
<Routing name="Routing">
<Routes pattern="$${ctx:ROUTINGKEY}">
<Route>
<RollingFile name="Rolling-${ctx:ROUTINGKEY}"
fileName="logs/test-${ctx:ROUTINGKEY}.log"
filePattern="./logs/test-${ctx:ROUTINGKEY}-%d{yyyyMMdd}-%i.log.gz">
<PatternLayout>
<pattern>%d{ISO8601} [%t] %p %c{3} - %m%n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy interval="6"
modulate="true" />
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
</RollingFile>
</Route>
</Routes>
</Routing>
</Appenders>
<Loggers>
<Root level="ALL">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="Routing" />
</Root>
</Loggers>
</Configuration>
In the code I use parts as:
In the script script (simulated), I do ThreadContext.put("ROUTINGKEY", "exampleA"); , in a second script I use "exampleB".
In main programm I set:
static {
System.setProperty("isThreadContextMapInheritable", "true"); // So all objects/loggers in same thread will inherit the ThreadC
}
Other classes/objects are created as usual with normal logger technique: private static Logger logger = LogManager.getLogger(Grassfrosch.class);; and using logger.info....
All logging was in correct logfile including the subordinate elements.
Back to the commerical product. Here I want to add the RoutingAppender (and the RollingFileAppender) dynamically with programming code in the groovy scripts.
Script "exampleA" should add the route for the own part by itself
Script "exampleB" should add the route for the own part by itself
I don't want to use the product config file itself due to product upgrades/patches, overwritten by the manufacturer.
So, I also implemented locally such logic "programmatically approach".
I create a FileAppender/RollingFileAppender
I create a Routes plus Route(s) object
I create a RoutingAppender, adding the routes etc.
Simplified code is:
private void fragment() {
Configuration cfg;
org.apache.logging.log4j.core.Logger rl = (org.apache.logging.log4j.core.Logger) LogManager.getRootLogger();
cfg = rl.getContext().getConfiguration();
String name = "exampleA";
PatternLayout layout = PatternLayout.newBuilder().withPattern("DYNAMIC[" + name + "] - %d [%p] - %m%n").build();
FileAppender fileAppender = FileAppender.newBuilder() //
.setName("FileAppender-" + name).withFileName("logs/FileAppender-" + name + ".log") //
.withAppend(true).setLayout(layout) //
// .setConfiguration(cfg)
.build();
fileAppender.start();
rl.addAppender(fileAppender);
Route route01 = Route.createRoute("FileAppender-" + name , name, null);
Routes routes = Routes.newBuilder().withPattern("$${ctx:ROUTINGKEY}").withRoutes(new Route[] { route01}).build();
RoutingAppender routingAppender = RoutingAppender.newBuilder().setName("EPI-Routing").withRoutes(routes) //
.setConfiguration(cfg) //
.build();
routingAppender.start();
rl.addAppender(routingAppender);
rl.getContext().updateLoggers();
}
With this code, both scripts "exampleA" and "exampleB", will written to both logfiles "FileAppender-exampleA" and "FileAppender-exampleB" and not as expected in the "log4j2.xml" example.
I guess, the problem is:
I have to add a FileAppender to existing configuration
due to I have to "reference" Route.createRoute.
Finally the FileAppender I have added to rootlogger (wrong).
I also try to add to script logger, but no success.
I stuck. Any hints?
Uwe

Log4j2: How to append to HTML log using HtmlLayout

I am creating html log files by using log4j2. When I am executing my code the first time the logs are formatted as a table. However, while executing a second time the data is not appended to the table, it is instead stored line by line under the table. Is there a way to append the logs to the existing table?
<Appenders>
<RollingFile name="fileLogger" fileName="/home/developers/Desktop/exam/app-info.html"
filePattern="app-info-%d{yyyy-MM-dd}.html">
<HTMLLayout charset="UTF-8" title="Howtodoinjava Info Logs" locationInfo="true" />
<Policies>
<TimeBasedTriggeringPolicy interval="10" modulate="true" />
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
</RollingFile>
<Console name="console" target="SYSTEM_OUT">
<PatternLayout pattern="[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{1} - %msg%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="com.howtodoinjava" level="info" additivity="true">
<appender-ref ref="fileLogger" level="info" />
</Logger>
<Root level="debug" additivity="true">
<appender-ref ref="console" />
<appender-ref ref="fileLogger" />
</Root>
</Loggers>
This is log file
Screenshot of log file
Unfortunately I think the answer is that you can't do what you want with the HtmlLayout provided by log4j2. It looks like the HtmlLayout was intended to generate a single html file per execution.
If you take a look at the source code for HtmlLayout, you'll see code like following:
#Override
public byte[] getHeader() {
final StringBuilder sbuf = new StringBuilder();
append(sbuf, "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\" ");
...
appendLs(sbuf, "<html>");
...
appendLs(sbuf, "<table ...");
...
return sbuf.toString().getBytes(getCharset());
}
....
#Override
public byte[] getFooter() {
final StringBuilder sbuf = new StringBuilder();
appendLs(sbuf, "</table>");
appendLs(sbuf, "<br>");
appendLs(sbuf, "</body></html>");
return getBytes(sbuf.toString());
}
The code is clearly expecting to create a new html document each time as you can see the opening html tags in getHeader and closing tags in the getFooter.
You could write your own layout to do what you want but the problem you're going to face is determining when to write your footer. You don't want to write the footer until you're completely done with the file, so you will have to somehow implement a way to detect this situation (assuming you want to have clean HTML in your log).
If you don't care about having clean HTML then just don't write any footer. In this case you could simply copy the HtmlLayout class and change it slightly to create a new layout that does not write any footer. For example:
//your package and imports go here
...
#Plugin(name = "NoFooterHtmlLayout", category = Node.CATEGORY, elementType = Layout.ELEMENT_TYPE, printObject = true)
public final class NoFooterHtmlLayout extends AbstractStringLayout {
//Various other methods copied from HtmlLayout go here
...
#Override
public byte[] getFooter() {
return new byte[0];
}
//Various other methods copied from HtmlLayout go here
...
}
and then in your log4j2 configuration you would use this:
<NoFooterHtmlLayout charset="UTF-8" title="My Title"
locationInfo="true" />
Hope this helps!

Modify existing log4j2 FileAppender configuration during runtime

I am migrating application (huge web application) from log4j1.x to log4j 2.11.0.
I need help/clarification to migrate following scenarios -
--------scenario1
log4j1.x:
<appender name="import_log_file" class="xxxx">
During runtime i want to change the file so i would just do appender.setFile(...new file...). And done.
log4j2:
how do I migrate above code?
Few ideas but not a straight answer:
Creating brand new appender via LoggerContext, Configuration might be a way but i want to update an existing appender's configuration and reload the log4j2 xml. How do I do that?
Another way could be redefining something like this
<appender name="import_log_file" class="xxxx">
And then setup "dynamic_name" property in threadcontext. But I am loosing original default file import.log
Any help is appreciated. Let me know if you have ideas.
During runtime i want to change the file
You can use the RoutingAppender together with a lookup to do this. See the log4j2 FAQ page for details.
Here is a very simple example of how to change the log file name at runtime:
package pkg;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;
public class Log4j2DiffFilePerCtxVarMain {
private static final Logger LOG = LogManager.getLogger();
public static void main(String[] args){
ThreadContext.put("myFileNameVar", "file1");
LOG.info("This should appear in file1.log");
ThreadContext.put("myFileNameVar", "file2");
LOG.info("This should appear in file2.log");
}
}
The configuration looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Routing name="myAppender">
<Routes pattern="$${ctx:myFileNameVar}">
<Route>
<File
fileName="logs/${ctx:myFileNameVar}.log"
name="myAppender-${ctx:myFileNameVar}">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</File>
</Route>
</Routes>
</Routing>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="myAppender" />
</Root>
</Loggers>
</Configuration>
The result of running the above code will be two files - file1.log an file2.log each with a single entry. The file1.log will contain the first log message and file2.log will contain the second message.
Hope this helps!

Log4j2: Empty log file in case of parallel tests

I have in my test automation project problem with logging. I'm using log4j2 logger with FileAppender. The way I'm using it is:
Logger logger = (Logger) LogManager.getLogger(loggerName);
Appender appender = FileAppender.newBuilder()
.withAppend(false)
.withBufferedIo(true)
.withFileName(DIR_NAME + File.separator + loggerName + ".log")
.withIgnoreExceptions(false)
.withImmediateFlush(true)
.withLocking(false)
.withLayout(PatternLayout.newBuilder().withPattern("%d{HH:mm:ss.SSS} [%-5level] %msg%n").withCharset(Charset.forName("UTF-8")).build())
.withName(loggerName)
.build();
appender.start();
logger.addAppender(appender);
It works when I'm running single test. All data are visible in console, the file is created and test log is written in the file. Problem occurs in case of tests are running in parallel - in different threads.
In this case, two different loggers and file appenders are created. Log files from both file appenders are created too and logs from both tests are visible in console. Everything seems to be fine, but every time one of these log files is empty. The empty log belongs to test which started later.
I suspect problem with caching. The first file appender holds all cache for writing so the second one is not able to write. Am I right? What is the solution for this?
Thank you.
You should be able to achieve what you want without using programmatic configuration. There are many reasons not to configure log4j2 programmatically, but the best one, in my opinion, is that in doing so you would make your code dependent on aspects of log4j2 that are not part of the public API. This means that if the implementation of log4j2 changes your code has to change as well. This creates more work for you in the long run.
So, with that in mind I will provide a demo of how to set up log4j2 using an XML config file such that it will generate separate logs for each test. I am assuming, since it was not specified in your question, that your goal is to create a log for each method with a Test annotation and that each of these methods is executed in parallel.
First, here is my TestNG class:
package testpkg;
import java.lang.reflect.Method;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class NewTest {
private static final Logger log = LogManager.getLogger();
#BeforeMethod
public void setThreadName(Method method){
ThreadContext.put("threadName", method.getName());
}
#Test
public void test1() {
log.info("This is the first test!");
log.warn("Something may be wrong, better take a look.");
}
#Test
public void test2() {
log.info("Here's the second test!");
log.error("There's a problem, better fix it");
}
}
As you can see here I have two Test methods and a BeforeMethod called setThreadName. The setThreadName method is, obviously, executed before each of the Test methods. It places a key named threadName into the log4j2 ThreadContext using the name of the method that is about to be run. This will be used as part of the log file name in the log4j2 config file.
Here is the log4j2.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Routing name="MyRoutingAppender">
<Routes pattern="$${ctx:threadName}">
<Route>
<File
fileName="logs/${ctx:threadName}.log"
name="appender-${ctx:threadName}"
append="false">
<PatternLayout>
<Pattern>[%date{ISO8601}][%-5level][%t] %m%n</Pattern>
</PatternLayout>
</File>
</Route>
</Routes>
</Routing>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="[%date{ISO8601}][%-5level][%t] %m%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="testpkg" level="TRACE" additivity="false">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="MyRoutingAppender" />
</Logger>
<Root level="WARN">
<AppenderRef ref="STDOUT" />
</Root>
</Loggers>
</Configuration>
As you can see I've set up the config file to use a RoutingAppender to dynamically generate appenders at runtime based on the ThreadContext key threadName and that threadName is also used in the fileName attribute of the FileAppender.
Here is my testNG config file:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="My suite" parallel="methods" thread-count="5" verbose="1">
<test name="testpkg" >
<classes>
<class name="testpkg.NewTest" />
</classes>
</test>
</suite>
As you can see here I've set it up so that each Test method within my class is run in parallel.
When executed this results in the following console output:
[RemoteTestNG] detected TestNG version 6.14.3
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-2] Here's the second test!
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-1] This is the first test!
[2018-05-04T21:54:54,709][WARN ][TestNG-test=testpkg-1] Something may be wrong, better take a look.
[2018-05-04T21:54:54,709][ERROR][TestNG-test=testpkg-2] There's a problem, better fix it
===============================================
My suite
Total tests run: 2, Failures: 0, Skips: 0
===============================================
You can clearly see that the output of the two methods is interleaved, so we know that the methods are indeed running in parallel.
The execution of the test class also creates two log files as expected. They are named test1.log and test2.log
Here are their contents:
test1.log:
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-1] This is the first test!
[2018-05-04T21:54:54,709][WARN ][TestNG-test=testpkg-1] Something may be wrong, better take a look.
test2.log:
[2018-05-04T21:54:54,703][INFO ][TestNG-test=testpkg-2] Here's the second test!
[2018-05-04T21:54:54,709][ERROR][TestNG-test=testpkg-2] There's a problem, better fix it
So we see here that as expected the logs from the first method went to test1.log and the logs from the second method went to test2.log
Enjoy!

Log4J2 JSONLAYOUT How to add custom Parameters

How to add customized parameters to Log4j2’s JSONLAYOUT?
Also is there a way to add pattern to the JSONLAYOUT’s message element?
I have tried the options listed here ->
logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout
Please help!
Solution is to explicitly add log4j2’s 2.10.0 version. This version supports custom parameter in JSONLayout!
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<JSONLayout compact="true" eventEol="true">
<KeyValuePair key="application" value="${sys:com.example.appname}"/>
</JSONLayout>
</Console>
<Async name="AsyncAppender" bufferSize="80">
<AppenderRef ref="Console"/>
</Async>
</Appenders>
As I understand it you're looking for a way to customize the format of the JSON output from the JSONLayout in a manner similar to how you can customize the PatternLayout by specifying "Conversion Patterns".
I believe the answer is that you can't customize the JSONLayout in the same manner. You can select the various pieces of information you want to be included in the message. For example, the documentation shows parameters like properties:
If true, the appender includes the thread context map in the generated JSON. Defaults to false.
So you can set various parameters to include certain kinds of information, but you don't have the direct control over the specific items that are included.
What you could do instead is to use ObjectMessage along with a JSON library to generate a JSON message. However, this would generate JSON within JSON (assuming you still wish to use JSONLayout with this approach). Here is some sample code to illustrate:
A class with a main method to generate a log message:
package example;
import java.util.HashMap;
import java.util.Map;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONObject;
public class JsonMessageExample {
private static final Logger log = LogManager.getLogger();
public static void main(String[] args) {
Map<String,String> msgMap = new HashMap<>();
msgMap.put("myKey", "myValue");
JSONObject message = new JSONObject(msgMap);
log.info(message);
}
}
The log4j2.xml config file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<JSONLayout/>
</Console>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
The output from the above:
{
"timeMillis" : 1510429852038,
"thread" : "main",
"level" : "INFO",
"loggerName" : "example.JsonMessageExample",
"message" : "{\"myKey\":\"myValue\"}",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.logging.log4j.spi.AbstractLogger",
"threadId" : 1,
"threadPriority" : 5
}
As you can see, the message name value pair has a value that is a JSON string. In order to parse this you would have to parse the outer object as JSON, pull the message field and then parse its value as JSON as well.
However, if you use a different layout such as a very basic PatternLayout like this: <PatternLayout pattern="%m%n"/>
You will be able to generate just one level of JSON output and therefore only have to parse once. However, you would have to write your logic to obtain all of the data that you need in your message and stuff it into your map (and JSON object) because now you're simply dumping the contents of the map.
Sample output using same java code with the layout changed to PatternLayout as described above:
{"myKey":"myValue"}
EDIT:
You could even do something like the following if you want to use the "Conversion Patterns" of PatternLayout, output logs in JSON format, and not have to write the logic to obtain some of the specifics:
<PatternLayout>
<pattern>{"timeMillis":"%d{UNIX_MILLIS}","thread":"%t","level":"%p","loggerName":"%c","message":%m}%n</pattern>
</PatternLayout>
Sample output:
{"timeMillis":"1510455694601","thread":"main","level":"INFO","loggerName":"example.JsonMessageExample","message":{"myKey":"myValue"}}

Resources