I'm exploring Log4j 2.14.0 and SLF4j 2.0 and trying to generate structured messages.
I've got my Appender set up with a slightly modified LogstashJsonEventLayoutV1.json,
<JsonTemplateLayout eventTemplateUri="classpath:LogstashJsonEventLayoutV1-test.json" properties="true" />
where I've removed the timestamp and hostname(I'm doing this as part of a unit test) and modified the config for "message" like so:
"message": {
"$resolver": "message",
"fallbackKey": "formattedMessage"}
When I log something
log4jLogger.atInfo().log(new MapMessage(Map.of("hello", "world")));
It's obviously generating JSONified log messages:
{"#version":1,"message":{"hello":"world"},"thread_name":"Test worker","level":"INFO","logger_name":"java.lang.Integer"}
In production my shop generally uses Log4J via SLF4J. I'd be willing to use the 2.0.0-alpha1 release of SLF4J to achieve this goal. How would I achieve the same thing via SLF4J's fluent API via addKeyValue?
logger.atDebug().addKeyValue("oldT", oldT).addKeyValue("newT", newT).log("Temperature changed.");
At the end of the day I just wrapped log4j--for this use case, there was no manna to be had for wrapping Slf4j when I could just target log4j.
Related
I h got an issue with the following serverless config.
this is my handler and the files/folders structure.
the issue is that after uploading my project to AWS when I test my lambda I got an error as follows:
lambda execution fails: "errorMessage": "Unable to import module 'app_monitor': No module named 'monitoring'"
{
"errorMessage": "Unable to import module 'src/app_monitor': No module named 'monitoring'",
"errorType": "Runtime.ImportModuleError",
"requestId": "bca3f67d-815f-452b-a2a6-c713ad2c6baa",
"stackTrace": []
}
have you got any clue how can I add this into serverless config.?
First, a quick tip on troubleshooting: When I ran into such issues it was helpful to go to the AWS console, look at the lambda function, and see what the uploaded file structure looks like on that end. Is the monitoring folder there?
Moreover, in order to specify how a specific function is packaged, you have to explicitly state that you want it to be individually packaged and not follow the general rules of the project as a whole.
You should try to add:
app_monitoring:
package:
individually: true
patterns:
- 'src/**'
More documentation on packaging configuration here
You may also have better luck with explicitly stating the patterns you need, I know I've had issues with globs in the past. So for example you can try:
patterns:
- 'src/app_monitoring.py'
- 'src/monitoring/get_lb.py'
We are trying to use our logback.xml that we use in GCP Cloud run which has amazing filtering features. Our logback.xml contains this for cloud run
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.orderlyhealth.api.logging.logback.GCPCloudLoggingJSONLayout">
<pattern>${CONSOLE_PATTERN}</pattern>
</layout>
</encoder>
</appender>
And our GCPCloudLoggingJSONLayout does a great job at setting all the things we need like clientId, customerRequestId, etc. etc. and we can filter across many many microservices on one customer or one customer request. We lose this in dataflow currently though. We tried adding logback.xml to src/main/resources and deploying the project seems to use it in the shell like so
{"message":"[main][-][:] o.a.b.r.d.DataflowRunner Template successfully created.\n",
"logger":"org.apache.beam.runners.dataflow.DataflowRunner",
"transactionId":null,"socket":null,"clntSocket":null,
"version":null,
"timestamp":{"seconds":1619694798,"nanos":4000000},
"thread":"main",
"severity":"INFO",
"instanceId":null,
"headers":{},
"messageInfo":{"message":"Message short enough. Displayed top level"}
}
thanks for any ideas on modifying dataflow logging.
Currently we see this instead which is not nearly as useful for tracing the customer request through systems
I don't think you can change how Dataflow logs to Cloud logging.
Instead, you can change how/what you log and let Dataflow pass them through to cloud logging. See Logging pipeline messages.
Or you can use cloud logging client libraries in your pipeline directly: https://cloud.google.com/logging/docs/reference/libraries.
Please take a look at How to override Google DataFlow logging with logback? for the latest version of this answer
I copied the current answer there to make it easier for folks who want to look:
Dataflow relies on using java.util.logging (aka JUL) as the logging backend for SLF4J and adds various bridges ensuring that logs from other libraries are output as well. With this kind of setup, we are limited to adding any additional details to the log message itself only.
This also applies to any runner executing a portable job since the container with the SDK harness has a similar logging configuration. For example Dataflow Runner V2.
To do this we want to create a custom formatter to apply to the root JUL logger. For example:
public class CustomFormatter extends SimpleFormatter {
public String formatMessage(LogRecord record) {
// implement whatever logic the is needed to add details to the message portion of the log statement
return super.formatMessage(record);
}
}
And then during start-up of the worker we need to update the root logger to use this formatter. We can achieve this using a JvmInitializer and implement the beforeProcessing method like so:
#AutoService(JvmInitializer.class)
public class LoggerInitializer implements JvmInitializer {
public void beforeProcessing(PipelineOptions options) {
LogManager logManager = LogManager.getLogManager();
Logger rootLogger = logManager.getLogger("");
for (Handler handler : rootLogger.getHandlers()) {
handler.setFormatter(new CustomFormatter());
}
}
}
I've configured my Grails app to read the log4j config from /conf/log4j.properties file instead of the more usual DSL in Config.groovy, by adding the following Spring bean:
log4jConfigurer(MethodInvokingFactoryBean) {
targetClass = "org.springframework.util.Log4jConfigurer"
targetMethod = "initLogging"
arguments = ["/conf/log4j.properties", 1000 * 60] // 2nd arg is refresh interval in ms
}
My goal is to log all the classes in the app itself at the DEBUG level, and all others at the WARN level. /conf/log4j.properties contains the following:
log4j.logger.com.myapp=DEBUG, CONSOLE
log4j.logger.grails.app=DEBUG, CONSOLE
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%-5p myapp %c{3} %d{HH:mm:ss,SSS} %t : %m%n
It seems the namespace com.myapp is used for regular classes in my app (e.g. those under src/groovy), whereas the namespace grails.app is used for Grails artefacts (controllers, services, taglibs, etc.). However the grails.app namespace also includes artefacts from plugins, which I don't want to log at the DEBUG level.
Is there a way to enable DEBUG logging only for the classes in my application?
You append your package unto the grails.app.controllers to get just your application.
info 'grails.app.controllers.mypackage'
Adding the following solved the problem
log4j.logger.grails.app.conf.com.myapp=DEBUG, CONSOLE
log4j.logger.grails.app.filters.com.myapp=DEBUG, CONSOLE
log4j.logger.grails.app.taglib.com.myapp=DEBUG, CONSOLE
log4j.logger.grails.app.services.com.myapp=DEBUG, CONSOLE
log4j.logger.grails.app.controllers.com.myapp=DEBUG, CONSOLE
log4j.logger.grails.app.domain.com.myapp=DEBUG, CONSOLE
Presumably if a Grails app has other types of artefacts that you want to log at the DEBUG level, an additional logger should be configured for each one.
I'm using the Grails routing plugin which allows defining Camel routes with a Groovy DSL syntax very similar to the Java DLS syntax.
Suppose I have the following RouteBuilder:
class MyRoute extends RouteBuilder {
from('activemq:route1')
.to('someProcessor1')
.to('direct:route2')
from('direct:route2')
.to('someProcessor2')
onException(Throwable.class).useOriginalMessage().handled(true)
.to('activemq:route.failed')
}
If I have a message that starts at activemq:route1, then moves through direct:route2 but fails in the someProcessor2, then I end up with the message as it started at activemq:route1 in my activemq:route.failed queue... but that's not what I want. If I have a failure in someProcessor2, I want the message as it started at direct:route2 (and likewise, if I have a failure in someProcessor1, I want the activemq:route1 message in my failed queue).
Is there any Apache Camel feature that allows me to "reset" the original message at the beginning of a RouteDefintion (i.e. from(<uri>))?
use something besides direct: to join your routes (seda, vm, activemq) and it will behave as you suggested...otherwise, you can also explicitly preserve the relevant state of the message in a header and restore it in the onException clause, etc.
Using a standard log4j configuration for my grails app, with a custom conversion pattern like that :
log4j = {
appenders {
console name:'stdout', layout:pattern(conversionPattern: '[%-7p][%d{dd/MM/yyyy HH:mm:ss,SSS}] %C %m%n')
}
root {
warn 'stdout'
additivity = true
}
error 'org.grails.plugins.springsecurity'
error 'org.codehaus.groovy.grails.web.servlet' // controllers
// ...
warn 'org.mortbay.log',
'org.apache.tomcat',
'org.apache.tomcat.util.digester'
debug 'grails.app'
}
My grails app start as expected .. with the good conversionPattern ... but only during few log lines ... to finally fallback to the default grails conversionPattern ... :-/
Any idea ?
I don't code in Grails but I do know log4j very well.
On the surface, it seems you need to inspect those lines that aren't formatted as expected. Chances are, they are not caught by the logger that uses your stdout appender.
From what I can piece together, it looks to me like maybe your warning logger is the only one that uses your stdout appender. Meaning anything other than warnings would not format as expected. Further, it's also possible that loggers present in your libraries catch some log statements but not others.
Basically, the best way to solve this is to modify your pattern to give you good information on your loggers (I would suggest replacing the %C pattern, which is very slow to execute, with %c to see the exact category that was used by the logger). Then, look at the differences between what's properly formatted and everything else.
Do they have a common level (info, debug, error, etc)?
Do they stem from the same package?
Are they 3rd party library calls?
Figure out what they have in common and you will find your error.