When running Apache Beam Google Dataflow jobs and using the SLF4J logger we don't get anything beyond the log message in Stack Driver.
Example of additional information would be function, line number etc.
Is there anyway to configure the logger like a log4j.xml or java logging properties file?
There is no way to customize logs messages in Dataflow other than what is shown in this logging pipelines messages
Have you looked at Cloud Logging? It has several features such as Custom logs / Ingestion API. In case you haven't, take a look at this guide to setup the SLF4J logging facade through Logback appender and Cloud Logging. Once you have configured Logback to use the Cloud Logging, you can use the SLF4J logging API. Another option is to use the Cloud Logging API with a default Java Logging API handler, which can be added programmatically or by using a configuration file, here is an example using logger.
Isaac Miliani, I tried the same option google cloud logging as provided in the google cloud docs,
Added logback.xml to src/main/resources (classpath).
Created loggingeventenhancer and enhancer class to add new labels.
Added markers to logger error, to find the type of error in Stackdriver.
But the logs in stackdriver doesnt have new labels added via logging appender. I think the logback.xml is not found by the maven compile command to deploy the job in dataflow.
Can you provide whats going on wrong here?
Related
Not able to enable the debug|trace level logging of the dataflow workers.
The documentation :https://cloud.google.com/dataflow/docs/guides/logging#SettingLevels
indicates the usage of DataflowWorkerLoggingOptions to programmatically overrides the default log level on the worker and enable the debug|trace level logging; however the interface is deprecated and no more present in bean-sdk 2.27.0 .
Has anyone been able to enable the worker level debugging in cloud dataflow; in any way.
The documentation is still up to date and the interface is still present and will work.
The interface is deprecated because the Java-based Dataflow worker is not used when running a pipeline using Beam's portability framework. Quoting the deprecation message:
#deprecated This interface will no longer be the source of truth for worker logging configuration once jobs are executed using a dedicated SDK harness instead of user code being co-located alongside Dataflow worker code. Please set the option below and also the corresponding option within org.apache.beam.sdk.options.SdkHarnessOptions to ensure forward compatibility.
So what you should do is follow the instructions that you linked and also set up logging in SdkHarnessOptions.
I have been trying to monitor a custom JMX metrics using stackdriver,
. I saw the below GIT page for fetching MBEAN metrics,
https://github.com/Stackdriver/stackdriver-agent-service-configs/blob/master/etc/collectd.d/jvm-sun-hotspot.conf
I require some additional custom metrics thats needs to be monitored from Stackdriver. Basically, i need
openejb.management:ObjectType=datasources,DataSource=jdbc_db
I tried modifying the conf file from Github by removing the OOb Mbeans and added the custom MBEAN, and i changed the target set to
<Target "set">
MetaData "stackdriver_metric_type" "custom.googleapis.com/jvm/jmx/datasource/jdbc_db"
I see a new metric is getting propagated to Stack driver, but other OOb metrics are affected, Can anyone provide a conf file which i can deplot in /stackdriver/collectd/etc/collectd.d/ location which will sends the custom MBean to Stachdriver
The Citrus Framework documentation indicates that integration test console output can be logged via the SLF4J logging system. It's not obvious whether this is automatic, or whether it needs to be enabled in some way. My experience indicates that it's not enabled as no log file containing what appears on the test run console has been produced.
My application uses Log4J with an associated log4j2-spring.xml file (in src/main/resources) to define log formats and files. When an integration test is run via Citrus, the application console output (and other information) is properly logged to the files specified in the Log4J config file. There is however, no Citrus console output logged anywhere (except on the console).
How do I enable the logging of the Citrus test console output? I created a separate log4j.xml that I placed in src/test/resources, but this seems to have been ignored.
Do I need to specify a separate logger in the config that's specific for Citrus output?
Citrus uses SLF4J which is a facade to several other logging frameworks out there. So you need to pick your favorite logging framework (in your case Log4J) and add a SLF4J logger binding for that logging framework. This is all described in the SLF4J user manual
I would suggest to add the SLF4J logger binding for Log4J as a test scoped dependency in your project. Also depending on your Log4J configuration setup you may need to add a Log4J logger configuration for com.consol.citrus and set a proper logging level for that in order to see the Citrus output logged by Log4J.
I can view the task manager's logs from web ui:
But suppose I submit a job, and use log4j2 to write some logs to a specific file, can I make some configurations so I can view the logs from web ui?
In the script :
$FLINK_HOME/bin/flink
You will find these 2 lines:
log=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-client-$HOSTNAME.log
log_setting=(-Dlog.file="$log" -Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j-cli.properties -Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml)
Either you configure log4j to store the logs in flink path, i.e: $FLINK_HOME/log, or modify $FLINK_HOME/bin/flink script by adding log4j logs path location.
Hope it helps :) !
Configuring log4j2's flume appender https://logging.apache.org/log4j/2.0/manual/appenders.html#FlumeAppender
for HA seems like a difficult task if you want a generic file that can be used for different clusters. Each flume 'agent' requires its own line in configuration like
<Agent host="192.168.10.101" port="8800"/>
How can I specify agents using system property. It is possible to do so. Also, how can redistribute load across all agents.
LoadBalancingLog4jAppender - a Log4j compliant appender that flume provides seems to solve the problems I am facing but I am really looking for log4j2 solution