How to configure Log4j2 Flume appender using system properties - log4j2

Configuring log4j2's flume appender https://logging.apache.org/log4j/2.0/manual/appenders.html#FlumeAppender
for HA seems like a difficult task if you want a generic file that can be used for different clusters. Each flume 'agent' requires its own line in configuration like
<Agent host="192.168.10.101" port="8800"/>
How can I specify agents using system property. It is possible to do so. Also, how can redistribute load across all agents.
LoadBalancingLog4jAppender - a Log4j compliant appender that flume provides seems to solve the problems I am facing but I am really looking for log4j2 solution

Related

Stackdriver adding custom JMX metrics, similar to jvm-sun-hotspot

I have been trying to monitor a custom JMX metrics using stackdriver,
. I saw the below GIT page for fetching MBEAN metrics,
https://github.com/Stackdriver/stackdriver-agent-service-configs/blob/master/etc/collectd.d/jvm-sun-hotspot.conf
I require some additional custom metrics thats needs to be monitored from Stackdriver. Basically, i need
openejb.management:ObjectType=datasources,DataSource=jdbc_db
I tried modifying the conf file from Github by removing the OOb Mbeans and added the custom MBEAN, and i changed the target set to
<Target "set">
MetaData "stackdriver_metric_type" "custom.googleapis.com/jvm/jmx/datasource/jdbc_db"
I see a new metric is getting propagated to Stack driver, but other OOb metrics are affected, Can anyone provide a conf file which i can deplot in /stackdriver/collectd/etc/collectd.d/ location which will sends the custom MBean to Stachdriver

Logging linenumbers with SLF4J and Google Dataflow

When running Apache Beam Google Dataflow jobs and using the SLF4J logger we don't get anything beyond the log message in Stack Driver.
Example of additional information would be function, line number etc.
Is there anyway to configure the logger like a log4j.xml or java logging properties file?
There is no way to customize logs messages in Dataflow other than what is shown in this logging pipelines messages
Have you looked at Cloud Logging? It has several features such as Custom logs / Ingestion API. In case you haven't, take a look at this guide to setup the SLF4J logging facade through Logback appender and Cloud Logging. Once you have configured Logback to use the Cloud Logging, you can use the SLF4J logging API. Another option is to use the Cloud Logging API with a default Java Logging API handler, which can be added programmatically or by using a configuration file, here is an example using logger.
Isaac Miliani, I tried the same option google cloud logging as provided in the google cloud docs,
Added logback.xml to src/main/resources (classpath).
Created loggingeventenhancer and enhancer class to add new labels.
Added markers to logger error, to find the type of error in Stackdriver.
But the logs in stackdriver doesnt have new labels added via logging appender. I think the logback.xml is not found by the maven compile command to deploy the job in dataflow.
Can you provide whats going on wrong here?

How are WorkerHarnessThreads managed in Cloud Dataflow?

Is the option numberOfWorkerHarnessThreads used by cloud-dataflow runner now?
Earlier the PipelineOptions property numberOfWorkerHarnessThreads was specified in the doc and was displayed in Dataflow Job Monitoring UI under Pipeline options. Both are missing now.
If this is not used, how are the worker threads managed now?
The option is still there. You can find it in DataflowPipelineDebugOptions.

In bluemix liberty apps how to use the Environment variables in server.xml

I have a Bluemix application pushed as a liberty buildpack. I would like to set some values in the server.xml, which are set using Environment variables for the app. Following is the snippet from my server.xml on how I tried to access the Environment variable. It is not working as expected. Please let me know if I miss something here.
<server>
<jndiEntry jndiName="myKey" value="${process.env.MY_ENV_VARIABLE}"/>
</server>
Also, if there is any other approach please suggest. Thank you.
I believe it should just be
<server>
<jndiEntry jndiName="myKey" value="${env.MY_ENV_VARIABLE}"/>
</server>
If you're using Liberty on Bluemix, often it's not necessary to configure things like JNDI entries, because auto configuration will automatically add entries for many types of bound services (such as databases and SSO).
Another nice practice for configuring Bluemix apps (which may or may not be right for this use case) is to use user-provided services for deployment-specific information. Your application would then parse the configuration information out of the VCAP_SERVICES environment variable.

How to properly configure Cygnus?

I was playing a bit with Cygnus, and I was wondering how to properly configure it. I’ve seen both agent_<id>.conf and cygnus_instance_<id>.conf files are needed. I understand the purpose of the first one, but not the second one. In addition, what about the grouping_rules.conf file? Is there any other configuration file?
File by file:
agent_<id>.conf file is the main configuration file for Cygnus. It inherits the syntax from Apache Flume, the base technology used by Cygnus, thus it is used to declare and specify the sources, channels and sinks of your Cygnus agent.
cygnus_instance_<id>.conf file is used to configure other Cygnus parameters that cannot be configured as part of the agent configuration, such as the logging file, the management interface port, etc. Cygnus service will run as many instances as cygnus_instance_<id>.conf files are configured. That's why an <id> must be provided, since this <id> will be used to find the proper agent_<id>.conf file.
grouping_rules.conf file is used when the Grouping Rules advanced feature is wanted to be used. Usually, this file may be empty (but it must exist) and Cygnus will run after all.
flume-env.sh file has been inherited from Apache Flume, and it is used in order to configure certain Flume paramters such as the classpath overwritting the default one, some Java options (-Xms, -Xmx, etc)...

Resources