In our current implementation one of the servers has the configuration for log4j2 set for RollingRandomAccessFile and in the other RollingFile. I would like to know what is the difference between the two and pros and cons of each if possible.
Thanks
The main difference is performance:
http://logging.apache.org/log4j/2.x/manual/async.html#FileAppender_vs._RandomAccessFileAppender
RandomAccessFileAppender is always buffered, while FileAppender provides a config switch (bufferedIO). Both have an "immediateFlush" config option in case you want to be sure your messages are on disk (e.g. audit logging). Finally, the default buffer size for RandomAccessFileAppender is larger: 256*1024 bytes vs 8*1024 bytes for FileAppender (both appenders buffer size can be set in configuration).
One more thing, the underlying implementation uses a RandomAccessFile in one case and an OutputStream in the other case (as the name suggests). There is a known issue with combining RandomAccessFileAppender with the unix logrotate utility (https://issues.apache.org/jira/browse/LOG4J2-354), and the Log4j team recommends you use the RollingRandomAccessFile for rollover and avoid use of the unix logrotate utility. Apparently FileAppender can be combined with logrotate - we haven't heard of any issues. (Since the question already mentions the Rolling... variant for both appenders you are already doing the right thing, I just thought I should mention it.)
Related
My environment : JBoss EAP 7 with a "log4j2.xml" in classpath (historic behavior).
I would like to introduce a way to have a (non mandatory) custom log4j2 configuration file (per EAR application) but still use (fall back) to (existing) "log4j2.xml" if the custom configuration file is missing.
To me, the only way to accomplish this was to use composite configuration by using "log4j2.configurationFile" property (within log4j2.component.properties) and set both the "log4j2.xml" and the custom configuration filename (separated by a comma).
But if the custom file is missing, even the generic "log4j2.xml" is ignored.
When looking at log4j2 (v2.12.1) code (https://github.com/apache/logging-log4j2/blob/log4j-2.12.1/log4j-core/src/main/java/org/apache/logging/log4j/core/config/ConfigurationFactory.java#L380) I can see that indeed if one config file is missing, none config file (of the list) is used (-> "return null")
Is there a way to accomplish the behavior I want?
Thanks
in storm-0.10.0 when I config the cluster.xml, I use ${storm.home}/logs/${logfile.name} like storm-0.9.0, but the value ${storm.home} and ${logfile.name} not identified, why?
Log4j2 has many different kinds of lookups. They are distinguished by their prefix. For system properties the prefix is sys:.
So try ${sys:storm.home}/logs/${sys:logfile.name}.
See also http://logging.apache.org/log4j/2.x/manual/lookups.html for more information.
Is it possible for HDFS Flume sink to roll whenever a single file (from a Flume source, say Spooling Directory) ends, instead of rolling after certain bytes (hdfs.rollSize), time (hdfs.rollInterval), or events (hdfs.rollCount)?
Can Flume be configured so that a single file is a single event?
Thanks for your input.
Reagarding your first question, it is not possible due to the sinks logic is disconnected from the sources logic. I mean, a sink only sees events being put into the channel which must be processed by him; the sink does not know if an event is the first or the last regarding a file.
Of course, you could try to create your own source (or extend an existing one) in order to add a header to the event with a value meaning "this is the last event". Then, another custom sink could behave depending on such a header: for instance, if the header is not set, then the events are not persisted but stored in memory until the header is seen; then all the information is persisted in the final backend as a bach. Other possibility is that custom sink persists the data in a file until the header is seen; then the file is closed and another one is opened.
Regarding your second question, it depends on the sink. The spooldir source behaves based on the deserializer parameter; by default its value is LINE, what means:
Specify the deserializer used to parse the file into events. Defaults to parsing each line as an event. The class specified must implement EventDeserializer.Builder.
But other custom Java classes can be configured, as said above; for instance, a deserialized for the whole file.
You can set rollsize to a small number combined with BlobDeserializer to load file by file instead of combining into blocks. This is really helpful when you have unsplittable binary files such as PDF or gz files.
This is part of the configuration that is relevant:
#Set deserializer to BlobDeserializer and set the maximum blob size to be 1GB.
#Notice that the blobs have to fit in memory so this doesn't work for files that cannot fit in memory.
agent.sources.spool.deserializer = org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder
agent.sources.spool.deserializer.maxBlobLength = 1000000000
#Set rollSize to 1024 to avoid combining multiple small files into one part.
agent.sinks.hdfsSink.hdfs.rollSize = 1024
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.rollInterval = 0
The answer to the question "Can Flume be configured so that a single file is a single event?" is yes.
Yo only have to configure the following property to be 1:
hdfs.rollCount = 1
I'm looking for a solution for your first question, because sometimes the file is too big and it's needed to split the file in several chunks.
You can use any event headers in hdfs.path. ( https://flume.apache.org/FlumeUserGuide.html#hdfs-sink )
If you are using Spooling Directory Source, you can enable putting the file name in the events using fileHeaderKey or basenameHeaderKey ( https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source ).
Can Flume be configured so that a single file is a single event?
It could be, however it is not recommended. The underlying implementation (protobuf) limits file sizes to 64m. Flume events are to be small in size due to its architecture and design. (Fault-tolerance, etc.)
I'm running Ant with output fed to a log file:
ant -logfile file.txt target-name
I'd also like to print some simple progress information to the console though. The answer seems to be a BuildEvent listener that writes to the console every time a new target is hit, but the documentation explicitly states:
A listener must not access System.out and System.err directly since ouput on these streams is redirected by Ant's core to the build event system.
Did I miss something? Is there a way to do this?
Ant replaces the System.out & System.err streams to remap messages printed there through it's own logging system.
That said, you can still get access to the ACTUAL OS streams by using java.io.FileDescriptor#out
Actually, the answer is Log4jListener.
There is a sample log4j configuration for logging into both console and file shown in the above link. You can then use an <echo> task with an appropriate level parameter to selectively decide what gets printed to console.
Thanks for the answers! I'm slow, but this is still something that I'd like to get right.
I've managed to get something working more or less like I want using carej's suggested approach with the java.io.FileDescriptor#out stream and an Ant scriptdef like this:
<scriptdef name="progress-text" language="javascript" >
output = new java.io.PrintStream(new java.io.FileOutputStream(java.io.FileDescriptor.err))
output.println(self.text)
</scriptdef>
Now I'm just left wondering how wize is this approach? Is there inherit risk in using the underlying OS streams directly?
EDIT:
2 Points which might be useful to anyone else with a similar question:
This article has a very good description of the Ant I/O system: http://codefeed.com/blog/?p=68
java.lang.System does something very similar to set System.out and System.err in the first place.
All of this gave me a little more confidence in this approach.
In most modern IDEs there is a parameter that you can set to ensure javac gets enough heap memory to do its compilation. For reasons that are not worth going into here, we are tied for the time being to JBuilder 2005/2006, and it appears the amount of source code has exceeded what can be handled by javac.
Please keep the answer specific to JBuilder 2005/2006 javac (we cannot migrate away right now, and the Borland Make compiler does not correctly support Java 1.6)
I realize how and what parameters should be passed to javac, the problem is the IDE doesn't seem to allow these to be set anywhere. A lot of configuration is hidden down in the Jbuilder Install\bin*.config files, I feel the answer may be in there somewhere, but have not found it.
did you find a good solution for that problem?
I have the same problem and the only solution I found is the following:
The environment variable JAVA_TOOL_OPTIONS can be used to provide parameters for the JVM.
http://java.sun.com/javase/6/docs/platform/jvmti/jvmti.html#tooloptions
I have created a batch file "JBuilderw.bat" with the following content:
set JAVA_TOOL_OPTIONS=-Xmx256m
JBuilderw.exe
Each time I start JBuilder using this batch file the env.var. JAVA_TOOL_OPTIONS will be set and javac.exe will receive the setting.
The JVM displays at the end the following message: "Picked up JAVA_TOOL_OPTIONS: -Xmx256m"
Drawback: all virtual machines started by JBuilder will get that setting. :(
Thanks,
JB
Have a look at http://javahowto.blogspot.com/2006/06/fix-javac-java-lang-outofmemoryerror.html
The arguments that you need to pass to JBuilder's javac is "-J-Xms256m -J-Xmx256m". Replace the 256m with whatever is appropriate in your case. Also, remove the quotes.
This should work for java 1.4, java 1.5 and forward.
BR,
~A
"I realize how and what parameters should be passed to javac, the problem is the IDE doesn't seem to allow these to be set anywhere."
I realized now that you know how to pass the right arguments ONLY not where/how to pass those arguments :-(
How about this : Can you locate where is the JAVA_HOME/bin directory that borland uses ? If yes, then you can rename the javac.exe(to say javacnew.exe) with a javac.bat which in turn will call the javacnew.exe (as well as pass the required arguments) ?
I don't know if this will help since I don't use Borland but in Eclipse, this is a setting that you attach to the program you're going to run. Each program you run in the IDE has configuration specific to it, including arguments to the VM. Is there something like that?
Do you have a jdk.config file located in JBuilder2005/bin/?
You should be able to modify vm parameters in that file like:
vmparam -Xms256m
vmparam -Xmx256m
Let me know if this works, I found it on a page talking about editing related settings in JBuilder 2005.
Edit the jbuilder.config file.
Put in comment those two lines:
vmmemmax 75%
vmmemmin 32m
has they ought to be <1Gb and with a > 1Gb PC , 75% is too big?