When I run flume using the command :
bin/flume-ng agent --conf conf --conf-file flume.conf --name agentName -Dflume.root.logger=INFO,console
it runs listing all its log data on the console. I would like to store all this log data (flume's log data) in a file. How do I do it?
You need to make a custom build of Flume which uses log4j2.
You configure log4j2 to use a rolling file appender that rolls every minute (or whatever the latency is that you desire) to a spooling directory.
You configure Flume to use a SpoolingDirectorySource against that spooling directory.
You can't use a direct Flume appender (such as what's in log4j2) to log Flume because you will get into deadlock.
You can't use log4j1 with a rolling file appender because it has a concurrency defect which means it may write new messages to an old file and the SpoolingDirectorySource then fails.
I can't remember if I tried the Log4j appender from Flume with this setup. That appender does not have many ways to configure it and I think it will cause you problems if the subsequent agent you're trying to talk to is down.
Another approach might be to patch log4j1 and fix that concurrency defect (there's a variable that needs to be made volatile)
(Yes, setting this up is a little frustrating!)
dont run with -Dflume.root.logger=INFO,console ,then flume will log in ./logs
Related
How can I tell if the settings files associated with a Mosquitto instance, have been properly applied?
I want to add a configuration file to the conf.d folder to override some settings in the default file, but I do not know how to check they have been applied correctly once the Broker is running.
i.e. change persistence to false (without editing the standard file).
Test it.
You can run mosquitto with verbose output enabled, which will generally give you feedback on what options were set, but don't just believe that.
To do that, stop running Mosquitto as a service (how you do this depends on you setup) and manually run it from the shell with the -v option. Be sure to point it at the correct configuration file with the -c option.
That's not enough to be sure that it's actually working properly. To do that you need to test it.
Options have consequences or we wouldn't use them.
If you configure Mosquitto to listen on a specific port, test it by trying to connect to that port.
If you configure Mosquitto to require secure connections on a port, test it by trying to connect to the port unsecured (this shouldn't work) and secured (this should work).
You should be able to devise relatively simple tests for any options you can set in the configuration file. If you care if it's actually working, don't just take it on faith; test it.
For extra credit you can bundle the tests up into a script so that you can run an entire test suite easily in the future and test your Mosquitto installation anytime you make changes to it.
Having duplicate configuration options with different values is a REALLY bad idea.
The behaviour of mosquitto is not defined in this case, which value should be honoured, the first found, the last? When using the conf.d directory, what order will the files be loaded in?
Also will you always remember that you have changed the value in a conf.d file in the future when you go looking?
If you want to change one of the defaults in the /etc/mosquitto/mosquitto.conf file then edit that file. (Any sensible package management system will notice the file has been changed and ask what to do at the point of upgrade)
The conf.d/ directory is intended for adding extra listeners.
Also be aware that there really isn't a default configuration file, you must always specify a configuration file with the -c command line option. The file at /etc/mosquitto/mosquitto.conf just happens to be the config file that is passed when mosquitto is started as a service when installed using most Linux package managers. (The default Fedora install doesn't even setup the /etc/mosquitto/conf.d directory)
I do something like this
-javaagent:/usr/local/lib/perfino/perfino.jar=server=ybperfino,name=${HSTNAMESHORT}-${APPNAME},group=${YBENV}/${HSTNAMESHORT},logMBean=10,logFile=${LOG_DIR}/perfinologs/${HSTNAMESHORT}-${APPNAME}.log
basically I want the log files to be created in the log directory for the app not the home directory for the userid
but it seems like the log file isn't being created either with logfile argument or with out !
using java11 if that makes any difference.
Found the answer - I had a competing java agent that was loading before it.
After I changed the order both java agents worked.
I have changed our Jenkins setup from everything running on one machine to a master-agent/slave setup. Before that everything worked fine, now I am facing issues that some programs I am calling that access files can't find them.
Case 1:
(Pls don't ask why it is so complicated, but the file structure is given and I can't change it)
I am calling a python script, that iself calls a batch file
filepath= os.path.abspath(os.path.join(pamFolder, "run.bat"))
p = subprocess.Popen(filepath, cwd=pamFolder, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
... and the batch file again a jar file with the actual program
java -XX:-UseGCOverheadLimit -cp "../..;../../libs/*" -jar ..\..\myjarfile.jar
Within the jar file there is an access to a file on the disk failing with an error message that the file can't be found:
ERR : The file was not found in the specified path 'U:\somefile.txt'. Please check this path for access and your configuration!
Case 2:
I am calling a batch file from Jenkins that is calling some other exe and in the end trying to open a file in Excel via the COM interface. Here I am getting the following exception (Excel can't access the file):
Unhandled Exception: System.Runtime.InteropServices.COMException: Microsoft Excel kann auf die Datei 'D:\Jenkins\workspace\myJob\someDir\someFile.xlsm' nicht zugreifen.
Question
As previously mentioned, both jobs were working in the previous setup. Both files DO exist.
I suspect that Jenkins / the programs are trying to find the files on the master where they are not available.
Is there any way to tell Jenkins that the called tools are fully executed on the slave node or in some other way tell them where to find these files?
EDIT
The job is already running on the slave. The console shows Running on [slave name] in D:/Jenkins/workspace/xxxxx.
The master is configured in a way that only jobs assigned to it run on the master. So pretty much all jobs should run on the slave.
EDIT2 / SOLUTION
It turned out that the 2 issues are caused by different things.
Case1: Solved this by using the UNC path
Case2: Solved by a mixture of giving the necessary permissions as described here and starting the slave service with a user with admin rights.
From my experience with this issue, usually it has to do with your SCM setup.
But as you stated that the files DO exist I think there might be the possibility that U:\ is a network share? Then consider changing your path to use a UNC path.
If that's not the case check if your jenkins slave as sufficient user rights to access said file.
You can tell Jenkins to run the job on the designated slave as follows:
Under Nodes > [SLAVE] > Configure, specify a label for the slave.
Under [Job] > Configure > Restrict where this project can be run, enter the label.
Now when you build, the console output of the job should read correctly along the lines Running on [SLAVE] (build_agent_01) in C:/jenkins and the files must be accessible.
I have a GUI that I package into a Window exe using install4j. In my GUI code, I used log4j2, specifying SYSTEM_OUT for STDOUT. However, when I run the exe in Window, I do not see any logging. I understand that the exe is not running in a console, but I don't understand how I can configure it to do what I want.
I tried to redirect stdout and stderr to a file, but that file is not created.
I know my log4j configuration works, because my GUI can run under JavaWebStart with no problem.
The redirection of stdout and stderr will not interfere with log4j or any other logging system unless you only log to the console.
In the log4j configuration, You have to configure an appender that logs to a writable file. If you install to the program files folder, the file cannot be in the installation directory because you will likely not have elevated privileges when running your application. Log to some folder in %APPDATA% instead.
I am running Jenkins from source using mvn jenkins-dev:run. It seems to be logging at the DEBUG level making page load very slow. How do I change the log level of Jenkins. I have tried the usual util.logging properties file, but I can't get it working.
mvn jenkins-dev:run -Dorg.slf4j.simpleLogger.defaultLogLevel=error
Default log level for all instances of SimpleLogger. Must be one of ("trace", "debug", "info", "warn", or "error"). If not specified, defaults to "info".
You can also edit ${MAVEN_HOME}/conf/logging/simplelogger.properties to make it consistent.
More info here: http://maven.apache.org/maven-logging.html
EDIT:
To also control the log level of the Jenkins war itself as the OP wanted i was able to do so using:
mvn jenkins-dev:run -Djava.util.logging.config.file=my-logging.properties
The contents of my-logging.properties are:
handlers = java.util.logging.FileHandler, java.util.logging.ConsoleHandler
.level= SEVERE
Now i only see two INFO messages comming from jetty . One could also configure the jetty logger via the same file if needed, i will not go into more detail about how to do this because i have little experience with it, but if i could take a guess, you would have to manipulate the correct class (eg: org.eclipse.jetty.LEVEL=WARN) used by Jenkins when embedding jetty.