I have a jenkins master-slave setup through JNLP connections. Everything is working fine except I can not find any logs on the slave nodes. There are logs on the master in $JENKINS-HOME/logs/slaves but none on the slave node.
Can you tell me on which path the log is or if there is even logging on the slave node?
Thank you very much!
Q
Jenkins stores all logs on master only, that's why you cannot find any log on nodes.
On Windows, the slave stores error logs in the same folder as the slave.jar file.
This reports things like:
Dec 19, 2018 2:38:14 PM hudson.remoting.Engine waitForServerToBack
"INFO: Failed to connect to the master. Will retry again".
That message will never appear be uploaded to Master.
I would like to see a similar log file on other slaves.
They are mostly transferred to Master by TCP.
For example, when a step starts, like a shell task, will do like this
call your shell content
# your script will be transform into a script file
script.sh > jenkins-log.txt
# running...
# after running
echo $? > jenkins-result.txt
during the running progress, data will be transport by TCP(pull or push)
jenkins-log.txt -> Filestream -> RemoteStream -> Master
and in master, you will see single log like it
jobs/xxx/branch/master/<id>/log
When job is done, master will send the command to clean the temp dir in the agent, so you can't see anything about logs.
One more thing, in our company, we are facing the problem of too much logs are being sent to Master like a DDOS, so a simple way to solve is to add a pipeline after the shell
limit by tail
xxx | tail -c 512k
or limit the size by head command
xxx | (head -n 1000;dd status=none of=/dev/null)
Related
I am trying to push our jenkins build logs to S3.
I used Groovy plugin and the following script in Build phase
// This script should be run in a system groovy script build step.
// The FilePath class understands what node a path is on, not just the path.
import hudson.FilePath
// Get path to console log file on master.
logFile = build.getLogFile()
// Turn this into a FilePath object.
logFilePathOnMaster = new FilePath(logFile)
logFileName = build.envVars["JOB_BASE_NAME"] + build.envVars["RT_RELEASE_STAGING_VERSION"] + '.txt'
// Create remote file path obj to build agent.
remoteLogFile = new FilePath(build.workspace, logFileName)
// Copy contents of master's console log to file on build agent.
remoteLogFile.copyFrom(logFilePathOnMaster)
And then I am using S3 plugin to push .txt files to S3.
But this script fetches the build log file from the master node.
How are the build logs transferred from slave to master node ?
Can I access the build log file on my slave node without master's involvement at all ?
The slave node must be preserving the build logs while building somewhere ? I cant seem to find it.
I am not much familiar with Groovy but here is the solution which worked for me using shell script.
I am using Jenkins 'Node and Label parameter plugin' to run our java process on a slave node. Job is triggered using 'Build >> Execute Shell' option. The log is collected into a file as below :
sudo java -jar xxx.jar | sudo tee -a ${JOB_NAME}/${BUILD_NUMBER}.log 2>&1
This log file is then pushed to S3 :
sudo aws --region ap-south-1 s3 cp ${JOB_NAME}/${BUILD_NUMBER}.log s3://bucket/JenkinsLogs/${JOB_NAME}/${BUILD_NUMBER}.log
Its working perfectly for us. Hope it helps you too.
Jenkins v2.89.3.
I'm trying to start an agent but am getting the following error.
$ java -jar agent.jar -jnlpUrl http://localhost:8080/computer/testagent/slave-agent.jnlp
Error: Unable to access jarfile agent.jar
The documentation from the wiki that I'm following is from here:
Launch agent headlessly -
https://wiki.jenkins.io/display/JENKINS/Distributed+builds#Distributedbuilds-Differentwaysofstartingagents
The handbook section for managing nodes doesn't appear to have been written yet:
https://jenkins.io/doc/book/managing/nodes/
If anyone know what I'm doing wrong, or another way I can start the agent from the slave (not from the master), please let me know.
In the "Run from agent command line..." CLI instructions that were given by the Jenkins master: The word 'agent.jar' is a hyperlink.
Hover over that hyperlink to copy the target URL.
Use that URL with wget to retrieve agent.jar from the Jenkins master by pasting the URL into your terminal window. e.g.,
[root#Jenkins-Agent-1 ~]# wget http://jm0:8080/jnlpJars/agent.jar
--2022-03-15 01:13:45-- http://jm0:8080/jnlpJars/agent.jar
Resolving jm0 (jm0)... 192.168.0.174
Connecting to jm0 (jm0)|192.168.0.174|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1524968 (1.5M) [application/java-archive]
Saving to: ‘agent.jar’
100%[====================================================================================>] 1,524,968 --.-K/s in 0.04s
And then finish the installation by executing the command that was given by your Jenkins master. e.g.,
[root#Jenkins-Agent-1 ~]# java -jar agent.jar -jnlpUrl http://jm0:8080/computer/JA1/jenkins-agent.jnlp -secret 73393f441b43921357d959cd664d9e69d67ff7ee1f9876070ab76ee46b420e19
I found that the Swarm plugin for Jenkins did exactly what I needed.
Install plugin on Jenkins
Download swarm jar on slave
java -jar path/to/swarm-client.jar -home http://localhost:8080
Result: Slave agent registers on master as seen from the Manage Nodes section of the Jenkins UI
https://wiki.jenkins.io/display/JENKINS/Swarm+Plugin
I use jenkins master-slave configuration for capturing Performance metrics of a product. We have observed that jenkins-slave tends to accumulate memory and thus influences the Performance metrics being captured.
To ensure consistency of the metrics being captured, we are thinking of restarting jenkins slave every day from the master, when there are no jobs running on the slave. Is this feasible?
How can we accomplish it?
Note: Using jenkins-slave as a service is not an option because we are having other security access issues with it.
I know this answer is coming in a bit late :
This is how I did the same for the same reasons, not sure if this is the best way to achieve this, but it solved many of our problems :
For Windows Machines :
Create a job that simply runs "shutdown -r -f" on windows machines.
It will restart the machines.
Now bringing it back online part. For similar reasons as yours, I
didn't use "jenkins-slave as a service". Instead I configured the
nodes to connect via JNLP client, and then added the slave.jar
command for each node in Window's task scheduler (to run on
startup)
Now the job restarts the machine and the Windows machine bring
itself online on Jenkins itself right after restart.
For Mac Machines :
The process is comparatively easier on mac. First, make a job to run
"shutdown -r now" on Mac node
The node should simply be setup to get connected via ssh. That will
take care of bringing it up online on Jenkins.
This was the "execute shell" part of my script to restart all the machines used for our automation :
distro=`uname`
if [ "$distro" = "Windows_NT" ] || [ "$distro" = "WindowsNT" ] ;then
echo "Restarting Windows Machine...."
shutdown -r -f
else
echo "Restarting Mac Machine...."
sudo shutdown -r now
fi
PS:
It's not exactly related to the question, but may be useful for the situation that you specified. It may be a good idea to add a batch script to clean temp files on startup of Windows machines.
Add following to a batch script (Say, cleanTemp.bat) in the startup folder of your Windows machine.
(For Windows 10, C:\Users\\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup)
rmdir %temp% /s /q
md %temp%
If you still need an answer:
https://wiki.apache.org/general/Jenkins#How_do_I_restart_a_Jenkins_Unix_Slave.3F
Although, I just did a disconnect and then I saw that the processes died in the slave. I did not have to kill them manually.
Then launch the slave again and that's it.
This is good from web UI. I have not searched for CLI for this yet.
Create a job e.g. "Reboot-Slave", and set it with shell "shutdown -r -t 0", and take the target slave name as a parameter. (in this way, the restart command will be executed directly on the target slave that you want to restart.)
Create another job e.g. "Reboot-Check-Slave-Online", in this job, you should call the 1st job and pass the target slave name as parameter, plus, you'd better write some logic to determine whether your slave finished the restarting and connected to Jenkins server again, you can implement it by adding an "Execute system groovy script" step in your job and write below code:
import hudson.model.*
def target_slave_param = "target_slave"
def resolver = build.buildVariableResolver
def target_slave = resolver.resolve(target_slave_param)
println "target_slave is: ${target_slave}"
def status = 0;
//do{
println "Searching for ${target_slave}";
slave = Hudson.instance.slaves.find({it.name == target_slave});
if (slave != null)
{
computer = slave.getComputer();
if (computer.isOffline())
{
println "Error! $target_slave is offline.";
status = 1;
}
else
{
println "OK: $target_slave is online";
}
}
else
{
println "Slave $target_slave not found!";
status = 1;
}
//}
Steps:
Install Node and Label parameter plugin
Check This project is parameterized option:
Use following command in Execute shell field:
(sudo bash -c "(sleep 30 && sudo shutdown -r now) &") &
Jenkins job is detached correctly and shows success execution.
I am running a background process through a script , this script is invoked when Jenkin starts building. However, the jenkins build gets stuck and on looking at the console it seems it is running the process and waits for it to complete.
This process will never complete, consider this as a server listening to its client.Every build I trigger kills the server process and restarts the process, so I am perfectly handling that scenario.
Is there any way , I can build jenkins successfully?
The exact details depend on your operating system (which you did not tell), but the Jenkins wiki has a page about this: https://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build
There is a trick you can do in order you free a Jenkins thread.
What you can do is to execute a bash script through a ssh connection and send it to the background while saving the pid of the process somewhere so you can make checks further.
The format of the command would be:
ssh -n _hostname_ "_commands_ & echo \$! > \"_path_to_pid_file_\"" &
Example with a never-ending program:
ssh -n myhost.domain.com "tail -f /var/log/my.log & echo \$! > \"$WORKSPACE/pid\"" &
This example will spawn the tail process listening forever for new changes in the /var/log/my.log file and store its pid in the $WORKSPACE/pid file.
When executed from a Jenkins job the ssh process will exit immediately while the commands sent to the background will remain in execution in the specified host.
I do this in order to maintain always one of the services I run in my build farm in-sync with the latest code modification of it in the repository.
Just have a job that ssh' into the target machine and then kill the process, update the service and re-launches it.
This could be a bit cumbersome but it works great!
A windows slave node connected to Jenkins server through "Java web start". The system information of the node doesn't have it's IP address.
I had to run through all the slaves node we had, and find which machine (ip address) corresponds to the slave node in Jenkins.
Is there a way to find the IP address of a slave node from Jenkins itself?
Through the Script Console (Manage Jenkins -> Nodes -> Select a node -> Script Console) of the node we can execute groovy script. Run the following command to get the IP address.
println InetAddress.localHost.canonicalHostName
The most efficient and platform-independent way to find out the IP is probably the following groovy code for the "global" Script Console on the master:
import hudson.model.Computer.ListPossibleNames
def node = jenkins.model.Jenkins.instance.getNode( "myslave" )
println node.computer.getChannel().call(new ListPossibleNames())
In the console, this yields (for example)
Result
[192.168.0.17]
The result is a list of strings, as there's potentially multiple IP addresses on one machine.
Since this does not require the node-specific consoles, it's easy to add a loop around the code that covers all nodes.
To answer this same question on a non-windows Jenkins slave:
Get the IP address:
println "ifconfig".execute().text
Get the hostname:
println "hostname".execute().text
From the Web Interface
Go to the node's Log link:
http://jenkins.mycompany.com:8080/computer/my_node_name/log
The first line should say something like:
JNLP agent connected from /10.11.12.123
screenshot
This is very similar to what deepak explained but I added images along the short steps.
In Jenkins UI click:
Manage Jenkins -> Nodes -> Select a node -> Script Console
then run println InetAddress.localHost.canonicalHostName
In your Jenkins job if its in groovy or else echo the ifonfig
sh "/sbin/ifconfig -a | grep inet"
To get the ip on a Windows slave:
Navigate to the Script Console (Manage Jenkins -> Nodes -> Select a node -> Script Console)
println "ipconfig".execute().text
Can also be found through the Jenkins UI:
Manage Jenkins --> Manage Nodes --> Click Node name --> Configure
This should display both the public and private ip address of that node