I recently had some issues regarding version upgrade for my Jenkins server. In order to update the version of the jenkins server, the first step I did was to create a backup:
sudo tar -zcvf /tmp/jenkins.tgz /var/lib/jenkins
Then, I copied the archived file, from server A and untar it on another server, server B. I can see all the files [workspace, config.xml, jobs] of server A to server B in var/lib/jenkins.
When I am logging into the jenkins box it showed:
Jenkins detected that you appear to be running more than one instance of Jenkins that share the same home directory '/var/lib/jenkins’. This greatly confuses Jenkins and you will likely experience strange behaviors, so please correct the situation.
This Jenkins:
490566619 contextPath="" at 8779#jm1597185631ybr.cloud.phx3.gdg
Other Jenkins:
1998724099 contextPath="" at 20292#jm1584048540yxl.cloud.phx3.gdg
So, I stopped the jenkins service using:
sudo service jenkins stop
Then I restarted the service using
sudo service jenkins restart
All the jobs started to appear suddenly. I have following questions:
Why did the jobs started to show up and not throw the error of running multiple instances?
If version is the only issue, why cannot the newly provisioned
server have the updated version? Is it when I copy the files from server A, the server B files gets overwritten and hence, shows the same error of the version upgrade?
Related
I've had this problem where I send a job to one of my nodes and sometimes the job would stuck on
[EnvInject] - Loading node environment variables.
The issue started when I moved my Jenkins machine from Amazon EC2 to Azure VM. Everytime it occured, I have stopped and relaunched the agent on the node, just so in a few minutes/hours it would freeze again. Usually I would stop the job after a few minutes, but if I let it run, in ~15 minutes I would receive the following:
FATAL: java.io.IOException: Unexpected termination of the channel
I have tried the following solutions, but with no help:
Rename the job
Create a new job (which runs just a basic command)
Uncheck the 'Use TCP_NODELAY flag on the SSH connection'
Adding '-Dhudson.slaves.ChannelPinger.pingInterval=2' to the JVM options
My architecture at the time was:
Jenkins as docker container on Azure VM (Ubuntu 18.04)
Jenkins ver 2.202
SSH Build Agents plugin version 1.31.1
Node on MacOS Mojave
The solution that sort of solved the problem, (now it occurs once in a few days, or even better) was adding ClientAliveInterval 120 to /etc/ssh/sshd_config and relaunch the com.openssh.sshd
sudo launchctl stop com.openssh.sshd
sudo launchctl start com.openssh.sshd
I noticed that all of my builds suddenly got stuck. If I click the build I get the following error: All eligible agents are disabled or offline
When I check my agent pools I get the following:
How can I bring them online/determine why they are offline?
Try logging into the VM, and run the run.cmd from an elevated powershell prompt in your agent directory.
Source: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops
Login to powershell.Navigate to the agent folder where it contains run cmd prompt and config cmd prompt.
Execute the following command
.\run.cmd
Check the date and time on your VM. If it's wrong, correct it.
Then run powershell as administrator and cd into the azagent folder that contains the run.cmd file (most likely C:\azagent\A4).
Execute the following command:
.\run.cmd
For me, issue was suddenly agents were reporting offline on the portal.
Restarted the agent service in the VM - didn't help
Removed config file also didn't help because logs were showing that dev.azure.com wasn't connecting
Simply just restarting VM helped, but still couldn't able to find the cause.
The VM that host the build agents can have a auto shut down time. Make sure your build vm is running by going to Virtual machines in the azure portal
My agent stopped going live when I updated our Azure Devops to the latest version 2019.1.3 iirc.
To solve this I did the following:
Went to services.msc on the agent machine. Stopped the existing service.
Went to agent folder and removed the existing configuration via "config.cmd remove"
Downloaded latest agent code from the Azure Devops Agent Pools portal page.
Replaced contents of old agent with new agent files.
Went back to agent folder and configured a new agent via "config.cmd". Told it to start as a service.
Voila! Service back running again.
One of my Azure agents started showing as offline. As a test, I ran curl https://dev.azure.com/{organization} and got back an error "curl: (35) schannel: failed to receive handshake, SSL/TLS connection failed".
I fixed the issue by downloading IISCryptoCli.exe from https://www.nartac.com/Products/IISCrypto/Download and then running "IISCryptoCli.exe /template default /reboot".
I solved this by connecting to the agent via Remote Desktop Connection and running the "Run Puppet Agent" application.
Please check the agent from agent tab under releases.
There should be a server listed for the agent under agent tab.
Now connect to that server and check weather the agent batch file or cmd window is open or not.(this agent for is the same file which is downloaded while configuring the agent.if you are struggling to find the file then follow new agent steps and download zip again and search with that file name.)
Now open the(filename -run) file on that server with admin right if it is already not running. This will make agent online in devops portal.
Now check the agent tab again in devops portal under releases.
This
I am trying to run a jenkins job inside a windows docker container. I have successfully created an image with windows server code docker image which will have MSBuildEngine 4.7.
The problem I am facing is I am not able to run a Jenkins job inside that container.
I am able to do it easily with linux environment.
The actual problem is, Jenkins first puts a shell file which will have the command to run the container and inspect it.
How do I tell Jenkins that my environment is not Linux and it is Windows.
Note: Searching in google does not help now a days. So I directly reached out here
I am working on this issue as well. I am finding that the (maybe just a) underlying issue is how Jenkins tells Docker to mount a volume to the container. I have yet to get around this issue.
edit:
There's a PR addressing this issue and I tested the fork with both Linux and Windows slaves to work as we intend.
Download Rbutcher's fork of the plugin:
git clone https://github.com/rbutcher/docker-workflow-plugin.git
Change to the working branch:
git checkout feat/windows_slaves
Build the plugin:
mvn -DskipTests clean install
Manually import into Jenkins:
Manage Jenkins> Manage Plugins> Advanced>Upload Plugin and select ./target/docker-workflow.hpi.
The session logging we have in {{/etc/profile}} can interfere with services that launch sub-shells as new users - specifically, it always launches an interactive terminal, regardless of the context, which can cause certain key processes (e.g. Jenkins) from being able to perform critical tasks.
We had a Jenkins version upgrade and after hte upgrade, Jenkins seems to not be able to restart. Here’s what’s happening
```ubuntu#hoatname:~$ sudo service jenkins status
Correct java version found
Jenkins Automation Server is not running
ubuntu#hostname:~$ sudo service jenkins start
Correct java version found
Starting Jenkins Automation Server jenkins jenkins#hostname:~$
jenkins#hostname:~$
jenkins#hostname:~$ sudo service jenkins status
[sudo] password for jenkins:
jenkins#hostname:~$ exit
exit
[fail]
ubuntu#hostname:~$
```
Essentially, it seems that “service jenkins start” is somehow causing a session to be created, which dumps it into a script. I suspect this is due to how /etc/profile contains a script-based session logger, and i suspect that Jenkins is attempting to execute this script when it su’s into its own jenkins user
What should I do to alleviate this?
I am currently using Jenkins on my development PC. I installed it on my development PC, because I had limited knowledge on this tool; so I tested on it in my development PC. Now, I feel comfortable with Jenkins as my long term "partner" in the build process and would like to "move" this Jenkins to a dedicated server.
Before this I have done few builds and have the artifacts archived from each build. In particular, the build number is very important to me for version control.
How can I export all the Jenkins information from my current PC to my new server?
Following the Jenkins wiki, you'll have to:
Install a fresh Jenkins instance on the new server
Be sure the old and the new Jenkins instances are stopped
Archive all the content of the JENKINS_HOME of the old Jenkins instance
Extract the archive into the new JENKINS_HOME directory
Do not forget to change the owner of the new Jenkins files : chown -R jenkins:jenkins $JENKINS_HOME
Launch the new Jenkins instance
Do not forget to change documentation/links to your new instance of Jenkins :)
JENKINS_HOME is by default located in ~/.jenkins on a Linux installation, yet to exactly find where it is located, go on the http://your_jenkins_url/configure page and check the value of the first parameter: Home directory; this is the JENKINS_HOME.
In case your JENKINS_HOME directory is too large to copy, and all you need is to set up same jobs, Jenkins Plugins and Jenkins configurations (and don't need old Job artifacts and reports), then you can use the ThinBackup Plugin:
Install ThinBackup on both the source and the target Jenkins servers
Configure the backup directory on both (in Manage Jenkins → ThinBackup → Settings)
On the source Jenkins, go to ThinBackup → Backup Now
Copy from Jenkins source backup directory to the Jenkins target backup directory
On the target Jenkins, go to ThinBackup → Restore, and then restart the Jenkins service.
If some plugins or jobs are missing, copy the backup content directly to the target JENKINS_HOME.
If you had user authentication on the source Jenkins, and now locked out on the target Jenkins, then edit Jenkins config.xml, set <useSecurity> to false, and restart Jenkins.
This worked for me to move from Ubuntu 12.04 (Jenkins ver. 1.628) to Ubuntu 16.04 (Jenkins ver. 1.651.2). I first installed Jenkins from the repositories.
Stop both Jenkins servers
Copy JENKINS_HOME (e.g. /var/lib/jenkins) from the old server to the new one. From a console in the new server:
rsync -av username#old-server-IP:/var/lib/jenkins/ /var/lib/jenkins/
Start your new Jenkins server
You might not need this, but I had to
Manage Jenkins and Reload Configuration from Disk.
Disconnect and connect all the nodes again.
Check that in the Configure System > Jenkins Location, the Jenkins URL is correctly assigned to the new Jenkins server.
Jenkins Server Automation:
Step 1:
Set up a repository to store the Jenkins home (jobs, configurations, plugins, etc.) in a GitLab local or on GitHub private repository and keep it updated regularly by pushing any new changes to Jenkins jobs, plugins, etc.
Step 2:
Configure a Puppet host-group/role for Jenkins that can be used to spin up new Jenkins servers. Do all the basic configuration in a Puppet recipe and make sure it installs the latest version of Jenkins and sets up a separate directory/mount for JENKINS_HOME.
Step 3:
Spin up a new machine using the Jenkins-puppet configuration above. When everything is installed, grab/clone the Jenkins configuration from the Git repository to the Jenkins home direcotry and restart Jenkins.
Step 4:
Go to the Jenkins URL, Manage Jenkins → Manage Plugins and update all the plugins that require an update.
Done
You can use Docker Swarm or Kubernetes to auto-scale the slave nodes.
Sometimes we may not have access to a Jenkins machine to copy a folder directly into another Jenkins instance. So I wrote a menu driven utility which uses Jenkins REST API calls to install plugins and jobs from one Jenkins instance to another.
For plugin migration:
GET request: {SOURCE_JENKINS_SERVER}/pluginManager/api/json?depth=1 will get you the list of plugins installed with their version.
You can send a POST request with the following parameters to install these plugins.
final_url=`{DESTINATION_JENKINS_SERVER}/pluginManager/installNecessaryPlugins`
data=`<jenkins><install plugin="{PLUGIN_NAME}#latest"/></jenkins>` (where, latest will fetch the latest version of the plugin_name)
auth=`(destination_jenkins_username, destination_jenkins_password)`
header=`{crumb_field:crumb_value,"Content-Type":"application/xml”}` (where crumb_field=Jenkins-Crumb and get crumb value using API call {DESTINATION_JENKINS_SERVER}/crumbIssuer/api/json
For job migration:
You can get the list of jobs installed on {SOURCE_JENKINS_URL} using a REST call, {SOURCE_JENKINS_URL}/view/All/api/json
Then you can get each job config.xml file from the jobs on {SOURCE_JENKINS_URL} using the job URL {SOURCE_JENKINS_URL}/job/{JOB_NAME}.
Use this config.xml file to POST the content of the XML file on {DESTINATION_JENKINS_URL} and that will create a job on {DESTINATION_JENKINS_URL}.
I have created a menu-driven utility in Python which asks the user to start plugin or Jenkins migration and uses Jenkins REST API calls to do it.
You can refer the JenkinsMigration.docx from this URL
jenkinsjenkinsmigrationjenkinsrestapi
Let us say we are migrating Jenkins LTS from PC1 to PC2 (irrispective of LTS version is same of upgraded).
It is easy to use ThinBackUp Plugin for migration or Upgrade of Jenkins version.
Step1: Prepare PC1 for migration
Manage Jenkins -> ThinbackUp -> Setting
Select correct options and directory for backup
If you need a job history and artifacts need to be added then please select 'Back build results' option as well.
Go back click on Backup Now.
Note: This Thinbackup will also take Plugin Backup which is optional.
Check the ThinbackUp folder must have a folder with current date and timestamp.
(wait for couple of minutes it might take some time.)
You are ready with your back, .zip it and copy to PARTICULAR (which will be 'Backup directory') directory in PC2.
Unzip ThinbackUp zipped folder.
Stop Jenkins Service in PC1.
Step2: Install Jenkins (Install using .war file or Paste archived version) in PC2.
Create Jenkins Service using command sc create <Jenkins_PC2Servicename> binPath="<Path_to_Jenkinsexe>/jenkins.exe"
Modify JENKINS_HOME/jenkins.xml if needed in PC2.
Run windows service <Jenkins_PC2Servicename> in PC2
Manage Jenkins -> ThinbackUp -> Setting
Make sure that you PARTICULAR path from step1 as Backup Directory in ThinBackup settings.
ThinbackUp -> Restore will give you a Dropdown list, choose a right backup (identify with date and timestamp).
Wait for some minutes and you have latest backup configurations including jobs history and plugins in PC2.
In case if there are additional changes needed in JENKINS_HOME/Jenkins.xml (coming from PC1 ThinbackUp which is not needed) then this modification need to do manually.
NOTE: If you are using Database setting of SCM in your Jenkins jobs then you need to take extra care as all SCM plugins do not support to carry Database settings with the help of ThinbackUp plugin.
e.g. If you are using PTC Integrity SCM Plugin, and some Jenkins jobs are using DB using Integrity, then it will create a directory JENKINS_Home/IntegritySCM, ThinbackUp will not include this DB while taking backup.
Solution: Directly Copy this JENKINS_Home/IntegritySCM folder from PC1 to PC2.