Currently Jenkins master can do a "preventive monitoring" of the nodes in the "/computer/configure" page but it is only possible to monitor 2 disk spaces :
$JENKINS_HOME
Temp Space
I would like to monitor other disk spaces. For example, the /var/lib/docker (default directory where all docker images are pulled) is quickly full and causes some error on the nodes.
As we use the EC2 slaves plugin, it will be useful to be able to specify a threshold for this space size in order to put the node "offline" when the threshold is reached.. The EC2 instance will then be destroyed and another "healthy" slave will be started if needed.
Is it currently possible to achieve this in Jenkins ?
Version : Jenkins ver. 2.89.1
Thank you !
You are right that Jenkins monitors only those directories out of the box. The code handling this is at hudson.node_monitors.TemporarySpaceMonitor and hudson.node_monitors.DiskSpaceMonitor.
It would be fairly easy to implement a new NodeMonitor that would accept a list of directories to watch and would use the Jenkins plugins framework facility to take a node down if the space goes below a defined limit. I could even imagine a new plugin that would offer a series of enhanced node monitors. The plan has even been to extract all node monitors into a dedicated plugin since long, but this wasn't completed for various reasons (cf. JENKINS-26466 for details).
I wrote such a plugin at https://github.com/jenkinsci/inodes-monitor-plugin if you are looking for an example on how to use the NodeMonitor API.
Related
I am referring to this page:
https://www.instana.com/docs/setup_and_manage/host_agent/updates/#update-interval
Is there a way to pass mode and time from outside as environment variables or any other way beside logging into the pod and manually changing the files inside etc/instana/com.instana.agent.main.config.UpdateManager.cfg file?
To whoever removed his/her answer: It was a correct answer. I don't know why you deleted it. Anyhow, I am posting again in case someone stumbles here.
You can control frequency and time by using INSTANA_AGENT_UPDATES_FREQUENCY and INSTANA_AGENT_UPDATES_TIME environment variables.
Updating mode via env variable is still unknown at this point.
Look at this page for more info: https://www.instana.com/docs/setup_and_manage/host_agent/on/docker/#updates-and-version-pinning
Most agent settings that one may want to change quickly are available as environment variables, see https://www.instana.com/docs/setup_and_manage/host_agent/on/docker. For example, setting the mode via environment variable is supported as well with INSTANA_AGENT_MODE, see e.g., https://hub.docker.com/r/instana/agent. The valid values are:
APM: the default, the agent monitors everything
INFRASTRUCTURE: the agent will collect metrics and entities but not traces
OFF: agent runs but collects no telemetry
AWS: agent will collect data about AWS managed services in a region and an account, supported on EC2 and Fargate, and with some extra configurations, on hosts outside AWS
On Kubernetes, it is also of course possible to use a ConfigMap to override files in the agent container.
I'm trying to deploy to a node using jenkins, and even though the job recognises the node, when attempting to run the job turns to pending and tries to look through every environment for the node.
I've recently set up a new jenkins job to deploy a spring batch project onto a server. We already have a job for another project to deploy to the same node, so the node is recognised, and when viewing that build it does list three jobs.
However, when trying to run this new job, it attempts to find the node against all of our existing labels (see code below for example output), but doesn't find the actual node it should be running on.
The example I've used is NEW_BATCH_DEPLOYMENT, this is listed with having 3 jobs on the environment, two are new jobs that haven't been run, one is a job that ran just before attempting the batch job and succeeded.
For debugging, we've attempted to deploy with NEW_BATCH_DEPLOYMENT_2, which gives us an error for "can't find node with label NEW_BATCH_DEPLOYMENT_2", and if we remove the node name, it simply runs with one of our default nodes.
Has anyone seen something similar to this, or have any idea of a solution? I've compared the new job against the working job and the only differences are the file paths for where to deploy to, and the git url to pull the projects down.
Jenkins version : 2.181
(pending—; ‘Env_1’ doesn’t have label ‘NEW_BATCH_DEPLOYMENT’; ‘Env_2’ doesn’t have label ‘NEW_BATCH_DEPLOYMENT’; ‘Env_2’ doesn’t have label ‘NEW_BATCH_DEPLOYMENT’;
I'd expect it to deploy to the node, but it just hangs with pending and doesn't reach the stage where it would output to the jenkins console.
As mentioned the other job with similar configuration works.
So just to confirm, that is the exact label you are using? "NEW_BATCH_DEPLOYMENT"? Or is that the name of the node? THe Label should be set in the Node configuration, under the "Labels" section, with no extra characters other than the label name.
I've had issues where it can't find the node label if there are spaces in the label (either on the job side or on the configuration/set-up side)
If the labels are correctly set up, it could also be that the node assigned to "NEW_BATCH_DEPLOYMENT" is offline.
Okay, we fixed it.
It seems that on a node level you can set restrictions on jobs, so when the node was set up, it was restricted to only run the one job. The issue is, the only way you can see this is using an admin login.
If anyone else has this issue I'd highly recommending checking the settings on the node to see if the node itself has any restrictions, rather than the job. You will need a Jenkins Admin to do this.
Now I would like to describe the following problem on my part here.
We have several test systems that previously had the problem that start the Jenkins Jobs simultaneously. I would like to avoid this by providing some kind of recognition. It's about distributing the started Jenkins jobs on our test machines.
Example:
Test 1 runs at the customer - Test 2 should recognize this
For example, if test-1 is occupied by Job1, it should be recognized at the start of Job 2 and then automatically routed to one of the free test machines.
Manage Jenkins> Manage Nodes > Node > Configure
You must set same label names for different nodes.
Restrict where this project can be run = new label name
(You must install 'Least Load plugin')
We use visual svn for version control. I have few cloud web servers where my websites are running.
I would like to create some repositories for the websites content. I checkout them in local editors (notepad ++), edit them and checkin to SVN. But when check-in to visualSVN, I would like them to get deployed to the webservers docroot. In some cases I would like to restart the webserver too.
Is it possible using Jenkins+deployment plugins. I am very new to jenkins, can somebody help me with some information how we can achieve this.
It is one of the scenarios Jenkins is designed for (Continuous Delivery, aka. CD). Your perfect plan might look like this:
Get a new instance of Jenkins up/running (for experiments) (if you're familiar with Docker it is one of the best ways to experiment with Jenkins);
Configure Subversion Plugin in Jenkins (integration with SVN);
Setup your first FreeStyle job in Jenkins that polls your Visual SVN server for changes (things you check-in to SVN) and learn how that works (* * * * * <~-
this polls changes from your source control an every minute, great for experiments);
Setup your second FreeStyle job that connects to one of your webservers (probably via SSH) and creates a file (simple "touch hello_world.log" is great to start with) in a special folder dedicated for that kind of tests (DO NOT MESS WITH YOUR PRODUCTUION CONTENT FOLDER(s));
Setup your third FreeStyle job that combines your experiences acquired in #1 and #2, and still writes to a test folder;
Compare results of the job output with your production deployment expectations (eq. files in place, content is processed the right way, configuration files are looking good and etc.);
Try it out on one of the production web servers, one folder/site at a time;
Apply your newly crafted delivery pipeline to the rest of servers/sites;
Learn how backup your Jenkins instance and actually make your first backup;
Try to restore your Jenkins instance from the backup made in the previous step;
Decide whether it is okay for you to maintain your own Jenkins instance or you will be better off with a hosted version of it (CloudBees Inc.);
Learn more about Pipeline in Jenkins and possibly (because it is not immediately obvious) migrate your FreeStyle job(s) to Pipeline DSL
and/or Jenkinsfile;
At times you might need to get back to "Get Started with Jenkins" manual and look up for the ideas or answers, it is okay - do not give up and feel free to post your questions here, at SO.
Hope these ideas will help you to get started.
We have a situation where something causes our Jenkins master to start leaking memory at a certain time each night. No jobs are running at this time, so I'm not interested in logs of build jobs.
We would like to figure out what is happening around this time, but we cannot see much in the jenkins.out.log, since it doesn't log a lot and lacks timestamps.
Does anyone know if there is a way to turn on more detailed operational (not just errors) logging in Jenkins?
You could use the logs accessible from the UI (Manage Jenkins » System Log) and set the log level to ALL for the default log recorder in Log Levels. This log is only accessible from the UI though.
Alternatively, you could increase the log level of the regular jenkins.log by passing --debug=9 to the Jenkins command. If you're on CentOS, changing the value of JENKINS_DEBUG_LEVEL in /etc/sysconfig/jenkins has the same effect.
You could also use the Jenkins/Hudson monitoring plugin to see snapshots of memory use etc inside the JVM.
Also you could install sar (sysstat) to see how the server itself is doing over time (if Linux).