How to run different tasks on 2 nodes in one job - task

I need to run different jobs on 2 nodes in one job.
#!/bin/bash
#Create the following folders:
mkdir -p /shared_folder/files/Source
mkdir -p /shared_folder/files/Target
#Generate the backups
#Under Source execute the following command:
cp -r /local/folder /shared_folder/files/Source
#Under Target execute the following command:
cp -r /local/folder /shared_folder/files/Target
#Delete all Target mobile configurations:
#Under Target execute the following command:
rm /local/folder/*.*
#Copy the mobile configurations from Source to Target:
#Under Target execute the following command:
cp /shared_folder/files/Source/*.* /local/folder/
Restart Tomcat Service in Target:
Under Target execute the following command:
/sbin/service tomcat restart
Below is my reference job which uses ${option.NodeFilter1} and ${option.NodeFilter2} to pass node names to the actual 2 jobs.
I don't know if I have to run this reference job locally and these 2 actuall jobs to be pushed to the machines or this reference job should also be pushed to the actuall machines. ?
After suggestion I've created this job reference. It references 2 actual jobs.
- id: 98e41ff6-93e3-455a-9744-29bb6e8aae27
project: SecureCloud
loglevel: INFO
sequence:
keepgoing: false
strategy: node-first
commands:
- jobref:
group: apps
name: Synchronize mobile configurations Source
args: -Nodefilter1 ${option.NodeFilter1}
nodeStep: 'true'
nodefilters:
filter: ${option.NodeFilter1}
description: Source
- jobref:
group: apps
name: Synchronize mobile configurations Destination
args: -NodeFilter2 ${option.NodeFilter2}
nodeStep: 'true'
nodefilters:
filter: ${option.NodeFilter2}
description: Destination
description: ''
name: Synchronize mobile configurations Master
uuid: 98e41ff6-93e3-455a-9744-29bb6e8aae27
group: apps
options:
NodeFilter1:
required: true
NodeFilter2:
required: true
This is one of 2 actuall jobs
- id: 47b65475-f14b-4e5e-8d4f-aa9bb3c8e2b6
project: SecureCloud
loglevel: INFO
sequence:
keepgoing: false
strategy: node-first
commands:
- script: |
#!/bin/bash
touch /home/user/source-#option.NodeFilter1#
scriptInterpreter: sudo
interpreterArgsQuoted: false
description: Synchronize mobile configurations
name: Synchronize mobile configurations Source
uuid: 47b65475-f14b-4e5e-8d4f-aa9bb3c8e2b6
nodefilters:
dispatch:
threadcount: 2
keepgoing: true
excludePrecedence: false
rankOrder: ascending
filter: ${option.NodeFilter1}
group: apps
options:
NodeFilter1:
required: true
description: Source Node
I get this error while trying to run this configuration:
Option input was not valid for [apps/Synchronize mobile configurations Source]: Option 'NodeFilter1' is required.
13:50:58 Execution failed: 637: [Workflow result: , step failures: {1=Dispatch failed on 1 nodes: [c000lrdp01na01.na01.mstrci.com: InvalidOptions: Invalid options: [NodeFilter1]]}, Node failures: {c000lrdp01na01.na01.mstrci.com=[InvalidOptions: Invalid options: [NodeFilter1]]}, flow control: Continue, status: failed

You can create two jobs, one each to perform the appropriate task on the separate nodes. Then create another job which executes both of those two jobs via the Job Reference feature. Make sure not to set a node filter in the third job.

Related

Configure Ansible playbook to skip Jenkins Initial setup

Hello I'm new to writing Ansible Playbooks but I'm trying to have my playbook install Jenkins. It installs Jenkins just fine but the issue becomes that it wants me to do the initial unlock before installing plugins, creating jobs etc. I've seen in here a few times people saying you just need to add this to your playbook and you should be good. When I add it and then run the playbook it still has this issue even if I do it from a brand new server. Wondering what everyone has done to get by this issue. Thanks for your assistance!
Code I've seen from other posts:
Gets error "Cannot get CSRF" when trying to install jenkins-plugin using ANSIBLE
- name: Jenkins Skip startUp for MI
lineinfile:
dest=/etc/sysconfig/jenkins
regexp='^JENKINS_JAVA_OPTIONS='
line='JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false"'
register: result_skip_startup_wizard
My Playbook
---
# jenkins
- name: Create jenkins group
group:
name: jenkins
state: present
- name: Create jenkins user
user:
name: jenkins
group: jenkins
state: present
- name: Import jenkins gpg key
rpm_key:
state: present
key: http://pkg.jenkins.io/redhat-stable/jenkins.io.key
validate_certs: no
- name: Download Jenkins repo
get_url:
url: http://get.jenkins.io/redhat-stable/jenkins-2.332.3-1.1.noarch.rpm
dest: /etc/yum.repos.d/
- name: Install java
yum:
name: java-11-openjdk
state: present
- name: Install Jenkins
package:
name: /etc/yum.repos.d/jenkins-2.332.3-1.1.noarch.rpm
state: latest
- name: Jenkins Skip startUp for MI
lineinfile:
dest=/etc/sysconfig/jenkins
regexp='^JENKINS_JAVA_OPTIONS='
line='JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false"'
register: result_skip_startup_wizard
- name: Start and Enable Jenkins
systemd:
name: jenkins
state: started
enabled: true
- name: Sleep for 30 seconds and continue with Jenkins buildout
wait_for: timeout=30
For reference this is what I see in the server when I check the file and then when I just grep for the process.
jenkins 8474 1 34 18:29 ? 00:00:20 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
You can see the changes though do get put in the file as mentioned from above. Which makes me think even after restarting the service its not seeing the new option. I even manually stopped jenkins and then started but it still did not pick it up.
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false"
A little late here but I figured I'd leave a comment in here as well as I discovered when I was testing that the setup depended on the version of Jenkins you were attempting to install. Versions I tested are the comment lines above the code.
On the latest line it is just an assumption on my part not a guarantee.
# testing for jenkins 2.319.1
- name: Jenkins Skip startUp for MI
lineinfile:
dest=/etc/sysconfig/jenkins
regexp='^JENKINS_JAVA_OPTIONS='
line='JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false"'
register: result_skip_startup_wizard
# below works for 2.332.1 or latest
- name: Jenkins Skip startUp for MI
lineinfile:
dest=/usr/lib/systemd/system/jenkins.service
regexp='^Environment="JAVA_OPTS=-Djava.awt.headless=true'
line='Environment="JAVA_OPTS=-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false"'
register: result_skip_startup_wizard

Azure DevOps - Static Web Apps failing within a Pipeline with container job

parameters:
- name: App_VariableGroup
type: string
default: my-defaults
values:
- my-defaults
trigger:
- main
pool:
vmImage: ubuntu-latest
container: ubuntu:20.04
variables:
- group: ${{ parameters.App_VariableGroup }}
steps:
- checkout: self
submodules: true
- script: |
echo Hello, world! \n
ls -al
displayName: 'Run a one-line script'
- task: AzureStaticWebApp#0
inputs:
app_location: $(publish_path)
api_location: ''
output_location: ''
skip_app_build: true
azure_static_web_apps_api_token: $(swa_deployment_token)
This code is failing with "container: ubuntu:20.04" and give the following error:
##[warning]Environment variable AGENT_CONTAINERMAPPING is a multiline string and cannot be added to the build environment.
/usr/bin/bash /__w/_tasks/AzureStaticWebApp_18aad896-e191-4720-88d6-8ced4806941a/0.200.0/launch-docker.sh
/__w/_tasks/AzureStaticWebApp_18aad896-e191-4720-88d6-8ced4806941a/0.200.0/launch-docker.sh: line 1: docker: command not found
##[error]Error: The process '/usr/bin/bash' failed with exit code 127
Finishing: AzureStaticWebApp
But the Task: AzureStaticWebApp#0 works fine with just the vmImage and no container.
I remember there is docker:dind concept that I used in gitlab-cicd but could anyone advice on what is going wrong here please?
One of the problems is that you need docker installed on your container. You may use this guide for how to do that.
However, there is another issue which seems to be a bug in the task itself causing it to fail, possibly related to not being able to load the AGENT_CONTAINERMAPPING environment variable. I ran into this bug myself with an Ubuntu container loaded with docker and other tools specific to my pipeline.
Please reference this bug I submitted to the Microsoft/azure-pipelines-task project for more details and to include your voice.

How do we conditionally run a CircleCI workflow?

I have followed the guide described in Conditional steps in jobs and conditional workflows and written the below code for my CircleCI pipeline.
version: 2.1
workflows:
version: 2.1
workflowone:
when:
condition: false
jobs:
- samplejob:
workflowtwo:
when:
condition: true
jobs:
- jobone
jobs:
samplejob:
docker:
- image: buildpack-deps:stable
steps:
- run:
name: Sample Job in WF 1
command: |
echo "This job is in workflowone and the workflow should not run"
jobone:
docker:
- image: buildpack-deps:stable
steps:
- run:
name: Sample Job in WF 2
command: |
echo "This job is in workflowtwo and the workflow should run"
When I run the above code the output is not what is expected. First workflow should not run because the condition is false. Both worflows start running when the pipeline in triggered. Can anyone point out the missing piece here?
According to the CircleCI docs, workflows (specifically) does not accept the condition key:
Note: When using logic statements at the workflow level, do not
include the condition: key (the condition key is only needed for job
level logic statements).
See here logic-statement-examples (scroll to the bottom of this section to see the note)

Azure pipelines UI to accept parameters (like Jenkins)

Jenkins has a UI concept with dropdown lists, etc. to allow users to specify variables at run time. This has proven essential in our builds to make decisions in the pipeline (ie. which agent to run on, code base to choose, etc). By allowing parameters we are able to have a single pipeline/definition handle the same task for many clients/releases/environments.
I have been watching many people over the past year ask for this - to eliminate the number of almost identical build definitions - is there a best practice to handle this? Would be nice to have a single build definition for a specific task that can be smart enough to handle parameters.
Edit : example of possible pseudo-code to build on levi-lu#MSFT's suggestion.
parameters:
- name: ClientName
displayName: Pool Image
default: Select client
values: powershell
valuesScript : [
assemble curl request to http://myUrl.com/Clients/GetAll
]
- name: TargetEnvironment
displayName: Client Environment
type: string
values: powershell
valuesScript: [
assemble curl request using above parameter value to
https://myUrl.com/Clients/$(ClientName)/GetEnvironments
]
trigger: none
jobs:
- job: build
displayName: Run pipeline job
pool:
vmImage: windows-latest
parameters:
ClientName : $(ClientName)
TargetEnvironment : $(TargetEnvironment)
steps:
- script: echo building $(Build.BuildNumber)
Runtime parameters is available now. You can now set runtime parameters at the beginning of your pipepline YAML using parameters. For below example:
parameters:
- name: image
displayName: Pool Image
default: ubuntu-latest
values:
- windows-latest
- vs2017-win2016
- ubuntu-latest
- ubuntu-16.04
- macOS-latest
- macOS-10.14
- name: test
displayName: Run Tests?
type: boolean
default: false
trigger: none
jobs:
- job: build
displayName: Build and Test
pool:
vmImage: ${{ parameters.image }}
steps:
- script: echo building $(Build.BuildNumber)
- ${{ if eq(parameters.test, true) }}:
- script: echo "Running all the tests"
Above example is from Microsoft official document. Click here to learn more about runtime parameters.
When you run above Yaml pipeline, You will be able to select the parameter's value from the dropdown list. See below screenshot.
Update: To set variables dynamically at runtime.
You can use the task.setvariable logging command to set variables dynamically in scripts.
For below example: $resultValue is the value from rest api call. And its value is assigned to variable VariableName
- powershell: |
$resultValue = call from Rest API
echo "##vso[task.setvariable variable=VariableName]$resultValue"
Check document here for more information.

jenkins-job-builder doesn't propagate a variable value

I am using jenkins-job-builder to create my pipeline project. But I have a problem with the variables values when I am trying to reuse or propagating.
It is my project configuration:
- project:
name: myproject
git_url: git#gitlabserver.cu:demos-products/myproject.git
jobs:
- '{name}-nfr-smoke-tests':
pipeline-next: '{name}-nfr-smoke-tests'
And here is my job-template:
- job-template:
name: "{name}-nfr-smoke-tests"
node: 'slave1'
scm:
- git:
skip-tag: false
url: 'git#gitlabserver.cu:test-products/{name}-nfr-tests.git'
branches:
- master
wipe-workspace: true
builders:
- shell: |
bundle install
bundle exec cucumber features/smoke.feature
publishers:
- trigger:
project: "{pipeline-next}"
threshold: SUCCESS
Ok, now when I run this configuration in jenkins and I check the job's construction, it says:
No such project ‘{name}-nfr-smoke-tests’. Did you mean ‘myproject-nfr-smoke-tests’?
Why the line: pipeline-next: '{name}-nfr-smoke-tests'doesn't propagates the value of variable name and just used it as a literal string? I am missing something.
You are missing 'name' under 'project' section in your job-template. Append the following lines:
- project:
name: project-name
The purpose of a project is to collect related jobs together, and provide values for the variables in a Job Template.
I found out that Jenkins Job Builder version 0.9.0-0.2 does not propagate the value, but for me version 1.3.0+2015.12.15.git136.959eb4b909-1 did. Perhaps updating your version of Jenkins Job Builder might help?

Resources