I am trying to implement a CI pipeline for a Machine Learning model in Azure DevOps by forking the MLOpsPython repo. At the task where the pipeline is invoked, defined like this:
- task: ms-air-aiagility.vss-services-azureml.azureml-restApi-task.MLPublishedPipelineRestAPITask#0
displayName: 'Invoke ML pipeline'
inputs:
azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
PipelineId: '$(AMLPIPELINE_ID)'
ExperimentName: '$(EXPERIMENT_NAME)'
WorkspaceName: 'prolab-ml-test-ws1'
PipelineParameters: '"ParameterAssignments": {"model_name": "$(MODEL_NAME)"}, "tags": {"BuildId": "$(Build.BuildId)", "BuildUri": "$(BUILD_URI)"}, "StepTags": {"BuildId": "$(Build.BuildId)", "BuildUri": "$(BUILD_URI)"}'
This stage fails with a 404 invoking the pipeline:
https://westeurope.aether.ms/api/v1.0/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/resourceGroups/my-rg/providers/Microsoft.MachineLearningServices/workspaces/wrong-workspace/PipelineRuns/PipelineSubmit/adcc2798-beae-4be7-aeb4-452411ca2f40
The workspace value in the url is incorrect - it appears to be the name of the resource group. How can I change the parameters for the MLPublishedPipelineRestAPITask task.
Ok, this was my own stupid fault. The workspace settings are taken from the service connection in Azure DevOps, not from the parameters in the yml defining the task. I had the wrong workspace name in the service connection.
Related
How to properly secure Jenkins multibranch pipeline for open source projects?
I work on an open source projects, with community contributions made
using pull requests. I want to build those PR, but if I do so, there
is a risk that those parts get modified and host malicious code or
steal credentials :
the pipeline
the pipeline's dependencies
the test suite
...
For what it worth, I did researches, and my beginner point of view is this:
1/ Find a way to prevent modified code to be executed for :
pipeline
pipeline's dependencies
=> is it even possible?
2/ execute build only on agent never on master
3/ find a way to properly block agents to access secrets (like github/slack tokens)
4/ Find a way to mitigated what a malicious test code can do
the agent's should use a subnet with only this agent and the master
the agent should run on a separate VM/container
2/ execute build only on agent never on master
:- Never use "Agent Any" instead use Agent labels on your pipeline. you can switch your node on pipeline execution by mentioning Agent at stage level.
3/ find a way to properly block agents to access secrets (like github/slack tokens)
:- Use Credentials method and declare it in environment section.
environment {
YOUR_PASSWORD = credentials('<CREDENTIAL_ID>')
}
I'm currently using Jenkins FreeStyle Project in my project,trying to migrate to Jenkins Pipeline, but I'm facing some issues:
1) I need to commit jenkinsfile in my project, but my deploy phase is just copy from target/project.war to jboss deployment folder, as shown below:
stage('Deploy') {
steps {
sh 'cp /var/lib/jenkins/workspace/project/project.war /opt/jboss/standalone/deployment/project.war'
}
}
The problem: currently the path is fixed and tomorrow if a change occurs and there is a need to deploy to another machine, An update should be made to the source code which should be avoided. In FreeStyle project i just update the JOB and everything works.
2) The project has 3 modules. The FreeStyle project was configured so that JOB A will call JOB B on finish. In pipeline how can this order be achieved:
- Start JOB A --> JOB B --> JOB C.
You can add the following into your script
1.Issue with copying:
Firstly, you avoid using the actual path(Location of file in workspace) to a
relative Path i.e. using project/*.war or **/*.war it will take it
from the workspace itself.
Second, Coming to the issue that you have
to change the target location like you said you have to change it
FreeStyle Project :) so you have to change it in the
JenkinsFile also :)
2.To call other jobs in your pipeline and the following
build job: 'Job2', parameters: [
new org.jvnet.jenkins.plugins.nodelabelparameter.NodeParameterValue
("TARGET_NODE", "description", nodeName)
]
If it does not have any parameter remove that section out.
There is something called Jenkins workflow which provides more power and control if your interested in it, you can look it up here https://dzone.com/refcardz/continuous-delivery-with-jenkins-workflow?chapter=1
you can use sshPulissher:send build artifacts over ssh
add this code in your jenkins pipeline
and configure your sshServer in manage jenkins
finally your war is transfert in your destination
I am building Jenkins for Test / QA automation scripts, lets name it TEST_JOB. For application, I have application source code Jenkins build, name it DEV_JOB.
My scenario is when DEV_JOB completes execution (successfully), execute TEST_JOB immediately. I am aware about setting up project upstream / downstream [ Build after other projects are built ] to accomplish this task. But here, Problem is DEV_JOB is on different server than TEST_JOB. Due to which, TEST_JOB fails to recognize DEV_JOB.
Now, how would I achieve this scenario?
You can use Jenkins API for remote trigger of Job.
Say you have job on DEV_JOB on JENKINS_1, add a penultimate step(or upstream/downstream project having only this step) which invokes TEST_JOB using remote API call of JENKINS_2 server.
Example command would be
$(curl --user "username:password" "http://JENKINS_2/job/TEST_JOB/buildWithParameters?SOMEPARAMETER=$SOMEPARAMETER")
username:password is a valid user on JENKINS_2.
Avoid using your own account here but rather a 'build trigger' account that only has permissions to start those jobs.
We are trying to define a set of jobs on Jenkins that will do really specific actions. JobA1 will build maven project, while JobA2 will build .NET code, JobB will upload it to Artifactory, JobC will download it from Artifactory and JobD will deploy it.
Every job will have a set of parameters so we can reuse the same job for any product (around 100).
The idea behind this is to create black boxes, I call a job with some input and I get always some output, whatever happens between is something that I don't care. On the other side, this allows us to improve each job separately, adding the required complexity, and instantly all products will get benefit.
We want to use Jenkins Pipeline to orchestrate the execution of actions. We are going to have a pipeline per environment/usage.
PipelineA will call JobA1, then JobB to upload to artifactory.
PipelineB will download package JobC and then deploy to staging.
PipelineC will download package JobC and then deploy to production based on some internal validations.
I have tried to get some variables from JobA1 (POM basic stuff such as ArtifactID or Version) injected to JobB but the information seems not to be transfered.
Same happens while downloading files, I call JobC but the file is in the job workspace not available for any other and I'm afraid that"External Workspace Manager" plugin adds too much complexity.
Is there any way rather than share the workspace to achieve my purpose? I understand that share the workspace will make it impossible to run two pipelines at the same time
Am I following the right path or am I doing something weird?
There are two ways to share info between jobs:
You can use stash/unstash to share the files/data between multiple jobs in a single pipeline.
stage ('HostJob') {
build 'HostJob'
dir('/var/lib/jenkins/jobs/Hostjob/workspace/') {
sh 'pwd'
stash includes: '**/build/fiblib-test', name: 'app'
}
}
stage ('TargetJob') {
dir("/var/lib/jenkins/jobs/TargetJob/workspace/") {
unstash 'app'
build 'Targetjob'
}
In this manner, you can always copy the file/exe/data from one job to the other. This feature in pipeline plugin is better than Artifact as it saves only the data locally. The artifact is deleted after a build (helps in data management).
You can also use Copy Artifact Plugin.
There are two things to consider for copying an artifact:
a) Archive the artifacts in the host project and assign permissions.
b) After building a new job, select the 'Permission to copy artifact' → Projects to allow copy artifacts: *
c) Create a Post-build Action → Archive the artifacts → Files to archive: "select your files"
d) Copy the artifacts required from host to target project.
Create a Build action → Copy artifacts from another project → Enter the ' $Project name - Host project', which build 'e.g. Lastest successful build', Artifacts to copy '$host project folder', Target directory '$localfolder location'.
The first part of your question(to pass variables between jobs) please use the below command as a post build section:
post {
always {
build job:'/Folder/JobB',parameters: [string(name: 'BRANCH', value: "${params.BRANCH}")], propagate: false
}
}
The above post build action is for all build results. Similarly, the post build action could be triggered on the current build status. I have used the BRANCH parameter from current build(JobA) as a parameter to be consumed by 'JobB' (provide the exact location of the job). Please note that there should be a similar parameter defined in JobB.
Moreover, for sharing the workspace you can refer this link and share the workspace between the jobs.
You could use the Pipelines shared groovy libraries plugin. Have a look at its documentation to implement libraries that multiple pipelines share and define shared global variables.
on a multi job I have two phases:
PhaseA running Build_job1, with a project name Build_job1, pulling stuff from git to dir: /var/lib/jenkins/workspace/Build_job1
PhaseB running Deploy_job2, that rsyncs /var/lib/jenkins/workspace/Build_job1/* to a bunch of servers.
For internal reasons I need to replicate the multijob, the build job and the deploy job to different environments (PROD, QA, Staging). I each case, the deploy job rsync will need to copy files from a different build directory (Build_QA, Build_Prod, Build_whatever etc.).
As Jenkins creates the dir per project name, I need the rsync command in the deploy job to get the project name as a parameter that is passed down from the build job.
help?
Are you wanting to pass down the current job's project name down to its children? If so, you can pass down this information via a Jenkins Set Environment Variables call "JOB_NAME" in conjunction with a predefined job parameter. For example, something like:
Param1=${JOB_NAME}
If the Multijob job name is "QA", you can pass that down to both the build and deploy phase jobs via a predefined parameter and then construct the final "Build_QA" path by doing something like "Build_${Param1}" or "Build_%Param1%".