Creating / Getting a Cloud Run Job using the Python API Client Library - google-cloud-run

I created a Cloud Run Job using command line:
gcloud --verbosity=debug beta run jobs create my-job \
--image=us-docker.pkg.dev/cloudrun/container/job:latest
When I can list the jobs using the API Client library, my-job is returned:
import googleapiclient.discovery
with googleapiclient.discovery.build('run', 'v1') as client:
request = client.namespaces().jobs().list(parent=f'namespaces/my-project')
response = request.execute()
print(response)
However, when I try to get the job using the following snippet, I get 404 "Requested entity was not found":
...
request = client.namespaces().jobs().get(name='namespaces/my-project/jobs/my-job')
response = request.execute()
...
I am also unable to create a job using the following snippet, this again return 404 "Requested entity was not found":
request = client.namespaces().jobs().create(parent=f'namespaces/my-project',
body={
"metadata": {
"name": "my-job2",
},
"spec": {
"template": {
"spec": {
"template": {
"spec": {
"containers": [{
"image": "us-docker.pkg.dev/cloudrun/container/job:latest"
}],
}
}
}
}
},
})
I have Cloud Run Admin permissions for the project.
What am I missing?

Looking into the API reference, it appears you are using the call correctly 1, 2. When doing the get call, you need to specify the correct regional endpoint. So, it might happen that you are using the global endpoint for the list, create and get calls. However, make sure that you use the regional endpoint for the get and create call.
The global endpoint has this documentation for v14: "For v1, this endpoint only supports Global List: use regional endpoints instead."
You can see an example using this command from cloud shell. (This assumes your region is 'us-central1', if not that needs to be updated).
curl -X GET
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token)
https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/my-project/jobs/my-job
curl -X GET
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token)
https://run.googleapis.com/apis/run.googleapis.com/v1/namespaces/my-project/jobs/my-job

Related

Excluding Draft PRs from Jenkins

I will like to be able to choose to only run Jenkins build on PRs who are not marked as draft. Is there currently a way to do that?
I found something like this: https://github.com/jenkinsci/github-branch-source-plugin/pull/416, but cannot seem to find any place in the Jenkins dashboard that will allow me to exclude draft PRs.
Thanks!
The GitHub API allows you to view details of a PR and see if the PR is a draft:
Request:
curl \
-H "Accept: application/vnd.github+json" \
-H "Authorization: token <TOKEN>" \
https://api.github.com/repos/OWNER/REPO/pulls/PULL_NUMBER
Response:
{
"url": "https://api.github.com/repos/octocat/Hello-World/pulls/1347",
"id": 1,
"node_id": "MDExOlB1bGxSZXF1ZXN0MQ==",
...
"auto_merge": null,
"draft": false, <---- This is what you want
"merged": false,
...
"deletions": 3,
"changed_files": 5
}
You will have to modify the Jenkins job itself to perform this API call, parse the response and get the value of draft, and continue or abort the build depending on the value.

Resource not found - Triggering BitBucket Pipeline using curl

I created a new project and added a repository to it in my workspace. Further, I added a bitbucket-pipelines.yml to build a pipeline. I am able to trigger the pipeline manually however while trying to execute it using BitBucket API using curl, I get the below error every time:
Can anyone suggest what I am missing here?
NOTE: The same curl command (below) is able to run other pipelines in different repositories in the same workspace, so do I need to enable something in my current repository to access pipeline using BitBucket APIs. TIA
Error:
{"type": "error", "error": {"message": "Resource not found"}}%
cURL command:
curl -X POST -is -u username:password \
-H 'Content-Type: application/json' \
https://api.bitbucket.org/2.0/repositories/workspace-name/repo-name/pipelines/ \
-d '
{
"target": {
"type": "pipeline_ref_target",
"ref_type": "branch",
"ref_name": "master",
"selector": {
"type": "custom",
"pattern": "create-tenant"
}
}}'
So the tokens I was using in the curl weren't added to Repository Settings -> User and group access. Got them added and was able to execute it successfully.

Jenkins Webhook Header as Argument to Shell script

I'm trying to trigger a Jenkins job through the webhook using the curl command whenever there is EC2 Spot Instance Interruption Warning with below sample event. All this will be done in AWS lambda from CloudWatch Event Trigger.
{
"version": "0",
"id": "1e5527d7-bb36-4607-3370-4164db56a40e",
"detail-type": "EC2 Spot Instance Interruption Warning",
"source": "aws.ec2",
"account": "123456789012",
"time": "1970-01-01T00:00:00Z",
"region": "us-east-1",
"resources": [
"arn:aws:ec2:us-east-1b:instance/i-0b662ef9931388ba0"
],
"detail": {
"instance-id": "i-0b662ef9931388ba0",
"instance-action": "terminate"
}
}
My aim is to get the instance-id from the event and pass it as a header to the jenkins webhook where the jenkins job gets triggered and this header has to be sent as an argument to the underlying python script in the jenkins job.
I tried the below approach which didn't give me success. And am not sure if this is how it is done. :)
curl -H 'param: instance' https://jenkins.url/generic-webhook-trigger/invoke\?token\=jenkins-job
Generic Jenkins webhook trigger is configured as below.
The final python script in the job is configured as below.
python maintenance.py $.param
I'm trying to get the final script similar to below. Please let me know if you know of any approach how can I get this done. TIA
python maintenance.py i-0b662ef9931388ba
I fixed this by adding the following.
Add string parameter under This project is parameterized as shown below.
Next under Generic Webhook Trigger I have added Header parameters
Now, we can directly pass this param in the build command using $param.
The curl command is now modified to below.
curl --location --request POST 'https://jenkins.url/generic-webhook-trigger/invoke\?token\=jenkins-job' --header 'Content-Type: application/json' --header "param: $EVENT_DATA" --data-raw ''
What did I learn from this?
I was too lazy to read the documentation and finally figured it out only after reading it.

Executing Dataflow Template via Google Cloud Scheduler

I am trying to execute a custom Dataflow Template via the Google Cloud Scheduler but when I execute the Job I get an UNAUTHENTICATED exception.
How do I give Google Cloud Scheduler access to use the Dataflow google API?
Here is the URL and POST body I am using:
https://dataflow.googleapis.com/v1b3/projects/<<PROJECT>>/templates:launch?gcsPath=gs://<<GCS_BUCKET>>/template
{
"jobName": "job-name-scheduled",
"parameters": {
"param1" : "parmval1"
"environment": {
"tempLocation": "gs://<<BUCKET>>/temp",
"region": "us-east1"
}
}
The Cloud Scheduler documentation points out that "Targeted HTTP endpoints must be publicly accessible".
Normally, for creating that kind of Dataflow job, you would submit something like this:
curl -X POST \
'https://dataflow.googleapis.com/v1b3/projects/<project>/templates:launch?gcsPath=gs://dataflow-templates/latest/Word_Count' \
-H 'Authorization: Bearer '$(gcloud auth application-default print-access-token) \
-H 'Content-Type: application/json' \
--data '{
"jobName": "scheduled_job",
"parameters": {
"inputFile" : "gs://dataflow-samples/shakespeare/kinglear.txt",
"output": "gs://<bucket>/output/my_output"
},
"environment": { "zone": "us-central1-f" }
}'
But, you can't send the authorization token through Cloud Scheduler.
For scheduling Dataflow jobs, you can see this answer instead.

How to create jenkins credentials via the REST API?

I need to create a jenkins credential ( https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin ) via a script. How can I do that using either the REST API or the cli ?
Note that I'm able to list the credentials using /credential-store/domain//api/json and /credential-store/domain//credential/8bd82461-e239-4db1-90bc-831ca3412e70/api/json etc.
This issue took me a while to figure, a lot of digging around, so I decided to let the solution here, if someone else needs it.
curl -X POST 'http://user:token#jenkins_server:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "identification",
"username": "manu",
"password": "bar",
"description": "linda",
"$class": "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"
}
}'
with latest jenkins you need a CRUMB to authenticate for this operation (ref https://stackoverflow.com/a/38314286)
CRUMB=$(curl -s 'http://user:token#jenkins_server:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -H $CRUMB -X POST 'http://user:token#jenkins_server:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "identification",
"username": "manu",
"password": "bar",
"description": "linda",
"$class": "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"
}
}'
Otherwise you get
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /credentials/store/system/domain/_/createCredentials. Reason:
<pre> No valid crumb was included in the request</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
if you need to create credentials but with pem file path you can use this:
prerequisites: ssh-credentials plugin
CRUMB=$(curl -s 'http://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -H $CRUMB -X POST 'http://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "'{{ii.ssh_user}}'",
"username": "'{{ii.ssh_user}}'",
"password": "",
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource",
"privateKeyFile": "'{{jenkins_home}}/{{ii.key_name}}.pem'",
},
"description": "'{{ii.ssh_user}}'",
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
}
}'
this command used in ansible but you can replace the {{variables}} with your own variables
if you need to add all the pem file content you need to change the lines to:
....
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource",
"privateKey": "{{private_key_content}}",
},
"description": "{{user}}",
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
...
Just adding my 2 cents here: if you want to create the credentials for a specific folder, then use the following:
curl -H $CRUMB -X POST 'http://user:token#jenkins_server:8080/job/MY_FOLDER_NAME/credentials/store/folder/domain/_/createCredentials' \
...
So, you need to use /job/My_Folder at the beginning of the query part and replace the /store/system with /store/folder
There is no specific API call for this, but you can do it via cli commands to the jenkins jar.
echo 'jenkins.model.Jenkins.instance.securityRealm.createAccount("username", "password")' | java -jar jenkins-cli.jar -s http://localhost/ groovy =
For granting them permissions you can create a task in Jenkins which is running every N minutes and executing a groovy script as described here:
https://wiki.jenkins-ci.org/display/JENKINS/Grant+Cancel+Permission+for+user+and+group+that+have+Build+permission
I have a groovy script that also sets user permission using Matrix-based security. The script was posted at Creating user in Jenkins via API
Unable to point to ssh keys in ~/.ssh on Jenkins host
Means this no longer works,
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource",
"privateKeyFile": "'{{jenkins_home}}/{{ii.key_name}}.pem'",
},
📌 Here is the official documentation for managing the Jenkins Credentials via REST API
https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials
Example of adding a deploy-key credential using the username wecoyote and the password secret123 in the testing domain of the /example-folder folder.
$ cat > credential.xml <<EOF
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>deploy-key</id>
<description>Test User</description>
<usernameSecret>false</usernameSecret>
<username>wecoyote</username>
<password>secret123</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
$ curl -X POST -u <username>:<password_or_token> -H content-type:application/xml -d #credential.xml \
https://jenkins.example.com/job/example-folder/credentials/store/folder/\
domain/testing/createCredentials
The expected responses are:
HTTP/200 Success, the credentials has been created.
HTTP/409 Failure, a credential with that id already exists.
HTTP/50x Could not parse the supplied domain XML body.

Resources