I created a Cloud Run Job using command line:
gcloud --verbosity=debug beta run jobs create my-job \
--image=us-docker.pkg.dev/cloudrun/container/job:latest
When I can list the jobs using the API Client library, my-job is returned:
import googleapiclient.discovery
with googleapiclient.discovery.build('run', 'v1') as client:
request = client.namespaces().jobs().list(parent=f'namespaces/my-project')
response = request.execute()
print(response)
However, when I try to get the job using the following snippet, I get 404 "Requested entity was not found":
...
request = client.namespaces().jobs().get(name='namespaces/my-project/jobs/my-job')
response = request.execute()
...
I am also unable to create a job using the following snippet, this again return 404 "Requested entity was not found":
request = client.namespaces().jobs().create(parent=f'namespaces/my-project',
body={
"metadata": {
"name": "my-job2",
},
"spec": {
"template": {
"spec": {
"template": {
"spec": {
"containers": [{
"image": "us-docker.pkg.dev/cloudrun/container/job:latest"
}],
}
}
}
}
},
})
I have Cloud Run Admin permissions for the project.
What am I missing?
Looking into the API reference, it appears you are using the call correctly 1, 2. When doing the get call, you need to specify the correct regional endpoint. So, it might happen that you are using the global endpoint for the list, create and get calls. However, make sure that you use the regional endpoint for the get and create call.
The global endpoint has this documentation for v14: "For v1, this endpoint only supports Global List: use regional endpoints instead."
You can see an example using this command from cloud shell. (This assumes your region is 'us-central1', if not that needs to be updated).
curl -X GET
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token)
https://us-central1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/my-project/jobs/my-job
curl -X GET
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token)
https://run.googleapis.com/apis/run.googleapis.com/v1/namespaces/my-project/jobs/my-job
I created a new project and added a repository to it in my workspace. Further, I added a bitbucket-pipelines.yml to build a pipeline. I am able to trigger the pipeline manually however while trying to execute it using BitBucket API using curl, I get the below error every time:
Can anyone suggest what I am missing here?
NOTE: The same curl command (below) is able to run other pipelines in different repositories in the same workspace, so do I need to enable something in my current repository to access pipeline using BitBucket APIs. TIA
Error:
{"type": "error", "error": {"message": "Resource not found"}}%
cURL command:
curl -X POST -is -u username:password \
-H 'Content-Type: application/json' \
https://api.bitbucket.org/2.0/repositories/workspace-name/repo-name/pipelines/ \
-d '
{
"target": {
"type": "pipeline_ref_target",
"ref_type": "branch",
"ref_name": "master",
"selector": {
"type": "custom",
"pattern": "create-tenant"
}
}}'
So the tokens I was using in the curl weren't added to Repository Settings -> User and group access. Got them added and was able to execute it successfully.
I am trying to execute a custom Dataflow Template via the Google Cloud Scheduler but when I execute the Job I get an UNAUTHENTICATED exception.
How do I give Google Cloud Scheduler access to use the Dataflow google API?
Here is the URL and POST body I am using:
https://dataflow.googleapis.com/v1b3/projects/<<PROJECT>>/templates:launch?gcsPath=gs://<<GCS_BUCKET>>/template
{
"jobName": "job-name-scheduled",
"parameters": {
"param1" : "parmval1"
"environment": {
"tempLocation": "gs://<<BUCKET>>/temp",
"region": "us-east1"
}
}
The Cloud Scheduler documentation points out that "Targeted HTTP endpoints must be publicly accessible".
Normally, for creating that kind of Dataflow job, you would submit something like this:
curl -X POST \
'https://dataflow.googleapis.com/v1b3/projects/<project>/templates:launch?gcsPath=gs://dataflow-templates/latest/Word_Count' \
-H 'Authorization: Bearer '$(gcloud auth application-default print-access-token) \
-H 'Content-Type: application/json' \
--data '{
"jobName": "scheduled_job",
"parameters": {
"inputFile" : "gs://dataflow-samples/shakespeare/kinglear.txt",
"output": "gs://<bucket>/output/my_output"
},
"environment": { "zone": "us-central1-f" }
}'
But, you can't send the authorization token through Cloud Scheduler.
For scheduling Dataflow jobs, you can see this answer instead.
I am trying to programatically update a cloud DataFlow job by using the REST API as described here
I have a PubSub to BigQuery job and my end goal is to replace the BigQuery output table.
I've tried updating the current job with a new job by using the replacedByJobId field but always getting this error:
{
"error": {
"code": 400,
"message": "(b7fd8310f1b85ccf): Could not modify workflow; invalid modifier value: 0",
"status": "INVALID_ARGUMENT"
} }
Request body:
{
"id": "jobid",
"projectId": "projectId",
"replacedByJobId = "newJobId", }
Is there another way to either replace a running job's parameters (OutputTable) or replace a running job with a new similar job?
In order to update a job you also need to provide a compatible replacement job. Note that update is currently only supported using the Java SDK.
You can find documentation on updating using the Java SDK at: Updating an Existing Pipeline: Launching Your Replacement Job.
java -jar pipeline/build/libs/pipeline-service-1.0.jar \
--project=my-project \
--zone=us-central1-f \
--streaming=true \
--stagingLocation=gs://my-bucket/tmp/dataflow/staging/ \
--runner=DataflowPipelineRunner \
--numWorkers=5 \
--workerMachineType=n1-standard-2 \
--jobName=ingresspipeline \
--update
I need to create a jenkins credential ( https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin ) via a script. How can I do that using either the REST API or the cli ?
Note that I'm able to list the credentials using /credential-store/domain//api/json and /credential-store/domain//credential/8bd82461-e239-4db1-90bc-831ca3412e70/api/json etc.
This issue took me a while to figure, a lot of digging around, so I decided to let the solution here, if someone else needs it.
curl -X POST 'http://user:token#jenkins_server:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "identification",
"username": "manu",
"password": "bar",
"description": "linda",
"$class": "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"
}
}'
with latest jenkins you need a CRUMB to authenticate for this operation (ref https://stackoverflow.com/a/38314286)
CRUMB=$(curl -s 'http://user:token#jenkins_server:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -H $CRUMB -X POST 'http://user:token#jenkins_server:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "identification",
"username": "manu",
"password": "bar",
"description": "linda",
"$class": "com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl"
}
}'
Otherwise you get
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /credentials/store/system/domain/_/createCredentials. Reason:
<pre> No valid crumb was included in the request</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
if you need to create credentials but with pem file path you can use this:
prerequisites: ssh-credentials plugin
CRUMB=$(curl -s 'http://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -H $CRUMB -X POST 'http://{{jenkins_admin_username}}:{{jenkins_admin_password}}#localhost:8080/credentials/store/system/domain/_/createCredentials' \
--data-urlencode 'json={
"": "0",
"credentials": {
"scope": "GLOBAL",
"id": "'{{ii.ssh_user}}'",
"username": "'{{ii.ssh_user}}'",
"password": "",
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource",
"privateKeyFile": "'{{jenkins_home}}/{{ii.key_name}}.pem'",
},
"description": "'{{ii.ssh_user}}'",
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
}
}'
this command used in ansible but you can replace the {{variables}} with your own variables
if you need to add all the pem file content you need to change the lines to:
....
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$DirectEntryPrivateKeySource",
"privateKey": "{{private_key_content}}",
},
"description": "{{user}}",
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey"
...
Just adding my 2 cents here: if you want to create the credentials for a specific folder, then use the following:
curl -H $CRUMB -X POST 'http://user:token#jenkins_server:8080/job/MY_FOLDER_NAME/credentials/store/folder/domain/_/createCredentials' \
...
So, you need to use /job/My_Folder at the beginning of the query part and replace the /store/system with /store/folder
There is no specific API call for this, but you can do it via cli commands to the jenkins jar.
echo 'jenkins.model.Jenkins.instance.securityRealm.createAccount("username", "password")' | java -jar jenkins-cli.jar -s http://localhost/ groovy =
For granting them permissions you can create a task in Jenkins which is running every N minutes and executing a groovy script as described here:
https://wiki.jenkins-ci.org/display/JENKINS/Grant+Cancel+Permission+for+user+and+group+that+have+Build+permission
I have a groovy script that also sets user permission using Matrix-based security. The script was posted at Creating user in Jenkins via API
Unable to point to ssh keys in ~/.ssh on Jenkins host
Means this no longer works,
"privateKeySource": {
"stapler-class": "com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$FileOnMasterPrivateKeySource",
"privateKeyFile": "'{{jenkins_home}}/{{ii.key_name}}.pem'",
},
📌 Here is the official documentation for managing the Jenkins Credentials via REST API
https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials
Example of adding a deploy-key credential using the username wecoyote and the password secret123 in the testing domain of the /example-folder folder.
$ cat > credential.xml <<EOF
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>deploy-key</id>
<description>Test User</description>
<usernameSecret>false</usernameSecret>
<username>wecoyote</username>
<password>secret123</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
$ curl -X POST -u <username>:<password_or_token> -H content-type:application/xml -d #credential.xml \
https://jenkins.example.com/job/example-folder/credentials/store/folder/\
domain/testing/createCredentials
The expected responses are:
HTTP/200 Success, the credentials has been created.
HTTP/409 Failure, a credential with that id already exists.
HTTP/50x Could not parse the supplied domain XML body.