I know a cloud task can be scheduled and can be configured using stream also to be executed.
As a developer I want to execute my spring cloud task using rest-api so that I can execute the task on demand.
Basically i have a work flow management system and we are using control-m agent. So now some of the jobs will be executed by control-m and some of the task will be deployed on spring cloud dataflow server. Now when one job completes then other job which is there on cloud has to be executed.
So for this I need the capability to call a rest api and execute the cloud task on demand.
I am sure this feature must be there but I am unable to find an example of documentation.
Can someone please help me.
Thanks in advance.
Please refer to the REST-API guide; specifically, you'd be using tasks/deployments endpoint to operate on an existing task.
Create:
dataflow:>task create foo --definition "timestamp"
Created new task 'foo'
Launch:
curl http://localhost:9393/tasks/deployments/foo\?arguments\=\&properties\= -d ""
p.s: all the supported REST-APIs are listed for your reference and they are accessible at: http://localhost:9393
Related
I want to execute a GCP cloud run job (not service) that periodically processes some data. I see that when i configure a cloud run job I can fill out "Container, Variables & Secrets, Connections, Security" field for container arguments. But I want to pass different arguments every time I execute them and I am wondering if there is a way. I haven't been able to find a way to do.
If there is no such way, am I supposed to use cloud run jobs only if I want them to do the same thing periodically?
But I want to pass different arguments every time I execute them and I am wondering if there is a way.
You can update the job before every execution.
I use gcloud CLI tool to run my jobs from a pipeline. I do multiple steps, first I use gcloud beta run jobs update my-job (with updates of variables) and after that I start an execution using gcloud beta run jobs execute my-job.
There is also a flag on the update command, --execute-now that you might want to use to start an execution when you update the job.
As stated in the official documentation:
Unlike a Cloud Run service, which listens for and serves requests, a Cloud Run job only runs its tasks and exits when finished. A job does not listen for or serve requests, and cannot accept arbitrary parameters at execution.
Therefore, there is no way for a Cloud Run job to receive parameters at execution time.
Also note that, at the moment, Cloud Run Jobs is in Preview:
Preview — Using Cloud Run jobs
This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.
I am trying to write a script to automate the deployment of a Java Dataflow job. The script creates a template and then uses the command
gcloud dataflow jobs run my-job --gcs-location=gs://my_bucket/template
The issue is, I want to update the job if the job already exists and it's running. I can do the update if I run the job via maven, but I need to do this via gcloud so I can have a service account for deployment and another one for running the job. I tried different things (adding --parameters update to the command line), but I always get an error. Is there a way to update a Dataflow job exclusively via gcloud dataflow jobs run?
Referring to the official documentation, which describes gcloud beta dataflow jobs - a group of subcommands for working with Dataflow jobs, there is no possibility to use gcloud for update the job.
As for now, the Apache Beam SDKs provide a way to update an ongoing streaming job on the Dataflow managed service with new pipeline code, you can find more information here. Another way of updating an existing Dataflow job is by using REST API, where you can find Java example.
Additionally, please follow Feature Request regarding recreating job with gcloud.
Our company policy requires the policy contraint "compute.requireShieldedVm" to be enabled. However, when running a Cloud Dataflow job, it is failing to create a worker with the error :
Constraint constraints/compute.requireShieldedVm violated for project projects/********. The boot disk's 'initialize_params.source_image' field specifies a non-Shielded image: projects/dataflow-service-producer-prod/global/images/dataflow-dataflow-owned-resource-20200216-22-rc00. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information."
Is there any way when running a Dataflow job to request that a ShieldedVm be used for the worker compute?
It is not possible to provide a custom image as there is no such parameter that one can provide during job submission as can be seen here Job Submission Parameters
Alternatively, if you are running a Python based dataflow job you can setup the environment through setup files. An example of which can be found here Dataflow - Custom Python Package Environment
I have a requirement to scale down OpenShift pods at the end of each business day automatically.
How might I schedule this automatically?
OpenShift, like Kubernetes, is an api-driven application. Essentially all application functionality is exposed over the control-plane API running on the master hosts.
You can use any orchestration tool that is capable of making API calls to perform this activity. Information on calling the OpenShift API directly can be found in the official documentation in the REST API Reference Overview section.
Many orchestration tools have plugins that allow you to interact with OpenShift/Kubernetes API more natively than running network calls directly. In the case of Jenkins for example there is the OpensShift Pipeline Jenkins plugin that allows you to perform OpenShift activities directly from Jenkins pipelines. In the cases of Ansible there is the k8s module.
If you were to combine this with Jenkins capability to run jobs on a schedule you have something that meets your requirements.
For something much simpler you could just schedule Ansible or bash scripts on a server via cron to execute the appropriate API commands against the OpenShift API.
Executing these commands from within OpenShift would also be possible via the CronJob object.
I want to build my maven project in Jenkins and copy all the the jar files to a remote Unix machine.
Also I want to connect to a LDAP data Store and start the services and test if the services are up and running
Basically I want to do the following tasks after my project is successfully build in Jenkins:-
1)Copy current version of my project to designated machine and location
2)Copy configure to connect to a designated integration test DS
3)Start the services in my project
4)Test that it is running.
Can I achieve this by Publish over SSH plugin provided in jenkins??
Or Shall I create some scripts which can automate the above tasks.The reason I am asking this is because I am not very familiar with Jenkins and Unix scripting.
Is there any good approach to do this task.
Thanks in advance.
Ansia
The Publish over SSH plugin will allow you copy files to remote server and execute arbitrary commands on the remote server.
Question is - do you know how you would achieve the following on the remote server?
2)Copy configure to connect to a designated integration test DS
3)Start the services in my project
4)Test that it is running
If yes, just enter those commands into Publish over SSH configuration. Or provide a script to be executed.
If you don't know how to achieve that, then that's a separate question.
Yes, you can use the publish over ssh plugin to copy the jars, and execute a script which launches your services. Take a look here to see how to launch a script "in the background" so it does not get killed when the session ends or to avoid blocking the Jenkins build by making it wait for the script to finish executing
Can't say much about LDAP as I haven't used it but depending on your needs I guess you could create a basic helper-jar with spring-ldap or any other similar library.