Bitbucket scheduled pipelines UI does not have an option for us to enter a cron expression and we can only run the pipeline hourly, daily or weekly. There is an option to create schedule via API call with cron expression in payload, however, unfortunately it does not accept a complex cron expression.
What could be the best way to achieve running the pipelines just on weekdays?
Looking for a better solution than these.
Have multiple daily pipelines mon-fri.
Have a daily pipeline and a check inside running logic for day.
Is there a better option?
Related
We are deploying/triggering Dataflow streaming jobs through Airflow using flex template. We want these streaming jobs to run, say for 24 hours (or until a certain clock time), then stop/cancel on its own. Is there a parameter in Dataflow (pipeline setting like max workers) that will do this?
I think there is no parameter and automatic approach to stop or drain a Dataflow job.
You can do that with an Airflow dag.
Example you can create a cron dag with Airflow (every 24 hours) having the responsability to stop or drain the Dataflow job, there is a built in operator to do that :
stop_dataflow_job = DataflowStopJobOperator(
task_id="stop-dataflow-job",
location="europe-west3",
job_name_prefix="start-template-job",
)
To stop one or more Dataflow pipelines you can use
DataflowStopJobOperator. Streaming pipelines are drained by default,
setting drain_pipeline to False will cancel them instead. Provide
job_id to stop a specific job, or job_name_prefix to stop all jobs
with provided name prefix.
How to :
Schedule job at specific dates across several months ?
it will be ~10 dates.
Second is there a tool that simulate cron, such that u can pass date/time and the tool tell you did cron triggers or not
cron does not schedule events in that manner. Jenkins scheduler is cron-based.
This answer provides several options.
Apparently there is now a released plugin - Schedule Build which appears to support calling the scheduleBuild action referenced in my groovy based answer.
See constraint (you can only schedule one build at a time) unless parameterized.
All jobs defined in my Jenkins use SCM polling. I want to keep all of the schedule rules same and control it in a single place. Is it possible?
You can use the cool Configuration slicing plugin to update the SCM Timer Trigger Slicer.
this will enable you to update multiple jobs at the same time.
The nicest choice for things like that is JobDSL. You have the power of a real programming language.
I want to use a lock in a workflow job in order to prevent jobs from running at the same time on the same node.
I want to use the functionality of the lock and latches plugin to control the parallel execution of jobs: When a Job A starts building on a specific node, Job B should wait until A is done, and then B can run.
How can I achieve that ? or is there another solution (in case locks are not supported in workflow jobs) ?
Thank you.
What exactly are you trying to prevent? The easiest way would be to set each node as having only 1 executor... If you do this, then the node will only ever run one job at a time. Note that some fly-weight tasks may run but generally these are non-significant and involve polling the remote SCM repository and such.
If you just mean within the same workflow, you can use various mix of the parallel step to split parallel sections and then combine the results.
I've got two Jenkins jobs, a continuous integration job that runs whenever a new change is submitted, and a build/deploy job that runs everyday at noon. I do not want the builds to run simultaneously, so I am using the Throttle Concurrent Builds plugin to force only these two jobs to queue up if either of them is already running. Any of my other jobs are allowed to run at the same time as these two jobs.
I would like the build/deploy job to always run at noon, but sometimes a developer submits a change just before noon causing the build/deploy job to wait until the CI job finishes.
Is there a way to block only the CI job during a defined time frame, like 11:30am until 1pm, so that the build/deploy job will not be blocked at noon?
IMHO the are few possibilities:
a first pre-step which check the date and exit with error if the date do NOT match the requirements
two jobs which disable and enable the job programmatically calling the REST
eg job 1
curl -X POST http://usr:pwd#host/job/joobname/disable
and job 2
curl -X POST http://usr:pwd#host/job/joobname/enable
at given time.
You can try to add slave node which is offline from 11:30am until 1pm and restrict change-caused build to this.
Second possibility is to enable SCM poll schedule to check for changes during allowed time-frame.