I'd like this job to wait a certain amount of seconds based off of a string parameter given to the job. The field only allows me to enter numbers, so I cannot use $(PARAM). I want to know what would be a good way to do this.
P.S:
Do jobs in quiet period hang executors?
You could use a pipeline job that takes in the parameter and then triggers your job, setting the quiet period in the build step (see quietPeriod at the bottom). Like this:
node {
build job: 'foo',
quietPeriod: params.QuietPeriod as int,
wait: false
}
Where you've defined QuietPeriod as the parameter with the number of seconds to wait.
For those wondering about the
'P.S: Do jobs in quiet period hang executors?'
part of the original question:
No, executors are not occupied during this quiet period.
Related
I would like to get stage or step duration for some jobs in my Jenkins Pipeline.
I have no idea how to get the duration, so I decided to calculate it. For that, I would like to get timestamp with this format : 1624869043 (because it's easier to calculate with this format).
What is the easiest way to do that ?
Use the timestamps plugin, with elapsed option and precision milliseconds.
just use timestamp option in your pipeline
options { timestamps() }
I've created a number of script syntax Jenkinsfiles with this header:
properties([
pipelineTriggers([
pollSCM('H/5 * * * *')
])
])
They almost always trigger at least twice for every commit. Once within 5 minutes of the commit, and again at +5 minutes from the first build. The builds are shorter than 5 minutes long, meaning the second build triggers after the first build has completed. Jenkins log shows both triggers as being identical, as though the second build is not aware of the first build.
I don't understand how this isn't a larger issue. It makes continuous deployment extremely dangerous. Am I missing something?
First of all, H/5 * * * * -> H means run in the next 5 mins whenever is possible. So if the build was run at 1st minute of the hour, second one may run at 6th-10th min whenever possible (Considers other jobs are running or not). And you didn't mention anything about stopping the concurrent builds also. So it will run as per your configuration.
You can stop building multiple parallel by adding this to your script.
options { disableConcurrentBuilds() }
Hope this helps.
Jenkins job trigger cron expression format is ? ? ? ? ?.
It cannot possible set the seconds unit.
But I want schedule seconds unit, like run at everyday 15:20:10.
Because there is a job with cron(0 12 * * *) and every minute job, and these two job must not to run simultaneously.
How can I do that?
Thanks for help.
you can have several options to do that :
use lock plugin to wait until your first job run will finish and only than trigger it.
use the Quiet period option in the Job configuration , add 10 seconds so the job will sleep 10 seconds after it trigger.
I have a pipeline that looks like
pipeline.apply(PubsubIO.read.subscription("some subscription"))
.apply(Window.into(SlidingWindow.of(10 mins).every(20 seconds)
.triggering(AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(20 seconds))
.withAllowedLateness(Duration.ZERO)
.accumulatingFiredPanes()))
.apply(RemoveDuplicates.create())
.apply(Window.discardingFiredPanes()) // this is suggested in the warnings under https://cloud.google.com/dataflow/model/triggers#window-accumulation-modes
.apply(Count.<String>globally().withoutDefaults())
This pipeline overcounts distinct values significantly (20x normal value). Initially, I was suspecting that the default trigger may have caused this issue. I have tweaked to use triggers that allow no lateness/discard fired panes/use processing time, all of which have similar overcount issues.
I've also tried ApproximateUnique.globally: it failed during pipeline construction because of an exception that looks like
Default values are not supported in Combine.globally() if the output PCollection is not windowed by GlobalWindows. There seems no way to add withoutDefaults to it (like we did with Count.globally).
Is there a recommended way to do COUNT(DISTINCT) in dataflow/beam streaming pipeline with reasonable precision?
P.S. I'm using Java Dataflow SDK 1.9.0.
Your code looks OK; it shouldn't overcount. Note that you are placing each element into 30 windows, so if you have a window-unaware sink (equivalent to collapsing all the sliding windows) you would expect precisely 30 times as many elements. If you could show a bit more of the pipeline or how you are observing the counts, that might help.
Beyond that, I have a few suggestions for the pipeline:
I suggest changing your trigger for RemoveDuplicates to AfterPane.elementCountAtLeast(1); this will get you the same result at lower latency, since later elements arriving will have no impact. This trigger, and your current trigger, will never fire repeatedly. So it does not actually matter whether you set accumulatingFiredPanes() or discardingFiredPanes(). This is good, because neither one would work with the rest of your pipeline.
I'd install a new trigger prior to the Count. The reason is a bit technical, but I'll try to describe it:
In your current pipeline, the trigger installed there (the "continuation trigger" of the trigger for RemoveDuplicates) notes the arrival time of the first element and waits until it has received all elements that were produced at or before that processing time, as measured by the upstream worker. There is some nondeterminism because it puns the local processing time and the processing time of other workers.
If you take my advice and switch the trigger for RemoveDuplicates, then the continuation trigger will be AfterPane.elementCountAtLeast(1) so it will always emit a count as soon as possible and then discard further data, which is very wrong.
I have a quartz job schedule like so:
_scheduler.ScheduleJob(job,
TriggerBuilder.Create().ForJob(job)
.WithSimpleSchedule(x =>
x.WithIntervalInSeconds(requeryTime.ToInt32())
.RepeatForever()
.WithMisfireHandlingInstructionNowWithRemainingCount()
)
.StartNow()
.Build())
To test I have the schedule run every 10 seconds. If I pause the service and then resume, I can still see all the jobs being called one after another. I want Quartz to ignore (do not queue) the jobs and just resume the schedule.
e.g. If the job runs every 10 secs and I pause the service for 1 minute, I get 6 jobs run in quick succession.
How do I go about ensuring Quartz ignores those jobs?
Short answer is to change the misfireThreshold in the configuration:
<add key="quartz.jobStore.misfireThreshold" value="60000" />
The default is 60000 milliseconds (60 seconds).
org.quartz.jobStore.misfireThreshold
The the number of milliseconds the scheduler will 'tolerate' a trigger
to pass its next-fire-time by, before being considered "misfired". The
default value (if you don't make an entry of this property in your
configuration) is 60000 (60 seconds).
Since your trigger runs every 10 seconds you can reduce the threshold to 9 seconds and it should work as you expect.
If you want to learn more about the strategies involved you can read this article and this one as well.
A similar question has been asked recenlty and Circular Reference gave a really good answer.