We have a jsr 352 java batch application that we are wanting to stop the execution early if a condition is met but not throw an exception. The basic layout of our job is as follows:
<job id=XXX>
<step id="XXX">
<listeners>
<listener ref=XXXXX /> <--Step listener
<listener ref=XXXX /> <-- Item process listener
</listeners>
<chunk checkpoint-policy="item" item-count="1">
<reader ref=XXXXXX />
<processor ref=XXXXXX />
<writer ref=XXXXXX />
</chunk>
</step>
</job>
During our step listener, in the beforeStep() method, if a specific condition is met we do not want to continue the rest of the batch job, we just want to gracefully stop. Up until now the only things that would end the batch job would be an exception being raised for an error scenario, or a successful completion of the job.
How would we accomplish skipping the rest of the step, since its the only step, if that condition is met inside the step listener? Also, is throwing an exception the recommended way to end the batch process during an error scenario, or is there a better/more recommended way?
You can have an external component to watch for the condition and stop the current job execution upon the condition. Calling JobOperator stop method will stop the job execution gracefully.
Or you can have your item reader checks for the condition, and returns null upon true which will naturally end the step execution.
Related
It may sound like a known issue but the problem is that when system reboots, the containers don't start and appear to be in the Exited status. We're using docker-compose to start up the containers (in total about ~10 containers launched as a PowerShell script).
The docker documentation says to use the restart_policy but that mainly deals with container crashes. https://docs.docker.com/compose/compose-file.
The restart always flag is also set in the config file and doesn't seem to help, have tried setting up the task scheduler however it's still the same issue.
I'm wondering if there's a way the containers will be started gracefully or if it could be set up in Task Scheduler?
You could create and schedule task to stop the containers at system startup first and create another task to schedule an event on the successful completion of the previous task.
The important thing for another task is to edit the new event filter in XML format and to update the original task upon the successful completion of which we want to trigger a new task.
<QueryList>
<Query Id="0" Path="Microsoft-Windows-TaskScheduler/Operational">
<Select Path="Microsoft-Windows-TaskScheduler/Operational">*[System[Provider[#Name='Microsoft-Windows-TaskScheduler'] and Task = 102]]</Select>
</Query>
</QueryList>
You need to edit query manually and to replace the following line in the XML filter:
*[System[Provider[#Name='Microsoft-Windows-TaskScheduler'] and Task = 102]]
with:
*[EventData [#Name='TaskSuccessEvent'][Data[#Name='TaskName']='\Original\Task']]
The event filter details for the new task are as follows:
Events Logs: Microsoft-Windows-TaskScheduler/Operational
Event source: TaskScheduler
Task category: Task completed (status 102)
The event ID of the original task with the completion status code id 102:
EventID: 102
Provider-Name: Microsoft-Windows-TaskScheduler
Channel: Microsoft-Windows-TaskScheduler/Operational
TaskName: \Original\Task
Finally, add the action details with the program executable path and script/command (passing it as the argument) and save your changes to be able to run with the highest privileges.
I am using global window with repeated forever after processing time trigger to process streaming data from pub-sub as below :
PCollection<KV<String,SMMessage>> perMSISDNLatestEvents = messages
.apply("Apply global window",Window.<SMMessage>into(new GlobalWindows())
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardMinutes(1))))
.discardingFiredPanes())
.apply("Convert into kv of msisdn and SM message", ParDo.of(new SmartcareMessagetoKVFn()))
.apply("Get per MSISDN latest event",Latest.perKey()).apply("Write into Redis", ParDo.of(new WriteRedisFn()));
Is there a way to make repeatedly forever apache beam trigger to only execute after the previous execution is completed ? The reason for my question is because the next trigger processing will need to read data from redis, written by the previous trigger execution.
Thank You
So the trigger here would fire at the interval you provided. The trigger is not aware of any downstream processing so it's unable to depend on such steps of your pipeline.
Instead of depending on the trigger for consistency here, you could add a barrier (a DoFn) that exists before the Write step and only gives up execution after you see the previous data in Redis.
You could try and explicitly declare a global window trigger, as the example below:
Trigger subtrigger = AfterProcessingTime.pastFirstElementInPane();
Trigger maintrigger = Repeatedly.forever(subtrigger);
I think that triggers would help you on your case, since it will allow you to create event times, which will run when you or your code trigger them, so you would only run repeatedly forever when a trigger finishes first.
I found this documentation which might guide you on the triggers you are trying to create.
I wanted to run a script if a split brain is detected. In the documentation, it is mentioned that we can do that by providing the path of the script like
resource <resource>
handlers {
split-brain <handler>;
...
}
But, below that, for the after-sb-0pri configuration, it is mentioned that
disconnect: Do not recover automatically, simply invoke the
split-brain handler script (if configured), drop the connection and
continue in disconnected mode.
So, my question is that, will the configured script be run only when after-sb-0pri is set to disconnect, or will that run for any set value
Document Link: https://linbit.com/drbd-user-guide/users-guide-drbd-8-4/#s-configure-split-brain-behavior
DRBD should invoke the split-brain handler whenever a split-brain is detected. I.E. Anything that would log a Split-Brain detected ... in the system/kernel logs. The documentation attempts to explain this at the beginning of chapter 5.17.1 with:
DRBD invokes the split-brain handler, if configured, at any time split
brain is detected.
Additionally, disconnect is the default value for after-sb-0pri. So, even if not explicitly set that will still be the behavior.
I'm a Java Batch newbie. I deployed a simple batch job that includes a JobListener on WebSphere Liberty 17.0.0.4 (note that I'm using IBM's JSR-352 implementation, not Spring Batch). The batch job itself runs as expected: it reads an input file, does a simple data transformation, and writes to a DB. But in the JobListener's afterJob() method for a successful execution, I see a batch status of STARTED and an exit status of null. (These are the same values I see logged in the beforeJob() method). I expected afterJob() to see a status of COMPLETED unless there were exceptions.
The execution id logged by afterJob() is the same value returned by JobOperator.start() when I kick off the job, so I know I'm getting the status of the correct job execution.
I couldn't find any examples of a JobListener that fetches a batch status, so there's probably a simple error in my JSL, or I'm fetching the batch status incorrectly. Or do I need to explicitly set a status somewhere in the implementation of the step? I'd appreciate any pointers to the correct technique for setting and getting a job execution's final batch status and exit status.
Here's the JSL:
<job ...>
<properties>...</properties>
<listeners>
<listener ref="jobCompletionNotificationListener"/>
</listeners>
<flow id="flow1">
<step id="step1">...</step>
</flow>
</job>
Here's the listener's definition in batch.xml:
<ref id="jobCompletionNotificationListener"
class="com.llbean.batch.translatepersonnames.jobs.JobCompletedListener"/>
Here's the JobListener implementation:
#Dependent
#Named("jobCompletedListener")
public class JobCompletedListener implements JobListener {
...
#Inject
private JobContext jobContext;
#Override
public void afterJob() {
long executionId = jobContext.getExecutionId();
JobExecution jobExecution = BatchRuntime.getJobOperator().getJobExecution(executionId);
BatchStatus batchStatus = jobExecution.getBatchStatus();
String exitStatus = jobExecution.getExitStatus();
logger.info("afterJob(): Job id " + executionId + " batch status = " + batchStatus +
", exit status = " + exitStatus);
...
}
}
I tried adding <end on="*" exit-status="COMPLETED"/> to the <job> and <flow> in the JSL, but that had no effect or resulted in a status of FAILED.
Good question. Let me tack on a couple points to #cheng's answer.
First, to understand why we implemented it this way, consider the case where the JobListener throws an exception. Should that fail the job? In Liberty we decided it should. But if the job already had a COMPLETED status, well, that would imply that it was... completed, and shouldn't be able to fail at that point.
So afterJob() is really more like "end of job" (or you could think of it as "after job steps").
Second, one reason to even ask this question is because you want to know, in the afterJob() method, whether the job executed successfully or not.
Well, in the Liberty implementation at least (which I work on, for IBM), you can indeed see this. A previous failure will have set the BatchStatus to FAILED while a successful (this far) execution will have its status still at STARTED.
(For what it's worth, this was an area we realized could use more attention and standardization a bit late in the 1.0 spec effort, and I hope we can address more in the future.)
If it helps and you're interested, you can see the basic logic in the flow leading up to and including the WorkUnitThreadControllerImpl.endOfWorkUnit call here.
It's because job listener's afterJob method is part of the job execution. So when you call getBatchStatus() inside afterJob() method, the job execution is still going on, and not yet completed, hence the batch status STARTED.
On a working Grails 2.2.5 system, we're occasionally losing connection to the MySQL database, for reasons that are not relevant here. The majority of the system recovers perfectly well from the outage. But any Quartz jobs (using Quartz plugin 0.4.2) are typically failing to run again after such an outage. This is a typical message which appears in the log at the point the job should run:
2015-02-26 16:30:45,304 [quartzScheduler_Worker-9] ERROR core.ErrorLogger - Unable to notify JobListener(s) of Job to be executed: (Job will NOT be executed!). trigger= GRAILS_JOBS.quickQuoteCleanupJob job= GRAILS_JOBS.com.aire.QuickQuoteCleanupJob
org.quartz.SchedulerException: JobListener 'sessionBinderListener' threw exception: Already value [org.springframework.orm.hibernate3.SessionHolder#593a9498] for key [org.hibernate.impl.SessionFactoryImpl#c8488d7] bound to thread [quartzScheduler_Worker-9] [See nested exception: java.lang.IllegalStateException: Already value [org.springframework.orm.hibernate3.SessionHolder#593a9498] for key [org.hibernate.impl.SessionFactoryImpl#c8488d7] bound to thread [quartzScheduler_Worker-9]]
at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1868)
at org.quartz.core.JobRunShell.notifyListenersBeginning(JobRunShell.java:338)
at org.quartz.core.JobRunShell.run(JobRunShell.java:176)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Caused by: java.lang.IllegalStateException: Already value [org.springframework.orm.hibernate3.SessionHolder#593a9498] for key [org.hibernate.impl.SessionFactoryImpl#c8488d7] bound to thread [quartzScheduler_Worker-9]
at org.quartz.core.QuartzScheduler.notifyJobListenersToBeExecuted(QuartzScheduler.java:1866)
... 3 more
What do I need to do to make things more robust, so that the Quartz jobs recover as well?
By default, a Quartz job will get a session bound to it. Disable that session binding and let your service handle the transaction / session. That's what we do and when we get our DB connections back up, jobs still work.
To disable session binding in your job, add :
def sessionRequired = false