Quartz.Net uses XML files to configure the jobs that are to be run and the schedules for running those jobs. As far as I understand, the only other option to XML configuration is to configure the jobs and schedules at compile time.
We have a large number of scheduled tasks (250 tasks that are each deployed into between 3 to 5 environments) and are looking to migrate these to use Quartz.Net.
[1] Are there any best practices to managing the job and schedule configuration information in Quarts.Net?
[2] Is there any tooling for the creation of the configuration files?
An example configuration for a job is:
<job>
<name>addDirectoryScanListener</name>
<group>directoryScanJobExample</group>
<description>Sample job for Quartz Server</description>
<job-type>Examples.DirectoryScanListenerExample, Examples</job-type>
</job>
<trigger>
<simple>
<name>addDirectoryScanListenerSimpleTrigger</name>
<group>directoryScanJobExampleSimpleTriggerGroup</group>
<description>Simple trigger to simply fire sample job</description>
<job-name>addDirectoryScanListener</job-name>
<job-group>directoryScanJobExample</job-group>
<misfire-instruction>SmartPolicy</misfire-instruction>
<repeat-count>0</repeat-count>
<repeat-interval>10000</repeat-interval>
</simple>
</trigger>
There are couple of built-in ways to configure Quartz.NET, here's a small comparison.
Code based configuration
You use the scheduler API to create/update/delete jobs and triggers
Schedules and jobs configured by hard coding naturally require recompilation, using a UI or such with scheduler API and persistent job store is a different story
XML Configuration
You can schedule jobs and watch XML file for changes
See Can quartz.net reconfigure jobs when config file changes? for example of watching for file changes
There is no tooling available as far as I know besides intellisense with Visual Studio when you have the XSD file associated with XML file
Then again, you have job store options. You can use any job store regardless of the way you configure your scheduler/jobs/triggers.
RAMJobStore
Fastest (but usually not a factor) and simplest to setup job store but no way to recover from missed trigger fires caused by server reboots etc
AdoJobStore
Stores triggers, jobs and their states between restarts and allows you to run Quartz.NET clustered for fail-over and scale
There are also couple web based administration UI available, see Any open-source admin UI for quartz.NET for details.
The Quartz.NET tutorial is always a good place to start too.
Related
Spring Cloud Data Flow is a great solution and currently I'm trying to find the possibility to preconfigure the tasks in order to trigger them manually.
The use-case very simple:
as a DevOps I should have ability to preconfigure the tasks, which includes creation of the execution graph and application and deploy parameters and save task with all parameters needed for execution.
as a User with role ROLE_DEPLOY I should have ability start, stop, restart and execution monitor the preconfigured tasks.
Preconfigured tasks is the task with all parameters needed for execution.
Is it possible to have such functionality?
Thank you.
You may want to review the continuous deployment section from the reference guide. There's built-in lifecycle semantics for orchestrating tasks in SCDF.
Assuming you have the Task applications already built and that the DevOps persona is familiar with how the applications work together, they can either interactively or programmatically build a composed-task definition and task/job parameters in SCDF.
The above step persists the task definition in the SCDF's database. Now, the same definition can be launched manually or via a cronjob schedule.
If nothing really changes to the definition, app version, or the task/parameters, yes, anyone with the ROLE_DEPLOY role can interact with it.
You may also find the CD for Tasks guide useful reference material. Perhaps repeat this locally (with desired security configurations) on your environment to get a hold of how it works.
We have been using spring batch for below use cases
Read data from file, process and write to target database (batch
kicks off when file arrives)
Read data from remote database, process and write to target database (runs on scheduled interval, triggered
by Autosys)
With the plan to move all online apps to spring-boot microservices and PCF, we are looking at doing a similar excercise on the batch side if it adds value.
In the new world, the spring cloud batch job task will be reading the file from S3 storage (ECSS3).
I am looking at good design here (stay away from too many pipes/filters and orchestration if possible), the input data ranges from 1MM to 20MM records
ECSS3 will notify on file arrival by sending an http request, the
workflow would be - clould stram httpsource->launch clould batch job task that will read from object store, process and save records to target database
Spring Clould Job Task triggered from PCF scheduler to read from remote database, process and save to target database
With the above design, I don't see the value of wrapping the spring batch job into clould task and running in the PCF with spring data flow
Am I missing something here ? Is PCF/SpringClouldDataFlow an overkill in this case ?
Orchestrating batch-jobs in a cloud setting could bring new benefits to the solution. For instance, the resiliency model that PCF supports could be useful. Spring Cloud Task (SCT) are typically run in a short-lived container; if it goes down, PCF will bring it back up and run in it.
Both the options listed above are feasible and it comes down to the use-case wrt the frequency in which you're processing the incoming data. It is really real-time or it can happily run on a schedule is something you'd have to determine to make the decision.
As for the applicability of Spring Cloud Data Flow (SCDF) + PCF, again, it comes down to your business requirements. You may not be using it now, but Spring Batch Admin is EOL in favor of SCDF's Dashboard. The following questions might help realize the SCDF + SCT value proposition.
Do you have to monitor the overall batch-jobs' status, progress, and health? Maybe you've requirements to assemble multiple batch-jobs as a DAG? How about visually composing a series of Tasks and orchestrate it entirely from the Dashboard?
Also, when the batch-jobs are used together with SCT, SCDF, and PCF Scheduler, you'd get the benefit to monitoring all of this from the PCF Apps Manager.
I want to send email batch at specific time like CRON.
I think whenever gem (https://github.com/javan/whenever) is not to fit in Cloud Foundry Environment. Because Cloud Foundry can't use crontab.
Please inform me what options are available to me.
There's a node.js app here that you could use to schedule a specific rake task.
I haven't worked with cloudfare so I'm not sure if it'll serve your needs, but you can also try some of the batch job processing tools rails has available: Delayed job and sidekiq. Those store data for recurring jobs either on your database (DJ) or in a separate redis database (Sidekiq) and both need keeping extra processes up and running, so review them deeply and the changes you'd need for your deployment process before using each one. There's also resque, and here's a tutorial to use it with rails for scheduling tasks.
There are multiple solutions here, but the short answer is that whatever you end up doing needs to implement its own scheduler. This is because there is no cron service available to your application when it runs on CF. This means there is nothing to trigger or schedule your actions. Any project or solution that depends on cron will not work when deploying to CF. Any project that implements it's own scheduler should work fine.
Some specific things I've seen people do successfully:
Use a web service that sends HTTP requests to your app on predefined intervals. The requests trigger your action. It's the services responsibility to let you define when to trigger and to send the HTTP requests. I'm intentionally avoiding mentioning any specific services, but you can find them by searching for "cron http service" or something like that.
Importing a library that has cron like functionality. I'm not familiar with Ruby, so I don't know the landscape there. #mlabarca has mentioned a couple that you might try out. Again, look to see that they implement the scheduling functionality and do not depend on cron. I'm more familiar with Java where you have Quartz and Spring, which has some scheduling functionality too.
Implement a "clock" process or scheduler. This would generally be a second app that you deploy on CF. It would be lightweight and probably not have a web interface. It could be as simple as do something, sleep, loop for ever repeating those two steps. It really depends on your needs. You could even get fancy and implement something like the first option above where you're sending some sort of request to your other apps to trigger the actual events.
There are probably other solutions as well, those are just some examples to get you started.
Probably also worth mentioning that the Cloud Controller v3 API will have first class features to run tasks. In this case, the "task" is some job that runs in a finite amount of time and exits (like a batch job). This is opposed to the standard "app" that when run on CF should continue executing forever (i.e. if it exits, it's cause of a crash). That said, I do not believe it will include a scheduler so you'd still need something to trigger the task.
I have been looking at using projects built using spring-cloud-task within spring-cloud-dataflow. Having looked at the example projects and the documentation, the indication seems to be that tasks are launched manually through the dashboard or the shell. Does spring-cloud-dataflow provide any way of scheduling task definitions so that they can run for example on a cron schedule? I.e. Can you create a spring-cloud-task app which itself has no knowledge of a schedule, but deploy it to the dataflow server and configure the scheduling there?
Among the posts and blogs I have looked at I noticed the following:
https://spring.io/blog/2016/01/27/introducing-spring-cloud-task
Some of the Q&A afterwards hints at this being a possibility, with the reference to triggers, but I think this was discussed before it was released.
Any advice would be greatly appreciated, many thanks.
There are few ways you could launch Tasks in Spring Cloud Data Flow. Following are the available options today.
Launch it using TriggerTask; with this you could either choose to launch it with fixedDelay or via a cron expression - example here.
Launch it via an event in streaming pipeline. Imagine a use-case where you would want to create a "thumbnail" as and when there's a new image (event) in s3-bucket or in a file-system directory; the "thumbnail" operation could be a task in this case - example here.
Lastly, in the upcoming releases, we will port over "scheduler" functionality from Spring XD to Spring Cloud Data Flow.
Yes, Spring Cloud Data Flow does provide a scheduling option. To enable it, you need to add below arguments while starting the server:
--spring.cloud.dataflow.features.schedules-enabled=true
I have one SQL Server database as the job store, two web applications that both can schedule jobs, and a Quartz.NET windows service to execute jobs.
I want the two web applications to just schedule jobs, while the windows service just to execute jobs.
Here comes the problem:
If I create IScheduler instances in the two web applications and in the windows service, they will execute jobs at the same time, there can be conflicts.
If I do not create IScheduler instances in the two web applications, how can I schedule jobs to the windows service from web applications?
Is there a way to let the IScheduler to just schedule jobs without executing jobs? (I can deploy the IJob assemblies to all these three applications)
You probably don't want to instantiate an IScheduler instance in the websites, precisely because creating a local instance also executes jobs.
I've implemented something similar to what you're looking to do.
First, make sure that in your service configuration file (app.config) that you configure the four keys quartz.scheduler.exporter.type, quartz.scheduler.exporter.port, quartz.scheduler.exporter.bindName, and quartz.scheduler.exporter.channelType.
Second, make sure that your web.config has the following four keys configured : quartz.scheduler.instanceName, quartz.scheduler.instanceId, quartz.scheduler.proxy, and quartz.scheduler.proxy.address.
Then when you create your StdSchedulerFactory() and use it to get a scheduler, you are not instantiating a new scheduler, but attaching to an existing scheduler. You can then do anything through the remote scheduler that you could do with a local one, but there is only a single instance that executes jobs.
In your config file, set key "quartz.threadPool.type" to "Quartz.Simpl.ZeroSizeThreadPool, Quartz". The quartz scheduler thus created can schedule/modify jobs/trigger/calendars but can not run the jobs.