I have two jobs, consider them to be the super simple jobs that just print a line and have no triggers or timeouts defines. They work fine when I call them from a controller class through: <name of my class>Job.triggerNow()
What I want is to trigger one job and, as it as it finishes, trigger a consequent different job.
I have tried using the quartzScheduler, but I can't seem to get a JobDetail from my job classes, so I'm not sure what is the correct way for doing this. I also want to pass some results from the first job onto the second one.
I know I can trigger the second job as the last line on my first job's execute method, but this is not desirable since its technically not part of the first job and couples things more than I would like.
Any help will be greatly appreciated. thanks
What it sounds like you are after is an asynchronous "pipeline" of work where there are different workers that are all in a line and pass data to be worked on from one to the next. This sort of architecture is amazingly flexible and applies to a large number of very common applications
The best way that I have found to get such an architecture in place with Grails is to use a message queue, like RabbitMQ for example, with a series of queues (one for each step in the pipeline), and then have the controller(s) put messages into the first step of the pipeline.
Then, you have a worker (just a service within the Grails app if you use the excellent RabbitMQ Grails plugin) listen to the queue that holds jobs for them to work on. As work comes into the queue, the worker will pop the job off, processes it, and then put a message into the queue of the next step in the pipeline.
I've found this to be the best way to architect just about any asynchronous pipeline, since it allows you to scale each piece separately as needed and doesn't have too much overhead. There are also ways to decouple the jobs from having to know about the next step in the pipeline, but I've found that in most cases this isn't really needed and just adds useless complexity.
Quartz is great for jobs that need to happen on a schedule, but a pipeline is much better at processing things as it comes in in a scaleable way
Please have a look #
JobListener
You can utilize
public void jobWasExecuted(JobExecutionContext context,
JobExecutionException jobException);
I built something similar to this in my web application using queue messaging technique with Redis. I simply define the dependency structure for all the jobs, and have a master job with the only purpose is to monitor/update the status of other jobs and trigger dependent jobs if needed.
Each job will have to report its status running/finish/cancel using the Redis queue. Master job pop each queue message and process it properly.
Related
I do have a job which requires external ressources and therefore it should not executed twice or more often. I used Block Queued Job Plugin to block the job if of a list of jobs is currently running.
This creates sometimes a build queue with some jobs blocked by the plugin ... which is correct.
But now I do need to reorder the build queue to give a specific build a chance to be executed.
There is usually just the fifo principle in place but I do need to overwrite this in specific situations manually.
simple queue plugin ... can not deal with blocked jobs
priority sorter .... sees to be outdated and not working for such a simple thing ...
Currently I write down the parameter handed over per job delete all and afterwards rebuild with the new order and with the parameters which were manually written down.
This is quit bad and I do need a working solution. Maybe I missed the right plugin.
We use delayed_job in a fairly high traffic Rails app, so there are always jobs in the queue.
After a deploy, when you restart dj, there is potential for the code in already queued jobs to become invalid. For example, if the deploy meant that a class used in the job no longer existed, the job would fail.
Is there a standard practice for getting round this?
Is there a standard practice for getting round this?
Yes, multi-stage deployments (or whatever is the proper name for this).
Step 1: Deploy a new version of code that allows both new and old jobs run at the same time. For example, if you want to remove some class, make new jobs not use it, but don't remove the class yet, so that it's available to old jobs.
Step 2: Wait until all old jobs are processed.
Step 3: Deploy a change that removes the class and the old job version.
Note: you could implement step 1 by copying old job code and giving it a new "version". If you had, say, ProjectJobs::Import, you copy it as ProjectJobs::Import2 (this version won't use class that you want to remove). Since they're different classes, DJ won't have any problems with picking appropriate implementation.
I would say, that you have to:
stop all workers right after they finish their jobs
deploy changes
start workers
I think that in your case, this code might be helpful.
I am looking for a way to be able run Active Job serially. Ideally, a long running Job 1 is scheduled to run at a certain time. A similarly running Job 2 is slated to run only after Job 1 completes. Job 3 then waits for Job 2 to run to completion before it starts and so on.
I have to admit that I am rather new to background jobs in Rails but I am already using Active Job with Sidekiq as the job runner for simple fire-and-forget tasks.
I like Active Job because it provides a simple enough interface to dive almost immediately into background jobs processing. I can use Sidekiq without having to define workers, for example.
For reference, I have achieved something similar but it was on .NET using the excellent Hangfire library which has continuations where you pass the ID of a parent job ensuring that the job will run only after the parent job has successfully completed.
It would be nice to have something as clean and simple as that using Sidekiq and Active Job but really any alternative ways to achieve the same thing are welcome. It doesn't have to be Sidekiq and Active Job.
The most straightforward way to to this is to call a third job from within a second job, and the second one from within a first job
I have several quartz scheduled jobs. With the help of attribute DisallowConcurrentExecution they do not execute simultaneously. But if they fire at one time, one job is missed. How to fix it (to put in queue)? I tried to use Mutex, but in result jobs were started and executed simultaneously, interrupting each other. I want one is done, the second will start after/ What to do?
You could use a JobChainingJobListener for this. It listens for your job to finish and then triggers the next job on the list and so on.
Take a look at the code to see how it works if you want to do this on your own.
I have some rails code that calls model1.func1(). A controller action calls this, where multiple people can be hitting it, as does a scheduled rake task. I want to make sure that model1.func1() cannot be called in parallel. If another thread needs to call at the same time, it should wait for model1.func() to finish. I guess I want to queue these calls. I was going to use sidekiq for this, but with only one worker. I read on a forum that
Sidekiq is not appropriate for the serial job and I don't want to make
it appropriate. Different tools are useful for different reasons,
jack of all trades master of none, etc.
What do you guys recommend instead?
I would consider beanstalkd with one worker process.