We use delayed_job in a fairly high traffic Rails app, so there are always jobs in the queue.
After a deploy, when you restart dj, there is potential for the code in already queued jobs to become invalid. For example, if the deploy meant that a class used in the job no longer existed, the job would fail.
Is there a standard practice for getting round this?
Is there a standard practice for getting round this?
Yes, multi-stage deployments (or whatever is the proper name for this).
Step 1: Deploy a new version of code that allows both new and old jobs run at the same time. For example, if you want to remove some class, make new jobs not use it, but don't remove the class yet, so that it's available to old jobs.
Step 2: Wait until all old jobs are processed.
Step 3: Deploy a change that removes the class and the old job version.
Note: you could implement step 1 by copying old job code and giving it a new "version". If you had, say, ProjectJobs::Import, you copy it as ProjectJobs::Import2 (this version won't use class that you want to remove). Since they're different classes, DJ won't have any problems with picking appropriate implementation.
I would say, that you have to:
stop all workers right after they finish their jobs
deploy changes
start workers
I think that in your case, this code might be helpful.
Related
I do have a job which requires external ressources and therefore it should not executed twice or more often. I used Block Queued Job Plugin to block the job if of a list of jobs is currently running.
This creates sometimes a build queue with some jobs blocked by the plugin ... which is correct.
But now I do need to reorder the build queue to give a specific build a chance to be executed.
There is usually just the fifo principle in place but I do need to overwrite this in specific situations manually.
simple queue plugin ... can not deal with blocked jobs
priority sorter .... sees to be outdated and not working for such a simple thing ...
Currently I write down the parameter handed over per job delete all and afterwards rebuild with the new order and with the parameters which were manually written down.
This is quit bad and I do need a working solution. Maybe I missed the right plugin.
The idea is when 2-3 concurrent commits are pushed to a branch, it shouldn't start all build jobs for a given step in circleCI. It should wait until the 1st job is finished and then only run the next one in the queue.
I have tried using the below links but no luck. Please help.
* https://circleci.com/orbs/registry/orb/gastfreund/dynamo-lock?version=1.0.1
* https://circleci.com/orbs/registry/orb/freighthub/lock
You can take a look at https://circleci.com/developer/orbs/orb/eddiewebb/queue#usage-examples as it seems quite robust, is still maintained and has pretty good documentation. It also lets you define a custom job and queue the entire workflow or just a specific job(s).
We are using Rails 4.2 and Sidekiq for processing jobs. Our application schedules jobs to be performed at some point in the future for our users, and thus we have probably thousands of currently scheduled jobs awaiting execution.
I am doing some substantial refactoring of the code underlying these jobs, changing the parameters and whatnot. My question is: when I deploy my new code, will the currently pending jobs -- which were scheduled using the old code -- be affected by my new code when they run?
I assume the answer is no, and that scheduled jobs include the code they are to process. But I'd feel a lot better with some confirmation. My googling did not reveal an answer.
Consider jobs stored in Redis to be exactly like data in a database. If you want to change them, you need to have a proper migration.
So the answer to your question is yes. The scheduled jobs will use the code that is deployed when they run, not when they were scheduled.
I have two jobs, consider them to be the super simple jobs that just print a line and have no triggers or timeouts defines. They work fine when I call them from a controller class through: <name of my class>Job.triggerNow()
What I want is to trigger one job and, as it as it finishes, trigger a consequent different job.
I have tried using the quartzScheduler, but I can't seem to get a JobDetail from my job classes, so I'm not sure what is the correct way for doing this. I also want to pass some results from the first job onto the second one.
I know I can trigger the second job as the last line on my first job's execute method, but this is not desirable since its technically not part of the first job and couples things more than I would like.
Any help will be greatly appreciated. thanks
What it sounds like you are after is an asynchronous "pipeline" of work where there are different workers that are all in a line and pass data to be worked on from one to the next. This sort of architecture is amazingly flexible and applies to a large number of very common applications
The best way that I have found to get such an architecture in place with Grails is to use a message queue, like RabbitMQ for example, with a series of queues (one for each step in the pipeline), and then have the controller(s) put messages into the first step of the pipeline.
Then, you have a worker (just a service within the Grails app if you use the excellent RabbitMQ Grails plugin) listen to the queue that holds jobs for them to work on. As work comes into the queue, the worker will pop the job off, processes it, and then put a message into the queue of the next step in the pipeline.
I've found this to be the best way to architect just about any asynchronous pipeline, since it allows you to scale each piece separately as needed and doesn't have too much overhead. There are also ways to decouple the jobs from having to know about the next step in the pipeline, but I've found that in most cases this isn't really needed and just adds useless complexity.
Quartz is great for jobs that need to happen on a schedule, but a pipeline is much better at processing things as it comes in in a scaleable way
Please have a look #
JobListener
You can utilize
public void jobWasExecuted(JobExecutionContext context,
JobExecutionException jobException);
I built something similar to this in my web application using queue messaging technique with Redis. I simply define the dependency structure for all the jobs, and have a master job with the only purpose is to monitor/update the status of other jobs and trigger dependent jobs if needed.
Each job will have to report its status running/finish/cancel using the Redis queue. Master job pop each queue message and process it properly.
I'm using Delayed Job to manage background work.
However I have some tasks that need to be executed at regular interval. Every hour, every day or every week for example.
For now, when I execute the task, I create a new one to be executed in one day/week/month.
However I don't really like it. If for any reason, the task isn't completely executed, we don't create the next one and we might lose the execution of the task.
How do you manage that kind of things (with delayed job) in your rails apps to be sure your regular tasks list remains correct ?
If you have access to Cron, I highly recommend Whenever
http://github.com/javan/whenever
You specify what you want to run and at what frequency in dead simple ruby, and whenever supplies rake tasks to convert this into a crontab and to update your system's crontab.
If you don't have access to frequent cron (like I don't, since we're on Heroku), then DJ is the way to go.
You have a couple options.
Do what you're doing. DJ will retry each task a certain number of times, so you have some leniency there
Put the code that creates the next DJ job in an ensure block, to make sure it gets created even after an exception or other bad event
Create another DJ that runs periodically, checks to make sure the appropriate DJs exist, and creates them if they don't. Of course, this is just as error prone as the other options, since the monitor and the actual DJ are both running in the same env, but it's something.
Is there any particular reason why you wouldn't use cron for this type of things?
Or maybe something more rubyish like rufus-scheduler, which is quite easy to use and very reliable.
If you don't need queuing, these tools are a way to go, I think.