Each Thing inserted into the database has an archive attribute. When set to 0, nothing will happen. However, if it is not, I want it to be added to a queue to be processed.
Archiving a Thing can take anywhere from 3 to 30 seconds, because a lot of requests are sent and handled. So my question is do I make it so:
When a Thing has archive set to 1, have it put in a queue to be processed by a Rake task every so often (every 15 minutes or so), and then have the archive attribute set to 2, to stop it from being processed again
Make a do_archive method on Thing, and when saving, do self.delay.do_archive, and let delayed_job handle all of that for me
Some Things do not need to be processed, and the archiving isn't a time-essential thing. My gut is that delayed_job is probably a better idea, as it's not time-specific and just goes through a queue as opposed to running a script every day at midnight.
I think you pretty much nailed it in your last paragraph. If it's something that is time dependent and not event driven, then cron makes more sense. But if there is an event that occurs and can queue it up, and it is not time dependent, then use a background job.
One thing you might want to consider is whether an actual messaging system makes more sense. While something like RabbitMQ might be overkill for where you are today, there are other simpler options. Sidekiq or Resque are two popular options that give you quite a bit more control over the background jobs and offer the simplicity of delayed_job and the robustness of a messaging system.
Related
In my Rails application, I have a long calculation requiring a lot of database access.
To make it short, my calculation took 25 seconds.
When implementing the same calculation within a background job (a big single worker), the same calculation take twice the same time (ie 50 seconds). I have try several technics to put the job in a background process put none add an impact on my performances => using DelayJob / Sidekiq / doing the process within my rails but in a thread created for the work, but all have the same impact on my performances *2.
This performance difference only exist in rails 'production' environment. It looks like there is an optimisation done by rails that is not done in my background job.
My technical environment is the following =>
I am using ruby 2.0 / rails 4
I am using unicorn (but I have same problem without it).
The job is using Rails.cache to store some partial computation.
I am using postgresql
Does anybody has an clue where this impact might come from ?
I'm assuming you're comparing the background job speed to the speed of running the operation during a web request? If so, you're likely benefiting from Rails's QueryCache, which caches db queries during a web request. Try disabling it like described here:
Disabling Rails SQL query caching globally
If that causes the web request version of the job to take as long as the background job, you've found your culprit. You can then enable the query cache on your background job to speed it up (if it makes sense for your application).
Background job is not something that need to used for speed-up things. It's main meaning is to 'fire and forget' and remove 25 seconds of calculating synchronously and adding some more of calculating asynchronously. So you can give user response that she's request is processing and return with calculation later.
You may take speed gain from background job by splitting big task on some small and running them at same time. In your case I think it's something impossible to use, because of dependency of operations in yours calculation.
So if you want to speed you calculation, you need to look into denormalization of your data structure, storing some calculated values for your big calculation on moment when source data for this calculation updated. So you will calculate less on user request for results and more on data storage. And it's good place for use background job. So you finish your update of data, create background task for update caches. And if user request for calculation comes before this task is finished you will still need to wait for cache fill-up.
Update: I think I am still need to answer your main question. So basically this additional time on background task processing is comes from implementation. Because of 'fire and forget' approach no one need that background task scheduler will consume big amount of processor time just monitoring for new jobs. I am not sure completely but think that if your calculation will be two times more complex, time gain will be same 25 seconds.
My guess is that the extra time is coming from the need for your background worker to load rails and all of your application. My clue is that you said the difference was greatest with Rails in production mode. In production mode, subsequent calls to the app make use of the app and class cache.
How to check this hypotheses:
Change your background job to do the following:
print a log message before you initiate the worker
start the worker
run your calculation. As part of your calculation startup, print a log message
print another log message
run your calculation again
print another log message
Then compare the two times for running your calculation.
Of course, you'll also gain some extra time benefits from database caching, code might remain resident in memory, etc. But if the second run is much much faster, then the fact that the second run didn't restart Rails is more significant.
Also, the time between the log message from steps 1 and 3 will also help you understand the start up times.
Fixes
Why wait?
Most important: why do you need the results faster? Eg, tell your user that the result will be emailed to them after it is calculated. Or let your user see that the calculation is proceeding in the background, and later, show them the result.
The key for any long running calculation is to do it in the background and encourage the user to not wait for the result. They should be able to do something else until they get the result.
Start the calculation automatically As soon as the user logs in, or after they do something interesting, start the calculation. That way, when (and if) the user asks for the calculation, the answer will either be already done or will soon be done.
Cache the result and bust the cache as needed Similar to the above, start the calculation periodically and automatically. If the user changes some data, then restart the calculation by busting the cache. There are also ways to halt any on-going calculation if data is changed during the calculation.
Pre-calculate part of the calculation Why are you taking 25 seconds or more for a dbms calculation? Could be that you should change the calculation. Investigate adding indexes, summary tables, de-normalizing, splitting the calculation into smaller steps that can be pre-calculated, etc.
I'm using delayed_job to create large numbers of jobs, nearly all at one time, to be done at a later time, if the number of jobs gets too high, after a certain amount of time, every job is cleared from the queue regardless of it's state.
the following rails project illustrates this issue:
https://github.com/hayksaakian/taskbreaker
to recreate the issue, create several tasks (say 5 to 15), each with around 100 goals
(from the web interface, or console)
then in console attempt to do these tasks with:
Task.attempt_tasks
What will happen is the following:
Many jobs will be created, the workers do their thing for several minutes, then poof every job disappears from the queue.
To verify this is the case, check any task, you'll notice that each accomplishment may not have an arbitrary_number equal to '10' (it should be 10 since we're incrementing by one for each of the delayed attempt_string calls). The arbitrary_array of each accomplishment is also not of length 10 (which it should be, given that we delayed 10 calls to attempt_array);
I'm not sure why this is happening as I'm seeing no errors, but i'm sure that it's happening.
see an example of the bad work at taskbreaker.herokuapp.com
Note i'm hosting on heroku if that's of any help. Also you'll need at least 5 workers to recreate the issue in any reasonable amount of time.
This was due to a race condition. Since the methods were happening concurrently, their reads and writes were overlapping, resulting in unexpected output.
When certain conditions are met, I'd like to schedule a worker to run a particular job in 5 minutes. The thing is, if the same conditions are met again, I want to check if there's something scheduled to run. If there is such a worker scheduled to run, then, I don't want to enqueue again, but if there isn't, it should be queued. I hope you guys understood what I'm trying to do. Can it be achieved? If yes, how?
Sounds like you want to use or implement a simple persisted lock. The code that enqueues the job can first check for the availability of the lock, acquire and enqueue if available, skip if not. The enqueued job can be responsible for releasing the lock. You'll want to account for failure, like adding a lock timeout. The redis-mutex gem may be a useful implementation of this idea.
Best practices promote jobs that are idempotent. This means that you should be writing them in such a way that it should be safe to run them more than once. Any subsequent call doesn't change the result of the first call. You achieve this by writing logic that does the proper checks, and acts accordingly. Since you don't provide a description of what your worker does, I can't be more specific.
For an example, here is a link to the Sidekiq's FAQ: Make your workers idempotent and transactional
The benefit of this approach is that you're playing along the convenient abstraction of scheduled workers, instead of fighting against it.
I am working on a multi-user tree editing app. It uses resque gem for background processes. To avoid runtime multiuser conflicts I want to use command pattern and store user actions in a resque queue so if someone is deleting a branch other user cannot edit children of that branch.
It works, but it is quite slow to pick the job first time from a queue, because resque worker checks for the jobs using 5 seconds interval. It slows down editing interface significantly. It is possibe to do something like this:
cmd = MyCommand.create!(:attr1 => 'foo', :attr2 => 'bar')
Resque.enqueue(MyCommand, cmd.id)
workers = Resque.workers.select {|w| w.queues.include?('my_queue') }
raise "Should be only one queue for commands!" if workers.size != 1
not_done = true
while not_done
not_done = workers[0].process
end
It does what I need, but I wonder if there is a more elegant way of doing this. Also, :process is a deprecated method for Worker instances.
I think your design approach is somewhat sound, but Redis/Resque may not be appropriate. What you want is a super fast in-memory queue that's similar to Resque, but that does not come with a polling delay.
I am pretty sure you can use MemCached for this, but there maybe other options. Any solution where your queued commands have to be pulled at certain interval would probably not provide acceptable performance for collaborative editing, unless it's OK to poll maybe every 100ms or even more often.
Finally, if you are placing every action on a single queue which can only process command serially (one at a time), you are inevitably going to end up in a situation where the queue may backup because too many commands are coming in, and they are not processing as fast. This is why a more scalable solution maybe with using versioning, where each element of the tree is versioned, and when updated/changed, all child elements are updated with a new version too. That way, an edit on an older version number is rejected.
Anyway.. good luck, sounds like a non-trivial problem to solve.
Like with browser games. User constructs building, and a timer is set for a specific date/time to finish the construction and spawn the building.
I imagined having something like a deamon, but how would that work? To me it seems that spinning + polling is not the way to go. I looked at async_observer, but is that a good fit for something like this?
If you only need the event to be visible to the owning player, then the model can report its updated status on demand and we're done, move along, there's nothing to see here.
If, on the other hand, it needs to be visible to anyone from the time of its scheduled creation, then the problem is a little more interesting.
I'd say you need two things. A queue into which you can put timed events (a database table would do nicely) and a background process, either running continuously or restarted frequently, that pulls events scheduled to occur since the last execution (or those that are imminent, I suppose) and actions them.
Looking at the list of options on the Rails wiki, it appears that there is no One True Solution yet. Let's hope that one of them fits the bill.
I just did exactly this thing for a PBBG I'm working on (Big Villain, you can see the work in progress at MadGamesLab.com). Anyway, I went with a commands table where user commands each generated exactly one entry and an events table with one or more entries per command (linking back to the command). A secondary daemon run using script/runner to get it started polls the event table periodically and runs events whose time has passed.
So far it seems to work quite well, unless I see some problem when I throw large number of users at it, I'm not planning to change it.
To a certian extent it depends on how much logic is on your front end, and how much is in your model. If you know how much time will elapse before something happens you can keep most of the logic on the front end.
I would use your model to determin the state of things, and on a paticular request you can check to see if it is built or not. I don't see why you would need a background worker for this.
I would use AJAX to start a timer (see Periodical Executor) for updating your UI. On the model side, just keep track of the created_at column for your building and only allow it to be used if its construction time has elapsed. That way you don't have to take a trip to your db every few seconds to see if your building is done.