This question already has answers here:
Scheduling tasks with rails
(2 answers)
Closed 9 years ago.
We have an application that needs to perform lots of short background tasks (each might schedule more). In the future we might need to run the tasks on multiple servers.
We also need the tasks (and their parameters) to be persistent (stored in a DB) and be able to monitor tasks (status / logs /etc)
Is there a ready made solution that works with Ruby ?
You can try delayed_job or resque.
Related
This question already has answers here:
how to remove a specific class of jobs from a sidekiq queue?
(3 answers)
how to delete a job in sidekiq
(8 answers)
Closed 6 months ago.
Hi friends I am using Sidekiq for background work, in this case to send an email at a certain time if the user has not completed some steps, but I want to stop or delete that Worker for a case where the user disconnects their email account.
Is there any way to eliminate all of a worker’s jobs for a specific queue in Sidekiq ?
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am developing a system in which the API should handle simultaneous, continuous by rails 4.0
In system, each user has 3 scripts to be run in background. The scripts grab the user's information from DB to call API repeatedly and process transaction. Currently I am using cronjob (whenever gem) to run scripts in the background for each individual user
So my problem is when the system has 1,000 people, I need to run 3000 cronjobs.
I think this system will have problems. Can anyone help me solve this problem?
At this point you have a system that performs some tasks periodically, and the amount of work your system has to handle (let's say, per hour) is less than the amount of work it could handle.
However, the amount of work increases with the number of users in your system so, as you have already guessed, there will be a moment when the situation will be critical. Your system will not be able to handle all the tasks it has to do.
One way to solve this problem is adding more machines to your system, that is, if you are currently using a single machine to run all your tasks, consider adding another one and split the job. You can split the job between the machines in a number of ways, but I would use a consumer-producer approach.
You will need to use a queue manager where your producer periodically sends a batch of tasks to be done (you can still use whenever gem for that) and a number of consumers (1 makes no sense, 2 would be OK by now but you could increase this number) get the tasks done one by one until there is none left.
The manager I like the most is Sidekiq but you can find some others that might match your needs better.
This question already has answers here:
Cannot play cached audio from service worker
(2 answers)
Closed 1 year ago.
Is there any restriction on what kind of files could be cached by a service worker?
Check out https://samdutton.github.io/samples/service-worker/prefetch-video/ which works around this issue by manually created ranged responses.
Fixing this is gated on figuring out what browsers should be doing here, and updating the service worker spec if needed.
Original answer: https://stackoverflow.com/a/37614302/6773912
There are no restrictions on the kind of files you can cache.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I have a Rails application that displays thousands of products.
The products are loaded from product feeds, so the source may be a large XML file or web service API calls.
I want to be able to re-use my models in my existing rails application in my import process.
What are my options in importing data into my Rails application?
I could use sidekiq to fire off rake tasks, but not sure if sidekiq is suitable for tasks that take 10+ minutes to run? Most use cases that I have seen is for sending of emails and other similiar light tasks
I could create maybe a stand-alone ruby script, but not sure how I could re-use my Rails models if I go this route.
Update
My total product could is around 30-50K items.
Sidekiq would be a great option for this as others have mentioned. 10+ minutes isn't unreasonable as long as you understand that if you restart your sidekiq process mid run that job will be stopped as well.
The concern I have is if you are importing 50K items and you have a failure near the beginning you'll never get to the last ones. I would suggest looking at your import routine and seeing if you can break it up into smaller components. Something like this:
Start sidekiq import job.
First thing job does is reschedule itself N hours later.
Fetch data from API/XML.
For each record in that result schedule a "import this specific data" job with the data as an argument.
Done.
The key is the second to last step. By doing it this way your primary job has a much better chance of succeeding as all it is doing is reading API/XML and scheduling 50K more jobs. Each of those can run individually and if a single one fails it won't affect the others.
The other thing to remember is that unless you configure it not to Sidekiq will rerun failed jobs. So make sure that "import specific data" job can be run multiple times and still do the right thing.
I have a very similar setup that has worked well for me for two years.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I plan on running our web app on Heroku. I am looking for a gem to handle a few background jobs. i.e. sending emails, to call a few methods which submit files to an encoding service via an API, etc.
A few that have, so far, come to mind are resque and delayed_job. I hear good things about resque and it also seems to be the more popular gem in its category. Ryan Bates has done an excellent screen cast on delayed_job. However, I hear delayed_job has had a few problems. i.e. not very stable in some areas. So I hear.
Heroku offers Redis-to-Go. They have a free plan which offers 5mb. If I go with resque, is this 5mb plan enough to handle background jobs? I don't want to end up spending more just for background jobs.
Just concerned that if I went with resque, I would need another db just to run background jobs. If I was using Redis for something else, then perhaps it would be worth it. Is it worth having another db just to handle background jobs?
Should I consider alternative gems? If so which ones?
Both delayed_job and resque work fairly well. resque should scale better as the volume of background requests increases.
resque's use of redis should be limited to the task request. Large data objects that are needed by the background tasks should be stored somewhere other than the background worker queue. For example, the files being sent to a background worker to be encoded should be stored in AWS S3 or some other persistent store, not the redis queue used by resque.
When using delayed_job or resque, you will need to run background workers which cost money. You might want to look at an autoscaling solution for dynamically starting and stopping background workers as needed.
See http://s831.us/h3pKE6 as an example.
We've used delayed_job very intensively, sending hundreds of concurrent emails, and it's worked very well. flawlessly. Yes, it'll cost $36/mo for the worker. But a single worker gets a lot of jobs done... several fairly complex emails (lot of dbase lookups) sent per second.