I have done a lot of searching and I am aware of grails-executor and the JMS plugin. I am looking for advice on the best way to implement a long running (as long as the application is running) service that runs in the background and accepts input on a blocking queue. It seems that there at two ways to satisfy my requirements... 1. JMS (which feels overly heavy handed) and 2. a service running on a thread that watches the queue... when something is added to it, it processes it and then waits for the next item. This service needs to have GORM capability so that it can create/save objects. My preference is to startup some type of service on a thread and use a blocking queue... Can anyone suggest the best way to do this? Should I just implement a class that gets called when grails bootstraps and have that class use the grails-executor to create a thread that just runs in the background? If anyone has used the jms plugin in grails, is it sufficiently lightweight that I should reconsider my position on this? Any and all advice is greatly appreciated. I am really NOT tied to any one solution, so all ideas will be considered and very much appreciated.
Thanks in advance!
I use the quartz plugin for a lot of similar "queue watching" functionality.
You can use Spring integration instead. With quartz you have to develop you enqueuing logic but with spring integration every thing is pre-developed.
Related
Problem: I have a program that will do a lot of post-processing on a file and then sending it ANOTHER web-service for more processing. Does my design smell and is this the right way to ‘tackle a problem’
Rails Accepts file and kicks off a resque_job to do work
The resque job will sends work via REST to another web-service cluster(very slow.. does MORE work) && places the task to be monitored in a monitor_queue within Rabbit MQ for completion . The Resque job will not wait another-webservice to complete task. It exits
There are some smell issues in my design, perhaps gut reactions that could be misguided?
Is it good design to have TWO message queues. The rational is that Rabbit_MQ has a built in method for creating worker_queues (they even give sample code). Resque_Job uses redis, and seems to be the accepted method of having ‘delayed jobs’ with Rails.
What I like about RabbitMQ: It has round-robin tasking abilities (so all threats get tasked) , and guarantees work will not be removed from a QUEUE without a message acknowledgement.
Resque seems to primary suggested solution for launching delayed_jobs within Rails.
Followup: When performing the polling, I was thinking a simple worker_queue of just iterating through the entire queue with seperate 'workers' makes the most sense? Do you agree.
I don't think this is a bad design. It's a type of Service Oriented Architecture, and even though you have separate queuing systems, they're completely separate applications and would only communicate through a specific interface, which has some pros and cons. I didn't quite understand the reasoning for using RabbitMQ, though. Also, a lot of new apps seem to be using Sidekiq; IMHO it is superior in every way to Resque.
I have a web app (ASP.NET). There are some scheduled tasks and background tasks that need to be run regularly (for example email queue, search indexer...). My question is should I create a windows service to handle all those jobs, or separate ones for each job? What is best practice?
Thank you.
I think it would be better to create separate services for each job. This will help if any functionality related to a job causes problem and stops the service it won't affect the other job.
I was wondering whether there is a possibility to call a method which is defined in daemons from a controller.
The reason behind this question is, I need to contact third party server which allows only one connection at a time. To make make a connection independent of passenger instance and to avoid multiple connection open and close I thought of using Deamon gem.
Is it there some other way to over come this problem?
Thanks for your help in advance.
You could use something like resque:
Resque (pronounced like "rescue") is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.
Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both.
I am developing my first web application using ASP.Net MVC, and I am in a situation where I would like a background service to process status notifications outside of the application, not unlike the reputation/badge system on stackoverflow.
What is the best way to handle something like this? Is it even possible in a shared-hosting environment like Godaddy, which I am using.
I don't need to communicate with the background worker directly, since I will be adding notification records to a database table with a column set to an "unprocessed" state. Then the worker will just scan the table on a regular schedule and processes what is ready.
Thanks for your advice.
Have you tried with quartz.net? I think it may fit your needs.
also take a look at this Simulate a Windows Service using ASP.NET to run scheduled jobs article.
it explains a nice way to schedule operations with no outer dependence.
The idea is to use Cache timeout to control the schedule. I've implemented it successfully on a project which required regular temp file cleaning. This cleaning is a bit heavy so we move this clean operation in a scheduled job (using the asp.net cache) to avoid having to deploy scheduled task or custom program.
To answer whether GoDaddy will support a seperate service you need to ask them.
However there are a number of creative ways that you can "get around" this issue on shared hosting.
Have a secure page that's purpose is to execute your background work. You could have scheduled task on a machine under your control that calls to this web page at set intervals.
Use a variation of the Background Worker Thread answer from #safi. Your background worker thread could check to see if another is already processing and stop, so that only one instance is running at a time.
If only one background task is enough for you then use the WebBackgrounder
And this is the article with detailed explanation.
I have a bit of code that needs to sit on a windows server 2003 machine and run every minute.
What is the recommended way of handling this? Is it ok to design it as a console service and just have the task scheduler hit it ever minute? (is that even possible?) Should I just suck it up and write it as a windows service?
Since it needs to run every single minute, I would suggest writing a Windows Service. It is not very complicated, and if you never did this before, it would be great for you to learn how it is done.
Calling the scheduled task every minute is not something I would recommend.
I would say suck it up and write it as a Windows service. I've not found scheduled tasks to be very reliable and when it doesn't run, I have yet to find an easy way to find out why it hasn't.
Windows Scheduled Tasks has been fairly reliable for our purposes and we favor them in almost all cases over Windows Services due to their ease of installing and advanced recovery features. The always on nature of a windows service could end up being a problem if a part of the code that was written ends ups getting locked up or looped in a piece of code that it shouldn't be in. We generally write our code in a fashion similar to this
Init();
Run();
CleanUp();
Then as part of the Scheduled Task we put a time limit on how long the process can run and have it kill the process if it runs for longer. If we do have a piece of code that is having trouble Scheduled Tasks will kill it and the process will start up in the next minute.
if you need to have it run every minute, I would build it as a windows service. I wouldn't use the scheduler for anything less than a daily task.
I would say that it depends on what it was doing, but in general I am always in favor of having the fewest layers. If you write it as a console service and use the task scheduler then you have two places to maintain going forward.
If you write it as a windows service then you only have one fewer places to check in case something goes wrong.
While searching for scheduled service help, i came across to a very good article by Jon Galloway.
There are various diadvantages if a windows service is used for scheduled task. I agreed with it. I would suggest to use Task Scheduled, simple in implementation. Please refer to detailed information of implementing the task scheduler. Hope this info helps in finalizing the implementation approach.
The only other point to consider, is that if you're job involves some kind of database interaction, consider looking into the integration/scheduling services provided by your database.
For example, creating an SSIS package for your SQL Server related service may seem a bit like overkill, but it can be integrated nicely with the environment and will have its own logging/error checking mechanisms already in place.
I agree, it is kind of a waste of effort to create even a console executable and schedule it to be run every minute. I would suggest exploring something like Quartz.Net. That way you can create a simple job and schedule it to run every minute.