Background
I'm currently working on a small Rails 5 project that needs to access and process an external API. There is a ruby wrapper gem available for the API, so accessing the data is not a problem.
Problem description
There are two parts of the equation that I am currently missing, and hoping someone out there can help me with.
1: I need to call the API, via Rails, every 15 minutes. How can I realize this? I was looking towards Active Job for this, but my research kind of stalled after getting no useful results.
2: The external API has different domain models and a different domain-specific language than my application. How can I map the different models without changes in Active Record?
1: I need to call the API, via Rails, every 15 minutes. How can I realize this? I was looking towards Active Job for this, but my research kind of stalled after getting no useful results.
The first problem you can solve using recurring tasks. The main idea is to run the process that will perform some operations every x minutes (or days or whatever fits your problem.
There are several tools that you can use. One of them is built-in the unix system and it is cron. You can read about it in system's manual. You can easily manage it using whenever gem. The main disadvantage is that you need an access to the system's cron which may be non-trivial on non-bare machines (for example Platform as a Service hosts such as Heroku).
You should also take a look at clockwork which does not rely on the system's cron. It uses approach where you have a separate process running all time and it keeps an eye on defined tasks.
In the second approach (having a separate process) you need to remember that time-consuming instructions may "lock" the process and postpone another tasks. In this case, you may want to use background processing such as sidekiq or delayed_job. The idea is to use one process for scheduling tasks at certain time and another process to process those tasks as soon as they appear in the queue.
2: The external API has different domain models and a different domain-specific language than my application. How can I map the different models without changes in Active Record?
You need to create a client that will consume the API and map its responses into models that you have in your application. This way, you don't need to make your model's scheme dependent on the API scheme. Take a look at resource_kit gem - this is a sample solution that uses this approach.
HI hdauven,
processing the API every 15 minutes will affect your server performance,so done it by using sidekiq, it is a background job and use sidetiq it will help you to perform the task every 15 min automatically
You are accessing API, Then why are you worrying about different domain.
I have a long running CPU bound task that I want to initialize from a link in my MVC application. When I click the link, I want the server to create a GUID to identify the job, return that GUID to the client, and perform the job after returning.
I set this up using ThreadPool.QueueWorkItem, but I've read this can be problematic in MVC. Is there a better option for this case? Is there a different approach I should be using?
In my experience it is better to perform long running CPU tasks not in ASP.NET application itself but in separate application. For example you can create separate Windows service to process tasks. To interchange data you can use for example message queue, database (probably the easiest way) or web service.
This approach has following advantages:
1) Integrity of background job. In IIS you can configure to restart worker processes periodically. If your background job is running at that moment it will be interrupted what could be undesirable.
2) Plan server load balancing. For example you can move your web service to separate server which will free web server and can provide better end user experience.
Take a look at this example to see how it can be implemented with Azure.
You can do a fire and forget, by creating an asynchronous task without waiting for it, and it will run successfully most of the time, but due IIS application life cycle management those task may be abruptly cut.
You can register an IRegisteredObject object in IIS, so IIS will such object know that the domain is being shutdown.
Please take a look to this article:
http://haacked.com/archive/2011/10/16/the-dangers-of-implementing-recurring-background-tasks-in-asp-net.aspx/
I am developing my first web application using ASP.Net MVC, and I am in a situation where I would like a background service to process status notifications outside of the application, not unlike the reputation/badge system on stackoverflow.
What is the best way to handle something like this? Is it even possible in a shared-hosting environment like Godaddy, which I am using.
I don't need to communicate with the background worker directly, since I will be adding notification records to a database table with a column set to an "unprocessed" state. Then the worker will just scan the table on a regular schedule and processes what is ready.
Thanks for your advice.
Have you tried with quartz.net? I think it may fit your needs.
also take a look at this Simulate a Windows Service using ASP.NET to run scheduled jobs article.
it explains a nice way to schedule operations with no outer dependence.
The idea is to use Cache timeout to control the schedule. I've implemented it successfully on a project which required regular temp file cleaning. This cleaning is a bit heavy so we move this clean operation in a scheduled job (using the asp.net cache) to avoid having to deploy scheduled task or custom program.
To answer whether GoDaddy will support a seperate service you need to ask them.
However there are a number of creative ways that you can "get around" this issue on shared hosting.
Have a secure page that's purpose is to execute your background work. You could have scheduled task on a machine under your control that calls to this web page at set intervals.
Use a variation of the Background Worker Thread answer from #safi. Your background worker thread could check to see if another is already processing and stop, so that only one instance is running at a time.
If only one background task is enough for you then use the WebBackgrounder
And this is the article with detailed explanation.
My team is having a debate which is better: a windows service or scheduled tasks. We have a server dedicated to running jobs and currently they are all scheduled tasks. Some jobs take files, rename them and place them in other directories on the network. Other jobs extract data from SQL, modify it, and ship it elsewhere. Other jobs ftp files out. There is a lot of variety, but all in all, they are fairly straightforward.
I am partial to having each of these run as a windows service instead of a scheduled task because it is so much easier to monitor a windows service than a scheduled task. Some are diametrically opposed. In the end, none of us have that much experience to provide actual factual comparisons between the two methods. I am looking for some feedback on what other have experienced.
If it runs constantly - windows service.
If it needs to be run at various intervals - scheduled task.
Scheduled Task - When activity to be carried out on some fixed/predefined schedule. It take less memory and resources of OS. Not required installation. It can have UI (eg. Send reminder mail to defaulters)
Windows Service - When a continue monitoring is required. It makes OS busy by consuming more. Require install/uninstallation while changing version. No UI at all (eg. Process a mail as soon as it arrives)
Use them wisely
Sceduling jobs with the build in functionality is a perfectly valid use. You would have to recreate the full functionality in order to create a good service, and unless you want to react to speciffic events, I see no reason to move a nightly job into a service.
Its different when you want to process a file after it was posted in a folder, thats something I would create a service for, thats using the filesystem watcher to monitor a folder.
I think its reinventing the wheel
While there is nothing wrong with using the Task Scheduler, it is itself, a service. But we have the same requirements where I work and we have general purpose program that does several of these jobs. I interpreted your post to say that you would run individual services for each task, I would consider writing a single, database driven (service) program to do all your tasks, and that way, when you add a new one, it is simply a data entry chore, and not a whole new progam to write. If you practice change control, this difference is can be significant. If you have more than a few tasks the effort may be comperable. This approach will also allow you to craft a logging mechanism best suited to your operations.
This is a portion of our requirments document for our task program, to give you an idea of where to start:
This program needs to be database driven.
It needs to run as a windows service.
The program needs to be able to process "jobs" in the following manner:
Jobs need to be able to check for the existence of a source file, and take action based on the existence or not of the source file. (i.e proceed with processing, vs report that the file isn't there vs ignore it because it is not critical that the file isn't there.
Jobs need to be able to copy a file from a source to a target location or
Copy a file from source, to a staging location, perform "processing", and then copy either the original file or a result of the "processing" to the target location or
Copy a file from source, to a staging location, perform "processing", and the processing is the end result.
The sources and destination that jobs might copy to and from can be disparate: UNC, SFTP, FTP, etc.
The "processing", can be, encrypting/decrypting a file, parsing a data file for correct format, feeding the file to the mainframe via terminal emulation, etc., usually implemented by calling a command line passing parameters to an .exe
Jobs need to be able to clean up after themselves, as required. i.e. delete intermediate or original files, copy files to an archive location, etc.
The program needs to be able to determine the success and failure of each phase of a job and take appropriate action which would be logging, and possibly other notification, abort further processing on failure, etc.
Jobs need to be configured to activate at certain set times, or at certain intervals (optionally during certain set hours) i.e. every 15 mins from 9:00 - 5:00.
There needs to be a UI to add new jobs.
There needs to be a button to push to fire off a job as if a timer event had activated it.
The standard Display of the program should show an operator what is going on and whether the program is functioning properly.
All of this is predicated on the premise that it is a given that you write your own software. There are several enterprise task scheduler programs available on the market, as well. Buying off the shelf may be a better solution for you.
What are the cons and pros of windows services vs scheduled tasks for running a program repeatedly (e.g. every two minutes)?
Update:
Nearly four years after my original answer and this answer is very out of date. Since TopShelf came along Windows Services development got easy. Now you just need to figure out how to support failover...
Original Answer:
I'm really not a fan of Windows Scheduler. The user's password must be provided as #moodforall points out above, which is fun when someone changes that user's password.
The other major annoyance with Windows Scheduler is that it runs interactively and not as a background process. When 15 MS-DOS windows pop up every 20 minutes during an RDP session, you'll kick yourself that didn't install them as Windows Services instead.
Whatever you choose I certainly recommend you separate out your processing code into a different component from the console app or Windows Service. Then you have the choice, either to call the worker process from a console application and hook it into Windows Scheduler, or use a Windows Service.
You'll find that scheduling a Windows Service isn't fun. A fairly common scenario is that you have a long running process that you want to run periodically. But, if you are processing a queue, then you really don't want two instances of the same worker processing the same queue. So you need to manage the timer, to make sure if your long running process has run longer than the assigned timer interval, it doesn't kick off again until the existing process has finished.
After you have written all of that, you think, why didn't I just use Thread.Sleep? That allows me to let the current thread keep running until it has finished and then the pause interval kicks in, thread goes to sleep and kicks off again after the required time. Neat!
Then you then read all the advice on the internet with lots of experts telling you how it is really bad programming practice:
http://msmvps.com/blogs/peterritchie/archive/2007/04/26/thread-sleep-is-a-sign-of-a-poorly-designed-program.aspx
So you'll scratch your head and think to yourself, WTF, Undo Pending Checkouts -> Yes, I'm sure -> Undo all today's work..... damn, damn, damn....
However, I do like this pattern, even if everyone thinks it is crap:
OnStart method for the single-thread approach.
protected override void OnStart (string args) {
// Create worker thread; this will invoke the WorkerFunction
// when we start it.
// Since we use a separate worker thread, the main service
// thread will return quickly, telling Windows that service has started
ThreadStart st = new ThreadStart(WorkerFunction);
workerThread = new Thread(st);
// set flag to indicate worker thread is active
serviceStarted = true;
// start the thread
workerThread.Start();
}
The code instantiates a separate thread and attaches our worker
function to it. Then it starts the thread and lets the OnStart event
complete, so that Windows doesn't think the service is hung.
Worker method for the single-thread approach.
/// <summary>
/// This function will do all the work
/// Once it is done with its tasks, it will be suspended for some time;
/// it will continue to repeat this until the service is stopped
/// </summary>
private void WorkerFunction() {
// start an endless loop; loop will abort only when "serviceStarted"
// flag = false
while (serviceStarted) {
// do something
// exception handling omitted here for simplicity
EventLog.WriteEntry("Service working",
System.Diagnostics.EventLogEntryType.Information);
// yield
if (serviceStarted) {
Thread.Sleep(new TimeSpan(0, interval, 0));
}
}
// time to end the thread
Thread.CurrentThread.Abort();
}
OnStop method for the single-thread approach.
protected override void OnStop() {
// flag to tell the worker process to stop
serviceStarted = false;
// give it a little time to finish any pending work
workerThread.Join(new TimeSpan(0,2,0));
}
Source: http://tutorials.csharp-online.net/Creating_a_.NET_Windows_Service%E2%80%94Alternative_1%3a_Use_a_Separate_Thread (Dead Link)
I've been running lots of Windows Services like this for years and it works for me. I still haven't seen a recommended pattern that people agree on. Just do what works for you.
Some misinformation here. Windows Scheduler is perfectly capable of running tasks in the background without windows popping up and with no password required. Run it under the NT AUTHORITY\SYSTEM account. Use this schtasks switch:
/ru SYSTEM
But yes, for accessing network resources, the best practice is a service account with a separate non-expiring password policy.
EDIT
Depending on your OS and the requirements of the task itself, you may be able to use accounts less privileged than Localsystem with the /ru option.
From the fine manual,
/RU username
A value that specifies the user context under which the task runs.
For the system account, valid values are "", "NT AUTHORITY\SYSTEM", or "SYSTEM".
For Task Scheduler 2.0 tasks, "NT AUTHORITY\LOCALSERVICE", and
"NT AUTHORITY\NETWORKSERVICE" are also valid values.
Task Scheduler 2.0 is available from Vista and Server 2008.
In XP and Server 2003, system is the only option.
In .NET development, I normally start off by developing a Console Application, which will run will all logging output to the console window. However, this is only a Console Application when it is run with the command argument /console. When it is run without this parameter, it acts as a Windows Service, which will stay running on my own custom coded scheduled timer.
Windows Services, I my mind, are normally used to manage other applications, rather than be a long running application. OR .. they are continuously-running heavyweight applications like SQL Server, BizTalk, RPC Connections, IIS (even though IIS technically offloads work to other processes).
Personally, I favour scheduled tasks over Window Services for repititive maintenance tasks and applications such as file copying/synchronisations, bulk email sending, deletion or archiving of files, data correction (when other workarounds are not available).
For one project I have been involved in the development of 8 or 9 Windows Services, but these sit around in memory, idle, eating 20MB or more memory per instance. Scheduled tasks will do their business, and release the memory immediately.
What's the overhead of starting and quitting the app? Every two minutes is pretty often. A service would probably let the system run more smoothly than executing your application so frequently.
Both solutions can run the program when user isn't logged in, so no difference there. Writing a service is somewhat more involved than a regular desktop app, though - you may need a separate GUI client that will communicate with the service app via TCP/IP, named pipes, etc.
From a user's POV, I wonder which is easier to control. Both services and scheduled tasks are pretty much out of reach for most non-technical users, i.e. they won't even realize they exist and can be configured / stopped / rescheduled and so on.
The word 'serv'ice shares something in common with 'serv'er. It is expected to always be running, and 'serv'e. A task is a task.
Role play. If I'm another operating system, application, or device and I call a service, I expect it to be running and I expect a response. If I (os, app, dev) just need to execute an isolated task, then I will execute a task, but if I expect to communicate, possibly two way communication, I want a service. This has to do with the most effective way for two things to communicate, or a single thing that wants to execute a single task.
Then there's the scheduling aspect. If you want something to run at a specific time, schedule. If you don't know when you're going to need it, or need it "on the fly", service.
My response is more philosophical in nature because this is very similar to how humans interact and work with another. The more we understand the art of communication, and "entities" understand their role, the easier this decision becomes.
All philosophy aside, when you are "rapidly prototyping", as my IT Dept often does, you do whatever you have to in order to make ends meet. Once the prototyping and proof of concept stuff is out of the way, usually in the early planning and discovering, you have to decide what's more reliable for long term sustainability.
OK, so in conclusion, it's highly dependent on a lot of factors, but hopefully this has provided insight instead of confusion.
A Windows service doesn't need to have anyone logged in, and Windows has facilities for stopping, starting, and logging the service results.
A scheduled task doesn't require you to learn how to write a Windows service.
It's easier to set up and lock down windows services with the correct permissions.
Services are more "visible" meaning that everyone (ie: techs) knows where to look.
This is an old question but I will like to share what I have faced.
Recently I was given a requirement to capture the screenshot of a radar (from a Meteorological website) and save it in the server every 10 minutes.
This required me to use WebBrowser.
I usually make windows services so I decided to make this one service too but it would keep crashing.
This is what I saw in Event Viewer
Faulting module path: C:\Windows\system32\MSHTML.dll
Since the task was urgent and I had very less time to research and experiment, I decided to use a simple console application and triggered it as a task and it executed smoothly.
I really liked the article by Jon Galloway recommended in accepted answer by Mark Ransom.
Recently passwords on the servers were changed without acknowledging me and all the services failed to execute since they could not logon.
So ppl claiming in the article comments that this is a problem. I think windows services can face same problem (Pls. correct me if I am wrong, I am jus a newbie)
Also the thing mentioned, if using task scheduler windows pop up or the console window pops up.
I have never faced that. It may pop up but it is at least very instantaneous.
Why not provide both?
In the past I've put the 'core' bits in a library and wrapped a call to Whatever.GoGoGo() in both a service as well as a console app.
With something you're firing off every two minutes the odds are decent it's not doing much (e.g. just a "ping" type function). The wrappers shouldn't have to contain much more than a single method call and some logging.
Generally, the core message is and should be that the code itself must be executable from each and every "trigger/client". So it should not be rocket science to switch from one to the other approach.
In the past we used more or less always Windows Services but since also more and more of our customers switch to Azure step by step and the swap from a Console App (deployed as a Scheduled Task) to a WebJob in Azure is much easier than from a Windows Service, we focus on Scheduled Tasks for now. If we run into limitations, we just ramp up the Windows Service project and call the same logic from there (as long as customers are working OnPrem..) :)
BR,
y
Windows services want more patience until it's done.
It has a bit hard debug and install. It's faceless.
If you need a task which must be done in every second, minute or hour,
you should choice Windows Service.
Scheduled Task is quickly developed and has a face.
If you need a daily or weekly task, you can use Scheduled Task.