I have an issue of Time out error when i click on the Export to Excel link in my application, due to heavy db transactions.
I want to change this functionality to the background so that user able to move forward while background process runs.
I want when user clicks on Export to Excel link background/new thread process runs and an email is sent to the user with a link to download a file.
I want to know which is best way to achieve this Creating new thread or Background process?
Thanks in Advance
You may want to use https://github.com/defunkt/resque or https://github.com/collectiveidea/delayed_job.
You can also watch:
http://railscasts.com/episodes/271-resque
http://railscasts.com/episodes/171-delayed-job-revised
http://railscasts.com/episodes/171-delayed-job (old one of above but free - but it is worth to buy a subscription)
There are three types of messages queues,
No queues - a new thread / process will be triggered for background task, eg: spawn
Database driven - corresponding task will be stored in a table in the database and will be deleted automatically on completion, eg: backgroundrb, delayed job
Message queues - Stored jobs in memory, eg: starling
I haven't used resque, but heard that it is bit complicated to configure.
Apart from running tasks in background, if you want to run some repetitive task on regular intervals, you can go for backgroundrb. Otherwise delayed job will be right choice.
Related
I use ASP.Net MVC 5 and I have a long running action which have to poll webservices, process data and store them in database.
For that I want to use TPL library to start the task async.
But I wonder how to do 3 things :
I want to report progress of this task. For this I think about SignalR
I want to be able to left the page where I start this task from and be able to report the progression across the website (from a panel on the left but this is ok)
And I want to be able to cancel this task globally (from my panel on the left)
I know quite a few about all of technologies involved. But I'm not sure about the best way to achieve this.
Is someone can help me about the best solution ?
The fact that you want to run long running work while the user can navigate away from the page that initiates the work means that you need to run this work "in the background". It cannot be performed as part of a regular HTTP request because the user might cancel his request at any time by navigating away or closing the browser. In fact this seems to be a key scenario for you.
Background work in ASP.NET is dangerous. You can certainly pull it off but it is not easy to get right. Also, worker processes can exit for many reasons (app pool recycle, deployment, machine reboot, machine failure, Stack Overflow or OOM exception on an unrelated thread). So make sure your long-running work tolerates being aborted mid-way. You can reduce the likelyhood that this happens but never exclude the possibility.
You can make your code safe in the face of arbitrary termination by wrapping all work in a transaction. This of course only works if you don't cause non-transacted side-effects like web-service calls that change state. It is not possible to give a general answer here because achieving safety in the presence of arbitrary termination depends highly on the concrete work to be done.
Here's a possible architecture that I have used in the past:
When a job comes in you write all necessary input data to a database table and report success to the client.
You need a way to start a worker to work on that job. You could start a task immediately for that. You also need a periodic check that looks for unstarted work in case the app exits after having added the work item but before starting a task for it. Have the Windows task scheduler call a secret URL in your app once per minute that does this.
When you start working on a job you mark that job as running so that it is not accidentally picked up a second time. Work on that job, write the results and mark it as done. All in a single transaction. When your process happens to exit mid-way the database will reset all data involved.
Write job progress to a separate table row on a separate connection and separate transaction. The browser can poll the server for progress information. You could also use SignalR but I don't have experience with that and I expect it would be hard to get it to resume progress reporting in the presence of arbitrary termination.
Cancellation would be done by setting a cancel flag in the progress information row. The app needs to poll that flag.
Maybe you can make use of message queueing for job processing but I'm always wary to use it. To process a message in a transacted way you need MSDTC which is unsupported with many high-availability solutions for SQL Server.
You might think that this architecture is not very sophisticated. It makes use of polling for lots of things. Polling is a primitive technique but it works quite well. It is reliable and well-understood. It has a simple concurrency model.
If you can assume that your application never exits at inopportune times the architecture would be much simpler. But this cannot be assumed. You cannot assume that there will be no deployments during work hours and that there will be no bugs leading to crashes.
Even if using http worker is a bad thing to run long task I have made a small example of how to manage it with SignalR :
Inside this example you can :
Start a task
See task progression
Cancel task
It's based on :
twitter bootstrap
knockoutjs
signalR
C# 5.0 async/await with CancelToken and IProgress
You can find the source of this example here :
https://github.com/dragouf/SignalR.Progress
I've written a number of actions in a RoR app, that perform different actions within process.
E.g.
- One action communicates with a third party service using their API and collects data.
- Another processes this data and places it into a relevant database.
- Another takes this new data and formats it in a specific way.
etc..
I would like to fire off the process at timed intervals, eg. Each hour. But I don't want to do the whole thing each time.
Sometimes I may just want to do the first two actions. At other times, I might want to do each part of the process.
So have one action run, and then when it's finished call another action. ETC..
The actions could take up to an hour to complete, if not longer, so I need a solution that won't timeout.
What would be the best way to achieve this?
You have quite a few options for processing jobs in the background:
Sidekiq: http://mperham.github.io/sidekiq/
Queue Classic: https://github.com/ryandotsmith/queue_classic
Delayed Job: https://github.com/collectiveidea/delayed_job
Resque: https://github.com/resque/resque
Just read through and pick the one that seems to fit your criteria the best.
EDIT
As you clarified, you want regularly scheduled tasks. Clockwork is a great gem for that (and generally a better option than cron):
https://github.com/tomykaira/clockwork
I'm working on an application, but at the moment I'm stuck on multithreading with rails.
I have the following situation: when some action occurs (it could be after a user clicks a button or when a scheduled task fires off), I'm starting a separate thread which parses some websites until the moment when I have to receive the SMS-code to continue parsing. At this moment I make Thread.stop.
The SMS-code comes as a POST request to some of my controllers. So I want to pass it to my stopped thread and continue its job.
But how can I access that thread?
Where is the best place to keep a link to that thread?
So how can I handle multithreading? There may be a situation when there'll be a lot of threads and a lot of SMS requests, and I need to somehow correlate them.
For all real purposes you can't, but you can have that other thread 'report' its status.
You can use redis-objects to create either a lock object using redis as its flag, create some type of counter, or just true, false value store. You can then query redis to see the corresponding state of the other thread, and exit if needed.
https://github.com/nateware/redis-objects
The cool part about this is it not only works between threads, but between applications.
I have a long running operation in my Grails application. My first solution is simply to perform the operation in the controller and let the user wait until the action is finished. This is not an acceptable solution, I want to show the user the progress of the action. The progress is simply text. The action can take from 10 seconds to roughly 30 minutes.
How can I show the progress to the user for my long running action?
First you might want to try the Executor plugin so you can run the job in the background. This works quite well.
Then I guess you have 2 options. Have the browser poll the server via Ajax for an update (as Tim pointed out the JProgress plugin would do something like this for you) or get even more bleeding edge and consider HTML5 WebWorkers for a kind of server push approach. WebWorkers are not available in
You will need something like a Task or Job domain class with a field percentageComplete. The controller will create and save the Task or Job and then spawn a new thread to execute it. Perhaps place the execution code in a service.
It will be up to your execution code to update the Task or Job's percentageComplete field as it completes its task. Then you can poll the job (via ajax) to see how the job is progressing.
Note: determining that percentage which is complete is very much to up to your specific code. You will probably have to just come up with a best guess based on the knowledge you have. Even for an operation where it is obvious how to determine percentage complete (like a file download), it is not certain (network issues, etc.)
Can you determin the state of your progress? Let's say in percent?
I would create the long operation as a quarz job (background) and query the state of job/ long running progress via ajax.
The JProgress plugin might help, but I've never tried it out...
hi
i'm going to set up a rails-website where, after some initial user input, some heavy calculations are done (via c-extension to ruby, will use multithreading). as these calculations are going to consume almost all cpu-time (memory too), there should never be more than one calculation running at a time. also i can't use (asynchronous) background jobs (like with delayed job) as rails has to show the results of that calculation and the site should work without javascript.
so i suppose i need a separate process where all rails instances have to queue their calculation requests und wait for the answer (maybe an error message if the queue is full), kind of a synchronous job manager.
does anyone know if there is a gem/plugin with such functionality?
(nanite seemed pretty cool to me, but seems to be only asynchronous, so the rails instances would not know when the calculation is finished. is that correct?)
another idea is to write my own using distributed ruby (drb), but why invent the wheel again if it already exists?
any help would be appreciated!
EDIT:
because of the tips of zaius i think i will be able to do this asynchronously, so i'm going to try resque.
Ruby has mutexes / semaphores.
http://www.ruby-doc.org/core/classes/Mutex.html
You can use a semaphore to make sure only one resource intensive process is happening at the same time.
http://en.wikipedia.org/wiki/Mutex
http://en.wikipedia.org/wiki/Semaphore_(programming)
However, the idea of blocking a front end process while other tasks finish doesn't seem right to me. If I was doing this, I would use a background worker, and then use a page (or an iframe) with the refresh meta tag to continuously check on the progress.
http://en.wikipedia.org/wiki/Meta_refresh
That way, you can use the same code for both javascript enabled and disabled clients. And your web app threads aren't blocking.
If you have a separate process, then you have a background job... so either you can have it or you can't...
What I have done is have the website write the request params to a database. Then a separate process looks for pending requests in the database - using the daemons gem. It does the work and writes the results back to the database.
The website then polls the database until the results are ready and then displays them.
Although I use javascript to make it do the polling.
If you really cant use javascript, then it seems you need to either do the work in the web request thread or make that thread wait for the background thread to finish.
To make the web request thread wait, just do a loop in it, checking the database until the reply is saved back into it. Once its there, you can then complete the thread.
HTH, chris