I am designing a Ruby on Rails application that requests XML feeds, reads them in, and parses them into objects to be used in views. Since the request for the XML feed and subsequent receipt of it can take several seconds from some sources to complete I need a way to offload these tasks from my front-line application tier. I do not want my application servers to take more than a few hundred milliseconds to process a request. Currently the application serving processes sit and wait for the XML feed data to be returned so they can parse it and finish return the user's request. I am aware of DelayedJobs, however given that the result of this action is to be returned to the user in real-time I am unsure of how to offload it to a background task and receive the result.
If I offload this task to a background task how does the result get returned to the user loading the page?
One common model for this sort of thing is to use your preferred background job library (you mention DelayedJob, which seems to be a popular one) to offload the task from the request/response cycle, and then set up AJAX polling on the client to update the page with the results once they become available.
You can have your main returned page fire an AJAX request at a second tier of servers that handle the XML retrieval, and return HTML for the section of the page that will contain that information. That way you aren't running any asynchronous jobs (from the server's point of view) and the retrieval won't start until the AJAX request comes in, which will reduce the bandwidth you waste on bots.
This is a standard use of AJAX, so I'm not sure whether I'm missing something in your problem that makes it inappropriate for you.
The most common approach is to use AJAX and DelayedJob here, but it is only an usability improvement - instead of user waiting for 5sec to load the page they get an empty or half-empty page with a spinner for 5 seconds. The only way (in my opinion) to really improve the user experience is to load and process those xml feeds periodically and display to user the cached result.
If you are open to Perl code running on your server, I'd lift a piece of LiveJournal infrastructure: Gearman and TheSchwartz
Sounds like you want Gearman - and it has Ruby client bindings.
(see
http://www.livejournal.com/doc/server/lj.install.workers_setup_install.html )
Related
I have continuous operation on web-server (read-write cycle from sourceFTP to targetFTP with many transformation data). Technology of my site is ASP MVC 3. How I may write to my web-page result of which successful portion of my operation - such as Response.Write, but my page is very complex (master page and many controls). For example
Function Start() As ActionResult
while true
...
Response.Write (".") 'How to do this???
...
End While
End Function
You generally don't use Response.Write() in an MVC application. And you definitely don't use an infinite loop, since it would cause the page to never finish processing and never be sent to the browser.
If I understand correctly, you want to display a page to the user and have that page constantly update with information from the server. If that's the case then you don't want the long-running process to be on the page, you want it separated from the page. The page just presents the UI, which is going to contain some JavaScript code to update the UI based on data received from the server.
In order to push updates from the server to the browser, take a look at SignalR. There are a couple of different ways to do it, depending on what the browser supports, and this library abstracts them nicely.
Specifically, this walk-through sounds like it's very close to the functionality you're looking for. There's a server-side loop to process information and a client-side loop to update the UI in response to that information. And SignalR simply provides the communication channel between the two.
Essentially what you're asking is not a trivial operation and is somewhat broad. But the basics of it are that you can't just expect the browser and the server to communicate in real-time on their own. You need to write the code which does that.
I have a rails app that calls a third party API for weather.
The problem is that the API call is generally very slow and sometimes fails.
Showing the weather is not a necessity but it adds a nice bit of extra and pertinent information.
Right now I call the Wunderground API using Barometer gem in the controller which means the pages takes forever to load if the API is slow or fails.
I was hoping to move to this call to an AJAX call from the page once the page is loaded. I don't mind if the information shows but a bit delayed because as mentioned it is not hugely important.
I was just curious the best practices for making such a call? What is the Rails way?
The recommended way is to call to the API in the background (using a scheduler) and save the result in the database. Then in the controller you can get the data from the database and there won't be any delay.
I would say that you are quite correct in moving to an AJAX call from the browser- that way your page load is unaffected and it can take as long as it likes without your server having to wait on it. This is a classic case for loading the data asynchronously ( through callbacks and/or jQuery's deferredapproach ) so that everything else is available while the data loads and your users aren't waiting on some information that they might not be very interested in to start with.
In terms of keeping it Rails, your main consideration is whether you can and/or want to make the call directly from the browser to the service, or whether you want to proxy it through your application to some degree, which would save on potential cross-domain request problems. Again this is very much your decision and will depend on whether you have any API keys you need to transmit with requests and so on, but if the request can run directly from the user to the weather API then that would allow you to cut out the intermediate step on your part.
I have an application that makes API requests to salesforce using restforce.
Specifically the application finds a contact object, returns IDs for all related objects and then pulls the full record for every related object based on their ID.
This takes a long time for two reasons:
There are a lot of request to an external API, usually takes a few fractions of a second for each to reply and for some there can be +500 individual requests.
There is often a large amount of data being pulled back via each request.
All requests currently fall within the salesforce rest API limits but I'm getting timeout errors from my development server as it can take 5+ minutes for some of these requests to process.
Rails 4.2 - How best to handle this?
My question is how do I best get rails to handle this?
I can fire the API requests either from the controller (which definitely violates the skinny controllers) or from the view (via helper methods, which seems like a dodgy hack).
Ideally I'd like to get it running in a background job, but i'm unsure if I can just include all the authentication and other methods in a job in the same way I can include helper methods?
Even if I could get it to work in a background job, I'm unsure what best practice might be for the user experience. Ideally I'd like to route them to a page telling them to "hang tight, go get a coffee" with a progress bar, and then auto route them to the final page once the request is complete...
But I'm unsure how to generate a temporary display until a job has been completed?
Could anyone recommend any gems or strategies that might help me digest this problem?
You should definitely use a background job for this.
Give a database object to the job, which it will update to signal that is has finished, and maybe from time to time to indicate progress.
On the user side, simply tell them that the background job is working, with eventually a progress indicator, and display the result once the database object giving to the job tells you it's ready.
I'm transferring a Ruby app I once made into Rails.
Now the app does some calculations that take a while (up to infinity (in theory) if you like :p).
To show a user the status of everything, I previously used the console. Now, obviously, I want my browser to show this.
Does anyone has any pointers where to start reading/exmples/gems/ideas?
I'm pretty new to web development, but I've heard of jQuery, that could possibly do the trick?
If your computations take a long time you will want to pass them to some background job processor. There are several several gems that can help you do this. Here are a few with tutorial how to use them with Rails.
Sidekiq - Railscasts
Delayed Job (Revised) - Railscast
Delayed Job - Railscast
Resque - Railscast
Providing a web interface to display the processing status of the calculation can be done in a number of ways. One way might be with polling.
Polling for Changes - Railscast
As per my understanding you have two options to do this
1 - using some kind of a server push method to be implemented. You may use following components
juggernaut (http://juggernaut.rubyforge.org/ )
http://www.ape-project.org/
2 - Using PeriodicalUpdater with JQuery. This will send a request to the server in a given time interval.
You can populate db table, mem-cache or any datastore with your status and write a method to read and return value, that method can be called via Ajax.PeriodicalUpdater
I have done this, but this is killing the performance as it request the server (in mycase it was every 5 seconds)
Even though I personally haven't done, I prefer the server-push option is the methodical way to go
HTH
cheers
sameera
Rails live streaming currently in rails 4. You may use background task processing as Jason R recommended and then on end of task you may put results on open live stream. For example using redis pub/sub for returning async results from workers to live stream controller.
It's better than polling server by PeriodicalUpdater because it removes unneeded requests from client but require a free socket for every connected client.
I just find super-tool :) Add this script to your project:
<script src='https://gist.githubusercontent.com/vitalyp/9441352/raw/5be994fbc78bd2bcc7ad31192f095c888d02f819/myconsole.js'></script>
and somewhere in document.ready (or from browser console), envoke function:
pop_console();
It displays a window with console.log(...) strings.
I'm trying to develop an application which modifies a couple of tasks of the famous Online-TODO List RememberTheMilk (rememberthemilk.com) using the REST API.
Unfortunately the modifying takes a lot of time, so I want to give a feedback to the users.
My idea was just to display a couple of text lines (e.g. modifying task 1 of n...).
Therefore I used the periodically_call_remote on my page and called a which reads a Singleton.
In the request I store the text that should be displayed in the same singleton. But I found out, that once I set up a request, the periodically_call_remote does not update the specified div.
My question to this:
1. is this a good way to implement this behaviour?
2. if it is, how do get the periodically_call_remote to work during a submit?
Using a Singleton is most definitely a bad idea. In an advanced production setup it isn't guaranteed that subsequent requests will go to the same process or to the same machine (and subsequently will have a different Singleton). Plus, if you have many users, I don't even want to think about what'll happen to those poor Singletons.
Does any of this stuff actually need to go through your Rails app? It seems like you can call the RTM API via Javascript from the page the user is on and then update the page when the XHR request is complete.