I'm working on mashup that scrapes a couple of sites for data. I want to scrape and cache the data on demand rather than index the entire sites.
The first time the data is fetched, the operation can be extremely slow--at least a couple of minutes.
What's the best practice for displaying pages line-by-line like this? Is there a way to display the page dynamically, showing data as it's fetched?
Related:
How to display HTML to the browser incrementally over a long period of time?
How to create an Incremental loading webpage
How to make sure an HTML table renders incrementally
Display the result on the webpage as soon as the data is available at server
I've used jquery to allow each expensive partial to be rendered on clicking a button:
view:
#book_forecast
= link_to 'See Sales Estimates' , book_forecast_work_update_path(:work => #work), :remote => true
book_forecast.js.erb:
$( "#book_forecast" ).html( "<%= escape_javascript( render( :partial => 'works/work_update/evaluation_forecast', :locals => { :work => #work} ) ) %>" );
work_update controller:
def book_forecast
#work = Work.find(params[:work])
respond_to do | format |
format.js
end
end
works/work_update/evaluation_forecast.html.haml:
# slow-loading code
The downside is that the user has to click on a button to render each partial, but on the other hand, using jquery instead of rendering as normal means the expensive code doesn't run when the page loads.
You can also use a 'loading' icon so that the user's got something to look at whilst the heavy code runs, something like:
same view:
#loading
%h2
.notice
Loading, please wait...
js:
$(document).ready(function () {
$('#loading')
.hide()
.ajaxStart(function() {
$(this).show();
})
.ajaxStop(function() {
$(this).hide();
})
;
It looks like Rails Live Streaming is one approach. Disadvantage is that it appears to be highly web server dependent and touchy. Here's a tutorial:
http://tenderlovemaking.com/2012/07/30/is-it-live.html
class MyController < ActionController::Base
include ActionController::Live
def index
100.times {
response.stream.write "hello world\n"
}
response.stream.close
end
end
Related
So I am trying to find a better way of refreshing the page. I have an app that builds an Excel spreadsheet using data you capture on the system. So what I was having it do is just reloading the page every 10 seconds till it's completed, so that the notice could be displayed properly.
Something like this (it's in the HAML syntax)
.pending_downloads
- if downloads_policy.pending?
.notification_notice
= image_tag 'spinner.gif'
Your data download is being prepared. This should only take a few minutes. It is safe to leave this page and return later.
= link_to "Cancel download.", download_path(downloads_policy.pending), :method => :delete, :class => "delete_link"
= javascript_tag("ToolkitApplication.periodical_reload();")
The Ajax (it's in coffeescript) for the periodical_reload(); method looks like this:
class #ToolkitApplication
this.periodical_reload = () ->
setInterval (->
window.location.reload()
), 10000
This approach I feel could be done better. I would like to have the ajax rather query the model every 3 seconds to see when the objects state has changed and then once it has changed then it will reload the window. So you dont get the page reloading like 10 times before the download is ready, every time I try reasearch if this is possible I get this rubyonrails guide which isnt really insightful with this sort of edge case. Is this possible and if so is there any good tutorials/blog posts/advice on how to do this? Google is yielding nothing.
So what i ended up doing was easy. Thanks to all that helped. in my controller i set a private method to check state
def any_uploads_status_changes?
return true if !Upload.exists?(params[:id])
return true if Upload.find(params[:id]).status
end
And then set another call in the controller called status:
def status
if any_uploads_status_changes?
render :text => 'window.location.reload();'
else
render :nothing => true
end
end
then set up an ajax request method(its in coffeescript syntax) ->
this.periodically_send_ajax_request = (url, method, interval) ->
setInterval (->
$.ajax
url: url
type: method
success: (result) ->
), interval
And then in the view just under condition called this request using js:
:javascript
ToolkitApplication.periodically_send_ajax_request("#{status_download_path(:id => downloads_policy.pending.id, :class => #model_class).html_safe }",'get', 2000);
just make sure that the path exists to the controller action
resources :downloads, :only => [:show, :destroy] do
member do
get :status
end
end
and there you go it will then query the controller according to what ever interval you specify and only if there are changes will it then reload the page. its a bit more code then simply reloading periodically but the user will appreciate it! :)
Note: I am presenting a logic here what I am doing.
What I am doing:
Think about the basic index action where we are listing products and with pagination. Now using remote-true option I have enabled ajax based pagination. So far things works perfectly fine. Take a look on sample code.
Products Controller:
def index
#products = Product.paginate(:order =>"name ASC" ,:page => params[:page], :per_page => 14)
respond_to do |format|
format.html # index.html.erb
format.json { render json: #products }
format.js
end
end
Index.html.erb
<h1>Products</h1>
<div id="products">
<%= render "products/products" %> // products partial is just basic html rendering
</div>
<script>
$(function(){
$('.pagination a').attr('data-remote', 'true')
});
</script>
Index.js.erb
jQuery('#products').html("<%= escape_javascript (render :partial => 'products/products' ) %>");
$('.pagination a').attr('data-remote', 'true');
So whats the problem:
Now I want to enable action caching on this. But index.js.erb file is not manipulating DOM. If I remove the remote-true functionality then things works fine with caching.
For action caching I have added this line on the top of the controller:
caches_action :index, :cache_path => Proc.new { |c| c.params }
Any suggestions?
Update:
Problem is jquery code is not executing. From this question
I found out what's wrong. jQuery actually surround the incoming script with a so that the browser evaluates the incoming code. But the caching mechansim merely saves the code as text and when one re-request, it returns the code as text but not evaluate it. Therefore, one needs to eval the code explicitly
But how to solve this problem??
After some trial and error, I think I have a work around.
Page links
Instead of
$(function() { $('.pagination a').attr('data-remote', 'true') });
use
$(function() {
$('.pagination a').click(function() {
$.ajax({
url: this.href,
dataType: 'script'
});
return false;
});
});
so response created by the app server will be run as javascript
Controller
next, change your caches_action line to
caches_action :index, cache_path: proc { |c| c.params.except(:_).merge(format: request.format) }
since ajax appends a _ params for some sort of timestamp
Hopefully this works :)
I don't see what the issue should be with using remote: true. Someone else suggested to use .ajax instead of remote: true, but that's exactly what the remote functionality does, so there shouldn't be any difference.
The other answer has code that explicitly uses jQuery.ajax, but the only difference in their code compared to what the remote functionality does is that they're specifying an explicit dataType. You can actually do that with remote: true though.
In your HTML link, you just need to specify data-type="script". Or, based on your posted JS, you'd do this:
$(function(){
$('.pagination a').attr('data-remote', 'true').attr('data-type', 'script');
});
EDIT: Also, I wrote more in-depth about the data-type attribute and how it works with Rails here: http://www.alfajango.com/blog/rails-3-remote-links-and-forms-data-type-with-jquery/
I think I found a solution to this problem. I have a controller that has caches_action for an action that uses format.js to fetch some data via ajax, and it was not working out of the box.
I found that, despite the request being transmitted to the server, and the server correctly parsing the request and "rendering" the index.js.erb template, nothing was updating in the DOM. Your solution with $.ajax and dataType:'script' fixed the problem for me, however, I didn't like having to do jquery to bind to a click on a link, which should happen by default... I was able to make it work correctly by changing my link_to to this:
= link_to "click me", user_action_path(params), remote: true, data:{type: 'script'}
Hope this helps!
I've been having the same problem but on my 3.0.3 application with :remote => true I have added :'data-type' =>: script and have been worked fine.
However, in my case, I don't see improvement when loaded by ajax the list.
Thanks for your time!
I get some reports data on my hand and I want to present these data on the web. The view of the HTML will be divided into two parts, the left part and the right part. There's a tree view in the left part consisting of the report names. In the right part presents the contents of the report.
What I want to achieve is when I click the report name on the left part, it will call an Action in the Controller, and passed the report name as parameter. The Action will fetch the data from the database and represent the data in the right part. And now I am stuck on how to realize this kind of view.
I've Googled a lot on the Internet and found Frameset, Partials or Ajax may capable of this. Because I've never developed web applications before and also new to Rails. So can anyone give me some advise or suggestion?
Things I've already known :
I've used Frameset to accomplish a view like this. But I found it needs a lot of .html files and all these .html files are static. And many people don't suggest it at all.
Then I've Googled Partials. But it seems Partials don't call the Action. It directly loads the _partials.html.erb to the main view. And besides, how can I control the layout? Using CSS?
I've never used Ajax.
If you want a fluid, seamless transition between one report and another, you should use both AJAX and Partials.
The way that it works is something like:
Make a left column in the html that has some links
Make the right column inside a partial
Assign the links to jQuery listeners to call the AJAX.
I'll put a bit of code here to show how it works:
Controller:
def index
reports = Report.all
if params[:report_id]
reports = Report.find(params[:report_id]
end
respond_to do |format|
format.html
format.js { render :template => "update_reports" }
end
end
update_reports.js.erb (in the same folder as the report views):
$('#report_viewer').html('<%= escape_javascript render :partial => "report_detail" %>');
In your view:
<div style=float:left>
<ul>
<li><%= link_to "Some report", "#", :class => "ajax" %></li>
</ul>
</div>
<div style=float:right id="report_viewer">
<%= render :partial => "report_detail" %>
</div>
<script type='text/javascript'>
$(document).ready(function() {
$(".ajax").click(function(e) {
$(this).ajax("your route to the action");
}
});
</script>
I think it's basically this, now let me explain a few things:
I don't remember if you have to do this, but in my case I created a new custom route to force the call to the action to be a json call instead of a html one. You can do this by adding :format => "js" to your route
You must name all your partials like "_yourname.html.erb". Rails won't recognize partials without the leading underscore.
In the controller, everything that comes after "format.js" is optional, you don't need to specify the template name, and if you don't Rails will look for the file index.js.erb.
The update_reports.js.erb file is basically a callback javascript that executes to update the current page. It finds the div where the partial is, and updates it rendering a new partial with the new report.
In the view, the link to change the report don't need to be a link at all if you're using the jQuery.click listener, but if it is a link, it must have the href as "#", or else the browser will just try to redirect to that location.
There are several ways to hook your link to the ajax function, I just chose the one I like it better, but you also could have a named function and call it in the html tag "onClick='yourFunction()'".
You need jQuery to call ajax like this. If you're sing Rails 3.0 or lower, you should replace the default Prototype with jQuery, because it's much better (IMHO), but I think prototype also have some ajax features.
It may seem complicated, but once you get the idea of it it'll become simple as writing any other action.
In the js callback file you could also add an animation to smooth the transition, like a fading. Look for the jQuery fade function for more info on this.
This is quite an open question so don't take this answer verbatim, but merely as a guide.
app/views/layouts/reports.html.erb
<!DOCTYPE html>
<html itemscope itemtype="http://schema.org/">
<head>
# omitted
</head>
<body>
<div id="body-container">
<div id="left-column">
<ul id="reports-list">
<% for report in #reports %>
<li><%= link_to report.name, report %></li>
<% end %>
</ul>
</div>
<div id="right-column">
<%= yield %>
</div>
</div>
</body>
</html>
app/controllers/reports_controller.rb
class ReportsController < ApplicationController
before_filter { #reports = Report.all }
def index
end
def show
#report = Report.find(params[:id])
end
def edit
#report = Report.find(params[:id])
end
def new
#report = Report.new
end
def update
# omitted
end
def create
# omitted
end
def destroy
#report = Report.find(params[:id])
end
end
routes.rb
YourApp::Application.routes.draw do
resources :reports
root to: "reports#index"
end
This would achieve the effect your after using just rails, of course adding ajax could add a better user experience.
I've become a bit confused about the idea of "rendering" a "template" due to the way an author speaks about it in a book I'm reading.
My original understanding of "rendering a template" was that it meant that Rails is providing the content that is viewed on the screen/presented to the viewer (in the way that a partial is rendered) but the book I'm reading seems to be using the concept of "rendering a template" to also mean something else. Let me explain in context
This book (rails 3 in action) sets up a page layout using the conventional layouts/application.html.erb file, and then it "yields" to different view pages, such as views/tickets/show.html.erb which adds more content to the screen. that's all straightforward..
Within this view views/tickets/show.html.erb, there is a rendering of a partial (which is also a straightforward concept).
<div id='tags'><%= render #ticket.tags %></div>
Now within this partial there is, using ajax, a call to a "remove" method in the "tags_controller.rb" which is designed to allow authorized users to remove a "tag" from a "ticket" in our mock project management application.
<% if can?(:tag, #ticket.project) || current_user.admin? %>
<%= link_to "x", remove_ticket_tag_path(#ticket, tag),
:remote => true,
:method => :delete,
:html => { :id => "delete-#{tag.name.parameterize}" } %>
<% end %>
Now here is the "remove" action in the tags controller (which disassociates the tag from the ticket in the database)...
def remove
#ticket = Ticket.find(params[:ticket_id])
if can?(:tag, #ticket.project) || current_user.admin?
#tag = Tag.find(params[:id])
#ticket.tags -= [#tag]
#ticket.save
end
end
end
At the end of this remove action, the author originally included render :nothing => true , but then he revised the action because, as he says, "you’re going to get it to render a template." Here's where I get confused
The template that he gets this action to render is "remove.js.erb", which only has one line of jquery inside it, whose purpose is to remove the "tag" from the page (i.e. the tag that the user sees on the screen) now that it has been disassociated from the ticket in the database.
$('#tag-<%= #tag.name.parameterize %>').remove();
When I read "rendering a template" I expect the application to be inserting content into the page, but the template rendered by the "remove" action in the controller only calls a jquery function that removes one element from the page.
If a "template" is "rendered", I'm expecting another template to be removed (in order to make room for the new template), or I'm expecting content to be "rendered" in the way that a partial is rendered. Can you clarify what is actually happening when a "template" is "rendered" in the situation with the jquery in this question? Is it actually putting a new page in front of the user (I expected some sort of physical page to be rendered)
You're nearly there! Rendering a template is indeed always about producing content, but for a slightly wider description of content. It could be a chunk of html, for example an ajax call to get new items might produce some html describing the new items, but it doesn't have to be.
A template might produce javascript as it does in your second example. Personally I am trying to avoid this and instead pass JSON back to the client and let the client side js perform the required work.
Another type of rendering you might perform is to produce some JSON. APIs will often do this, but you might also do this on a normal page. For example rather than rendering some javascript to delete tag x you might render the json
{ to_delete: "tag-123"}
and then have your jQuery success callback use that payload to know which element to remove from the DOM, by having this in your application.js file
$('a.delete_tag').live('ajax:success', function(data){
var selector = '#' + data.to_delete;
$(selector).remove()
}
(Assuming that your delete links had the class 'delete_tag')
Rendering JSON like this isn't really a template at all, since you'd usually do this via
render :json => {:to_delete => "tag-#{#tag.name.parameterize}"}
although I suppose you could use an erb template for this (I can't imagine why though).
My understanding is that js.erb is "rendered" by executing the javascript functions within it. Very often something like the below is done:
jQuery(document).ready(function() {
jQuery('#element').html('<%= escape_javascript(render pages/content) %>');
});
There's a really succinct overview of rendering at http://guides.rubyonrails.org/layouts_and_rendering.html that may help as it also goes into the details of the ActionController::Base#render method and what happens behind the scenes when you use render :nothing (for example). Render but can be used for files or inline code as well -- not just 'templates' in the traditional sense.
If you google "will_paginate" and "ajax", the top result is this blog post: But the original author of will_paginate says to not use this method (bad for SEO/spiders) ...
But I cant get the original authors method to work (his javascript kills all my links). An other gentleman suggests a similar method to mislav's (the original will_paginate author) concept. But I cant get that to work either.
so .... what is the best way to paginate using AJAX, and stay SEO friendly? (for RAILS >2.1)
Tomh's answer is correct. Just for shiggles, I prototyped a quick implementation. Here's a screencast that shows it using Ajax when Javascript is enabled (your users) and still having pretty URLs when Javascript is disabled (Google). And here are a few code snippets to get you rolling on it.
config/routes.rb:
map.connect 'items/:page', :controller => "items", :action => "index", :page => 1
app/controllers/items_controller.rb:
class ItemsController < ApplicationController
def index
#items = Item.paginate(:all, :page => params[:page])
respond_to do |format|
format.html
format.js do
render :update do |page|
page.replace_html :items, :partial => "items"
page << "ajaxifyPagination();"
end
end
end
end
end
app/views/items/index.html.erb:
<h1>Listing items</h1>
<div id="items">
<%= render :partial => "items" %>
</div>
app/views/items/_items.html.erb:
<%= will_paginate #items %>
<table>
<% for item in #items %>
<tr>
<td><%= item.id %></td>
</tr>
<% end %>
</table>
layout:
<%= javascript_include_tag :defaults %>
public/javascripts/application.js:
$(document).ready(function() {
ajaxifyPagination();
});
function ajaxifyPagination() {
$(".pagination a").click(function() {
$.ajax({
type: "GET",
url: $(this).attr("href"),
dataType: "script"
});
return false;
});
}
My example uses jQuery (with jRails), but it's straightforward to do with Prototype as well.
Seo friendly and unobtrusive javascript goes hand in hand. What you can do is the following.
Code the entire site as if only html is enabled (incl your pagination thing)
Use respond_to and serve only the list of items if the request comes in from js
Using onDomReady from whatever library you pick you attempt to catch all pagination links and add an onclick event which triggers an ajax call to that new view and returns the result. You put that result into the container containing the data you are paginating. The onclick then returns false.
To give your users a better user experience you can add some features like active links etc to the same javascript method.
Using this approach the pagination will work for JS and non-js as the non-js users (including Googlebot) will traverse your pagination as normal. Only in the event that the user has javascript enabled, the container with data will be updated with new results.
Unfortunately, I don't think you can use Ajax in the way you want and still stay SEO friendly as far as the paginated content. The problem is that the robots of Google and friends, as far as I know, won't go through your content using XHR requests so they simply won't see that content.
That said, if the paginated items each have their own static, SEO-friendly pages (or are otherwise statically available on your site), the content will still find its way into their engines. This is the way you'll probably want to go.
There is a railscasts on this topic which helped me out http://railscasts.com/episodes/174-pagination-with-ajax
I'm running rails 3.2, so I added the pagination.js there mentioned to app/assets/javascripts folder
pagination.js
$(function() {
$(".pagination a").live("click", function() {
$(".pagination").html("Loading...");
$.getScript(this.href);
return false;
});
});
And then created
home.js.erb
$('#div_tags_list').html('<%= escape_javascript(render partial: '/customersb2b/user_customer_numbers_list').html_safe %>')
$('#receipts_list').html('<%= escape_javascript(render partial: '/customersb2b/feed').html_safe %>')
Since I have two distinct listings on my homepage.
This is all I had to do to put will_paginate working with Ajax.
As for the SEO concerns, well, I don't know much about it, but the URL http://localhost:3000/customers?_=1366372168315&feed_page=1&tags_page=2 still works
There is a great way to do this easily if not worried about spiders. Took me 5 minutes. Check out:
https://github.com/ronalchn/ajax_pagination/wiki/Adding-AJAX-to-will_paginate
If you get an error about a missing 'history' file, install:
https://github.com/wweidendorf/jquery-historyjs
but also be aware of:
rails ajax_pagination couldn't find file 'history'