Rendering large database record set as JSON in rails - ruby-on-rails

Question

Is there a way to present a large record set in JSON format without consuming large amount of memory? For example I have the following query:
...
records = Records.where(query)
respond_to do |format|
format.html
format.json { render :json => records.to_json }
end
There are times that records will contain thousands of entries and JSON is strictly used for getting the data without using pagination and such data must fit inside the memory for it to be returned. A Record entry will also contain a lot of fields (I am using MongoDB/Mongoid) and it is necessary to include such fields.

Answer 1

It's almost always a bad idea to return every resource in a database.
You can respond with limited set of results and provide total number of records.
For example:
{
total: 503
records: [
{ id: 1 },
{ id: 2 }
]
}
And add possibility to use limit and offset parameters to iterate through all pages.
There is chapter named Pagination and partial response in free e-book Web API Design describes it.
I would recommend you to read that book. It contains only 30 pages.
upd:
In your case you are able to paginate results using limit and skip mongoid's methods.
records = Records.where(query).limit(params[:limit]).skip(params[:offset])
respond_to do |format|
format.html
format.json { render :json => { total: records.count, records: records }.to_json }
end