I have a Rails + React app that is ~3mb (too big!)
About 2mb of that is a single instance variable, #songs, that's passed to the react component like this:
<%= react_component('SongApp', songData: #songs, songId: #song_id, allBooks: Book.reactify) %>
By passing the data in like that, I can immediately use it as this.props.songData in the component.
I want to compress this data (which is a JSON string)—so the client device doesn't need to download so much—and decompress within the React component on the client device. How can I do this?
Use Rack::Deflater
Rack::Deflater middleware compresses responses at runtime using deflate or trusty ol’ gzip. Inserted correctly into your Rack app, it can drastically reduce the size of your HTML / JSON controller responses
Add it to config/application.rb thusly
module YourApp
class Application < Rails::Application
config.middleware.use Rack::Deflater
end
end
Answer Number 2
For the response to be in gzip format we don't have to change the render method call.
If the request has the header Accept-Encoding: gzip, Rails will automatically compress the JSON response using gzip.
If you don't want the user to send a request with preset header., you can add the header to the request manually in the controller before rendering the response:
request.env['HTTP_ACCEPT_ENCODING'] = 'gzip'
render :json => response.to_json()enter code here
check this gist
Related
Here is the code which is working using Net::HTTP::Post
request = Net::HTTP::Post.new(url)
...
form_data = [
['attachments[]', File.open('file1.txt')],
['attachments[]', File.open('file2.txt')]
]
request.set_form form_data, 'multipart/form-data'
http.request(request)
Now, I am trying to use httparty like below but it is not working.
body = { attachments: [ File.open('file1.txt'), File.open('file2.txt') ] }
HTTParty.post(url, body: body)
The response I am getting from web service call is below:
#<HTTParty::Response:0x557d7b549f90 parsed_response={"error"=>true, "error_code"=>"invalid_attachment", "error_message"=>"Attachmen
t(s) not found or invalid."}, #response=#<Net::HTTPBadRequest 400 Bad Request readbody=true>, #headers={"server"=>["nginx"], "date"=>[
"Mon, 20 May 2019 07:41:50 GMT"], "content-type"=>["application/json"], "content-length"=>["102"], "connection"=>["close"], "vary"=>["
Authorization"], "set-cookie"=>["c18664e1c22ce71c0c91742fbeaaa863=uv425hihrbdatsql1udrlbs9as; path=/"], "expires"=>["Thu, 19 Nov 1981
08:52:00 GMT", "-1"], "cache-control"=>["no-store, no-cache, must-revalidate", "private, must-revalidate"], "pragma"=>["no-cache", "no
-cache"], "x-ratelimit-limit"=>["60"], "x-ratelimit-remaining"=>["59"], "strict-transport-security"=>["max-age=63072000; includeSubdom
ains;"]}>
It looks like it is not able to read the contents of files. Does HTTParty support this or I need to use some other gem?
Something like this should work, I just tested it, worked for me no problem.
HTTParty.post(url,
body: { attachments: [
File.read('foo.txt'),
File.read('bar.txt')] })
With HTTParty you can pass IO/Files as parameters the same way (multipart is automatically set to true if there's a file in parameters).
But keep in mind that files should be closed after upload, otherwise you may run out of file descriptors before GC collects them:
files = ['file1.txt', 'file2.txt'].map{|fname| File.open(fname) }
begin
HTTParty.post(url, body: { attachments: files })
ensure
files.each(&:close)
end
That should work for you if net/http variant does (and is actually the same as your code).
Other thing to look at is content type detection by filename - because file upload consists of filename, content type and data itself.
Error 400 with "invalid_attachment" you're getting suggests that more probably it's related to content type or other validation on server side (so make sure you're testing with the same files and nothing else changes other than http lib), also check httparty to be a recent version
I've written a test program which sends the same multipart request using both Net::HTTP and HTTParty. Then it compares and prints the request strings so that we can compare them. The only substantive difference between the two requests is that HTTParty attempts to guess and set the Content-Type header (e.g. text/plain for a file named file1.txt), whereas Net::HTTP always uses application/octet-stream.
HTTParty definitely does read the files and send them in the request. So, I suggest you investigate if the server is returning an error due to the Content-Type (maybe the content type in your particular request is not supported).
For your reference, here is the test program and specific results.
I am trying to upload a csv file to Rails and parse it into a db. I have tried using both Paw and Postman to send the http request, specifying POST, attaching the csv file, and specifying Content-Type as application/csv
The request header:
POST /skate_parks/import HTTP/1.1
Content-Type: text/csv
Host: localhost:3000
Connection: close
User-Agent: Paw/2.3.4 (Macintosh; OS X/10.11.5) GCDHTTPRequest
Content-Length: 11663
Name,Address,Suburb,Postcode,State,Business Category,LGA,Region,
Aireys Inlet Skate Park,Great Ocean Road,Aireys Inlet,3231,VIC,Skate Parks,Surf Coast,Barwon S/W, etc...
The controller skate_parks_controller.rb
def import
SkatePark.import(params[:body])
end
The model
class SkatePark < ApplicationRecord
require 'csv'
def self.import(file)
CSV.foreach("file", headers: true) do |row|
skate_park_hash = row.to_hash
skate_park = SkatePark.where(name: skate_park_hash["name"])
if skate_park.count == 1
skate_park.first.update_attributes(skate_park_hash)
else
SkatePark.create!(skate_park_hash)
end
end
end
end
The error
Started POST "/skate_parks/import" for ::1 at 2016-05-26 13:48:34 +1000
Processing by SkateParksController#import as HTML
Completed 500 Internal Server Error in 3ms (ActiveRecord: 0.0ms)
Errno::ENOENT (No such file or directory # rb_sysopen - file):
app/models/skate_park.rb:6:in `import'
app/controllers/skate_parks_controller.rb:7:in `import'
The problem is params[:body] is nil, so you're essentially calling SkatePark.import(nil). Rails doesn't put the raw POST body into params like you've apparently assumed it does.
You have two options. The better option, in my opinion, is to upload the data as multipart/form-data. Rather than putting the raw data into the POST body, you'll do the same thing a browser does when a user chooses a file in an <input type="file">, which is to say you'll encode it as form data. When you do that, you will be able to access the data through params, as described in the Form Helpers Rails Guide under "Uploading Files." (Since you apparently aren't using a form, you can skip to "What Gets Uploaded" to see how to handle the data you receive.)
To test this with Postman, follow the instructions for "form-data" under "Request body" in the Sending Requests docs, which I'll excerpt here for posterity:
multipart/form-data is the default encoding a web form uses to transfer data. This simulates filling a form on a website, and submitting it. The form-data editor lets you set key/value pairs (using the key-value editor) for your data. You can attach files to a key as well.
Your other option is to access the POST body directly via request.raw_post as described here: How to access the raw unaltered http POST data in Rails? This is not very "Railsy," however, and among other things will be harder to test.
I have an api written in rails which on each request responds with a JSON response.
The response could be huge, so i need to compress the JSON response using gzip.
Wondering how to do this in rails controller?
I have added the line
use Rack::Deflater
in config.ru
Should I also be changing something in the line which renders JSON?
render :json => response.to_json()
Also, how do i check if the response is in gzip format or not..??
I did a curl request from terminal, I see only the normal plain JSON.
My post Content Compression with Rack::Deflater describes a couple of ways to integrate Rack::Deflater. The easiest would be to just update config/application.rb with:
module YourApp
class Application < Rails::Application
config.middleware.use Rack::Deflater
end
end
and you'll automatically compress all controller responses with deflate / gzip if the client explicitly says they can handle it.
For the response to be in gzip format we don't have to change the render method call.
If the request has the header Accept-Encoding: gzip, Rails will automatically compress the JSON response using gzip.
If you don't want the user to send a request with preset header., you can add the header to the request manually in the controller before rendering the response:
request.env['HTTP_ACCEPT_ENCODING'] = 'gzip'
render :json => response.to_json()
You can query Curl by setting a custom header to get gzipped response
$ curl -H "Accept-Encoding: gzip, deflate" localhost:3000/posts.json > posts_json.gz
then, then decompress it to view the actual response json
$ gzip -d posts_json.gz
$ cat posts_json
If it doesn't work. post back with output of rake middlewares to help us troubleshoot further.
In some cases you can consider to write huge response into a file and gzip it:
res = {} # huge data hash
json = res.to_json
Zlib::GzipWriter.open('public/api/huge_data.json.gz') { |gz| gz.write json }
and update this file regularly
Consider not putting Rack middlewares in config.ru when using Rails
Rails has it's own middleware stack manager since Rails 2.
The correct way is:
# config/application.rb or config/environment.rb depends on your Rails version
config.middleware.use Rack::Deflater
Don't use #djcp's solution when using Rack::ETag
Short answer:
module MyApp
class Application < Rails::Application
config.middleware.insert_before Rack::ETag, Rack::Deflater
end
end
The order of Rack::Deflater and Rack::ETag matters because Rack::Deflater uses Zlib::GzipWriter to compress the response body and it would compress with a timestamp by default, which means the compressed response body would change every second even if the original response body is the same.
To reproduce this problem, run the following script:
require 'rack/etag'
require 'rack/deflater'
require 'rack/content_length'
#app = Rack::Builder.new do
use Rack::ETag
use Rack::Deflater
use Rack::ContentLength
run ->(*) { [200, {}, ['hello world']] }
end
def puts_etag
puts #app.call({ 'HTTP_ACCEPT_ENCODING' => 'gzip' })[1]['ETag']
end
puts_etag
sleep 1
puts_etag
One can simply swap the lines of Rack::ETag and Rack::Deflater and get the expected output.
Rails uses Rack::ETag by default and config.middleware.use is just appending. To insert Rack::Deflater before Rack::Etag, use config.middleware.insert_before instead.
🍻
I've setup unicorn in rails 3.1 and http streaming works until I enable Rack::Deflater.
I've tried both with and without use Rack::Chunked. In curl I can see my response while in chrome I get the following errror: ERR_INVALID_CHUNKED_ENCODING
The result is same in other browsers (firefox, safari) and between development (osx) and production (heroku).
config.ru:
require ::File.expand_path('../config/environment', __FILE__)
use Rack::Chunked
use Rack::Deflater
run Site::Application
unicorn.rb:
listen 3001, :tcp_nopush => false
worker_processes 1 # amount of unicorn workers to spin up
timeout 30 # restarts workers that hang for 30 seconds
controller:
render "someview", :stream => true
Thanks for any help.
The problem is that Rails ActionController::Streaming renders directly into a Chunked::Body. This means the content is first chunked and then gzipped by the Rack::Deflater middleware, instead of gzipped and then chunked.
According to the HTTP/1.1 RFC 6.2.1, chunked must be last applied encoding to a transfer.
Since "chunked" is the only transfer-coding required to be understood
by HTTP/1.1 recipients, it plays a crucial role in delimiting messages
on a persistent connection. Whenever a transfer-coding is applied to a
payload body in a request, the final transfer-coding applied must be
"chunked".
I fixed it for us by monkey patching ActionController::Streaming _process_options and _render_template methods in an initializer so it does not wrap the body in a Chunked::Body, and lets the Rack::Chunked middleware do it instead.
module GzipStreaming
def _process_options(options)
stream = options[:stream]
# delete the option to stop original implementation
options.delete(:stream)
super
if stream && env["HTTP_VERSION"] != "HTTP/1.0"
# Same as org implmenation except don't set the transfer-encoding header
# The Rack::Chunked middleware will handle it
headers["Cache-Control"] ||= "no-cache"
headers.delete('Content-Length')
options[:stream] = stream
end
end
def _render_template(options)
if options.delete(:stream)
# Just render, don't wrap in a Chunked::Body, let
# Rack::Chunked middleware handle it
view_renderer.render_body(view_context, options)
else
super
end
end
end
module ActionController
class Base
include GzipStreaming
end
end
And leave your config.ru as
require ::File.expand_path('../config/environment', __FILE__)
use Rack::Chunked
use Rack::Deflater
run Roam7::Application
Not a very nice solution, it will probably break some other middlewares that inspect/modify the body. If someone has a better solution I'd love to hear it.
If you are using new relic, its middleware must also be disabled when streaming.
I am working on a Rails App that Uses OmniAuth to gather Oauth/OAuth2 credentials for my users and then posts out to those services on their behalf.
Creating simple posts to update status feeds work great.. Now I am to the point of needing to upload files. Facebook says "To publish a photo, issue a POST request with the photo file attachment as multipart/form-data." http://developers.facebook.com/docs/reference/api/photo/
So that is what I am trying to do:
I have implemented the module here: Ruby: How to post a file via HTTP as multipart/form-data? to get the headers and data...
if appearance.post.post_attachment_content_type.to_s.include?('image')
fbpost = "https://graph.facebook.com/me/photos"
data, headers = Multipart::Post.prepare_query("title" => appearance.post.post_attachment_file_name , "document" => File.read(appearance.post.post_attachment.path))
paramsarray = {:source=>data, :message=> appearance.post.content}
response = access_token.request(:post, fbpost, paramsarray, headers)
appearance.result = response
appearance.save
end
I but I am getting a OAuth2::HTTPError - HTTP 400 Error
Any assistance would be Incredible... As I see this information will also be needed for uploading files to SoundCloud also.
Thanks,
Mark
Struggled with this myself. The oauth2 library is backed by Faraday for it's HTTP interaction. with a little configuration it supports uploaded files out of the box. First step is to add the appropriate Faraday middleware when building your connection. An example from my code:
OAuth2::Client.new client_id, secret, site: site do |stack|
stack.request :multipart
stack.request :url_encoded
stack.adapter Faraday.default_adapter
end
This adds the multipart encoding support to the Faraday connection. Next when making the request on your access token object you want to use a Faraday::UploadIO object. So:
upload = Faraday::UploadIO.new io, mime_type, filename
access_token.post('some/url', params: {url: 'params'}, body: {file: upload})
In the above code:
io - An IO object for the file you want to upload. Can be a File object or even a StringIO.
mime_type - The mime type of the file you are uploading. You can either try to detect this server-side or if a user uploaded the file to you, you should be able to extract the mime type from their request.
filename - What are are calling the file you are uploading. This can also be determined by your own choosing or you can just use whatever the user uploading the file calls it.
some/url - Replace this with the URL you want to post to
{url: 'params'} - Replace this with any URL params you want to provide
{file: upload} - Replace this with your multipart form data. Obviously one (or more) of the key/value pairs should have an instance of your file upload.
I'm actually using successfully this code to upload a photo on a fb page :
dir = Dir.pwd.concat("/public/system/posts/images")
fb_url = URI.parse("https://graph.facebook.com/#{#page_id}/photos")
img = File.open("myfile.jpg")
req = Net::HTTP::Post::Multipart.new(
"#{fb_url.path}?access_token=#{#token}",
"source" => UploadIO.new(img, "application/jpg", img.path),
"message" => "some messsage"
)
n = Net::HTTP.new(fb_url.host, fb_url.port)
n.use_ssl = true
n.verify_mode = OpenSSL::SSL::VERIFY_NONE
n.start do |http|
#result = http.request(req)
end