I want to take a large JSON PUT request and process it entirely in the background with Sidekiq. As such, I do not want to automatically parse the incoming JSON from the request's body.
Whats the best practice for accomplishing this on a controller/action-based level?
As far as I know this is done via Rack middleware, so I don't think you can just disable it for one controller/action right out of the box. You can patch ActionDispatch::ParamsParser similar to what this guy did:
http://www.jkfill.com/2015/02/21/skip-automatic-params-parsing/
You, of course, can write your own middleware that parses the parameters, or specify a custom parser for JSON MIME type.
At last, you can disable the JSON parsing all together. In your config/application.rb add:
config.middleware.swap ActionDispatch::ParamsParser, ActionDispatch::ParamsParser, Mime::JSON => nil
Related
In rails 5 development mode, I keep getting this Net::ReadTimeout(Net::ReadTimeout) error in my custom session middleware that does a http request to my local mock custom session engine. Also the error only triggered after I made a code change and rails renders an error page. This is extremely annoying and really slows down the development process since we all have to refresh twice to see the result of code change.
After tracing down the code in those middlewares, It appears that my custom session middleware kick off the http request before the reloader complete reloading.
I wondered if we can halt/stop the rack middleware request from keep going down the rack until the reloading is done.
Rails version: 5.1
Ruby version: 2.4.1
I put the following log messages before and after the reloader is done reloading
application.rb
ActiveSupport::Reloader.to_run do
puts 'Reloading'
end
ActiveSupport::Reloader.to_complete do
puts 'DONE Reloading'
end
custom_session_siddleware.rb
def call(env)
...
puts 'Session Processing'
http = Net::HTTP.new(uri, port)
...
#app.call(env)
end
The out put after I make a code change and refresh
Reloading
Session Processing
DONE Reloading
Here are all my middlewares
use Rack::Sendfile
use ActionDispatch::Static
use ActionDispatch::Executor
use ActiveSupport::Cache::Strategy::LocalCache::Middleware
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use RequestStore::Middleware
use ActionDispatch::RemoteIp
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use ActionDispatch::Reloader
use ActionDispatch::Callbacks
use ActiveRecord::Migration::CheckPending
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use Rack::Head
use Rack::ConditionalGet
use Rack::ETag
use MyCustomSession::CustomSessionMiddleware
If your custom middleware is making a request back to the running Rack application, it's going to be very easy for you to have a bad time. But it can be made to work.
By the time your initial request is inside the Reloader, the actual reload has already occurred -- but that's only the unload.
Following the unload, any application (or engine) classes will need to be autoloaded again the next time they're accessed... and I think that's where you're running into an issue: the inner request needs to load something, but it can't do so while the outer request is active.
To fix this, the outer request needs to inform the system that it's in a safe place for another thread/request to load new code. Specifically, you need to wrap the Net::HTTP request (only -- not the #app.call(env)) with permit_concurrent_loads.
I have a very simple number crunching Ruby function that I want to make available via a web API. The API is essentially a single endpoint, e.g. http://example.com/crunch/<number> and it returns JSON output.
I can obviously install Rails and implement this quickly. I require no more help from a 'framework' other than to handle HTTP for me. No ORM, MVC and other frills.
On the far end, I can write some Ruby code to listen on a port and accept GET request and parse HTTP headers etc. etc. I don't want to re-invent that wheel either.
What can I use to expose a minimal API to the web using something with the least footprint/dependencies. I read about Sinatra, Ramaze, etc., but I believe there can be a way to do something even simpler. Can I just hack some code on top of Rack to do what I am trying to do?
Or in other words, what will be the simplest Ruby equivalent of the following code in nodejs:
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
var ans = crunch(number);
res.end(ans);
}).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');
You seem like you want to use Rack directly. "Rack from the Beginning" is a decent tutorial that should get you started.
It'll probably look something like this:
class CrunchApp
def self.crunch(crunchable)
# top-secret crunching
end
def self.call(env)
crunchy_stuff = input(env)
[200, {}, crunch(crunchy_stuff)]
end
private
def self.input(env)
request = Rack::Request.new(env)
request.params['my_input']
end
end
Rack::Server.start app: CrunchApp
But I must say, using that instead of something like Sinatra seems silly unless this is just a fun project to play with things. See their 'Hello World':
require 'sinatra'
get '/hi' do
"Hello World!"
end
Ruby-Grape is a good option for your use case. It has a minimal implementation over Rack that allow the creation of simple REST-API endpoints.
Cuba is another good option with a thin layer over Rack itself.sample post
If you are familiar with Rails you can use the Rails API gem which is very well documented with minor overhead. Remember also that Rails-API will be part of Rails 5.
Last, but not last you can implement it on Rack directly.
I have a custom rack middleware used by my Rails 4 application. The middleware itself is just here to default Accept and Content-Type headers to application/json if the client did not provide a valid information (I'm working on an API). So before each request it changes those headers and after each request it adds a custom X-Something-Media-Type head with a custom media type information.
I would like to switch to Puma, therefore I'm a bit worried about the thread-safety of such a middleware. I did not play with instances variables, except once for the common #app.call that we encounter in every middleware, but even here I reproduced something I've read in RailsCasts' comments :
def initialize(app)
#app = app
end
def call(env)
dup._call(env)
end
def _call(env)
...
status, headers, response = #app.call(env)
...
Is the dup._call really useful in order to handle thread-safety problems ?
Except that #app instance variable I only play with the current request built with the current env variable :
request = Rack::Request.new(env)
And I call env.update to update headers and forms informations.
Is it dangerous enough to expect some issues with that middleware when I'll switch from Webrick to a concurrent web server such as Puma ?
If yes, do you know a handful way to make some tests en isolate portions of my middleware which are non-thread-safe ?
Thanks.
Yes, it's necessary to dup the middleware to be thread-safe. That way, anything instance variables you set from _call will be set on the duped instance, not the original. You'll notice that web frameworks that are built around Rack work this way:
Pakyow
Sinatra
One way to unit test this is to assert that _call is called on a duped instance rather than the original.
I'm developing a new Ruby on Rails 3.2 application.
This application will receive a periodic json callback with statistics.
The callback i receive is not entirely valid. The json rules are separated by newlines.
The callback POSTs have a content-type header of application/json, and contain exactly one JSON string per line, with each line representing one event. Please note that currently the POST headers define this post as application/json, though it’s not; each line is a valid JSON string, but the overall POST body is not. For example:
This is a example of the callback:
{"email":"foo#bar.com","timestamp":1322000095,"unique_arg":"my unique arg","event":"delivered"}
{"email":"foo#bar.com","timestamp":1322000097,"unique_arg":"my unique arg","event":"click"}
{"email":"foo#bar.com","timestamp":1322000096,"unique_arg":"my unique arg","event":"open"}
When i receive this callback my Rails application crashes with a "MultiJson::DecodeError"
743: unexpected token at '{"email":"foo#bar.com","timestamp":1322000096,"unique_arg":"my unique arg","event":"open"}'
I think the application detects the application/json header and try automatic to parse it.
How can i convert this to a valid JSON object, so i can use this in my controller?
Thanks.
Put
gem 'yajl-ruby' in Gemfile and
require 'yajl/json_gem' config/application.rb
In gem file declare
gem 'yajl-ruby'
It is used to Parse and encode multiple JSON objects to and from streams or strings continuous.
And require 'yajl/json_gem' in application.rb
Put Elements in Brackets:
[{"email":"foo#bar.com","timestamp":1322000095,"unique_arg":"my unique arg","event":"delivered"},
{"email":"foo#bar.com","timestamp":1322000097,"unique_arg":"my unique arg","event":"click"},
{"email":"foo#bar.com","timestamp":1322000096,"unique_arg":"my unique arg","event":"open"}]`
Unable to reproduce your problem by copy-and-pasting the error message token using both ActiveSupport::JSON.decode and MultiJson.decode in a Rails shell. Rails 3.2.1
Any other clues?
This issue is likely due to response body encoded in ISO-8859-1 instead of UTF-8. Possible fix on client side is be convert it back to UTF-8 using ruby 'iconv' encoding methods.
How does one create a POST request using TCPSocket in Ruby? Is there a special format to making a post? I have the following but I get a parse error (it's for a rails server):
require 'socket'
s = TCPSocket.open("localhost", 3000)
s.puts("POST /<controller>/<action> HTTP/1.1")
s.puts("Host: localhost:3000")
s.puts("Content-Type: application/x-www-form-urlencoded")
s.puts("Content-Length: 103\r\n\r\n")
The Host: field should not include the port number.
Found this article that may be of some use to you. I especially like Eric Hodel's comment about how to do it with Net::HTTP. I know you specified that you wanted to do TCPSocket.send (presumably because you're working on something slightly more interesting than just sending POSTs), but if you aren't doing something more complicated you may be able to use Net::HTTP and rejoice at how easy it is.