Is there a way in Rails to specify the maximum allowed content-length so that requests that exceed this value are rejected immediately?
I have a login form on my application that is the only POST available to an unauthenticated user. This has been identified as a potential vulnerability to a slow POST DoS attack. One of the mitigations is to limit the allowed request size.
I cannot seem to find the knob to turn which will allow me to automatically reject the request if the content-length exceeds a particular value.
We're using the Puma web server if that affects the answer.
Puma has two parameters actually, the number of threads and the number of workers. If we slightly change the default puma.rb, it will look like that:
workers Integer(ENV['WORKERS_NUMBER'] || 1)
max_threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 1)
min_threads_count = max_threads_count
threads min_threads_count, max_threads_count
Related
We implemented connection pooling in our client code to invoke a server which closes(sends Connection:close in response headers) a connection after 2.5mins. Due to server behaviour we sometimes/intermittently get NoHttpResponseException. And this may occur at high TPS or at low TPS as well.
We are using apache http client version 4.5.11. And there is one validateAfterInactivity setting in PoolingHttpClientConnectionManager which is by-default set to 2000ms. But i think we may get same exception if we try to get the connection in 2000ms period.
We can choose to set aggressive value for validateAfterInactivity but i heard that it can degrade the performance by ~20 to 30ms for each request.
is retrying this exception a good solution ?
And also align to same context, can we retry in case of java.net.SocketException: Connection reset ?
#ok2c any suggestion here ?
Thanks in advance.
NoHttpResponseException is considered safe to retry for idempotent methods.
In your particular case however I would consider limiting the TTL (total to live) of client connections to 2.5 minutes to match that of the server endpoints.
I have an application which makes GET request of length 18k characters. If this requests goes through HAProxy then I get immediately 400. If I hit directly my service everything is fine. Is there a parameter in HAProxy which sets maximum length of URL request in HAProxy?
Thanks in advance
I am using nginx to proxy to a unicorn upstream running a Ruby on Rails application. I want to be able to limit the total amount of the backend resources a singe user (IP address) can consume. By backend resources, I mean the number of active requests a user can have running on the upstream unicorn processes at once.
So for example, if an IP address already has 2 writing connections to a particular upstream, I want any further requests to be queued by nginx, until one of the previously open connections is complete. Note that I don't want requests to be dropped - they should just wait until the number of writing connections drops below 2 for the user.
This way, I can ensure that even if one user attempts many requests for a very time consuming action, they don't consume all of the available upstream unicorn workers, and some unicorn workers are still available to service other users.
It seems like ngx_http_limit_conn_module might be able to do this, but the documentation is not clear enough for me to be sure.
Another way to think about the problem is that I want to protect against DoS (but not DDoS, i.e. I only care about DoS from one IP at a time), by making the server appear to any one IP address as if it has the ability to process N simultaneous requests. But in reality the server can process 10*N requests, but I am limiting the simultaneous requests from any one IP to 1/10th of the server's real capacity. Just like a normal server behaves, when the number of simultaneous workers is exceeded requests are queued until previous requests have completed.
You can user limit_req module
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
It doesn't limit number of connections, but it limits requests per second. Just use large burst to delay request and not to drop them.
Here's example.
http {
limit_req_zone $binary_remote_addr zone=one:10m rate=2r/s;
...
server {
...
location / {
limit_req zone=one burst=50;
}
You know that average request processing time is say 1 second, so setting limit to 2r/s allows only two workers being busy with this particular ip address (approximately, of course). If request takes 0.5 sec to complete, you can set 4r/s.
If you know the time consuming url, then you can make use of the limit_req module to implement 2r/s for the long requests, and no limit for short requests.
http {
...
#
# Limit request processing rate or connection
# If IP address is 127.0.0.1, the limit_conn_zone will not count it.
#
geo $custom_remote_addr $custom_limit_ip {
default $binary_remote_addr;
127.0.0.1 "";
}
limit_req_zone $custom_limit_ip zone=perip:10m rate=2r/s;
...
server {
...
# By default, do not enforce the maximum allowed number of connections for the remote IP
set $custom_remote_addr 127.0.0.1;
# If the URI matches a super time consuming requests, limit to 2r/s.
if ($uri ~* "^/super-long-requests") {
set $custom_remote_addr $remote_addr;
}
limit_req zone=perip burst=50;
...
}
}
I'm running an app on heroku where users can get the HTTP status (200,301,404, etc) of several URLs that they can paste on a form.
Although it runs fine on my local rails server, when I upload it on heroku, I cannot check more than 30 URLs (I want to check 200), as heroku time outs after 30seconds giving me an H12 Error.
def gethttpresponse(url)
httpstatusarray = Hash.new
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
httpstatusarray['url'] = url
httpstatusarray['status'] = response.code
return httpstatusarray
end
At the moment I'm using Net:HTTP, and it seems very heavy. Is there anything I can change on my code or any other gem I could use to get the HTTP status/headers on a more efficient (fast) way?
i noticed that
response.body holds the entire HTML source code of the page which i do not need. is this loaded on the response object by default?
If this is the most efficient way to check HTTP Status, would you agree that this needs to become a background job?
Any reference on gems that work faster, reading material and thoughts are more than welcome!
If a request takes longer than 30 seconds then the request is timed out as you're seeing here. You're entirely dependent on how fast the server the other end responds. For example, if say it were itself a Heroku application on 1 dyno then it will take possibly 10 seconds to unidle the application therefore leaving only 20 seconds for the other URLs to be tested.
I suggest you move each check to it's own background job and then poll the jobs to know when they complete and update the UI accordingly.
Recently I added a Varnish instance to a Rails application stack. Varnish in it's default configuration can be convinced from caching a certain resource using the Cache-Control Header like so:
Cache-Control: max-age=86400, public=true
I achieved that one using the expires_in statement in my controllers:
def index
expires_in 24.hours, public: true
respond_with 'some content'
end
That worked well. What I did not expect is, that the Cache-Control header ALSO affects the browser. That leads to the problem that both - Varnish and my users browser cache a certain resource. The resource is purged from varnish correctly, but the browser does not attempts to request it again unless max-age is reached.
So I wonder wether I should use 'expires_in' in combination with Varnish at all? I could filter the Cache-Control header in a Nginx or Apache instance in front of Varnish, but that seems odd.
Can anyone enlighten me?
Regards
Felix
That is actually a very good and valid question, and a very common one with reverse proxies.
The problem is that there's only one Cache-Control property and it is intended for the client browser (private cache) and/or a proxy server (shared cache). If you don't want 3rd party proxies to cache your content at all, and want every request to be served by your Varnish (or by your Rails backend), you must send appropriate Cache-Control header from Varnish.
Modifying Cache-Control header sent by the backend is discussed in detail at https://www.varnish-cache.org/trac/wiki/VCLExampleLongerCaching
You can approach the solution from two different angles. If you wish to define max-age at your Rails backend, for instance to specify different TTL for different objects, you can use the method described in the link above.
Another solution is to not send Cache-Control headers at all from the backend, and instead define desirable TTLs for objects in varnish vcl_fetch(). This is the approach we have taken.
We have a default TTL of 600 seconds in Varnish, and define longer TTLs for pages that are definitely explicitly purged when changes are made. Here's our current vcl_fetch() definition:
sub vcl_fetch {
if (req.http.Host ~ "(forum|discus)") {
# Forum pages are purged explicitly, so cache them for 48h
set beresp.ttl = 48h;
}
if (req.url ~ "^/software/") {
# Software pages are purged explicitly, so cache them for 48h
set beresp.ttl = 48h;
}
if (req.url ~ "^/search/forum_search_results" ) {
# We don't want forum search results to be cached for longer than 5 minutes
set beresp.ttl = 300s;
}
if(req.url == "/robots.txt") {
# Robots.txt is updated rarely and should be cached for 4 days
# Purge manually as required
set beresp.ttl = 96h;
}
if(beresp.status == 404) {
# Cache 404 responses for 15 seconds
set beresp.http.Cache-Control = "max-age=15";
set beresp.ttl = 15s;
set beresp.grace = 15s;
}
}
In our case we don't send Cache-Control headers at all from the web backend servers.