I'm running an app which accepts data from GPS loggers, being this data sent via SMS, HTTP or TCP. I was using the HTTP connection to transmit the data, which was straightforward to receive through a GET request on my Rails app.
For battery saving purposes, we changed the connection to pure TCP and nginx is not accepting these requests for now:
From nginx's access.log:
HTTP:
xx.xx.xx.xxx - - [03/Mar/2013:20:17:45 -0500] "GET /?imei=123456789012345&rmc=$GPRMC,035106.000,A,4145.1451,N,08506.8784,W,1.56,176.49,010213,,*13,3656mV,AUTO HTTP/1.1" 302 247 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22"
TCP:
xx.xx.xx.xxx - - [03/Mar/2013:18:31:22 -0500] "imei=123456789012345&rmc=$GPRMC,233012.000,A,4221.6614,N,07106.1014,W,17.25,218.94,030313,,*21,4102mV,AUTO,310,260,ADB7,13EF,057,310,260,0000,0000,044,310,260,0000,0000,055,310,260,0000,0000,05A,310,260,0000,0000,059,310,260,0000,0000,05E,0,0,0000,0000,000,2" 400 172 "-" "-"
Does anyone know a way to filter these TCP requests on Nginx and process them as HTTP GET requests to be available through a Rails app?
Thanks in advance
Answer:
I was able to achieve this by creating a custom proxy, using em-proxy and faraday:
require 'em-proxy'
require 'faraday'
Proxy.start(:host => "0.0.0.0", :port => 8080, :debug => false) do |conn|
conn.on_data do |data|
http = Faraday.new(:url => 'http://example.com') do |faraday|
faraday.request :url_encoded # form-encode POST params
faraday.response :logger # log requests to STDOUT
faraday.adapter Faraday.default_adapter # make requests with Net::HTTP
end
http.get "/upload?#{data}"
end
end
You should look at websockets. Nginx now has native support for them, but there is an older module for Nginx that does the work as well.
Here's an older question about this with both solutions provided.
Nginx is a HTTP server so it expects clients to speak HTTP. It is not possible to have a client speaking a non-HTTP protocol to nginx in the manner you're describing (ok, maybe you could write a custom module to do it, but that seems silly).
I think what you need to do is write your own proxy server which will convert the raw TCP protocol into HTTP requests. Your GPS loggers would talk to the proxy which would then talk to nginx. This ought to be fairly simple since the TCP payload appears to just be a urlencoded string, so you could just use those directly.
Related
I am getting occasional errors of this type in production and staging:
Unsafe redirect to "https://${ip}:${port}/businesses/new", pass allow_other_host: true to redirect anyway.
It is being caused by Curl requests from a random IP:
User-Agent: "curl/7.64.1"
Accept: "*/*"
Host: "${ip}:${port}"
Version: "HTTP/1.1"
I do not get it with any other URLs processed within the app.
When I try the same curl request in development I get this (which is correct):
Rails - [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked host: "${ip}:${port}"
I cannot find where the difference is that makes this throw an exception in production.
Any insights on this issue would be appreciated.
Using the method described in this answer (clojure.java.io/input-stream), how would I go about setting the User-Agent request header?
You can't. clojure.java.io is a very simplistic API, for doing the easiest stuff. If you want any real customization you'll need to use a real HTTP library.
You can, but only to a certain extent, setting a property.
From this answer https://stackoverflow.com/a/47300260/483566 I tried running netcat in a terminal: nc -l -p 8080
On the REPL, I tried the following:
$ lein repl
nREPL server started on port 42819 on host 127.0.0.1 - nrepl://127.0.0.1:42819
REPL-y 0.4.3, nREPL 0.6.0
Clojure 1.10.0
OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (System/setProperty "http.agent" "Clojure REPL")
nil
user=> (slurp "http://localhost:8080/")
Netcat is not actually serving content, so the REPL will block, but if you see the terminal where netcat runs, you'll see something like:
GET / HTTP/1.1
User-Agent: Clojure REPL Java/11.0.4
Host: localhost:8080
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
For actual customization, you probably want to use an HTTP client library that provides access to the HTTP headers (most will do).
I am constantly getting Failed to open TCP connection to :80 (Cannot assign requested address - connect(2) for nil port 80) (Errno::EADDRNOTAVAIL) while using the ruby faraday gem. I don't have alot of experience with ruby on rails.
I have a docker ruby on rails service running on elastic beanstalk that is using puma with ssl. CMD ["bundle", "exec", "puma", "-C", "config/puma.rb"] I have a network load balancer configured with it to forward 443 -> 8443 (i've experimented with both self signed certs and real wild card certs).
ssl_bind '0.0.0.0', '8443', {
key: '/var/app/ssl/something.key',
cert: '/var/app/ssl/something.crt'
}
This configuration works as expected
Puma starting in single mode...
* Version 3.12.0 (ruby 2.5.1-p57), codename: Llamas in Pajamas
* Min threads: 5, max threads: 5
* Environment: staging
* Listening on tcp://0.0.0.0:3000
* Listening on ssl://0.0.0.0:8443?cert=/var/app/ssl/something.crt&key=/var/app/ssl/something.key&verify_mode=none
and I can get the healthz status page with both types of certs. Using httpie and --verify=no for the self signed.
$ http https://backend.something.com/healthz
HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Type: application/json; charset=utf-8
{
"name": "my-backend-service",
"version": "0.0.1"
}
I have another ruby on rails backend service that makes api requests to this service using faraday. Ive removed some of the request/response code from my actual.
def connection(baseUrl, options = {})
conn = Faraday.new(url: baseUrl) do |c|
# dont really know if this is needed or not
# http.use_ssl? is always false
c.adapter :net_http do |http|
http.verify_mode = OpenSSL::SSL::VERIFY_NONE # if http.use_ssl?
end
end
I don't know if this http.verify_mode is actually working. I cant really find that method anywhere around here on ruby-doc.org
If you try to make a request it will just end up faiing.
conn = connection(https://backend.something.com)
response = conn.post '/foo', params[:foo].to_json
The logs show from starting to parameters and then the http.rb:939 error. I realize the parameters aren't valid here but they aren't my problem.
Started POST "/foo"
Processing by FooController#create as */*
Parameters: {"paramter"=>"something", "paramter"=>"something", "paramter"=>"something"}
ERROR -- : /usr/local/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect': Failed to open TCP connection to :80 (Cannot assign requested address - connect(2) for nil port 80) (Errno::EADDRNOTAVAIL)
If I make the same request from httpie or curl to this service I get the expected results over both http/https.
$ http POST https://backend.something.com parameter="something" parameter="something" parameter="something"
HTTP/1.1 201 Created
Cache-Control: max-age=0, private, must-revalidate
Content-Type: application/json; charset=utf-8
If you inspect the conn object it kinda still seems like it is sitting on its default value for #url_prefix = http:/. Found in the docs above, but I don't know if I'm looking at the correct thing the correct ruby way. I imagined that Faraday.new(url: baseUrl) would parse the correct schema, which is https.
#<Faraday::Connection:0x0000565351701430 #parallel_manager=nil, #headers={"User-Agent"=>"Faraday v0.15.4"}, #params={}, #options=#<Faraday::RequestOptions (empty)>, #ssl=#<Faraday::SSLOptions (empty)>, #default_parallel_manager=nil, #builder=#<Faraday::RackBuilder:0x0000565351700f58 #handlers=[FaradayMiddleware::EncodeJson, FaradayMiddleware::ParseJson, Faraday::Adapter::NetHttp]>, #url_prefix=#<URI::HTTP http:/>, #manual_proxy=false, #proxy=nil, #temp_proxy=nil>
It seems that the the address you are trying to connect to is invalid. From what I can see in the error:
ERROR -- : /usr/local/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect': Failed to open TCP connection to :80 (Cannot assign requested address - connect(2) for nil port 80) (Errno::EADDRNOTAVAIL)
it is nil. Ensure that the host is properly set.
I am having trouble making a proxy call. How on earth do you make this happen?
Here is what I have so far:
proxy_addr = '162.243.105.128'
proxy_port = 6170
Net::HTTP::Proxy(proxy_addr, proxy_port).start('www.google.com') {|http| http}
I get:
#<#<Class:0x007f85d8a092d0> www.google.com:80 open=false>
When:
Net::HTTP::Proxy(proxy_addr, proxy_port).start('www.google.com') {|http| http.get('www.google.com')}
I get
#<Net::HTTPNotImplemented 501 Tor is not an HTTP Proxy readbody=true>
How do I make this work!
Tor is a SOCKS proxy, not an HTTP proxy.
I ran into the same issue. I don't think it is possible with plain Net::HTTP. Install http://socksify.rubyforge.org/
require 'socksify/http'
http = Net::HTTP::SOCKSProxy(addr, port)
puts http.get(URI('http://echoip.com'))
I keep getting the two following errors from my server, I assumed they were just bots looking for potential targets, but does anyone know specifically why I'm getting these? I'm using the SslRequirement plugin to make sure all hits to the login/signup page are redirected to SSL, so all of these weird https requests to root should just be redirected to regular http.
A ActionController::UnknownHttpMethod occurred in application#index:
quit, accepted HTTP methods are get, head, put, post, delete, and options
/usr/local/lib/ruby/gems/1.9.1/gems/actionpack-2.3.4/lib/action_controller/request.rb:35:in `request_method'
PATH_INFO : /
REMOTE_ADDR : 99.19.208.249
REMOTE_PORT : 6376
REQUEST_METHOD : CONNECT
REQUEST_URI : /
SERVER_PORT : 443
SERVER_PROTOCOL : HTTP/1.0
SERVER_SOFTWARE : Apache
A ActionController::UnknownHttpMethod occurred in application#index:
CONNECT, accepted HTTP methods are get, head, put, post, delete, and options
/usr/local/lib/ruby/gems/1.9.1/gems/actionpack-2.3.4/lib/action_controller/request.rb:35:in `request_method'
HTTPS : on
HTTP_X_FORWARDED_PROTO : https
PATH_INFO : /
REMOTE_ADDR : 91.209.196.76
REMOTE_PORT : 50751
REQUEST_METHOD : quit
REQUEST_URI : /
SERVER_PORT : 443
SERVER_PROTOCOL : HTTP/0.9
The CONNECT command is used by HTTP proxy servers to indicate that the client wants to just connect a socket directly to another server; this is usually used for tunneling TLS over an HTTP proxy, but could be used for tunneling almost any protocol.
QUIT is not an HTTP command, but it is an SMTP command. It is possible that you are getting these commands from a bot that is trying to find open relays for sending spam; it's trying to figure out if you have an open SMTP relay, or an open HTTP proxy that allows the CONNECT command which could also be used to tunnel SMTP traffic.
So, likely you're just being hit by a spam botnet trying to find open relays. My advice would be to drop such requests as early as possible, and not worry about them.
CONNECT: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.9
QUIT usually means 'close the connection'.
# Avoid annoying ActionController::UnknownHttpMethod exceptions like:
#
# ActionController::UnknownHttpMethod) "CONNECT, accepted HTTP methods are get, head, put, post, delete, and options"
#
# Install this file in app/metal and these requests will receive a 405
# "Method Not Allowed" status and will be logged under `info'.
class IgnoreUnknownHttpMethod
def self.call(env)
[
if ActionController::Request::HTTP_METHODS.include?(env["REQUEST_METHOD"].downcase)
404 # Not Found
else
Rails.logger.info("Ignoring unknown HTTP method; #{env.inspect}")
405 # Method Not Allowed
end, {"Content-Type" => "text/plain"}, []]
end
end
Credit https://gist.github.com/remvee/600569
I just noticed a few attempt to connect to my servers as follow, we see that it ends with a QUIT...
198.20.87.98 - - [22/Dec/2015:21:43:42 -0800] "GET / HTTP/1.1" 444 5666 "-" "-"
198.20.87.98 - - [22/Dec/2015:21:43:42 -0800] "GET /robots.txt HTTP/1.1" 444 5666 "-" "-"
198.20.87.98 - - [22/Dec/2015:21:43:42 -0800] "GET /sitemap.xml HTTP/1.1" 444 5666 "-" "-"
198.20.87.98 - - [22/Dec/2015:21:43:58 -0800] "quit" 405 5461 "-" "-"
As a side note, my server returns 444 which is not a legal HTTP code. It means NO RESPONSE and I do that because their "Agent String" is empty.
Looking at the IP address, I found a search engine that searches for things on the internet. Not a hacker trying to break in per se. (There intend is not evil, it seems.) I suppose that some of the applications they are checking for must understand the QUIT. So as a robot, it makes sense that you would want to try what would otherwise looks like weird commands such as CONNNECT and QUIT. There is no real limit to the commands an application can support, even if those commands are not official HTTP commands.
If your application does not understand these methods, then nothing will happen so you should not bother too much about these.
If your application receives those methods, then you may want to write a little something at the start to get the method and compare to GET and POST (and whatever else you support, like DELETE and PUT) and if it matches none of these, then reply with a 405 error code: "Method Not Allowed".
https://www.rfc-editor.org/rfc/rfc7231#section-6.5.5
If you cannot change your application and you are not sure whether it could react to a CONNECT / QUIT hit, then you could look into using a setup such as mod_security for Apache2.
as per jturkel at https://gist.github.com/remvee/600569.
In Rails 3.2 for non http actions I added the following at the end of config/application.rb and solved the problem for quit.
# silence ActionController::UnknownHttpMethod exceptions
ActionDispatch::ExceptionWrapper.rescue_responses.merge!( 'ActionController::UnknownHttpMethod' => :method_not_allowed )