Is it possible, using Suave.IO and F#, to create a simple one-line TCP server along the lines of the http server below?
startWebServer defaultConfig (Successful.OK "Hello World!")
(the TCP server could do something simple like reply with the string it received)
Thanks, Ian
I don't think so. If you look at their API Reference, they support 2 protocols, HTTP and HTTPS:
type Protocol =
| HTTP
| HTTPS of obj
member secure : bool
Related
I have a Lua proxy that needs to route requests. Each request destination is established based on the response from another HTTP request with a header from the initial request. My understanding is that HAProxy is an event-driven software, so blocking system calls are absolutely forbidden and my code is blocking because is doing an HTTP request.
I read about yielding after the request but I think it won't help since the HTTP request is already started. The library for doing the request is https://github.com/JakobGreen/lua-requests#simple-requests
local requests = require('requests')
core.register_fetches('http_backend', function(txn)
local dest = txn.sf:req_fhdr('X-dest')
local url = "http://127.0.0.1:8080/service";
local response = requests.get(url.."/"+dest);
local json = response.json()
return json.field
end )
How do I convert my code to be non-blocking?
You should consider using HAProxy's SPOE which was created exactly for these blocking scenarios.
I managed to do it using Lua. The thing I was making wrong was using require('requests') this is blocking. Ideally for HA never use a Lua external library. I have to deal with plain sockets and do an HTTP request and very important to use HA core method core.tcp() instead of Lua sockets.
I am new with Erlang and my doubt is how to create a JSON Object in Erlang and pass that object in REST API call. I have read so many posts but didn't get any satisfying answer.
Edit
Here I am calling the API:
offline_message(From, To, #message{type = Type, body = Body}) ->
Type = xml:get_text(Type),
Body = xml:get_text(Body),
Token = gen_mod:get_module_opt(To#jid.lserver, ?MODULE, auth_token, fun(S) -> iolist_to_binary(S) end, list_to_binary("")),
PostUrl = gen_mod:get_module_opt(To#jid.lserver, ?MODULE, post_url, fun(S) -> iolist_to_binary(S) end, list_to_binary("")),
to = To#jid.luser
from = From#jid.luser
if
(Type == <<"chat">>) and (Body /= <<"">>) ->
Sep = "&",
Post = {
"token":binary_to_list(Token),
"from":binary_to_list(from),
"to":binary_to_list(to),
"body":binary_to_list(Body)
},
?INFO_MSG("Sending post request to ~s with body \"~s\"", [PostUrl, Post]),
httpc:request(post, {binary_to_list(PostUrl), [], "application/json", binary_to_list(Post)},[],[]),
ok;
true ->
ok
end.
Is everything ok here regarding JSON String. I am trying to modify this module.
How to create a JSON Object in Erlang
There are no such things as objects in erlang, so the simple answer is: you can't. However, the things you send over the wire are just strings, and you can certainly create strings using erlang.
To make things easier, you can use an erlang module like jsx to create the json formatted strings that you want to send in your request. However, in order to use erlang modules you will have to learn a little about rebar3, which is erlang's package installer (see What is the easiest way for beginners to install a module?).
Remember that an http request is just a string that is formatted a certain way. An http request starts with a line like:
POST /some/path HTTP/1.1
Then there are some lines of text called headers, which look like:
User-Agent: Mozilla-yah-yah-bah
Content-Type: application/json
Content-Length: 103
Then there are a couple of newlines followed by some additional text, which is called the post body, which can be in several different formats (the format should be declared in the Content-Type header):
Format Content-Type
------ -----------
"x=1&y=2" application/x-www-form-urlencoded
"{x:1, y:2}" application/json
"more complex string" multipart/form-data
To make it easier to assemble an http request and send it to a server, erlang has a builtin http client called inets, which you can read about in the docs here. For an example that uses inets, see here. Because inets is a bit cumbersome to use, alternatively you can use a third party http client like hackney. Once again, though, you will need to be able to install hackney with rebar3.
Once you send the request, it's up to the server to decipher the request and take the necessary course of action.
I have some remote REST API running through HTTP2. It runs through SSL using certificate. The goal is to send and receive data via HTTP2 with SSL certificate via Proxy.
There are http-2 & net-http2 gems that allow to send requests with HTTP2. But what about proxy? In a standard Net::HTTP library, there is a child class, Net::HTTP::Proxy that duplicates behavior of parent's Net::HTTP class except the fact it sends requests via proxy-server. But HTTP2 gems does not support it.
The closes idea I came up is to make something similar to Proxy implementation of http1.1 - to write "Host:" and "Proxy-Authorization: " fields to the socket, that Net-Http2 gem uses:
#socket.writeline sprintf('CONNECT %s:%s HTTP/%s',
#address, #port, HTTPVersion)
#socket.writeline "Host: #{#address}:#{#port}"
if proxy_user
credential = ["#{proxy_user}:#{proxy_pass}"].pack('m')
credential.delete!("\r\n")
#socket.writeline "Proxy-Authorization: Basic #{credential}"
end
#socket.writeline ''
But it ends up with:
SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
I might miss some technical knowledge to achieve this, so any help related to direction of research is appreciated.
After all, I finished my idea with an example present in net/http standard library and created a pull request for Net-http2 gem: https://github.com/ostinelli/net-http2/pull/11
The idea was correct, all we have to do is to send proxy "CONNECT" message using any tcp socket, with an address where we want to connect, so it creates a TCP tunnel which bypass all the data in and out, doesn't matter if it is HTTP1.1 or HTTP2, or anything else.
Here is a part of code:
def self.proxy_tcp_socket(uri, options)
proxy_addr = options[:proxy_addr]
proxy_port = options[:proxy_port]
proxy_user = options[:proxy_user]
proxy_pass = options[:proxy_pass]
proxy_uri = URI.parse("#{proxy_addr}:#{proxy_port}")
# create a regular TCP socket (with or w/o SSL, if needed)
proxy_socket = tcp_socket(proxy_uri, options)
# The majority of proxies do not explicitly support HTTP/2 protocol,
# while they successfully create a TCP tunnel
# which can pass through binary data of HTTP/2 connection.
# So we’ll keep HTTP/1.1
http_version = '1.1'
buf = "CONNECT #{uri.host}:#{uri.port} HTTP/#{http_version}\r\n"
buf << "Host: #{uri.host}:#{uri.port}\r\n"
if proxy_user
credential = ["#{proxy_user}:#{proxy_pass}"].pack('m')
credential.delete!("\r\n")
buf << "Proxy-Authorization: Basic #{credential}\r\n"
end
buf << "\r\n"
proxy_socket.write(buf)
validate_proxy_response!(proxy_socket)
proxy_socket
end
I'm trying to find out if a play 2 (with scala) app is running on http or https
I tried with routes.Application.index.absoluteURL(request), like this
def chatUri(username: String)(implicit request: RequestHeader): String = {
val uri = routes.Application.index.absoluteURL(request)
but I get the following error:
/home/sas/tmp/websocket-chat/app/controllers/Application.scala:51: overloaded method value absoluteURL with alternatives:
[error] (secure: Boolean)(implicit request: play.api.mvc.RequestHeader)java.lang.String <and>
[error] (play.mvc.Http.Request)java.lang.String
[error] cannot be applied to (play.api.mvc.RequestHeader)
[error] val rootUri = Uri(routes.Application.index.absoluteURL(request))
I tried to transform the RequestHeader into a Request, but I get the following error
val rootUri = Uri(routes.Application.index.absoluteURL(request.asInstanceOf[Request[Any]]))
(secure: Boolean)(implicit request: play.api.mvc.RequestHeader)java.lang.String <and>
[error] (play.mvc.Http.Request)java.lang.String
[error] cannot be applied to (play.api.mvc.Request[Any])
[error] val rootUri = Uri(routes.Application.index.absoluteURL(request.asInstanceOf[Request[Any]]))
Any idea how can I achieve it?
Must say I'm surprised about problems with getting absolute url in Scala, AFAIR in Java it works well, anyway... I doubt if it will help you to determine the protocol (edit: as #MariusSoutier wrote)
As there's no built-in support for SSL in Play 2 most probably you are using (or you should use) some HTTP server on front of your application, let's say Apache. There are some samples and posts describing the proccess:
Take a look at topic: How to config PlayFramework2 to support SSL? Nasir gives there a sample of configuring the Apache as a proxy for Play
There's also nice description of configuring Apache as a proxy (warning the posts describes the Play 1.x, however Apache part will be the same)
Finally you need to to set the proper headers which will be forwarded to your app
so after setting the headers (as showed in point 3) you'll be able to check it in your controller:
def index = Action { request =>
val proto = request.headers("X-FORWARDED-PROTO")
Ok("Got request [" + request + "] with schema: " + proto )
}
or the same in Java controller:
public static Result index() {
String proto = request().getHeader("X-FORWARDED-PROTO");
return ok("Got request [" + request() + "] with schema: " + proto);
}
First off, by creating an absolute URL, you cannot find out if the app is running on http or https - take a look at the method signature:
def absoluteURL (secure: Boolean = false)(implicit request: RequestHeader): String
That's right, you have to tell this method whether or not you want secure.
I think this is because Play was designed to working behind a reverse proxy which makes it transparent to use encrypted requests. That means Play shouldn't have to care about this. absoluteURL can only enforce https URLs, for example to make sure a login page uses https.
Depending on your reverse proxy, you can set a custom http header that tells you what is used. RequestHeader doesn't have the information.
I want to build a small script in python which needs to fetch an url. The server is a kind of crappy though and replies pure ASCII without any headers.
When I try:
import urllib.request
response = urllib.request.urlopen(url)
print(response.read())
I obtain a http.client.BadStatusLine: 100 error because this isn't a properly formatted HTTP response.
Is there another way to fetch an url and get the raw content, without trying to parse the response?
Thanks
It's difficult to answer your direct question without a bit more information; not knowing exactly how the (web) server in question is broken.
That said, you might try using something a bit lower-level, a socket for example. Here's one way (python2.x style, and untested):
#!/usr/bin/env python
import socket
from urlparse import urlparse
def geturl(url, timeout=10, receive_buffer=4096):
parsed = urlparse(url)
try:
host, port = parsed.netloc.split(':')
except ValueError:
host, port = parsed.netloc, 80
sock = socket.create_connection((host, port), timeout)
sock.sendall('GET %s HTTP/1.0\n\n' % parsed.path)
response = [sock.recv(receive_buffer)]
while response[-1]:
response.append(sock.recv(receive_buffer))
return ''.join(response)
print geturl('http://www.example.com/') #<- the trailing / is needed if no
other path element is present
And here's a stab at a python3.2 conversion (you may not need to decode from bytes, if writing the response to a file for example):
#!/usr/bin/env python
import socket
from urllib.parse import urlparse
ENCODING = 'ascii'
def geturl(url, timeout=10, receive_buffer=4096):
parsed = urlparse(url)
try:
host, port = parsed.netloc.split(':')
except ValueError:
host, port = parsed.netloc, 80
sock = socket.create_connection((host, port), timeout)
method = 'GET %s HTTP/1.0\n\n' % parsed.path
sock.sendall(bytes(method, ENCODING))
response = [sock.recv(receive_buffer)]
while response[-1]:
response.append(sock.recv(receive_buffer))
return ''.join(r.decode(ENCODING) for r in response)
print(geturl('http://www.example.com/'))
HTH!
Edit: You may need to adjust what you put in the request, depending on the web server in question. Guanidene's excellent answer provides several resources to guide you on that path.
What you need to do in this case is send a raw HTTP request using sockets.
You would need to do a bit of low level network programming using the socket python module in this case. (Network sockets actually return you all the information sent by the server as it as, so you can accordingly interpret the response as you wish. For example, the HTTP protocol interprets the response in terms of standard HTTP headers - GET, POST, HEAD, etc. The high-level module urllib hides this header information from you and just returns you the data.)
You also need to have some basic information about HTTP headers. For your case, you just need to know about the GET HTTP request. See its definition here - http://djce.org.uk/dumprequest, see an example of it here - http://en.wikipedia.org/wiki/HTTP#Example_session. (If you wish to capture live traces of HTTP requests sent from your browser, you would need a packet sniffing software like wireshark.)
Once you know basics about socket module and HTTP headers, you can go through this example - http://coding.debuntu.org/python-socket-simple-tcp-client which tells you how to send a HTTP request over a socket to a server and read its reply back. You can also refer to this unclear question on SO.
(You can google python socket http to get more examples.)
(Tip: I am not a Java fan, but still, if you don't find enough convincing examples on this topic under python, try finding it under Java, and then accordingly translate it to python.)
urllib.urlretrieve('http://google.com/abc.jpg', 'abc.jpg')