I’m using jersey-client v1.18.1
I need to make 2 sequential requests where the 1st request has an InputStream and then must pass that same InputStream along to the 2nd request (eg. sort of like a proxy). The 2nd request will then write the InputStream to disk and send back to the 1st request the fully qualified path to the location on disk where the 2nd request wrote the file.
The following code-snippet outlines what I have tried, but cannot get to work properly. I’m currently receiving the error:
"com.sun.jersey.api.client.ClientHandlerException: A message body writer for Java type, class org.seleniumhq.jetty9.server.HttpInputOverHTTP, and MIME media type, application/octet-stream, was not found”
I believe I have all the proper Maven dependencies in my project for the MIME and message body writers.
1st Request originating on Host 1 going to Host 2
Client client = Client.create();
client.resource(uri_for_request_1)
client.path(“request_1_servlet");
client.queryParam(“uri_for_request_2", uri_for_request_2);
client.queryParam("targetFilename", targetFilename);
ClientResponse response = client.accept(MediaType.APPLICATION_JSON).entity(inputStream).post(ClientResponse.class);
2nd Request originating on Host 2 going to Host 3
Client client = Client.create();
client.resource(request.getParamater(“uri_for_request_2"))
client.path(“request_2_servlet");
client.queryParam("targetFilename", request.getParamater(“targetFilename");
ClientResponse response = client.accept(MediaType.APPLICATION_JSON).entity(request.getInputStream()).post(ClientResponse.class);
Host 3
Writes InputStream to file and sends back to Host 2 fully qualified path.
Host 2
Sends back to Host 1 fully qualified path.
Variations I’ve tried on post calls:
client.accept(MediaType.APPLICATION_JSON).type(MediaType.APPLICATION_OCTET_STREAM).entity(inputStream).post(ClientResponse.class);
client.accept(MediaType.APPLICATION_JSON).type(MediaType.APPLICATION_OCTET_STREAM).post(ClientResponse.class, inputStream);
I can confirm the 1st request is being made on Host 1 and reaches Host 2. It is the 2nd request on Host 2 that fails during the post() call.
Related
I have been scouring the internet with no luck. I have a basic LUA script for HAProxy, which looks like this:
core.Info("LUA script for parsing request ID element - loaded");
function parseId(txn, salt)
local payload = txn.sf:req_body()
-- parses hex value from element named "ID". Example payload: {"Platform":"xyz.hu","RecipientId":"xyz.hu","Channel":"xyz","CallbackURL":"http://x.x.x.x:123","ID":"5f99453d000000000a0c5164233e0002"}
local value = string.sub(string.match(payload, "\"ID\":\"[0-9a-f]+\""), 8, -2)
core.Info("ID : " .. value)
return value
end
-- register HAProxy "fetch"
core.register_fetches("parseId", parseId)
What it does is what it says: takes a 32 characater long ID from an incoming request. In the HAProxy config file, the result is used for sticky-session handling:
stick-table type string len 32 size 30k expire 30m
stick on "lua.parseId" table gw_back
This produces two lines of log for each request:
ID: xyz which is logged from the LUA script
The detailed request data which is logged from the HAProxy config file using "log-format", e.g.:
Jan 20 22:13:52 localhost haproxy[12991]: Client IP:port = [x.x.x.x:123], Start Time = [20/Jan/2022:22:13:52.069], Frontend Name = [gw_front], Backend Name = [gw_back], Backend Server = [gw1], Time to receive full request = [0 ms], Response time = [449 ms], Status Code = [200], Bytes Read = [308], Request = ["POST /Gateway/init HTTP/1.1"], ID = [""], Request Body = [{"Platform":"xyz.hu","RecipientId":"xyz.hu","Channel":"xyz","CallbackURL":"http://x.x.x.x:123","ID":"61e9d03e000000000a0c5164233e0002"}]
I wanted to extend logging due to some strange issues happening sometimes, so I wanted to one (or both) of below approaches:
Pass the "ID" value back from the LUA script into the HAProxy config as a variable, and log it along with the request details. I can log the full request body, but don't want to due to GDPR and whatnot.
Get some request details in the LUA script itself, and log it along with the ID.
So, basically, to be able to connect the ID with the request details. If multiple requests are coming to same URL very quickly, it is difficult to find which of them belongs to a specific ID. However I couldn't accomplish these.
For the first one, I added this line into the LUA before returning the "value" variable:
txn:set_var("req_id", value)
I was hoping this would create a variable in HAProxy called "req_id", and I can log it with "log-format", but all I got was empty string:
ID = [""]
For the second one, I'm at a complete loss. I'm not able to find ANY documentation on these. I have been scouring the internet with no luck. E.g. the txn.sf:req_body() function which I know is working, I simply cannot find it documented anywhere, so I'm not sure what other functions are available to get some request details.
Any ideas for either or both of my approaches? I'm attaching my full HAProxy config here at the end, just in case:
global
log 127.0.0.1 len 10000 local2 debug
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
lua-load /opt/LUA/parseId.lua
stats socket /etc/haproxy/haproxysock level admin
defaults
log global
option httplog
option dontlognull
mode http
timeout connect 5000
timeout client 50000
timeout server 50000
# Request body is temporarily logged in test environment
log-format "Client IP:port = [%ci:%cp], Start Time = [%tr], Frontend Name = [%ft], Backend Name = [%b], Backend Server = [%s], Time to receive full request = [%TR ms], Response time = [%Tr ms], Status Code = [%ST], Bytes Read = [%B], Request = [%{+Q}r], ID = [%{+Q}[var(txn.req_id)]], Request Body = [%[capture.req.hdr(0)]]"
frontend gw_front
bind *:8776
option http-buffer-request
declare capture request len 40000
http-request capture req.body id 0
http-request track-sc0 src table gw_back
use_backend gw_back
backend gw_back
balance roundrobin
stick-table type string len 32 size 30k expire 30m
stick on "lua.parseId" table gw_back
# Use HTTP check mode with /ping interface instead of TCP-only check
option httpchk POST /Gateway/ping
server gw1 x.x.x.x:8080 check inter 10s
server gw2 y.y.y.y:8080 check inter 10s
listen stats
bind *:8774 ssl crt /etc/haproxy/haproxy.cer
mode http
maxconn 5
stats enable
stats refresh 10s
stats realm Haproxy\ Statistics
stats uri /stats
stats auth user:password
I have a Lua proxy that needs to route requests. Each request destination is established based on the response from another HTTP request with a header from the initial request. My understanding is that HAProxy is an event-driven software, so blocking system calls are absolutely forbidden and my code is blocking because is doing an HTTP request.
I read about yielding after the request but I think it won't help since the HTTP request is already started. The library for doing the request is https://github.com/JakobGreen/lua-requests#simple-requests
local requests = require('requests')
core.register_fetches('http_backend', function(txn)
local dest = txn.sf:req_fhdr('X-dest')
local url = "http://127.0.0.1:8080/service";
local response = requests.get(url.."/"+dest);
local json = response.json()
return json.field
end )
How do I convert my code to be non-blocking?
You should consider using HAProxy's SPOE which was created exactly for these blocking scenarios.
I managed to do it using Lua. The thing I was making wrong was using require('requests') this is blocking. Ideally for HA never use a Lua external library. I have to deal with plain sockets and do an HTTP request and very important to use HA core method core.tcp() instead of Lua sockets.
How Configurations for Http Connector - Http Transaction for multipart/form-data?
this is my screen shoot setting,
screen shoot
with this setting, the request was send but the file disappeared from request. anyone can help me?
multi-part form encoded is supported out of the box by Convertigo, except it is supported on the HTML connector and not the HTTP connector so to upload a file :
Create a HTML connector. The hostname, port and isHttps properties are not mandatory to be modified here
Create a HTML transaction, name it "DoUpload"
On this Transaction create a "TransactionStarted" handler (RightClick->new>Handler->Start of Transaction
In this handler create a HTTP-Upload statement (RightClick->new->statement->HTTP_upload_request)
Configure statement properties :
- FilePath to the file to upload (JS property)
- filename to the name of the uploaded file (JSproperty)
- host, port, isHttps to the correct setting for HTTP or https
- URI to the uri path (JS property)
Be sure to set the transaction's "Maintain Connector State" property to "true"
See : http://help.convertigo.com/7.2.1/topic/com.twinsoft.convertigo.studio.help/help/helpRefManual/HTTPuploadrequest.html
This should do the trick.
I want to build a small script in python which needs to fetch an url. The server is a kind of crappy though and replies pure ASCII without any headers.
When I try:
import urllib.request
response = urllib.request.urlopen(url)
print(response.read())
I obtain a http.client.BadStatusLine: 100 error because this isn't a properly formatted HTTP response.
Is there another way to fetch an url and get the raw content, without trying to parse the response?
Thanks
It's difficult to answer your direct question without a bit more information; not knowing exactly how the (web) server in question is broken.
That said, you might try using something a bit lower-level, a socket for example. Here's one way (python2.x style, and untested):
#!/usr/bin/env python
import socket
from urlparse import urlparse
def geturl(url, timeout=10, receive_buffer=4096):
parsed = urlparse(url)
try:
host, port = parsed.netloc.split(':')
except ValueError:
host, port = parsed.netloc, 80
sock = socket.create_connection((host, port), timeout)
sock.sendall('GET %s HTTP/1.0\n\n' % parsed.path)
response = [sock.recv(receive_buffer)]
while response[-1]:
response.append(sock.recv(receive_buffer))
return ''.join(response)
print geturl('http://www.example.com/') #<- the trailing / is needed if no
other path element is present
And here's a stab at a python3.2 conversion (you may not need to decode from bytes, if writing the response to a file for example):
#!/usr/bin/env python
import socket
from urllib.parse import urlparse
ENCODING = 'ascii'
def geturl(url, timeout=10, receive_buffer=4096):
parsed = urlparse(url)
try:
host, port = parsed.netloc.split(':')
except ValueError:
host, port = parsed.netloc, 80
sock = socket.create_connection((host, port), timeout)
method = 'GET %s HTTP/1.0\n\n' % parsed.path
sock.sendall(bytes(method, ENCODING))
response = [sock.recv(receive_buffer)]
while response[-1]:
response.append(sock.recv(receive_buffer))
return ''.join(r.decode(ENCODING) for r in response)
print(geturl('http://www.example.com/'))
HTH!
Edit: You may need to adjust what you put in the request, depending on the web server in question. Guanidene's excellent answer provides several resources to guide you on that path.
What you need to do in this case is send a raw HTTP request using sockets.
You would need to do a bit of low level network programming using the socket python module in this case. (Network sockets actually return you all the information sent by the server as it as, so you can accordingly interpret the response as you wish. For example, the HTTP protocol interprets the response in terms of standard HTTP headers - GET, POST, HEAD, etc. The high-level module urllib hides this header information from you and just returns you the data.)
You also need to have some basic information about HTTP headers. For your case, you just need to know about the GET HTTP request. See its definition here - http://djce.org.uk/dumprequest, see an example of it here - http://en.wikipedia.org/wiki/HTTP#Example_session. (If you wish to capture live traces of HTTP requests sent from your browser, you would need a packet sniffing software like wireshark.)
Once you know basics about socket module and HTTP headers, you can go through this example - http://coding.debuntu.org/python-socket-simple-tcp-client which tells you how to send a HTTP request over a socket to a server and read its reply back. You can also refer to this unclear question on SO.
(You can google python socket http to get more examples.)
(Tip: I am not a Java fan, but still, if you don't find enough convincing examples on this topic under python, try finding it under Java, and then accordingly translate it to python.)
urllib.urlretrieve('http://google.com/abc.jpg', 'abc.jpg')
I have written a web-service using Erlang and Mochiweb. The web service returns a lot of results and takes some time to finish the computation.
I'd like to return results as soon as the program finds it, instead of returning them when it found them all.
edit:
i found that i can use a chunked request to stream result, but seems that i can't find a way to close the connection. so any idea on how to close a mochiweb request?
To stream data of yet unknown size with HTTP 1.1 you can use HTPP chunked transfer encoding. In this encoding each chunk of data prepended by its size in hexadecimal. Last chunk is a zero-length chunk, with the chunk size coded as 0, but without any data.
If client doesn't support HTTP 1.1 server can send data as binary chunks and close connection at the end of the stream.
In MochiWeb it's all works as following:
HTTP response should be started with Response = Request:respond({Code, ResponseHeaders, chunked}) function. (By the way, look at the code comments);
Then chunks can be send to client with Response:write_chunk(Data) function. To indicate client the end of the stream chunk of zero length should be sent: Response:write_chunk(<<>>).
When handling of current request is over MochiWeb decides should connection be closed or can be reused by HTTP persistent connection.