Now, i come with a strange question. I send http request with httpclient. In the request, when i use domain name, like dynamic.12306.cn, or i use ip address and put the ip information in windows/system32/driver/ext/hosts like 122.227.2.27 dynamic.12306.cn, the request is quick to return. But if i only use the ip and don't put any info in to hosts, it is very slow.
The the two above, I will show example below:
Case 1. The speed is fast. The request url is https://dynamic.12306.cn/otsweb/main.jsp
or The request url is https://122.227.2.27/otsweb/main.jsp and put the 122.227.2.27 dynamic.12306.cn into hosts,
Case 2. The speed is slow. The request url is https://122.227.2.27/otsweb/main.jsp an don't put any info into hosts.
I open the debug mode of httpclient, and i find when i use the method of case 2, it is very slow to connect to server.
The logs:
2013/03/17 10:19:10:665 CST [DEBUG] BasicClientConnectionManager - Get connection for route {s}->https://122.227.2.27
2013/03/17 10:19:11:234 CST [DEBUG] DefaultClientConnectionOperator - Connecting to 122.227.2.27:443
2013/03/17 10:19:20:796 CST [DEBUG] RequestAddCookies - CookieSpec selected: best-match
it will cost several seconds to connect server.
But if i use the method of case 1.
The logs:
2013/03/17 10:30:13:876 CST [DEBUG] BasicClientConnectionManager - Get connection for route {s}->https://dynamic.12306.cn
2013/03/17 10:30:14:403 CST [DEBUG] DefaultClientConnectionOperator - Connecting to dynamic.12306.cn:443
2013/03/17 10:30:14:499 CST [DEBUG] RequestAddCookies - CookieSpec selected: best-match
it is fast to connect server.
try listen to DNS query when you r doing the request. I got the same issue, which turned out the hosting website appended a hostname right after the ip.
Related
I have been scouring the internet with no luck. I have a basic LUA script for HAProxy, which looks like this:
core.Info("LUA script for parsing request ID element - loaded");
function parseId(txn, salt)
local payload = txn.sf:req_body()
-- parses hex value from element named "ID". Example payload: {"Platform":"xyz.hu","RecipientId":"xyz.hu","Channel":"xyz","CallbackURL":"http://x.x.x.x:123","ID":"5f99453d000000000a0c5164233e0002"}
local value = string.sub(string.match(payload, "\"ID\":\"[0-9a-f]+\""), 8, -2)
core.Info("ID : " .. value)
return value
end
-- register HAProxy "fetch"
core.register_fetches("parseId", parseId)
What it does is what it says: takes a 32 characater long ID from an incoming request. In the HAProxy config file, the result is used for sticky-session handling:
stick-table type string len 32 size 30k expire 30m
stick on "lua.parseId" table gw_back
This produces two lines of log for each request:
ID: xyz which is logged from the LUA script
The detailed request data which is logged from the HAProxy config file using "log-format", e.g.:
Jan 20 22:13:52 localhost haproxy[12991]: Client IP:port = [x.x.x.x:123], Start Time = [20/Jan/2022:22:13:52.069], Frontend Name = [gw_front], Backend Name = [gw_back], Backend Server = [gw1], Time to receive full request = [0 ms], Response time = [449 ms], Status Code = [200], Bytes Read = [308], Request = ["POST /Gateway/init HTTP/1.1"], ID = [""], Request Body = [{"Platform":"xyz.hu","RecipientId":"xyz.hu","Channel":"xyz","CallbackURL":"http://x.x.x.x:123","ID":"61e9d03e000000000a0c5164233e0002"}]
I wanted to extend logging due to some strange issues happening sometimes, so I wanted to one (or both) of below approaches:
Pass the "ID" value back from the LUA script into the HAProxy config as a variable, and log it along with the request details. I can log the full request body, but don't want to due to GDPR and whatnot.
Get some request details in the LUA script itself, and log it along with the ID.
So, basically, to be able to connect the ID with the request details. If multiple requests are coming to same URL very quickly, it is difficult to find which of them belongs to a specific ID. However I couldn't accomplish these.
For the first one, I added this line into the LUA before returning the "value" variable:
txn:set_var("req_id", value)
I was hoping this would create a variable in HAProxy called "req_id", and I can log it with "log-format", but all I got was empty string:
ID = [""]
For the second one, I'm at a complete loss. I'm not able to find ANY documentation on these. I have been scouring the internet with no luck. E.g. the txn.sf:req_body() function which I know is working, I simply cannot find it documented anywhere, so I'm not sure what other functions are available to get some request details.
Any ideas for either or both of my approaches? I'm attaching my full HAProxy config here at the end, just in case:
global
log 127.0.0.1 len 10000 local2 debug
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
lua-load /opt/LUA/parseId.lua
stats socket /etc/haproxy/haproxysock level admin
defaults
log global
option httplog
option dontlognull
mode http
timeout connect 5000
timeout client 50000
timeout server 50000
# Request body is temporarily logged in test environment
log-format "Client IP:port = [%ci:%cp], Start Time = [%tr], Frontend Name = [%ft], Backend Name = [%b], Backend Server = [%s], Time to receive full request = [%TR ms], Response time = [%Tr ms], Status Code = [%ST], Bytes Read = [%B], Request = [%{+Q}r], ID = [%{+Q}[var(txn.req_id)]], Request Body = [%[capture.req.hdr(0)]]"
frontend gw_front
bind *:8776
option http-buffer-request
declare capture request len 40000
http-request capture req.body id 0
http-request track-sc0 src table gw_back
use_backend gw_back
backend gw_back
balance roundrobin
stick-table type string len 32 size 30k expire 30m
stick on "lua.parseId" table gw_back
# Use HTTP check mode with /ping interface instead of TCP-only check
option httpchk POST /Gateway/ping
server gw1 x.x.x.x:8080 check inter 10s
server gw2 y.y.y.y:8080 check inter 10s
listen stats
bind *:8774 ssl crt /etc/haproxy/haproxy.cer
mode http
maxconn 5
stats enable
stats refresh 10s
stats realm Haproxy\ Statistics
stats uri /stats
stats auth user:password
I am currently developing an application where we need some request to hit our server ASAP. To speed up the request process we have to eliminate handshake (as it takes extra) and have a permanent connection.
The application is using the Alamofire framework to make all request to our server and the setup is the following:
We have a session manager set up with default configuration and http header.
lazy var sessionManager: Alamofire.SessionManager = {
let configuration = URLSessionConfiguration.default
configuration.httpAdditionalHeaders = Alamofire.SessionManager.defaultHTTPHeaders
let manager = Alamofire.SessionManager(configuration: configuration)
return manager
}()
The session manager is persistent across all requests. Each request is made using the following code:
self.sessionManager.request(request.urlString, method: request.method, parameters: request.parameters)
.responseJSON { [weak self] response in
// Handle the response
}
request.urlString is the url of our server "http://ourserver.com/example"
request.method is set to post
request.parameters is a dictionary of paramaters
The request is working fine and we get a valid response. The problem arises on the keep alive timer, which is set by our server to 300 seconds. The device holds the connection for a maximum of 30 seconds on wifi and closes it almost instantly over GSM.
Server Debug
We did some debugging on our server and found the following results
Tests:
Test 1:
iPhone connects to the Internet via WiFi
Test 2:
iPhone connects to the Internet via 3G
Behaviour:
Both cases: app makes an HTTP/1.1 request to a web server with “Connection: keep-alive”; The Server (server ip = 10.217.81.131) responds with “Keep-Alive: timeout=300, max=99”
The client side (test 1 - app over WiFi) sends TCP FIN on the 30th second and the connection closes
The client side (test 2 – app over 3G) sends immediately (zero seconds) a TCP FIN request after it receives the HTTP/1.1 OK message from its first HTTP POST
Test 1 logs on the server side:
At 23.101902 the app makes an HTTP/1.1 POST request to the server with “Connection: keep-alive”
At 23.139422 the server responds HTTP/1.1 200 OK with “Connection: Keep-Alive” and “timeout=300” (300 seconds)
The Round-Trip-Time (RTT) is reported as 333.82 msec (this highlights the margin of error we have on the following timestamps):
The app, however, closes the connection in 30 seconds (approx. given the Internet transport variations – the difference between the 54.200863 and the 23.451979 timestamps):
The test is repeated numerous times with an approx. time of 30 seconds being always monitored
Test 2 logs on the server side:
The HTTP/1.1 POST request from the app:
The HTTP OK server response with keep-alive being accepted and set at 300 seconds:
The RTT is at 859.849 msec
The app closes immediately the connection, where immediately is 21.197918 – 18.747780 = 2.450138 seconds
The tests are repeated while switching from WiFi to 3G and back with the same results being recorded.
Client Debug
Using WiFi
First Attempt (connection established)
Optional(
[AnyHashable("Content-Type"): text/html,
AnyHashable("Content-Encoding"): gzip,
AnyHashable("Content-Length"): 36,
AnyHashable("Set-Cookie"): user_cookieuser_session=HXQuslXgivCRKd%2BJ6bkg5D%2B0pWhCAWkUPedUEGyZQ8%2Fl65UeFcsgebkF4tqZQYzVgp2gWgAQ3DwJA5dbXUCz4%2FnxIhUTVlTShIsUMeeK6Ej8YMlB11DAewHmkp%2Bd3Nr7hJFFQlld%2BD8Q2M46OMRGJ7joOzmvH3tXgQtRqR9gS2K1IpsdGupJ3DZ1AWBP5HwS41yqZraYsBtRrFnpGgK0CH9JrnsHhRmYpD40NmlZQ6DWtDt%2B8p6eg9jF0xE6k0Es4Q%2FNiAx9S9PkhII7CKPuBYfFi1Ijd7ILaCH5TXV3vipz0TmlADktC1OARPTYSwygN2r6bEsX15Un5WUhc2caCeuXnmd6xy8sbjVUDn72KELWzdmDTl6p5fRapHzFEfGEEg2LOEuwybmf2Nt6DHB6o6EA5vfJovh2obpp4HkIeAQ%3D; expires=Sun, 08-Jan-2017 12:51:43 GMT; path=/,
AnyHashable("Keep-Alive"): timeout=300, max=100,
AnyHashable("Connection"): Keep-Alive,
AnyHashable("X-Powered-By"): PHP/5.3.10-1ubuntu3.11,
AnyHashable("Server"): Apache/2.2.22 (Ubuntu),
AnyHashable("Vary"): Accept-Encoding,
AnyHashable("Date"): Sun, 08 Jan 2017 10:51:43 GMT])
Second Attempt (within 30 sec, the connection is still alive)
Optional([AnyHashable("Content-Type"): text/html,
AnyHashable("Content-Encoding"): gzip,
AnyHashable("Content-Length"): 36,
AnyHashable("Keep-Alive"): timeout=300, max=99,
AnyHashable("Connection"): Keep-Alive,
AnyHashable("X-Powered-By"): PHP/5.3.10-1ubuntu3.11,
AnyHashable("Server"): Apache/2.2.22 (Ubuntu),
AnyHashable("Vary"): Accept-Encoding,
AnyHashable("Date"): Sun, 08 Jan 2017 11:00:18 GMT])
Then after 30 seconds the connection drops (FI)
Using 3G
First Attempt
Optional([AnyHashable("Content-Type"): text/html,
AnyHashable("Content-Encoding"): gzip,
AnyHashable("Content-Length"): 36,
AnyHashable("Connection"): keep-alive,
AnyHashable("X-Powered-By"): PHP/5.3.10-1ubuntu3.11,
AnyHashable("Server"): Apache/2.2.22 (Ubuntu),
AnyHashable("Vary"): Accept-Encoding,
AnyHashable("Date"): Sun, 08 Jan 2017 11:04:31 GMT])
Then the connection drops almost instantly.
Now that I looked at the code a second time, I think I see the problem. The underlying NSURLSession class defaults to ignoring the keep-alive header, because some servers "support" it, but in practice, break badly if you actually try to use it, IIRC.
If you want a session to support keep-alive, you have to explicitly set HTTPShouldUsePipelining in the session configuration to YES.
Note that there is still no guarantee that the connection will stay up, depending on how aggressively iOS decides to power manage the radio, but at least you'll have a prayer. :-)
I am currently trying to write an auxiliary module for Metasploit. The module basically tries multiple default credentials to get access to the router's management page. The authentication is done via web, i.e. HTTP POST.
Currently, the module works as expected for plain HTTP connections, i.e. unsecured connections, however every connection attempt via HTTPS (port 443), returns nil. Below is the function used within the Metasploit class to retrieve the login page:
def get_login_page(ip)
begin
response = send_request_cgi(
'uri' => '/',
'method' => 'GET'
)
# Some models of ZyXEL ZyWALL return a 200 OK response
# and use javascript to redirect to the rpAuth.html page.
if response && response.body =~ /changeURL\('rpAuth.html'\)/
vprint_status "#{ip}- Redirecting to rpAuth.html page..."
response = send_request_cgi(
'uri' => '/rpAuth.html',
'method' => 'GET'
)
end
rescue ::Rex::ConnectionError
vprint_error "#{ip} - Failed to connect to Web management console."
end
return response
end
When trying to connect via HTTPS, the first send_request_cgi call returns nil. No exception are caught or thrown. I have tried with 3 different hosts to make sure the issue was not with a specific endpoint. All my 3 attempts failed to return a response. At every attempt, I set the RPORT option to 443;
RHOSTS 0.0.0.0 yes The target address range or CIDR identifier
RPORT 443 yes The target port
Note that I have replaced the real IP with 0.0.0.0. Using a web browser, I can actually connect to the router via HTTPS with no issue (other than having to add an exception since the certificate is untrusted) and am presented the login page. With Wireshark, I tried to look at the generated traffic. I can clearly see that nothing is sent by the router. I notice the 3-way handshake being completed and the HTTP GET request being made:
GET / HTTP/1.1
Host: 0.0.0.0
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Content-Type: application/x-www-form-urlencoded
Content-Length: 0
There are 3-4 ACK after and then a FIN/PUSH sent by the server.
Based on this page on Metasploit's GitHub, I was under the impression that connections to HTTPS websites were handled by the underlying framework. I have not seen any articles/tutorial/source that leads me to believe otherwise. The doc about the send_request_cgi does not specify any specific requirement to establish a HTTPS connection. Other posts did not had the exact same issue I'm having. At this point I suspect either the OS, the framework or me forgetting to enable something. Other modules I have looked at either only targets HTTP websites - which I doubt - or do not have any special handling for HTTPS connections.
Any help determining the cause would be greatly appreciated.
Version of Metasploit:
Framework: 4.9.3-2014060501
Console : 4.9.3-2014060501.15168
Version of OS:
SMP Debian 3.14.5-1kali1 (2014-06-07)
As per this post on SecurityStreet, the solution was to set SSL to true in the DefaultOptions in the initialize function:
def initialize
super(
...
'DefaultOptions' =>
{
...
'SSL' => true
}
)
...
end
Connections to routers using HTTPS worked afterwards.
i run in some problems with my jersey rest api and a client.
This is how im using the methods on a server side:
#POST
#Path("/seed")
#Produces(MediaType.APPLICATION_JSON)
#Consumes(MediaType.APPLICATION_JSON)
public Response addSeed(Seed seed) throws InterruptedException {
if (!Validator.isValidSeed(seed)) {
return Response.status(400).entity("{\"message\":\"Please verify your JSON!\", \"stat\":\"failed\"}")
.build();
}
save(seed);
return Response.status(200).build();
}
If i run a Jersey client in a while(true) loop, there are connections open and won't close. So im running into a problem i have a lot of connections open and my network crashes. So i can't use my server any more. After the connections are closed i can connect to the server.
This is a client:
ClientConfig config = new DefaultClientConfig();
Client client = Client.create(config);
WebResource service = client.resource(getBaseURI()).path("api/seed");
while (true) {
ClientResponse cr = service.header("Content-Type", "application/json").post(ClientResponse.class, seed);
System.out.println(cr);
cr.close();
My Questions are:
What can i do on the server side, to prevent clients open a new connection?
How can i specify a max number of connections?
And how should i implement the jersey client to reuse open connection?
I don't know of a way to limit Jersey resources at the web-app level. If you upgrade to GlassFish EE, you can make your resources EJBs #Stateless #StatelessDeployment(maxInstances=16)
The pile up of connections could be because of Keep-Alive settings. In Tomcat 6 there are two you can tune your connector with:
maxKeepAliveRequests, which defaults to 100. It's the maximum number of HTTP requests which can be pipelined until the connection is closed by the server. Setting this attribute to 1 will disable HTTP/1.0 keep-alive, as well as HTTP/1.1 keep-alive and pipelining. Setting this to -1 will allow an unlimited amount of pipelined or keep-alive HTTP requests.
keepAliveTimeout, which defaults to connectionTimeout which defaults to 60k ms. It it the number of milliseconds this Connector will wait for another HTTP request before closing the connection.
i want to check my server connection to know if its available or not to inform the user..
so how to send a pkg or msg to the server (it's not SQL server; it's a server contains some serviecs) ...
thnx in adcvance ..
With all the possibilities for firewalls blocking ICMP packets or specific ports, the only way to guarantee that a service is running is to do something that uses that service.
For instance, if it were a JDBC server, you could execute a non-destructive SQL query, such as select * from sysibm.sysdummy1 for DB2. If it's a HTTP server, you could create a GET packet for index.htm.
If you actually have control over the service, it's a simple matter to create a special sub-service to handle these requests (such as you send through a CHECK packet and get back an OKAY response).
That way, you avoid all the possible firewall issues and the test is a true end-to-end one. PINGs and traceroutes will be able to tell if you can get to the machine (firewalls permitting) but they won't tell you if your service is functioning.
Take this from someone who's had to battle the network gods in a corporate environment where machines are locked up as tight as the proverbial fishes ...
If you can open a port but don't want to use ping (i dont know why but hey) you could use something like this:
import socket
host = ''
port = 55555
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host, port))
s.listen(1)
while 1:
try:
clientsock, clientaddr = s.accept()
clientsock.sendall('alive')
clientsock.close()
except:
pass
which is nothing more then a simple python socket server listening on 55555 and returning alive