Siege - Why do I get all these responses? - load-testing

I'm confused about what Siege is doing under the hood. I use Siege every now and then to create some traffic for my web services. Today I noticed that Siege lists more responses than I would have I expected.
E.g. I use Siege as
siege -c 1 -t 5s http://www.github.com/index.html
I'd expect to see only one response with the 'index.html' HTML resource. But instead I get
** SIEGE 4.0.2
** Preparing 1 concurrent users for battle.
The server is now under siege...
HTTP/1.1 301 0.22 secs: 0 bytes ==> GET /index.html
HTTP/1.1 301 0.74 secs: 0 bytes ==> GET /index.html
HTTP/1.1 200 0.72 secs: 84938 bytes ==> GET /index.html
HTTP/1.1 200 0.59 secs: 25628 bytes ==> GET /
HTTP/1.1 200 0.14 secs: 97194 bytes ==> GET /images/modules/site/org_example_nasa.png?sn
HTTP/1.1 200 0.06 secs: 8182 bytes ==> GET /images/modules/site/home-ill-platform.png?sn
HTTP/1.1 200 0.06 secs: 10324 bytes ==> GET /images/modules/site/home-ill-projects.png?sn
HTTP/1.1 200 0.07 secs: 11500 bytes ==> GET /images/modules/site/home-ill-work.png?sn
HTTP/1.1 200 0.06 secs: 6137 bytes ==> GET /images/modules/site/home-ill-build.png?sn
HTTP/1.1 200 0.12 secs: 152050 bytes ==> GET /assets/github-1997dc5b96e7febe96e26200e8d35ecd91516cd11316d8bbff113b04ea81b23b.js
HTTP/1.1 200 0.09 secs: 88940 bytes ==> GET /assets/frameworks-44ae517d6facdc7480be70913e6d4abb86971b42006cf0f0dd6597b480cc272f.js
HTTP/1.1 200 0.06 secs: 6224 bytes ==> GET /assets/compat-8e19569aacd39e737a14c8515582825f3c90d1794c0e5539f9b525b8eb8b5a8e.js
HTTP/1.1 200 0.60 secs: 2062 bytes ==> GET /u/3777891?v=3&s=80
HTTP/1.1 200 0.15 secs: 6504 bytes ==> GET /u/1739496?v=3&s=200
HTTP/1.1 200 0.06 secs: 12695 bytes ==> GET /assets/site-a1797477c6773085524735a03e97c502b07acafa1b95efd6cda07775c7bb7105.css
HTTP/1.1 200 0.10 secs: 98986 bytes ==> GET /assets/github-ac9c637b29122a4699fcd4d205b2d09efa4d4962d369158f7d907123061143f1.css
HTTP/1.1 200 0.08 secs: 21615 bytes ==> GET /assets/frameworks-a44e0bdd1666101af23963e4027cd7a0a1eea1339e0e7422524f2e7f3900e86b.css
The first 2 are redirects, then comes the expected HTML. But why do I get all the other images, JS and CSS resources? I can't imagine that Siege actually parses the HTML and secondarily requests resources from there. I'm lost.

HTML parsing is a feature which has been added in Siege 4.0.0.
You can edit Siege configuration file and disable response parsing. Here is an extract from the configuration file:
# Parser
# This directive allows you to turn on the html parser. With this
# feature enabled, siege will harvest resources like style sheets,
# images, javascript, etc. and make additional requests for those
# items.
#
# HTML parsing was added to version 4.0.0 It is enabled by default.
# When the parser is enabled, care must be given to other features.
# For example, we allow to set accept-encoding to anything you'd like
# but if you want to parse those pages, then you MUST set the encoding
# to a supported one.
#
# With the default options set, you should be able to enable the parser
# with success.
#
# Use this feature to enable it. (true = on, false = off)
#
# Example: parser = true
#
parser = true
Simply set parser = false and it disables the parser.

Related

Kong: kong-spec-expose plugin cannot load the documentation(302 permanently moved)

I have hard times configuring this kong-spec-expose plugin.It is supposed to automatically configure the routes with Swagger. After some time I managed to configure it but when i try to access the documentation of a certain route it is always giving me the 302 permanently moved.So i tested it with curl and here I will leave a link for the kong plugin and a screenshot of the request..
https://docs.konghq.com/hub/optum/kong-spec-expose/
* Trying 127.0.0.1:8000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /api/employee-controller/profile/user/12/base/specz HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Moved Temporarily
< Date: Wed, 11 Jan 2023 09:50:38 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Kong-Response-Latency: 409
< Server: kong/2.8.1
Actually I tried configuring the plugin on multiple routes then on service level but nothing seems to work out.. then tried to analyse where this request is redirected but with no results.

Wireshark: Dump client-server dialogue

If you use "Follow TCP stream" in wireshark you get a very nice display for the client server dialogue.
One color is the client, the other color is the server.
Is there a way to dump this to a ascii without loosing who said what?
For example:
server> 220 "Welcome to FTP service for foo-server."
client> USER baruser
server> 331 Please specify the password.
client> supersecret
I want to avoid screenshots. Adding "server>" and "client>" to the lines is error prone.
It may not be possible with the GUI version, but it's achievable with the console version tshark:
tshark -r capture.pcap -qz follow,tcp,ascii,<stream_id> > stream.txt
Replace <stream_id> with an actual stream ID (eg: 1):
tshark -r capture.pcap -qz follow,tcp,ascii,1 > stream.txt
This will output an ASCII file. How is it better than saving it directly from the GUI version? Well:
The data sent by the second node is prefixed with a tab to differentiate it from the data sent by the first node.
Since the output in ascii mode may contain newlines, the length of each section of output plus a newline precedes each section of output.
This makes the file easily parsable. Example output:
===================================================================
Follow: tcp,ascii
Filter: tcp.stream eq 1
Node 0: xxx.xxx.xxx.xxx:51343
Node 1: yyy.yyy.yyy.yyy:80
786
GET ...
Host: ...
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: */*
User-Agent: ...
Referer: ...
Accept-Encoding: ...
Accept-Language: ...
Cookie: ...
235
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: ...
Expires: -1
X-Request-Guid: ...
Date: Mon, 31 Aug 2015 10:55:46 GMT
Content-Length: 0
===================================================================
786\n is the length of the first output section from Node 0.
\t235\n is the legnth of the response section from Node 1 and so on.

Nginx + passenger serving 4 requests/s for a static page

I deployed my app to Digitalocean with Passenger and Nginx. I used apache bench to see how many requests per second I can get on a static page (simple hello world rails view), but I am only getting 4 requests/s.
ab -n 100 http://107.170.100.242/fo
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 107.170.100.242 (be patient).....done
Server Software: nginx/1.8.0
Server Hostname: 107.170.100.242
Server Port: 80
Document Path: /fo
Document Length: 5506 bytes
Concurrency Level: 1
Time taken for tests: 22.662 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 632600 bytes
HTML transferred: 550600 bytes
Requests per second: 4.41 [#/sec] (mean)
Time per request: 226.617 [ms] (mean)
Time per request: 226.617 [ms] (mean, across all concurrent requests)
Transfer rate: 27.26 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 181 226 65.4 204 445
Waiting: 181 226 65.4 204 445
Total: 181 227 65.4 204 446
It should be literally thousands per second as I am using Nginx. I have been researching this for the entire day without results, can someone please direct me to the right path to solve this?
This would be the nginx config directive that will cause it to bypass the app server and serve the static files directly:
root /var/www/my_app/public;
Are you sure that is right?

POST request treated as GET in Heroku environment

I have weird case. I have a RoR app, which provides REST API which I'm connecting to from Java application.
I'm developing RoR locally, and deploying it on Heroku environment.
Regardless how (I tried from Java APP, Mozilla REST client, etc.) I try to send POST HTTP request that should be handled by create action in api controller. On localhost - everything is working as expected. On Heroku production env - the POST request is treated as normal GET.
Here are my routes for this resource:
api_v1_items GET /api/v1/items(.:format) api/v1/items#index {:format=>:json}
POST /api/v1/items(.:format) api/v1/items#create {:format=>:json}
api_v1_item GET /api/v1/items/:id(.:format) api/v1/items#show {:format=>:json}
PATCH /api/v1/items/:id(.:format) api/v1/items#update {:format=>:json}
PUT /api/v1/items/:id(.:format) api/v1/items#update {:format=>:json}
DELETE /api/v1/items/:id(.:format) api/v1/items#destroy {:format=>:json}
So I'm trying to do POST request to /api/v1/items passing all necessary parameters.
In localhost the response is correct:
Started POST "/api/v1/items?token=l4XOHrhDApPqTp1u4TxBjQ" for 127.0.0.1 at 2014-05-15 22:11:49 +0200
Processing by Api::V1::ItemsController#create as JSON
Parameters: {"height"=>10.0, "item_name"=>"Super item", "width"=>20.0, etc...
However the same request fired at Heroku its treated as GET:
2014-05-15T20:27:58.137541+00:00 app[web.1]: Started GET "/api/v1/items?token=iEdDkDLiDUlWi0mDbr6XYw" for 89.74.57.51 at 2014-05-15 20:27:58 +0000
2014-05-15T20:27:58.223620+00:00 app[web.1]: Processing by Api::V1::ItemsController#index as JSON
Any idea? Of course both repos are in sync. Checked few times.
This is really weird... maybe some kind of Heroku cache magic?
HTTP/1.1 301 Moved Permanently
301 redirects are not Heroku magic. Your DNS (or possibly your app) is likely forwarding all apex requests (mydomain.com) to the www subdomain.
Using subdomains is preferred:
Heroku Dev Center: Custom Domains
I experienced a similar error when not using a custom domain just because of an easily overlooked error: I was using heroku.com instead of herokuapp.com
wrong:
http://my-app.heroku.com
right:
http://my-app.herokuapp.com
I suspect it's very similar in cause to the issue mentioned in Catsby's answer.
Well, I tried CURL, and it appeared error is silly.
I was posting at http://mydomain.com, where it's routed as GET.
When I fire at http://www.mydomain.com - it works.
Heroku magic.
Below are curl's and results for your reference. Maybe somebody will be able to explain why it works like this...
POST at mydomain.com
curl -v -H "Accept: application/json" -H "Cont"width":20.0,"item_desc":"The super item","std_pack":40,"sku":"A1004","depth":20.0}}' http://mydomain.com/api/v1/items?token=dSWeyKjjtZu0ZSs6b2J-yw
* Adding handle: conn: 0x7fe70b803000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fe70b803000) send_pipe: 1, recv_pipe: 0
* About to connect() to mydomain.com port 80 (#0)
* Trying 78.46.51.229...
* Connected to mydomain.com (78.46.51.229) port 80 (#0)
> POST /api/v1/items?token=dSWeyKjjtZu0ZSs6b2J-yw HTTP/1.1
> User-Agent: curl/7.30.0
> Host: mydomain.com
> Accept: application/json
> Content-type: application/json
> Content-Length: 174
>
* upload completely sent off: 174 out of 174 bytes
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 15 May 2014 21:20:58 GMT
* Server Apache is not blacklisted
< Server: Apache
< Location: http://www.mydomain.com/api/v1/items?token=dSWeyKjjtZu0ZSs6b2J-yw
< Content-Length: 273
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved here.</p>
</body></html>
* Connection #0 to host mydomain.com left intact
POST at www.mydomain.com
Maciejs-MacBook-Pro:merchbag maciejsimm$ curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"item":{"height":10.0,"item_name":"Super duper item","width":20.0,"item_desc":"The super","std_pack":40,"sku":"A1005","depth":20.0}}' http://www.mydomain.com/api/v1/items?token=dSWeyKjjtZu0ZSs6b2J-yw
* Adding handle: conn: 0x7fc191003000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fc191003000) send_pipe: 1, recv_pipe: 0
* About to connect() to www.mydomain.com port 80 (#0)
* Trying 50.17.185.176...
* Connected to www.mydomain.com (50.17.185.176) port 80 (#0)
> POST /api/v1/items?token=dSWeyKjjtZu0ZSs6b2J-yw HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.mydomain.com
> Accept: application/json
> Content-type: application/json
> Content-Length: 133
>
* upload completely sent off: 133 out of 133 bytes
< HTTP/1.1 201 Created
< Cache-Control: max-age=0, private, must-revalidate
< Content-Type: application/json; charset=utf-8
< Date: Thu, 15 May 2014 21:24:17 GMT
< Etag: "41231ae0f50a604cd7316a014d19b3f2"
* Server WEBrick/1.3.1 (Ruby/2.0.0/2014-05-08) is not blacklisted
< Server: WEBrick/1.3.1 (Ruby/2.0.0/2014-05-08)
< Set-Cookie: request_method=POST; path=/
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Request-Id: ba05dd74-bf52-47d5-b8a9-d0516aff5804
< X-Runtime: 0.020289
< X-Ua-Compatible: chrome=1
< X-Xss-Protection: 1; mode=block
< Content-Length: 234
< Connection: keep-alive
<
* Connection #0 to host www.mydomain.com left intact
{"id":15,"partner_id":1,"sku":"A1005","item_name":"Super duper item","item_desc":"The super","std_pack":40,"height":10,"width":20,"depth":20,"image":null,"created_at":"2014-05-15T21:24:17.753Z","updated_at":"2014-05-15T21:24:17.761Z"}
I had this same issue when sending a POST request to heroku using HTTP instead of HTTPS. Every time heroku routed my POST requests as GET requests. Once I updated the url to use HTTPS, my POST requests were routed by heroku as POSTs and not GETs, resolving the issue. The redirection issues mentioned in the previous posts are likely the root cause of the issue I experienced as well.

Why do I get 400 Bad Request header on a thin ruby website hosted on heroku

I look after www.lesscss.org. The source is here https://github.com/cloudhead/lesscss.org.
This is a thin web application and runs on heroku. Accessing the website in a browser is fine.
We have had a bug that curl -I lesscss.org gives a 400 request - https://github.com/cloudhead/less.js/issues/1232
and it does
$ curl -I lesscss.org
HTTP/1.1 400 Bad Request
Server: nginx
Date: Tue, 19 Mar 2013 01:15:50 GMT
Connection: close
I've done alot of searching and haven't found a reason why or how to rectify the above.. shouldn't it be a 302 redirect or a 200 ok?
[Edit]
with verbose option
$ curl -Iv http://www.lesscss.org
* About to connect() to www.lesscss.org port 80 (#0)
* Trying 107.21.106.77...
* 0x8001f140 is at send pipe head!
* STATE: CONNECT => WAITCONNECT handle 0x80057408; line 1032 (connection #0)
* Connected to www.lesscss.org (107.21.106.77) port 80 (#0)
* STATE: WAITCONNECT => DO handle 0x80057408; line 1151 (connection #0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.lesscss.org
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x80057408; line 1236 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x80057408; line 1352 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x80057408; line 1363 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Server: nginx
Server: nginx
< Date: Wed, 20 Mar 2013 09:34:37 GMT
Date: Wed, 20 Mar 2013 09:34:37 GMT
< Connection: keep-alive
Connection: keep-alive
<
* STATE: PERFORM => DONE handle 0x80057408; line 1533 (connection #0)
* Connection #0 to host www.lesscss.org left intact
* Expire cleared
The toto engine that you're using has this line in it:
return [400, {}, []] unless #request.get?
So anything that is not a get will result in a 400. Toto could probably afford to allow head requests through (and combine with Rack::Head to drop request bodies on the floor if needed)
With -I you're making HEAD requests and not GET. Remove the -I flag and it will work. You can see how it works with the -v flag:
curl -Iv lesscss.org

Resources