I am sending logs to elasticsearch via rsyslog. For the parsing of those logs, I am using liblognorm rule.
I want to create fields of nginx logs,
here is a log entry,
127.0.0.1 - kibanaadmin [13/Jun/2017:14:18:17 +0530] "GET /ui/favicons/favicon-32x32.png HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0"
Here is the pattern file,
version=2
rule=:%clientip:ipv4% - %user:word% [%timestamp:char-to:]%] %auth:word% "%verb:alpha% %request:word%" %response:number% %bytes:number% "%referrer:word"%" "%agent:char-to:{"extradata":"("}"
The reason for parsefailure is I believe due to the date-time format.
Can somebody help in creating a rule for parsing nginx logs ?
For the aforementioned Nginx access log line you need a rule like this:
version=2
rule=:%clientip:ipv4%%-:whitespace%-%-:whitespace%%user:word%%-:whitespace%[%timestamp:char-to{"extradata":"]"}%]%-:whitespace%"%verb:word%%-:whitespace%%request:word%%-:whitespace%%protocol:char-to{"extradata":"\""}%"%-:whitespace%%response_code:number{"format":"number"}%%-:whitespace%%bytes:number{"format":"number"}%%-:whitespace%"%referrer:char-to{"extradata":"\""}%"%-:whitespace%"%agent:char-to{"extradata":"\""}%"
which produces the following output (pretty printed):
{
"agent": "Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko\/20100101 Firefox\/53.0",
"referrer": "-",
"bytes": 0,
"response_code": 304,
"protocol": "HTTP\/1.1",
"request": "\/ui\/favicons\/favicon-32x32.png",
"verb": "GET",
"timestamp": "13\/Jun\/2017:14:18:17 +0530",
"user": "kibanaadmin",
"clientip": "127.0.0.1"
}
To test your log file along with a Liblognorm rule file more easily you can create a nginx.rule file for your rules and run the command:
lognormalizer -H -p -r nginx.rule < nginx.log
You can find more information about lognormalizer command here.
Related
I was installed this library, I am working with sails.js
npm install sails-hook-swagger-generator --save
from their main website https://github.com/theoomoregbee/sails-hook-swagger-generator
I have created this file ./swagger/swagger.json
I want to see the swagger from the browser, how can I see the endpoints?
I have in routes something like that
'GET /api/v1/information': { action: 'actions/getinformation' },
I can not see nothing from the browser, what is the problem?
EDIT 1
from their answer https://github.com/theoomoregbee/sails-hook-swagger-generator/issues/27
you can simply change the directory you want to place the generated
swagger json may be assets, then clone
https://github.com/swagger-api/swagger-ui following the instructions
here
Just simply extract the dist folder (which is needed for just the
swagger ui) and update this line
https://github.com/swagger-api/swagger-ui/blob/1a95b9e9c972b95227a2976553fea988e58d7ff2/dist/index.html#L44
to be the path to the generated swagger json within your assets folder
i dont know what should i put in that line, i am putting the path of my file swagger.json
/disk/team/work/proyect/swagger
but when i try this url in the browser
http://127.0.0.1:8080/
i am getting this [2020-09-01T19:39:35.791Z] "GET /" Error (404): "Not found"
[2020-09-01T19:39:35.961Z] "GET /" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36"
[2020-09-01T19:39:35.963Z] "GET /" Error (404): "Not found"
uploading a file via a form/POST request in my Flask app works fine when the app is executed directly but fails when Shinyproxy hosts the Flask app. I tracked the issue down to the point that Shinyproxy is not executing the form POST request properly:
app.py:
from flask import Flask, request, render_template
app = Flask(__name__, static_url_path="/static")
#app.route('/', methods=['GET', 'POST'])
def index():
if request.method == 'GET':
return render_template('index.html')
else:
return "POST request received"
app.run(host='0.0.0.0', port=3838)
index.html:
<html>
<body>
<form method=post enctype=multipart/form-data>
<input type="file" name="file"/>
<input type = "submit" value="Upload">
</form>
</body>
</html>
returns, after clicking on "Upload", what one would expect, the GET and the POST request:
10.81.71.42 - - [23/Dec/2019 14:37:28] "GET / HTTP/1.1" 200 -
10.81.71.42 - - [23/Dec/2019 14:37:30] "POST / HTTP/1.1" 200 -
Running the exact same app in Shinyproxy via the
Dockerfile
FROM python:3
RUN pip install flask werkzeug
RUN mkdir /templates
COPY ["index.html", "/templates"]
COPY app.py /
EXPOSE 3838
CMD ["python", "app.py"]
returns the same but without the POST line:
172.17.0.1 - - [23/Dec/2019 14:39:42] "GET / HTTP/1.1" 200 -
The shinyproxy.log says
2019-12-23 14:42:33.682 DEBUG 17832 --- [XNIO-2 I/O-1] io.undertow.server.handlers.proxy : Sent request ClientRequest{path='/', method=POST, protocol=HTTP/1.1} to target 10.81.71.42 for exchange HttpServerExchange{ POST /proxy_endp
oint/b933863e-9fad-4d00-a657-034ede313e34/ request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9], Accept-Language=[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.
7], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], Origin=[http://192.168.76.81:8080], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36], Connectio
n=[keep-alive], Content-Length=[188], Content-Type=[multipart/form-data; boundary=----WebKitFormBoundarymPv6WtCTpGZQqRbF], Cookie=[JSESSIONID=CZxjGBM7BW597wysbBABgNWZL2x7qvsfujWVERgR], Referer=[http://192.168.76.81:8080/app_direct/flask_t
est/], Upgrade-Insecure-Requests=[1], Host=[192.168.76.81:8080]} response {Expires=[0], Cache-Control=[no-cache, no-store, max-age=0, must-revalidate], X-XSS-Protection=[1; mode=block], X-Content-Type-Options=[nosniff], Pragma=[no-cache]}
}
2019-12-23 14:42:33.683 DEBUG 17832 --- [XNIO-2 I/O-1] io.undertow.request.io : Fixed length stream closed with with 188 bytes remaining
2019-12-23 14:42:33.683 DEBUG 17832 --- [XNIO-2 I/O-1] i.u.client.http.HttpClientExchange : request terminated for request to localhost/127.0.0.1:20000 /
2019-12-23 14:42:33.683 ERROR 17832 --- [XNIO-2 I/O-1] io.undertow.proxy : UT005028: Proxy request to /proxy_endpoint/b933863e-9fad-4d00-a657-034ede313e34/ failed
io.undertow.server.TruncatedResponseException: null
Does anybody have an idea how I can host this app in shinyproxy (or otherwise)? I want to enable concurrent users to upload files and not using the same docker container/not interfere with each other.
Thanks and kind regards,
shosaco
That is a bug in Shinyproxy 2.2.0 and 2.3.0. Reverting to ShinyProxy 2.1.0 solves the problem, see https://github.com/openanalytics/shinyproxy/issues/184
I need send POST request to my nginx frontend server which should redirect it to upstream servers.
In details:
send request to http://192.168.0.10/foo/bar/blah and URL in this request should be changed to http://192.168.0.21[22,23]:8080/foo/blah
upstream myapp {
server 192.168.0.21:8080;
server 192.168.0.22:8080;
server 192.168.0.23:8080;
}
server {
listen 80;
server_name localhost;
location /foo/bar/blah/ {
rewrite ^/foo/blah^/ /$1 break;
proxy_pass http://myapp;
}
but in nginx error log I see that my request changed from POST to GET and also seems didn't change URL:
"POST /foo/bar/blah HTTP/1.1" 301 185 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64)
"GET /foo/bar/blah/ HTTP/1.1" 404 117 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64)
How can I keep my POST request and change the URL?
Also about my rewrite rule
rewrite ^/foo/blah^/ /$1 break;
I found a lot of examples for changing URL and all of them looks the same.
And it is really strange for me, how this rewrite rule can change URL from /foo/bar/blah/ to /foo/blah/:
in documentaion says:
http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite
If the specified regular expression matches a request URI, URI is
changed as specified in the replacement string.
but in my case request URI is /foo/bar/blah/ so regular expression /foo/blah^/ doesn't matches URI, so this rule shouldn't work.
Am I right?
Would be helpful any advises.
UPD:
fixed:
location = /foo/bar/blah {
proxy_pass http://myapp/foo/blah;
}
My guess, you don't need rewrite.
location /foo/bar/ {
proxy_pass http://myapp/;
}
This should remove /foo/bar part from proxied URL.
I have an app that lives in a sub-URI, http://myhost/app2. I'm having a hard time figuring out how I can configure the proper locale routing for this.
I've put this in routes.rb I've tried with both:
Rails.application.routes.draw do
get '' => redirect("/#{I18n.default_locale}")
scope "/:locale", locale: /#{I18n.available_locales.join("|")}/ do
root 'index#home'
end
end
as well as:
Rails.application.routes.draw do
prefix = Rails.application.config.relative_url_root
get "#{prefix}" => redirect("#{prefix}/#{I18n.default_locale}")
scope "#{prefix}/:locale", locale: /#{I18n.available_locales.join("|")}/ do
root 'index#home'
end
end
In my production.rb file, I have: config.relative_url_root = "/app2"
My nginx.conf file (taken from the Phusion Passenger docs):
http {
...
server {
...
location ~ ^/app2(/.*|$) {
alias /home/dani/app2/public$1;
passenger_base_uri /app2;
passenger_app_root /home/dani/app2/;
passenger_document_root /home/dani/app2/public/;
passenger_enabled on;
}
}
}
When I try accessing http://myhost/app2 in production, I get a Rails error page that says "The page you were looking for doesn't exist."
My production.log file looks like this with every request I make:
I, [2014-09-01T12:22:56.050499 #15230] INFO -- : Started GET "/app2" for myipaddress at 2014-09-01 12:22:56 -0400
I looked at the nginx access.log file and I see the following:
[01/Sep/2014:12:26:04 -0400] "GET /app2 HTTP/1.1" 301 91 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Firefox/31.0"
[01/Sep/2014:12:26:04 -0400] "GET /en HTTP/1.1" 404 1351 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Firefox/31.0"
So redirection from /app2 to /en is occurring but myhost/en doesn't resolve to anything on this setup, it should be myhost/app2/en.
Any advice would be helpful.
Note: Everything works fine in development.
I am using Pow and Nginx to serve my Rails 4 app in development. A simple file upload is returning 500 error and the request is not reaching the Rails controller. I assume this is the case because there is no mention of the request in the Rails log. Without any mention of the error, I am not sure what is going wrong here.
I started with an Ajax file upload but replaced it with a simple form which is also not working.
Tried this
and then went to this
Current avatar.html.erb
<%= form_for #user, html: { multipart: true }, method: "post", url: '/settings/avatar/update', class: "", id: "update_avatar" do |f| %>
<%= f.file_field :avatar, class: 'js-upload-photo-button js-change-avatar-btn', accept: 'image/png,image/gif,image/jpeg,image/jpg' %>
<%= f.submit "Upload" %>
<% end %>
Request headers in Chrome
Expanded Request Headers section
nginx.log
127.0.0.1 - - [01/Feb/2014:11:28:26 +0530] "POST /settings/avatar/update HTTP/1.1" 500 643 "https://allotrop.dev/settings/avatar" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
pow access.log
[Sat Feb 01 2014 11:28:26 GMT+0530 (IST)] INFO [127.0.0.1] GET allotrop.dev /500.html
There are other questions here about Rails returning 500 error with no mention in log rails 500 error no production log entry
, Rails 3.2.13, 500 error in development with no log
and How to properly diagnose a 500 error (Rails, Passenger, Nginx, Postgres)
But these are one-off errors and do not seem related to the problem I am facing. It would be great if anyone can point me in the right direction.
Update 1
Relevant line from routes.rb
post '/settings/avatar/update', to: 'settings#update_avatar'
Update 2
Found my nginx error log. BTW, if you are using Homebrew, it is at /usr/local/Cellar/nginx/1.4.0/logs/error.log
nginx/error.log
2014/02/01 13:05:54 [crit] 8787#0: *85813 open() "/usr/local/var/run/nginx/client_body_temp/0000000010" failed (13: Permission denied), client: 127.0.0.1, server: *.dev, request: "POST /settings/avatar/update HTTP/1.1", host: "allotrop.dev", referrer: "https://allotrop.dev/settings/avatar"
The problem is not with Rails but with Nginx which is pretty evident from the nginx error.log. This question helped me understand what I was dealing with - Rails 3 + carrierwave + nginx = permission denied.
Nginx uses the client_body_temp_path directive to specify the location where it will temporarily store the uploaded files from the user request. Homebrew had set it by default to /usr/local/var/run/nginx. This folder also contains fastcgi_temp, proxy_temp, scgi_temp and uwsgi_temp for me. Nginx worker processes run with user nobody and they were not able to access these folders. I chowned all these folders to the nobody user, but that did not help.
Finally, I did
client_body_temp_path /tmp/nginx/; inside the HTTP module of my nginx.conf to make it work.
Doing a ls -l shows
drwx------ 2 nobody wheel 68 Feb 1 14:44 nginx
I am not sure why this worked inside /tmp and not inside the original /var/run/nginx. I belive I will face similar issue when I use other temp folders or in production. Will update this thread if and when that happens.
I recommend symlinking the other relevant logs like the nginx access and error log, pow access and app log to the /log directory of your Rails app. It helps in looking up errors in one of these when you face a tricky bug.