I have a scenario where one kestrel host is behind two (if it makes a difference) nginx proxies, and for the life of me I cannot get HttpContext.Connection.RemoteIpAddress to point to the original client IP address.
By adding some logging right after building the app,
var app = builder.Build();
app.UseCors();
app.UseForwardedHeaders();
app.UseHttpLogging();
app.Use(async (context, next) =>
{
app.Logger.LogInformation("Request RemoteIp: {RemoteIpAddress}",
context.Connection.RemoteIpAddress);
await next(context);
});
I can then see the following output on the console
[18:24:37 INF] Request:
Protocol: HTTP/1.1
Method: GET
Scheme: http
PathBase:
Path: /api/what-is-my-ip
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Connection: keep-alive
Host: --ACTUAL HOSTNAME USED IN URL--
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en;q=0.5
Cookie: [Redacted]
Upgrade-Insecure-Requests: [Redacted]
X-Forwarded-For: --CORRECT-REAL-IP-OF-CLIENT--, 172.18.0.1
X-Forwarded-Proto: http
X-Real-IP: --CORRECT-REAL-IP-OF-CLIENT--
[18:24:37 INF] Request RemoteIp: ::ffff:172.18.0.4
From what I can tell, the X-Forwarded-For header is correct (although I am missing one hop*), but the last line always displays the local ip of the machine that runs Kestrel and completely ignores the X-Forwarded-For header (the first IP address in that list is what I want to access, and from what I understand is where the original client's IP should appear after multiple hops).
*I think I am missing one hop because the requests are routed as follows:
Internet Client
--> Docker host that runs NGINX (172.18.0.1) - 1st proxy_pass
--> Docker container that serves UI with NGINX (172.18.0.3) 2nd proxy_pass
--> Kestrel container (172.18.0.4)
I would expect to finally see an X-Forwarded-For list on Kestrel that contained the 2nd proxy_pass server as well, ie
Original Client IP, 172.18.0.1, 172.18.0.3
Anyhow, if it helps my configuration regarding forwarding are
builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
options.ForwardLimit = null;
options.KnownProxies.Add(IPAddress.Parse("172.18.0.1"));
options.KnownProxies.Add(IPAddress.Parse("172.18.0.3"));
}
And both NGINX proxies are setup as such
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
I don't understand what I am doing wrong, unless HttpContext.Connection.RemoteIpAddress is not meant to resolve to the first address of the X-Forwarded-For header when it is present -- but then why on earth do I need to configure the ForwardedHeadersOptions for?!
Related
I have a small project that uses Django Channels (websockets) and it works fine locally. I've added SSL to my docker-based deployment with the letsencrypt-companion-container docker image, which, exposes 443, handles all the SSL, then funnels requests/responses vi port 80 into my A/WSGI adapter (uvicorn or daphne).
When under SSL, my client code is getting this error in the JS console:
`WebSocket connection to 'wss://my_server.com/ws/echo/' failed: Error during WebSocket handshake: Unexpected response code: 404`
It isn't clear to me why. How do I make this work?
Here is my nginx.conf, if it helps:
upstream wsgi {
server web:8000;
}
upstream asgi {
server web:8000;
}
server {
listen 0.0.0.0:80;
location / {
proxy_pass http://wsgi;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /ws {
proxy_pass http://asgi;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;
}
location /staticfiles/ {
alias /home/app/web/staticfiles/;
}
}
I was facing a similar issue, I got a 403 error on my django server. After checking this and setting my ALLOWED_HOSTS = "*" the server worked.
Being honest it doesn't look like a best practice to me, but while I find something better, is a way to have the server up and running.
Make sure also that you include this configuration on your nginx
I have setup two nginx instances as follows:
nginx (https) → docker:[nginx (http) → uwsgi]
The front facing nginx process exposes the https service, which passes down all requests via proxy_pass to the docker nginx process. It also redirects all requests to http → https.
The problem is that the docker nginx process has the following line in a location block in its default server instance:
server {
...
location = / {
return 301 $scheme://$http_host${request_uri}login/;
}
}
With the intention of redirecting / to the login page. This works fine except that the redirection always points to an http://... url. E.g. A request to http://myserver.com/, gets redirected to https://myserver.com/, then it gets passed down to the docker nginx which returns a 301 with the following url: http://myserver.com/login/. I want it to be https://myserver.com/login/ or whatever schema the front-facing server may offer.
This is how I setup the front-facing nginx process:
server {
listen 443 ssl http2 default_server;
...
location / {
proxy_pass http://localhost:81;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_cache_bypass $http_upgrade;
proxy_redirect https:// http://;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Scheme $scheme;
}
}
server {
listen 80 default_server;
...
location = / {
return 301 https://$http_host${request_uri};
}
}
Is this kind of redirection even possible?
Also, in case you wonder, I also tried all possible combinations of X-Forwarded-Proto, X-Scheme and proxy_redirect as other answers suggest, namely:
Nginx does redirect, not proxy
how to handle nginx reverse proxy https to http scheme redirect
One trick that I've found is that you can disable absolute redirects (e.g. instead of redirecting to http://localhost/your_url_with_trailing_slash_now it will redirect to /your_url_with_trailing_slash_now).
add the following to any point within the server block (of the nginx instance that does this 301 redirect, in your case the nginx-docker instance):
server {
absolute_redirect off;
...
More info can be found here: https://serverfault.com/questions/227742/prevent-port-change-on-redirect-in-nginx
I am trying to setup a basic mockup (for training) with 2 docker containers :
One with nginx
One with jenkins
nginx is used as a reverse container (the next step wil be https frontend).
jenkins is supposed to be publicly accessible as www.devportal.org:90/ci
I have partially succeeds. I mean that I can access it with the given address. But ...
When I try to connect with the admin user, I get the login page. But when filling the user / password, I am redirected to http://www.devportal.org/ci/.
If I add the port to this address, I obtain the instance home page, connected as the admin user.
When I select the manage page, I see a warning message saying that my reverse proxy configuration is broken.
Besides, when going to configure system, the page is displayed but it never finishes loading.
I tried various configuration for nginx, but none of them gave me the correct result.
The Jenkins Location URL is set to the instance address known by nginx.
As far as both are within containers linked to a dedicated docker network where jenkins is known as jenkins_ci, the address is http://jenkins_ci:8080/ci/.
And if I run :
docker exec nginx_revproxy wget http://jenkins_ci:8080/ci/
I actually get the home page.
Here is my nginx configuration (in : /etc/nginx/conf.d/reverse_proxy.conf)
# Reverse proxy configuration for one jenkins server
# Nginx listen public traffic on port 90 and forward to jenkins container address on port 8080
server {
listen 90;
access_log /var/log/nginx/reverse-access.log;
error_log /var/log/nginx/reverse-error.log;
location ^~ /ci/ {
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://jenkins_ci:8080/ci/;
# Fix the "It appears that your reverse proxy set up is broken" error.
proxy_set_header X-Forwarded-Host $host;
proxy_read_timeout 90;
proxy_redirect http://jenkins_ci:8080/ http://www.devportal.org:90/;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
}
}
When getting an error page at connexion I see this on nginx container log
172.17.0.1 - - [19/Jul/2019:12:32:04 +0000] "GET /ci/ HTTP/1.1" 404 153 "http://www.devportal.org:90/ci/login?from=/ci/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0" "-"
2019/07/19 12:32:04 [error] 7#7: *40 "/usr/share/nginx/html/ci/index.html" is not found (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "GET /ci/ HTTP/1.1", host: "www.devportal.org", referrer: "http://www.devportal.org:90/ci/login?from=/ci/"
On the jenkins container log I also have :
Jul 19, 2019 2:18:25 PM hudson.diagnosis.ReverseProxySetupMonitor getTestForReverseProxySetup
WARNING: http://www.devportal.org/ci/manage vs. http:
Additional information
With nginx conf:
server {
listen 90;
server_name www.devportal.org;
access_log /var/log/nginx/reverse-access.log;
error_log /var/log/nginx/reverse-error.log;
location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
#rewrite all static files into requests to the root
#E.g /static/12345678/css/something.css will become /css/something.css
rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
}
location /ci/ {
proxy_pass http://jenkins_ci:8080;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Fix the "It appears that your reverse proxy set up is broken" error.
proxy_set_header X-Forwarded-Host $host;
proxy_read_timeout 90;
proxy_redirect http://jenkins_ci:8080/ http://www.devportal.org:90/;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
proxy_set_header Connection ""; # Clear for keepalive
}
}
and
<jenkins.model.JenkinsLocationConfiguration>
<adminAddress>[my mail address]</adminAddress>
<jenkinsUrl>http://www.devportal.org:90/ci</jenkinsUrl>
</jenkins.model.JenkinsLocationConfiguration>
in
jenkins.model.JenkinsLocationConfiguration.xml
I make some progress regarding the error message in jenkins log :
Jul 22, 2019 11:59:35 AM hudson.diagnosis.ReverseProxySetupMonitor getTestForReverseProxySetup
WARNING: http://www.devportal.org/ci/manage vs. http://www.devportal.org:90/ci/manage
Now, I only loose the port number (I lost the whole domain name before).
Note that it is the same with or without proxy_redirect.
So Jenkins URL should be set to what your user see as standing in description:
Optionally specify the HTTP address of the Jenkins installation, such as
http://yourhost.yourdomain/jenkins/. This value is used to let Jenkins know how
to refer to itself, ie. to display images or to create links in emails. This is
necessary because Jenkins cannot reliably detect such a URL from within itself
so in your case: http://www.devportal.org:90/
That line should be probably removed as I think it's not needed:
proxy_redirect http://jenkins_ci:8080/ http://www.devportal.org:90/;
And you should remove '/ci' from proxy pass like this
proxy_pass http://jenkins_ci:8080;
I'm trying make load balancer with nginx about 4 hours and I don't have more ideas how to deal with it.
My plan is:
client -> nginx load balancer -> gitlab or taiga.io
For now my nginx conf:
upstream gitlab {
server localhost:8081;
}
server {
listen 80;
server_name git.localhost.com
client_max_body_size 300M;
location / {
proxy_pass http://localhost:8081/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
Gitlab working on localhost:8081 fine.
When I goes to git.localhost.com i see only 502 error with error log:
6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: git.localhost.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8081/", host: "git.localhost.com"
What I'm doing wrong?
I tried this: Gitlab 5.3 behind nginx reverse proxy and still does not works.
There must be some discrepancy between http://localhost:8081 (which works) and the location to which nginx is proxying your requests (which doesn't work).
Currently your upstream block is not being used.
In proxy_pass, try replacing http://localhost:8081/ with http://gitlab.
ActionCable doesn't work in production. Works well in development, but not in production.
Running Nginx with Puma on Ubuntu 14.04. I have checked that redis-server is up and running.
Rails -v 5.0.0.1
production.log:
INFO -- : Started GET "/cable/"[non-WebSocket] for 178.213.184.193 at 2016-11-25 14:55:39 +0100
ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: close, HTTP_UPGRADE: )
INFO -- : Finished "/cable/"[non-WebSocket] for 178.213.184.193 at 2016-11-25 14:55:39 +0100
Request from client:
GET ws://mityakoval.com/cable HTTP/1.1
Host: mityakoval.com
Connection: Upgrade
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
Origin: http://mityakoval.com
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4,uk;q=0.2,nb;q=0.2
Cookie: _vaktdagboka_session=******
Sec-WebSocket-Key: *******
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Protocol: actioncable-v1-json, actioncable-unsupported
Response:
HTTP/1.1 404 Not Found
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 25 Nov 2016 13:52:21 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache
X-Request-Id: d6374238-69ef-476e-8fc5-e2f8bbb663de
X-Runtime: 0.002500
nginx.conf:
upstream puma {
server unix:///home/mityakoval/apps/vaktdagboka/shared/tmp/sockets/vaktdagboka-puma.sock;
}
server {
listen 80 default_server deferred;
# server_name example.com;
root /home/mityakoval/apps/vaktdagboka/current/public;
access_log /home/mityakoval/apps/vaktdagboka/current/log/nginx.access.log;
error_log /home/mityakoval/apps/vaktdagboka/current/log/nginx.error.log info;
location ^~ /assets/ {
gzip_static on;
expires max;
add_header Cache-Control public;
}
try_files $uri/index.html $uri #puma;
location #puma {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://puma;
}
location /cable {
proxy_pass http://puma;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
error_page 500 502 503 504 /500.html;
client_max_body_size 10M;
keepalive_timeout 10;
}
cable.yml:
redis: &redis
adapter: redis
url: redis://127.0.0.1:6379
production: *redis
development:
adapter: async
test:
adapter: async
in production.rb:
config.action_cable.allowed_request_origins = ["http://mityakoval.com"]
in routes.rb:
mount ActionCable.server, at: '/cable'
UPDATE:
Don't forget to restart nginx :) That was the problem for me.
You should change the value of proxy_pass property from http://puma to http://puma/cable.
Therefore, the correct location section for the /cable will be:
location /cable {
proxy_pass http://puma/cable;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
While other posts have correctly posted solutions I thought I'd post a bit more info on how to identify where the problem is/where to fix it for other nginx noobs.
You will know your nginx config needs the proxy_set_header Upgrade at the path you're mounting action cable if the rails error contains HTTP_UPGRADE:. (Meaning nothing is passed to HTTP_UPGRADE). After fixing the problem my logs are showing HTTP_UPGRADE: websocket
Gotchya 1: As mentioned by the op, make sure you restart nginx after making a change(I was incorrectly doing this).
Gotchya 2: Also look for include statements in the nginx config as your config could be split across multiple files. The location /cable { section should be inside of server { which in my case was missing because it was in a different config file from an includes statement which I didn't notice for a while.
Similar error but different problem: Your rails logs will contain an additional error in the logs right before the one the OP mentioned saying the origin is not allowed, that is when your rails config needs to be updated as another answer mentions updating config.action_cable.allowed_request_origins.
The logging is subject to change with rails but hopefully this helps clarify where the problem and a few gotchya's I encountered as someone who knows nothing about nginx.
Super late to this conversation, however, for anyone who is facing the same error message using Rails5, Action Cable, etc. & DEVISE you simply solve it like suggested here. It all comes down to the web socket server not having a session, hence the error message.
app/channels/application_cable/connection.rb
module ApplicationCable
class Connection < ActionCable::Connection::Base
identified_by :current_user
def connect
self.current_user = find_verified_user
logger.add_tags 'ActionCable', current_user.name
end
protected
def find_verified_user
verified_user = User.find_by(id: cookies.signed['user.id'])
if verified_user && cookies.signed['user.expires_at'] > Time.now
verified_user
else
reject_unauthorized_connection
end
end
end
end
app/config/initializers/warden_hooks.rb
Warden::Manager.after_set_user do |user,auth,opts|
scope = opts[:scope]
auth.cookies.signed["#{scope}.id"] = user.id
auth.cookies.signed["#{scope}.expires_at"] = 30.minutes.from_now
end
Warden::Manager.before_logout do |user, auth, opts|
scope = opts[:scope]
auth.cookies.signed["#{scope}.id"] = nil
auth.cookies.signed["#{scope}.expires_at"] = nil
end
Solution was developed by Greg Molnar
The resolution that needs a NGINX configuration changes to accept this action cable request.
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
Add the above lines to your location block in the nginx site configuration, then restart nginx.
My solution was to add these lines to my production.rb file:
config.action_cable.url = 'ws://your_site.com/your_action_cable'
config.action_cable.allowed_request_origins = [ 'http://your_site.com' ]
You can change you nginx config about /cable
proxy_set_header X-Forwarded-Proto http;
I used you nginx config and add this change on myu server, it works fine.
Worked with:
location ^~ /cable {
...
}
Location requires ^~
Your cable.yml file should look like this:
production:
adapter: redis
url: <%=ENV['REDIS_URL']%>
Then you should have this key set up in the environment, should look something like this:
REDIS_URL: 'redis://redistogo:keyblahblahblhblah'
Also, you should have this in production.rb:
config.web_socket_server_url = "wss://YOUR_URL.com/cable"