Nginx variables ignore case - docker

I'm setting up Nginx using Docker's service discovery. My service name is webAdmin.
Relevant section of the current Nginx config reads
resolver 127.0.0.11 valid=10s; # Docker DNS server
if (!-f $request_filename) {
set $upstream_admin_server webAdmin:8000;
proxy_pass http://$upstream_admin_server;
break;
}
When visiting the appropriate server, Nginx returns a 404. The logs reveal that Nginx is attempting to resolve a lower case version of my service name.
2019/08/26 21:53:46 [error] 3756#3756: *1569 webadmin could not be resolved (3: Host not found), client: 10.0.0.29, server: admin.mysite.com, request: "GET /favicon.ico HTTP/1.1", host: "admin.mysite.com"
When I avoid using a variable the config reads
resolver 127.0.0.11 valid=10s; # Docker DNS server
if (!-f $request_filename) {
proxy_pass http://webAdmin:8000;
break;
}
Nginx is then able to resolve the service name and correctly route my request.
I attempted to use quotes, single and double but neither seem to have any effect. The Nginx docs for set don't seem to offer any clues.
Why is my variable being converted to lower case?

When Nginx attempts to resolve a name it actually forces the name to lower case. Source can be found here.
I assume this decision was made with the knowledge that DNS names are supposed to be "case insensitive". But it results in inconsistent behavior between an explicitly declared resolver and the default resolver.
For now it seems that the best option is to avoid the use of capitalization in service names. (ie. webAdmin -> web_admin)
Thanks to Adiii for the guidance!

Its working fine with me using nginx:alpine. I test it using the below configuration and changing the value for LocalhOst.
server {
listen 80;
server_name localhost;
location / {
set $upstream_admin_server LocalhOst:80;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://$upstream_admin_server/index.html;
}
location /index.html {
root /usr/share/nginx/html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
You can test it below command.
docker run --rm --name my-custom-nginx-container -p 80:80 -v $PWD/nginx.conf:/etc/nginx/conf.d/default.conf -it nginx:alpine

Related

nginx reverse proxy server does not fetch full bundle from the upstream server in docker stack, net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)

I deployed an nginx:1.22.1 instance alongside a static react app server on a worker node in a docker swarm. This is docker swarm mode, not classic swarm.
The advertise address that I listed when joining the swarm is internal to the data center, I do not know if that matters because I can still access these services with the public addresses.
Both containers are pinned to the same worker node and communicate over a user-created overlay network.
I can retrieve the full bundle directly from the react app server over the public network.
I cannot retrieve the full bundle through the nginx reverse-proxy server over the public network.
When I attempt to fetch the bundle using chrome browser as the user-agent I get 2 errors:
net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK).
The app bundle is cutoff mid js function as if a chunk of data was not transmitted.
Rarely, the upstream server will send html and not a js bundle. But I receive that whole response body and it is not truncated like the js bundle.
I have played with all kinds of configuration and cannot get it to work.
(most relevant)
This is my configuration under /etc/nginx/conf.d/default.conf
resolver 127.0.0.11 valid=10s;
error_log /dev/stdout info;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
ssl_certificate /etc/nginx/certs/nginx.pem;
ssl_certificate_key /etc/nginx/certs/key.pem;
client_max_body_size 100M;
proxy_buffers 8 1024k;
proxy_buffer_size 1024k;
proxy_max_temp_file_size 1024m;
location / {
set $reactapp reacthost;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://$reactapp:3000/;
proxy_redirect off;
}
root /usr/share/nginx/html;
}
error_page 500 502 503 504 /50x.html;
}
I use the variable $reactapp for service discovery after nginx server start. See NGINX blog here.
Note that the nginx:1.22.1 instance runs with user nginx after it is deployed to the stack. I only see this below message when I deploy via docker stack. If I start the container directly using docker engine, I do not see it.
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
However, I can exec into the container as the nginx user, access /var/cache/nginx/, and create a directory.
I do not know if:
my server / location configuration is plain bad.
The NGINX server cannot write a part of the container it needs to write when the service is deployed in stack mode.
If I cannot access the server properly over the public network via the overlay network.
Prior to using docker stack I was able to use this reverse proxy.
The two containers were on the same host without swarm mode running.
The containers communicated over a bridge network.
The reverse proxy server port was published on the public interface of the server it was deployed on.
The NGINX server started after the upstream server.
Because there is no depends_on key honored in stack mode I have to allow DNS service discovery after the NGINX server starts up. Placing them on an overlay gives me more flexibility in how I do my deployments, but this has become a bit muddled. There are enough differences between the two environments that it has become difficult to get the stack to behave as I expect.

How to use nginx(intsalled on docker) reverse proxy gitlab(installed on docker too)

I installed gitlab according to the official documentation.
sudo docker run --detach \
--hostname git.stupidpz.com \
--publish 8443:443 --publish 880:80 --publish 822:22 \
--name gitlab \
--restart always \
--volume $GITLAB_HOME/config:/etc/gitlab \
--volume $GITLAB_HOME/logs:/var/log/gitlab \
--volume $GITLAB_HOME/data:/var/opt/gitlab \
--shm-size 256m \
gitlab/gitlab-ee:latest
Now I want to use Nginx(installed By Myself) to reverse proxy gitlab instead of the nginx that comes with the gitlab container.
According to official documentation I added some code in gitlab.rb
# Define the external url
external_url 'http://git.stupidpz.com'
# Disable the built-in nginx
nginx['enable'] = false
# Disable the built-in puma
puma['enable'] = false
# Set the internal API URL
gitlab_rails['internal_api_url'] = 'http://git.stupidpz.com'
# Define the web server process user (ubuntu/nginx)
web_server['external_users'] = ['nginx']
Then gitlab cannot be accessed, I found some error logs in this file /var/log/gitblab/gitlab_workhorse/current
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2023-01-25T20:57:21Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2023-01-25T20:57:31Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2023-01-25T20:57:41Z","uri":""}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial tcp 127.0.0.1:8080: connect: connection refused","level":"error","method":"GET","msg":"","time":"2023-01-25T20:57:51Z","uri":""}
Did nothing else except for adding some code in gitlab.rb.
I wonder where this dial tcp 127.0.0.1:8080 comes from?
I hope you can help me, or give me a correct demo.Many thanks.This problem has been bothering me for two days
Now i figure out why i could not make it works,I mixed up Using an existing Passenger/NGINX installation and Using a non-bundled web-server
If you just need to use your own nginx to proxy gitlab(both of them was installed on docker)
you just need to add two lines to gitlab.rb.
# Disable the built-in nginx
nginx['enable'] = false
# Define the web server process user (ubuntu/nginx)
web_server['external_users'] = ['nginx']
and here is nginx's conf
upstream gitlab-workhorse {
server unix://var/opt/gitlab/gitlab-workhorse/sockets/socket fail_timeout=0;
}
server {
listen *:80;
server_name git.example.com;
server_tokens off;
root /opt/gitlab/embedded/service/gitlab-rails/public;
client_max_body_size 250m;
access_log /var/log/gitlab/nginx/gitlab_access.log;
error_log /var/log/gitlab/nginx/gitlab_error.log;
# Ensure Passenger uses the bundled Ruby version
passenger_ruby /opt/gitlab/embedded/bin/ruby;
# Correct the $PATH variable to included packaged executables
passenger_env_var PATH "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/usr/local/bin:/usr/bin:/bin";
# Make sure Passenger runs as the correct user and group to
# prevent permission issues
passenger_user git;
passenger_group git;
# Enable Passenger & keep at least one instance running at all times
passenger_enabled on;
passenger_min_instances 1;
location ~ ^/[\w\.-]+/[\w\.-]+/(info/refs|git-upload-pack|git-receive-pack)$ {
# 'Error' 418 is a hack to re-use the #gitlab-workhorse block
error_page 418 = #gitlab-workhorse;
return 418;
}
location ~ ^/[\w\.-]+/[\w\.-]+/repository/archive {
# 'Error' 418 is a hack to re-use the #gitlab-workhorse block
error_page 418 = #gitlab-workhorse;
return 418;
}
location ~ ^/api/v3/projects/.*/repository/archive {
# 'Error' 418 is a hack to re-use the #gitlab-workhorse block
error_page 418 = #gitlab-workhorse;
return 418;
}
# Build artifacts should be submitted to this location
location ~ ^/[\w\.-]+/[\w\.-]+/builds/download {
client_max_body_size 0;
# 'Error' 418 is a hack to re-use the #gitlab-workhorse block
error_page 418 = #gitlab-workhorse;
return 418;
}
# Build artifacts should be submitted to this location
location ~ /ci/api/v1/builds/[0-9]+/artifacts {
client_max_body_size 0;
# 'Error' 418 is a hack to re-use the #gitlab-workhorse block
error_page 418 = #gitlab-workhorse;
return 418;
}
# Build artifacts should be submitted to this location
location ~ /api/v4/jobs/[0-9]+/artifacts {
client_max_body_size 0;
# 'Error' 418 is a hack to re-use the #gitlab-workhorse block
error_page 418 = #gitlab-workhorse;
return 418;
}
# For protocol upgrades from HTTP/1.0 to HTTP/1.1 we need to provide Host header if its missing
if ($http_host = "") {
# use one of values defined in server_name
set $http_host_with_default "git.example.com";
}
if ($http_host != "") {
set $http_host_with_default $http_host;
}
location #gitlab-workhorse {
## https://github.com/gitlabhq/gitlabhq/issues/694
## Some requests take more than 30 seconds.
proxy_read_timeout 3600;
proxy_connect_timeout 300;
proxy_redirect off;
# Do not buffer Git HTTP responses
proxy_buffering off;
proxy_set_header Host $http_host_with_default;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://gitlab-workhorse;
## The following settings only work with NGINX 1.7.11 or newer
#
## Pass chunked request bodies to gitlab-workhorse as-is
# proxy_request_buffering off;
# proxy_http_version 1.1;
}
## Enable gzip compression as per rails guide:
## http://guides.rubyonrails.org/asset_pipeline.html#gzip-compression
## WARNING: If you are using relative urls remove the block below
## See config/application.rb under "Relative url support" for the list of
## other files that need to be changed for relative url support
location ~ ^/(assets)/ {
root /opt/gitlab/embedded/service/gitlab-rails/public;
gzip_static on; # to serve pre-gzipped version
expires max;
add_header Cache-Control public;
}
## To access Grafana
location /-/grafana/ {
proxy_pass http://localhost:3000/;
}
error_page 502 /502.html;
}
last but not least,you need to add another bash to your nginx's container,
-v /var/opt/gitlab:/var/opt/gitlab
This will let your nginx container connect to gitlab container.Otherwise you will get "cannot find var/opt/gitlab/gitlab-workhorse/sockets/socket".
It looks like you are installing a GitLab instance as a custom git server on a remote host. There are 3 pieces of this that must work.
DNS setup, remote host's ports and firewall setup.
Working installation of GitLab on the remote host.
Valid SSL certificates, and a correct nginx config for HTTPS.
The first step really depends on your virtual machine and container's setup, but essentially, make sure it (the VM or container) has a public port that responds to requests.
These variables must be set in the remote host's environment as such
--volume $GITLAB_HOME/config:/etc/gitlab
--volume $GITLAB_HOME/logs:/var/log/gitlab
--volume $GITLAB_HOME/data:/var/opt/gitlab \
The above URL covers all the GitLab install steps once you have signed in and verified that it was installed correctly and that it runs as expected on that remote host.
Only then, install and configure nginx. Since GitLab likely will transfer credentials and other secure data, you will need to setup https on nginx.
An example of an Nginx configuration can be found here. There is also a tool by Mozilla that makes building a custom nginx config easier, found here.
The error you show has this URL "127.0.0.1:8080". It is likely you have supplied this URL to the gitlab.rb config somewhere, and that might be a mistake. I cannot be sure without the whole config file however.
Also, it is likely the GitLab image will need to run its own nginx instance, so that the said container when launched may do its job and act as a git server. To reverse proxy this GitLab instance, you may need to install nginx onto your host machine and point it to GitLab Image's nginx.
You may be able to do away with the second nginx instance by appending a new server {} block into the Gitlab Image's nginx config. I would not recommend this.

Why doesn't Nginx reverse proxy on WSL2 see Rails app in Docker Container using hostname?

I've got several Rails websites running in Docker dev containers. Docker is running in WSL (Ubuntu 20.04) on Windows 11. Nginx is running in Ubuntu as a reverse proxy, IIS is turned off in Windows. The Ubuntu /etc/hosts file is automatically populated from the hosts file in Windows. It is set up like this because others on the team are running Linux on Macs but I switch between Rails and .Net development.
An example website is mysite1.localhost which is exposed on port 8081 on Docker and there is an entry of '127.0.0.1 mysite1.localhost' in both hosts files.
The problem I have is browsing (Chrome on Windows) localhost:8081 returns 200 from the website, great, but using the hostname mysite1.localhost returns 502 Bad Gateway.
I am assuming Nginx doesn't know about Docker or something like that?
Here is the mysite1.conf for Nginx:
server {
listen 80;
listen [::]:80;
server_name mysite1.localhost;
resolver 127.0.0.1;
location ~* "^/shared-nav" {
proxy_set_header Accept-Encoding "";
proxy_pass http://localhost:3000/stuff$is_args$args;
}
location / {
ssi on;
ssi_silent_errors off;
log_subrequest on;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8081;
add_header Cache-Control "no-cache";
if ($request_filename ~* ^.*?/([^/]*?)$) {
set $filename $1;
}
if ($filename ~* ^.*?\.(eot)|(ttf)|(woff)|(woff2)$) {
add_header Access-Control-Allow-Origin *;
}
}
}
I can see two problems in nginx/error.log:
2023/02/02 08:50:10 [warn] 2841#2841: conflicting server name "mysite1.localhost" on 0.0.0.0:80, ignored
2023/02/02 08:50:12 [error] 2845#2845: *52 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: mysite1.localhost, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8081/", host: "mysite1.localhost"
It doesn't seem to matter whether or not docker is running.
For the conflicting server name waring, I've tried looking for temporary files that need deleting but cannot find anything.
Most of the other questions I've looked at involve solving problems with containerized Nginx where as this is sitting in WSL.
Please let me know if I can better explain the problem, thanks for any help.
One thing I didn't take into consideration was using VS Code to dev in the containers means that the ports were forwarded to Windows - although the containers are running in WSL, VS Code is running in Windows.
So, I could either turn IIS back on and use it as a reverse proxy or try Nginx for Windows. I've opted for the latter as it means I can share the same config files as the Linux guys and will see how it works out, for now I can browse the websites by hostname.
If anyone else needs to work with this set up, I'm happy to pass on my experiences.

NGinx and multiple docker containers but one subdomain

My Goal
I want to use a single NGinx docker container as a proxy.
I want it to respond to traffic on my domain: "sub.domain.com" and listen on port 80 and 443.
When traffic comes in on /admin I want it to direct all traffic to one docker container (say... admin_container:6000).
When traffic comes in on /api I want it to direct all traffic to another docker container (say... api_container:5500).
When traffic comes in on any other path (/anything_else) I want it to direct all traffic to another docker container (say... website_container:5000).
Some Helpful Context
Real quick, let me provide some context in case it's helpful. I have a NodeJS website running in a docker container. I'd like to also have an Admin section, created with ASP.NET Core that runs in a second docker container. I'd like both of these websites to share and make use of a single ASP.NET Core Web Api project, running in a third docker container. So, one NodeJS project and two ASP.NET Core projects, that all live on a single subdomain:
sub.domain.com/
Serves the main website
sub.domain.com/admin
Serves the Admin website
sub.domain.com/api
Serves API endpoints and handles Database connectivity
What I have So Far
So far I have the NGinX reverse proxy set up and a single docker container for the NodeJS application. Currently all traffic on :80 is redirected to 443. All traffic on 443 is directed to the NodeJS docker container, running privately on :5000. I'll admit I'm not great with NGinx and don't fully understand how this works.
The NGinx.conf file
worker_processes 2;
events { worker_connections 1024; }
http {
sendfile on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
upstream docker-nodejs {
server nodejs_prod:5000;
}
server {
listen 80;
server_name sub.domain.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name sub.domain.com;
ssl_certificate /etc/nginx/ssl/combined.crt;
ssl_certificate_key /etc/nginx/ssl/mysecretkeyfile.key;
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_pass http://docker-nodejs;
}
}
}
Quick Note: In the above code "nodejs_prod:5000" refers to the container who's name is "nodejs_prod" and listens on port 5000. I don't understand how that works, but it is working. Somehow docker is creating DNS entries in the private network, one for each container name. I'm actually using docker-compose.
My Actual Question: How will this NGinx.conf file look when I have 2 more websites (each a docker container). It's important that /admin and /api are sent to the correct docker containers, and not handled by the "catch all" location. I'm imagining that I'll have a "catch all" location which captures all traffic that DOESN'T START WITH /admin OR /api.
Thank you!
In your nginx config you want to add location rules, such as
location /admin {
proxy_pass http://admin_container:6000/;
}
location /api {
proxy_pass http://api_container:5500/;
}
This redirects /admin to admin_container port 6000, and /api to api_container and port 5500.
docker-compose creates a network between all it's containers. Which is why http://api_container:5500/ points to the container named api_container and the port 5500. This can be used for communication between containers. You can read more about it here https://docs.docker.com/compose/networking/

Docker port is not working over https after setting up an SSL over ubuntu nginx

I set up the Letsencrypt certificate directly to an AWS EC2 Ubuntu instance running Nginx and a docker server using port 9998. The domain is set up on Route 53. Http is redirected to https.
So https://example.com is working fine but https://example.com:9998 gets ERR_SSL_PROTOCOL_ERROR. If I use the IP address like http://10.10.10.10:9997 is working and checked the server using port 9998 okay.
The snapshot of the server on docker is:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
999111000 img-server "/bin/sh -c 'java -j…" 21 hours ago Up 21 hours 0.0.0.0:9998->9998/tcp hellowworld
It seems something is missing between Nginx and the server using port 9998. How can I fix it?
Where have you configured the ssl certificate ? Only Nginx?
The reason why you cannot visit https://example.com:9998 using ssl protocal is that that port provides http service rather than https.
I suggest not to publish 9998 of hellowworld and proxy all the traffic with nginx (if Nginx is also started with docker and in the same network).
Configure https in Nginx and the origin sever provides http.
This is a sample configuration https://github.com/newnius/scripts/blob/master/nginx/config/conf.d/https.conf
server {
listen 80;
server_name example.com;
return 301 https://example.com$request_uri;
}
server {
listen 443;
server_name example.com;
access_log logs/example.com/access.log main;
error_log /var/log/nginx/debug.log debug;
ssl on;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
location / {
proxy_pass http://apache:80;
proxy_set_header Host $host;
proxy_set_header CLIENT-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~ /.well-known {
allow all;
proxy_pass http://apache:80;
}
# Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store(Mac).
location ~ /\. {
deny all;
}
}

Resources