We have an nginx running in a docker (docker image nginx:1.19.2-perl). In principle, permissions are correct everything works fine when we start but then after sometime (maybe hours or days), suddenly users gets a forbidden 403 when accessing couple of files. And in the nginx logs we see a permission error. This seems to happen always over the same two files. But again, it does not happen always, after a restart it works fine again.
[error] 16#16: *101701 open() "/usr/share/nginx/html/plugins/jquery.easy-pie-chart/dist/jquery.easypiechart.js" failed (13: Permission denied), client: xxxx, server: xxx, request: "GET /plugins/jquery.easy-pie-chart/dist/jquery.easypiechart.js HTTP/1.1", host: "yyyy", referrer: "http://xxxx/"
[error] 16#16: *101702 open() "/usr/share/nginx/html/plugins/angular-chart.js/dist/angular-chart.js" failed (13: Permission denied), client: xxx, server: yyy, request: "GET /plugins/angular-chart.js/dist/angular-chart.js HTTP/1.1", host: "xxx", referrer: "http://xxx/"
Any idea what might be happening?.
We did not see this issue before updating nxginx (previous version nginx:1.12.0-perl) and before udpating host OS (before Ubuntu 16.04.3 now Ubuntu 16.04.6)
Edit:
Another mystery, if we do an 'ls' of the directory where the file is located the problem is solved, until the error happens again in few hours
Just in case my previous answer doesn't work you can try loading the scrips via CDN instead of from your filesystem.
I think this can be related with some missing config in you Dockerfile. Try adding this to your Dockerfile:
RUN chgrp -R root /var/cache/nginx /var/run /var/log/nginx && \
chmod -R 770 /var/cache/nginx /var/run /var/log/nginx
Related
On a Digital Ocean droplet running Ubuntu 21.10 impish I am deploying a bare bones Rails 7.0.0.alpha2 application to production. I am setting up nginx as the reverse proxy server to communicate with Puma acting as the Rails server.
I wish to run puma as a service using systemctl without sudo root privileges. To this effect I have a puma service setup in the users home folder located at ~/.config/systemd/user, the service is enabled and runs as I would expect it to run.
systemctl status --user puma_master_cms_production
reports the following
● puma_master_cms_production.service - Puma HTTP Server for master_cms (production)
Loaded: loaded (/home/comtechmaster/.config/systemd/user/puma_master_cms_production.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-11-18 22:31:02 UTC; 1h 18min ago
Main PID: 1577 (ruby)
Tasks: 10 (limit: 2338)
Memory: 125.1M
CPU: 2.873s
CGroup: /user.slice/user-1000.slice/user#1000.service/app.slice/puma_master_cms_production.service
└─1577 puma 5.5.2 (unix:///home/comtechmaster/apps/master_cms/shared/tmp/sockets/puma_master_cms_production.sock)
Nov 18 22:31:02 master-cms systemd[749]: Started Puma HTTP Server for master_cms (production).
The rails production.log is empty.
The puma error log shows the following
cat log/puma_error.log
=== puma startup: 2021-11-18 22:31:05 +0000 ===
The pid files exist in the application roots shared/tmp/pids folder
ls tmp/pids
puma.pid puma.state
and the socket that nginx needs but is unable to connect to due to permission denied exists
ls -l ~/apps/master_cms/shared/tmp/sockets/
total 0
srwxrwxrwx 1 comtechmaster comtechmaster 0 Nov 18 22:31 puma_master_cms_production.sock
nginx is up and running and providing a
502 bad gateway
response. The nginx error log reports the following error
2021/11/18 23:18:43 [crit] 1500#1500: *25 connect() to unix:/home/comtechmaster/apps/master_cms/shared/tmp/sockets/puma_master_cms_production.sock failed (13: Permission denied) while connecting to upstream, client: 86.160.191.54, server: 159.65.50.229, request: "GET / HTTP/2.0", upstream: "http://unix:/home/comtechmaster/apps/master_cms/shared/tmp/sockets/puma_master_cms_production.sock:/500.html"
sudo nginx -t reports the following
sudo nginx -t
nginx: [warn] could not build optimal proxy_headers_hash, you should increase either proxy_headers_hash_max_size: 512 or proxy_headers_hash_bucket_size: 64; ignoring proxy_headers_hash_bucket_size
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successfu
just to be pedantic both an ls and a sudo ls to the path reported in the error shows
ls /home/comtechmaster/apps/master_cms/shared/tmp/sockets/
puma_master_cms_production.sock
as expected so I am stumped to understand why nginx running as root using sudo service nginx start is being denied access to a socket that exists, that is owned by the local user rather than root.
I expect the solution is going to be something totally obvious but I can not see what
This problem ended up being related to the folder permissions for the users home folder and specifically a change in the way Ububntu 20.10 sets permissions differently to previous versions of ubuntu, or at least a difference in the way the DigitalOcean setup scripts behave.
This was resolved with a simple command line chmod o=rx from the /home against the user folder concerned e.g.
cd /home
chmod o=rx the_home_folder_for_user
I was try to mount a folder into "/usr/share/nginx/html/" and the Docker consoler shows an error of "[error] 28#28: *1 directory index of /usr/share/nginx/html/ is forbidden". I use this command to mounted volume "docker-share dilrukshi$ docker run -d -p 8080:80 --name web -v ~/docker-share/html:/usr/share/nginx/html nginx" and also I used nginx/1.19.3 Official NGINX Docker Image. In a web page show, a "403 Forbidden" error and also "/usr/share" folder doesn't have "/nginx/html". Wha's wrong with? and How can I fix it?
Docker consoler
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
172.17.0.1 - - [20/Oct/2020:07:09:41 +0000] "GET / HTTP/1.1" 403 555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36" "-"
2020/10/20 07:09:41 [error] 28#28: *1 directory index of "/usr/share/nginx/html/" is forbidden, client: 172.17.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost:8080"
Browser Error
it looks like a problem with the folder permissions. Try to execute the following:
chmod -R 755 ~/docker-share/html
When you map a host folder into the container, the files' ownership is maintained. e.g.
If you execute the following
docker run -it --rm -v "~/docker-share/html:/usr/share/nginx/html" nginx:alpine
You'll get something like this:
total 12
drwx--x--x 2 1000 1000 4096 Oct 20 07:48 .
drwxr-xr-x 3 root root 4096 Jan 9 2020 ..
-rwx--x--x 1 1000 1000 83 Oct 20 07:48 index.html
In my case the folder is owned by 1000 (in your case you'll find your uid). The nginx container will use the nginx user (uid: 101) for its workers.
I used docker-compose to run a local gitlab server.
git:
container_name: git-server
image: gitlab/gitlab-ce:latest
hostname: 'gitlab.example.com'
ports:
- '8090:80'
- '22:22'
volumes:
- "$PWD/srv/gitlab/config:/etc/gitlab"
- "$PWD/srv/gitlab/logs:/var/log/gitlab"
- "$PWD/srv/gitlab/data:/var/opt/gitlab"
networks:
- net
I want to setup custom-hooks for a project repo i created in the gitlab webUI so that it triggers a jenkins job. As per gitlab documentation, this is the path for repos in omnibus installations where i will have to create custom-hooks directory
/var/opt/gitlab/git-data/repositories/<group>/<project>.git
But inside of /var/opt/gitlab/git-data/repositories , I don't see a group directory or project directory at all
root#gitlab:~# ls -lt /var/opt/gitlab/git-data/repositories
total 0
drwxr-s---. 3 git root 16 Apr 18 04:05 #hashed
drwxr-sr-x. 3 git root 17 Apr 18 04:00 +gitaly
root#gitlab:~#
I tried searching using find. But it returned nothing. I tried searching by name of files in my project repo, but that didn't return anything as well.
In the gitlab webUI, I am able to see it all. But in the server, none of the file and dir exists.
How is it that I am not able to find any of the file in my repos when i ssh to gitlab-server?
Since I am not able to go this way, I tried by creating a post-receive.d directory under the global hooks directory /opt/gitlab/embedded/service/gitlab-shell/hooks and then adding my post-receive file as below
#!/bin/bash
# Get branch name from ref head
if ! [ -t 0 ]; then
read -a ref
fi
IFS='/' read -ra REF <<< "${ref[2]}"
branch="${REF[2]}"
if [ "$branch" == "master" ]; then
crumb=$(curl -u "jenkins:1234" -s 'http://jenkins:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
curl -u "jenkins:1234" -H "$crumb" -X POST http://jenkins:8080/job/maven/build?delay=0sec
if [ $? -eq 0 ] ; then
echo "*** Ok"
else
echo "*** Error"
fi
jenkins is the name of the container which is in the same network as that of gitlab server.
gitlab docs says then I will have to change permission of the file to git and then make it executable. I did so. But it didn't work either. Also, I find all of the git directories is owned by root in my container.
After pushing code, i figured the hook I put in the /opt/gitlab/embedded/service/gitlab-shell/hooks/post-receive.d directory is not working and in the logs, I see below error right after I push code changes to my maven repo
==> /var/log/gitlab/nginx/gitlab_error.log <==
2020/04/18 04:57:31 [crit] 832#0: *256 connect() to unix:/var/opt/gitlab/gitlab-workhorse/socket failed (13: Permission denied) while connecting to upstream, client: <my_public_ip>, server: gitlab.example.com, request: "GET /jenkins/maven.git/info/refs?service=git-receive-pack HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/jenkins/maven.git/info/refs?service=git-receive-pack", host: "gitlab.example.com:8090"
Here, gitlab.example.com is mapped to my public ip in the /etc/hosts file of my host on which I am running docker.
If you run the following command inside of the container you should see your group repos
gitlab-rake gitlab:storage:rollback_to_legacy
inside of /var/opt/gitlab/git-data/repositories , I don't see a group directory or project directory at all
The documentation "Install GitLab using docker-compose" includes the following volumes:
volumes:
- '$GITLAB_HOME/gitlab/config:/etc/gitlab'
- '$GITLAB_HOME/gitlab/logs:/var/log/gitlab'
- '$GITLAB_HOME/gitlab/data:/var/opt/gitlab'
That means if you see locally some repos in $GITLAB_HOME/gitlab/data/git-data/repositories, you should see the same in /var/opt/gitlab/git-data/repositories/.
Assuming, of course, that you have created at least one projet/repo in your GitLab instance.
I just started using Docker and still not used to it yet. I added images from https://github.com/markoshust/docker-magento
However when i tried to run magento setup page, i see these errors:
phpfpm_1 | - - 12/Jun/2018:18:49:04 +0000 "GET /setup/index.php/navigation" 403
phpfpm_1 | [12-Jun-2018 18:49:04] WARNING: [pool www] child 8 said into stderr: "NOTICE: Access to the script '/var/www/html/setup/index.php/navigation' has been denied (see security.limit_extensions)"
app_1 | 2018/06/12 18:49:04 [error] 7#7: *1 FastCGI sent in stderr: "Access to the script '/var/www/html/setup/index.php/navigation' has been denied (see security.limit_extensions)" while reading response header from upstream, client: 172.18.0.1, server: localhost, request: "GET /setup/index.php/navigation HTTP/1.1", upstream: "fastcgi://unix:/sock/docker.sock:", host: "magento2.test", referrer: "http://magento2.test/setup/"
A little search online suggests changing the config in www.conf but i'm not sure how to do that with Docker, i tried editing the image's template and deleted the containers and try creating the containers again but still getting these errors. How can i verify the configs are correct and if i'm doing it wrong how to correctly change the config files? Thanks
You may access the container using docker exec and do whatever you want.
Example:
docker container exec -it <container_id> bash
If still you don't have permissions to change folders or files, try using -u root.
Example:
docker container exec -u root -it <container_id> bash
You may also do a bind mount to the container folder.
Example:
docker container run -v /host/folder:/container/folder <image>
This will map all contents and put on your host for easy updating files.
I am getting the following errors in nginx.error.log file:
2017/01/19 23:51:57 [crit] 809#809: *1 connect() to
unix:///home/deploy/Production01/appname/shared/tmp/sockets/Production01-puma.sock
failed (2: No such file or
home/deploy/Production01/appname/shared/tmp/sockets/Production01-puma.sock:/axis-cgi/jpg/image.cgi",
host: "185.123.188.98", referrer: "1" 2017/01/19 23:51:57 [info]
809#809: *1 client 217.118.84.185 closed keepalive connection
and in the puma.log I have:
=== puma startup: 2017-01-19 23:53:00 +0200 ===
* Listening on unix:///home/deploy/apps/Production01/shared/tmp/sockets/Production01-puma.sock
deploy.rb
puma.rb
nginx.conf
I am using capistrano and everything is deployed correctly. On my local machine the demo project is working fine.
This is my first deploy and I guess I have not do something correctly in the config files.
Could you advice?
You need to symlink your nginx files.
Try this on your server
cd ~
sudo ln -nfs home/deploy/Production01/appname/current/config/nginx.conf /etc/nginx/sites-available/appname
sudo ln -nfs /etc/nginx/sites-available/appname /etc/nginx/sites-enabled/
sudo service nginx restart