I started a RoR project. I use docker to easily install what I need (database, external tools...). In the root of my project, I've got a data folder containing all the docker volumes. The issue is when I run rails generate controller X, I don't know why but rails wants to access to the data folder (which is completely useless) and so, I have a permission denied for the data folder;
ruby-2.5.1/gems/rb-inotify-0.9.10/lib/rb-inotify/notifier.rb:192:in `initialize': Permission denied # dir_initialize - /home/mcdostone/X/Y/data/pgadmin/storage/root (Errno::EACCES)
from ruby-2.5.1/gems/rb-inotify-0.9.10/lib/rb-inotify/notifier.rb:192:in `new'
ll -ah /home/mcdostone/X/Y/data/
total 16K
drwxr-xr-x 4 root root 4.0K Aug 21 10:52 ./
drwxr-xr-x 17 mcdostone mcdostone 4.0K Aug 21 12:59 ../
drwxr-xr-x 4 root root 4.0K Aug 21 10:54 pgadmin/
drwx------ 19 999 root 4.0K Aug 21 10:52 postgres/
Any idea to avoid that unless changing the folder's permissions ? I don't want to break my containers.
There are three possible solutions for this problem:
Simply move the directory into the parent directory.
Change your Docker setup so that it uses the same user id that your host system uses (you can control this on some images via environment variables)
The gem rb-inotify cannot exclude properly. It is a known (but closed) issue (48). So you could fix / add the behaviour so that folders with permission errors will be simply ignored or that you can control ignored folders. I'm sure they will happily accept a PR.
Related
I have an azure web app with a custom container running apache, php 7.4 with laravel and a mounted azure file storage.
Serving .png files from a mounted storage does work for my local build in docker desktop and it works on a testystem on a physical machine.
Storage is mounted to /var/www/data_persistent and laravel serves data from /var/www/api/public. To preserve the data that is displayed images are written to the storage to be served.
I have a link from inside the public folder to one in the storage
ln -fs /var/www/data_persistent/maps /var/www/api/public/maps
On Azure, if I request an image in any folder of maps I get a 400 response, if the file is completely empty it works.
this is how it looks on the webapp:
total 28
4 drwxr-xr-x 2 root root 4096 Aug 11 13:17 css
0 -rwxr-xr-x 1 root root 0 Jan 28 2022 favicon.ico
4 drwxr-xr-x 3 root root 4096 Aug 11 13:17 images
4 -rwxr-xr-x 1 root root 1785 Feb 4 2022 index.php
4 drwxr-xr-x 2 root root 4096 Aug 11 13:17 js
0 lrwxrwxrwx 1 root root 29 Aug 11 13:20 maps -> /var/www/data_persistent/maps
and this is inside maps:
total 29943
0 drwxrwxrwx 2 nobody nogroup 0 Jul 10 23:41 .
0 drwxrwxrwx 2 nobody nogroup 0 May 29 16:23 ..
0 drwxrwxrwx 2 nobody nogroup 0 Jul 28 15:36 1
2715 -rwxrwxrwx 1 nobody nogroup 2779239 Jul 28 15:47 1.tif
0 drwxrwxrwx 2 nobody nogroup 0 Jul 28 12:34 7
27229 -rwxrwxrwx 1 nobody nogroup 27881853 Jul 28 12:43 7.tif
Writing to the file storage works, the folders and files you see are the result of uploads from frontend and a script.
Any help is appreciated.
UPDATE 17.08.2022:
I have tried quite some options, like
EnableSendfile off
in the apache conf in sites-enabled but no luck until now.
Interestingly the Apache log show status code 200 when serving the images.
ALSO serving html files does work from the mounted storage, but only if they are valid html content (!).
The problem seems to be the setting EnableMMAP in apache.
Add EnableMMAP Off to <Directory "${APACHE_DOCUMENT_ROOT}"> in apache2.conf
Example for this block of /etc/apache2/apache2.conf
<Directory "${APACHE_DOCUMENT_ROOT}">
Options Indexes FollowSymLinks
EnableMMAP Off
AllowOverride None
Require all granted
</Directory>
Source: https://azureossd.github.io/2020/09/15/unable-to-download-static-content-php-on-azure-app-service-linux/
I had the same root issue so thanks for sharing your solution. Azure support pointed me here to resolve my issue. I'll share additional information for future reference and hopefully to help others.
In my case I just added this line to the Apache config file because all my DocumentRoot is in /home:
EnableMMAP Off
I didn't test it, but enabling EnableSendfile probably also doesn't work based on the Apache docs.
Important performance note
I got very low performance with this configuration:
Linux Docker container
Website (PHP) stored and served from the persistent /home dir
The same website stored in the Docker image instead of in /home gave a 6x request time speedup on the B1 plan. A B3 plan showed a 1.5x speedup. I didn't test other web app plans.
Investigation
My starting point with this problem was that I wanted to use a single Docker image to host multiple PHP websites instead of storing the websites inside separate Docker images. The Web App persistent storage seemed like a great fit for this.
The symptoms I noticed when my website was hosted in /home were:
HTML and JS files worked fine
image files returned status 400
I increased the log level of Apache with LogLevel trace8 and noticed that files that served through mod_deflate worked fine while those that were served directly didn't. Disabling mod_deflate made the HTML and JS files also fail with status 400. Using mod_deflate means that files aren't served directly from disk and that the
Unfortunately I didn't connect the dots myself and I spent a lot of time trying various combinations of Docker images and Apache configurations...
EnableMMAP is a performance optimization however so it might be worthwhile to actually store the different web apps in separate Docker images instead of using a single generic one with an external mount point.
I'm also considering using Nginx instead of Apache to see if that changes anything:
https://learn.microsoft.com/en-us/answers/questions/835092/app-service-stack-image.html
https://techcommunity.microsoft.com/t5/apps-on-azure-blog/use-php-fpm-log-to-troubleshoot-php-8-linux-azure-app-service/ba-p/3275136
This is my jenkinx/jenkins_home/workspace folder looks like. (while doing ls -la)
drwxrwxrwx 24 nfsnobody nfsnobody 4096 Sep 29 18:26 workspace
There is a folder inside this workspace. This folder was created by Jenkins automatically when i build a job. This job name is Sandbox_Test-Job
Here's the folder
drwxr-xr-x 2 nfsnobody nfsnobody 4096 Sep 29 18:26 Sandbox_Test-Job
As you can see host machine's user does not have write permission to this folder and the script in the host machine is unable to write to Sandbox_Test-Vinod_M
I will have to manually set the permission first for this folder before the script can write. How can we make sure that when Jenkins create this job folder for each job, the folder has to have write permission for the user in the host?
First, you want to run ls -n to find the real UID/GID of the files/dirs instead of the display names. Next, check to see if that user appears in your /etc/passwd
nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
You would need to find the displayed UID in ls -n, not nfsnobody - 65534.
It's unlikely "nfsbody is the owner of the files ( RHEL NFS reference , Linux Home Server HOWTO - Fedora ), more likely the files were written to an nfs volume shared across systems where the UID for jenkins ( run id jenkins) is not the same across them.
Align the UIDs (non-trivial as you must fix the passwd plus the existing ownership UID) and things will then be OK.
Suggested reading from SUSE and ServerFault.
If you're lucky - all the files have one UID and you just need to sync the UID in the passwds, or maybe blow them all away if it's just the workspaces.
ps: Not really a Jenkins issue, better guidance to be found on Serverfault or SuperUser.
pps: there is some help on S/O worth reading as well (search "nfsnobody'):
nfsnobody User Privileges, chown: invalid user: ‘nfsnobody’ in fedora 32 after install nfs
I have a jenkins running on CentOS 8 and a simple job who should write something in a text file.
echo "Hello" > /home/dev/Git/Test.txt
The target directory should be Git:
[dev#h2899618 ~]$ ls -l /home/dev/
insgesamt 12
drwxrwxr-x 3 dev dev 4096 7. Okt 20:54 Git
The user dev has the following permissions:
[dev#h2899618 ~]$ /usr/bin/id
uid=1002(dev) gid=1002(dev) Gruppen=1002(dev),10(wheel),48(apache),991(jenkins),994(docker)
Jenkins uses the following permissions (validated with executing /usr/bin/id inside of the job):
uid=996(jenkins) gid=991(jenkins) groups=991(jenkins)
The job failed with the message:
[Test4Jenkins] $ /bin/sh -xe /tmp/jenkins3692200929413510467.sh
+ /usr/bin/id
uid=996(jenkins) gid=991(jenkins) groups=991(jenkins)
+ echo Hallo
/tmp/jenkins3692200929413510467.sh: line 3: /home/dev/Git/Test.txt: Permission denied
So why jenkins hasn´t the permission to write this directory?
So, you have drwxrwxr-x 3 dev dev 4096 permissions which means that user dev can do whatever, users that are in the group dev can do whatever, others, including jenkins do not have write bit and cannot modify directory content. However, adding this bit by executing chmod o+w /home/dev/ will resolve the issue nut in the worst way because you are giving to one user permissions on the home folder of another user. Better use some dedicated directory, e.g. /var/lib/share. Or even better try to achioeve your goals in such a way that does not require Jenkins to access anything but it's own home folder =) good luck!
P.S. you probably wanted to add jenkins user to group dev, not dev to jenkins user
I have an application running inside docker which programmatically creates files. Most of the time files are created properly and then deleted. Sometimes a file with unknown permission is created
drwxrwxrwx 0 root root 4096 Apr 17 21:41 ..
-????????? ? ? ? ? ? __CG__NarwhalLayoutDomainEntitiesTheme.php
-rwxrwxrwx 1 root root 8314 Apr 19 12:46 __CG__NarwhalLayoutDomainEntitiesTheme.php.5ad802fc34f6b0.60147712
When that happens though I can see the file in Windows Explorer when I try to delete it I get this error:
"You need permission to perform this action"
"You require permission from the computer's administrator to make changes to this file"
I even tried as administrator but it still won't work and when I try to delete the file from inside the container shell. I get this error:
rm: cannot remove '__CG__NarwhalLayoutDomainEntitiesTheme.php': No such file or directory
This problem goes away when I restart the docker but I don't want to restart docker every few minutes.
Looks like this is an ongoing problem with docker for windows. I found a temporary solution here https://github.com/docker/for-win/issues/525
I fixed the problem by simply pointing my doctrine proxies to a folder inside the container such as /tmp and is not mounted.
I have a Rake at the end of which the server moves the processed file from a folder to another one, using FileUtils.mv, like this:
FileUtils.mv('/path-to-upload-folder/'+filename, '/path-to-imported-folder/'+filename) if File.exist?('/path-to-upload-folder/'+filename)
If i run this command from within the rails server (I have an action that is a copy of the rake task, just for simplicity of debugging inside a controller), everything goes fine (probably because I run the server with root privileges with rvmsudo).
When running from the Rake task, I get a permission denied error, like this
Errno::EACCES: Permission denied # sys_fail2 -
The source folder is called uploads, and the destination folder is the imported folder. Following the permissions and the user and groups of the folder
drwxr-xr-x 2 malatini malatini 4096 lug 14 14:26 imported/
drwxr-xr-x 2 www-data www-data 135168 lug 14 14:26 uploads/
where malatini is my current user.
I know that for running a raw mv from the two folders I need to be a sudo user, but why can I run the same command from within the Rails servers without any problem?
I also tried to change permissions and owners/groups of the destination folder, but with no luck.
Reading here I suppose the problem is the user that is running the rake task. The same problem happens either if I manually run the rake task, either if it is run as a cron job.
I am running under
Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
Any suggestion?
EDIT:
As #Nic Nilov suggested I tried to change owner and group of source and destination foleders, and actually I managed to perform the mv, and hence rake task, changing owner and group of both folders to malatini
drwxr-xr-x 2 malatini malatini 4096 lug 14 14:26 imported/
drwxr-xr-x 2 malatini malatini 135168 lug 14 14:26 uploads/
but in this way, apache (which is in charge of moving files to uploads folder) is not able to write to uploads folder. No other configuration is working (not just the group of uploads folder, nor the 777 to imported folder).
You basically answered your own question. This is a permission issue, related to the OS user under which the move operation is attempted. It works from Rails since, as you say, you run it with rvmsudo.
Two ways around that would be to either run your rake task from a privileged user or set ownership on both folders such that the current user you run rake under is allowed to perform mv.
On your folders both user and group set to be the same. You could set their group to the group of the user running rake, e.g.:
chgrp malatini ./uploads
This would make malatini group the owner of both folders:
drwxr-xr-x 2 malatini malatini 4096 lug 14 14:26 imported/
drwxr-xr-x 2 www-data malatini 135168 lug 14 14:26 uploads/
Which should allow the mv operation.
UPDATE
When running rake under a privileged user and doing that from cron and to avoid keeping the password stored anywhere you can use NOPASSWD directive.
See for more details this askubuntu answer.