I'm trying to set up continuous deployment of a hugo site using Jenkins on an Ubuntu 16.04 server. I've installed hugo using snap as was recommended on their site. I've been able to get the site working on my main user account, but when I try to have Jenkins do it, it hits this error:
cannot create user data directory: /var/lib/jenkins/snap/hugo/1766: Permission denied
Which seems a little odd because the Jenkins has write permissions to that folder, if I'm reading the permissions correctly:
jenkins#computer: ~$ ls -al /var/lib/jenkins/snap/hugo/1766
total 8
drwxr-xr-x 2 jenkins jenkins 4096 May 11 01:33 .
drwxr-xr-x 4 jenkins jenkins 4096 May 11 01:33 ..
Am I missing something here? Are there other things I need to do?
when I try to have Jenkins do it
Make sure the Jenkins job is actually running as jenkins: add a step in your job to print id -a.
Chances are: it is not running as Jenkins, but as your main account, which does not have the right to write in that folder.
"cannot create user data directory": it is possible 1766 is the id of the user account, created as the user.
Check if creating (as your main account) that 1766 folder, with a chmod 777 (for testing) helps.
Note hugo issue 3143 does mention:
Snaps are read-only for security. We want to prevent a hostile party from sneakily changing the software on your machine, so you cannot modify a snap that is installed on your system. This also means you can always check the signature on the snap, even long after you installed it, to make sure it is still exactly the software you intended. If you want to modify a snap, you can usually build your own version of it, especially if it is open source.
So where can a snap write data? Each snap gets its own set of writable directories which have specific properties. There are two directories which the snap can write to independent of the user. One of these is versioned - each time the snap is upgraded the data is saved and the new snap revision can upgrade its copy. The other ‘common’ data directory is not versioned and is used for big blobs of data that you don’t want to duplicate across revisions of the snap:
/var/snap/<name>/current/ ← $SNAP_DATA is the versioned snap data directory
/var/snap/<name>/common/ ← $SNAP_COMMON will not be versioned on upgrades
Typically, configuration is stored in one of these, along with system-wide data for the snap.
There are also an equivalent two writable directories for each snap in the user home, which can be used to store snap data that is specific to one user or another, separately:
~/snap/<name>/current/ ← $SNAP_USER_DATA that can be rolled back
~/snap/<name>/common/ ← $SNAP_USER_COMMON unversioned user-specific data
You can use the snap edition of Hugo to write (i.e. run hugo new site etc.) anywhere inside your $HOME directory, say ~/tmp/htest, but not in the system-wide /tmp directory.
That matches the "Hugo Install from snap" section:
Hugo-as-a-snap can write only inside the user’s $HOME directory—and gvfs-mounted directories owned by the user—because of Snaps’ confinement and security model.
Use sudo snap install hugo --classic to disable the default security model if you want hugo to be able to have write access in other paths besides the user’s $HOME directory.
I solved this ultimately by:
uninstalling Jenkins with sudo apt-get remove --purge jenkins
creating a standard user account named jenkins
reinstalling Jenkins and
setting the Jenkins Workspace Root Directory to /home/jenkins/workspace/${ITEM_FULL_NAME}
Could also probably be cured by making a directory /home/jenkins that is owned by the Jenkins profile
Related
I'm trying to figure out how it's supposed to work based on these two articles:
https://medium.com/#mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b
https://vsupalov.com/docker-shared-permissions/
I am setting up a dockerfile based on an image that has user Ubuntu already set (the subject of the first linked article above) so that the container runs by default under user ubuntu. This is adhering to best practice.
The problem I'm having is that the code directories COPYed in the dockerfile are all owned by root, and calling cmake .. required for the docker build fails because of this. I understand that the COPYs will by default run as root, and that even if I use the --chown flag with COPY, any parent directories implicitly created by the COPY would be owned by root regardless of any --chown flag used.
Doesn't the fact that the container already has a Ubuntu user mean that calling RUN adduser --uid 1000 ubuntu in the dockerfile (the suggestion from the second linked article above) would be problematic (it'd be at best redundant)?
Then that would mean that we would not want to RUN adduser so does this mean the only remaining option is to actually just sprinkle tons of chowns everywhere in the dockerfile? I refuse to do this.
By specifying USER ubuntu near the top of the dockerfile prior to any of the COPYs it appears to work to eliminate most of the required sudo chown calls. There will still be some required, if some part of the directory tree being copied to started out owned by root and need to be owrked with later.
When I attempt to create ssh keys on my Debian instance :
ssh-keygen -f /home/serverkeys/.ssh/id_rsa
I am getting the error : Saving key "/home/serverkeys/.ssh/id_rsa" failed: No such file or directory
I am using root user and when I check permissions on the directory :
ls -ld
I get
drwxr-xr-x 2 root root 4096 Oct 17 08:32 .
I have a number of questions :
1)What am I missing and why is the system not able to automatically create the .ssh directory specified ?
2)Do I need to create another user and switch from using root. I am not sure if the new user will inherit permissions on all packages I have so far since I have installed them with root user profile?
I intend to use the ssh keys for integration between Jenkins and Gitlab as outlined here Jenkins_Gitlab .In the example however the Jenkins instance and Gitlab instance are installed on different servers and yet for my setup they are installed on the same Debian instance and also my Gitlab repo is public.
I am unsure as to whether I still need the ssh key because Jenkins and Gitlab are running on same Debian instance eg would Jenkins still need ssh keys in order to checkout the code from Gitlab.
When you want to save your ssh key in a specified folder write the folder name without quotation that works for me.
Enter file in which to save the key (/home/rakiiibul/.ssh/id_rsa): /home/rakiiibul/Desktop/sshkey
Since you are already using a root user you don't need to create another user.here is screenshot of that
I am running Docker Toolbox v. 1.13.1a on Windows 7 Pro Service pack 1 x64OS.
with Virtual Box Version 5.1.14 r112924
when I try to run any docker image e.g. official postgres image from Docker Hub with volumes disabled, it works fine!
But when I enable the volumes it fails.
I tried all official documentations
The VM has shared folder as required and has full access to it also
shared folder screenshot
In case of my example of postgresql it crashes with following log
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... LOG: could not link file "pg_xlog/xlogtemp.27" to "pg_xlog/000000010000000000000001": Operation not permitted
FATAL: could not open file "pg_xlog/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data"
I know its the problem with folder permissions. But kinda stuck!
A ton of thanks in advance
I've been busy with this problem all day and my conclusion that it's currently simply not possible to run postgresql inside a docker container while keeping your data persistent in a separate volume.
I even tried running the container without linking to a volume and copying the data that was originally in /var/lib/postgresql into a folder of my host OS (Windows 10 Home), then copy that into the folder that got then linked to the container itself.
Alas, I got the next error:
FATAL: data directory "/var/lib/postgresql/data/pgadmin" has wrong ownership
HINT: The server must be started by the user that owns the data directory.
In conclusion: There's something going wrong with the ownership and the correct user owning it and to be able to fix it, you'll need a unix commandline on Windows that is able to run docker (something currently not possible with Bash on Ubuntu on Windows that is running using Ubuntu 16.04 binaries).
Maybe, in the future, you'll be able to run the needed commands (found here, under Arbitrary --user Notes), but these are *nix commands and powershell (started by Kitematic) can't run those. Bash for Ubuntu for Windows could run those, but that shell has no connection to the docker daemon/service on windows...
TL;DR: Lost a day of work: It is currently impossible on Windows.
I have been trying to fix this issue also ..
At first I thought it was a symlink problem (because the first error fails on " could not link .. operation not permitted)
To be sure symlink is permitted you have to :
share a folder in virtualbox
run virtualbox as administrator (if you account is in administrator group) Right click virtualbox.exe and select run as Administrator
if your account is not administrator, add the symlink privilege with secpol.msc > "Local Policies-User Rights Assignments" add your user to "Create symbolic links"
enable symlink for your shared folder in virtualbox :
VBoxManage setextradata VM_NAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/SHARED_FOLDER_NAME 1
Alternatively you can also use the c:\User\username folder which is shared and symlink enabled by default dockertools installation
Now I can create symlinks in the shared folder from the docker container .. but I still have the same error "could not link ... operation not permitted"
So the reason must be somewhere else ... in the file permissions as you said but I do not see why ?
I used this procedure to install Jenkins:
https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+RedHat+distributions
After it was up and running I discovered the /var/lib/jenkins partition on my server is very small. I want to move it, but I do not want to change the user that it runs under. I am new to Linux and I'm stumped. How do I move it for example to my Home/Public folder? The "Jenkins" user doesn't seem to have a Home folder. Its running as a daemon on startup, so I have no idea where to configure those settings.
Can I create a Home folder for the Jenkins user? How?
I read this article:
https://wiki.jenkins-ci.org/display/JENKINS/Administering+Jenkins
but do not understand HOW to "set the new Jenkins home". I have used the export command, and restarted the service, but the old path still shows up in the Manage Jenkins screens.
I've read the 2-3 similar questions on stackoverflow also, but there's always a big missing piece for me. Where to find that file where I change the path permanently?
Here's an easy way to solve your problem. First, move the Jenkins directory from /var/lib/jenkins to /home/jenkins. Then create a symlink at /var/lib/jenkins pointing to /home/jenkins. And of course, stop the Jenkins service before doing that.
sudo service jenkins stop
sudo mv /var/lib/jenkins /home
sudo ln -s /home/jenkins /var/lib/jenkins
sudo service jenkins start
I managed to change the home location for Jenkins by modifying content of /etc/sysconfig/jenkins file as follows:
JENKINS_HOME="/home/jenkins"
Okay, I reread your question a little bit more closely, lets see if we can figure this out. I am going to list some info that you may or may not know.
The jenkins installation and jenkins home are not the same thing. One is where the war file and other parts that jenkins needs to run live. jenkins_home is where your data is stored. By default, jenkins_home lives in ~/.jenkins. When you start jenkins, it looks for an environment variable to tell it where to find those files.
Jenkins runs as a seperate user, which, by default, is jenkins. This way it doesn't get in the way of you. The jenkins user will not have access to YOUR home directory, so that would be a poor solution. Ideally, it would have its own home directory, /home/jenkins. Your home directory could then be /home/jenkins/.jenkins. You say that folder doesn't exist- if you don't have access to it to create it yourself, that is perfectly fine, you can specify ANY folder. However, the jenkins user must have ownership of that folder to read and write to it.
It looks like Jenkins on redhat will be running with tomcat by default. The documentation for how to set environment variables for tomcat is https://wiki.jenkins-ci.org/display/JENKINS/Tomcat
This all gets set up with a script.https://wiki.jenkins-ci.org/display/JENKINS/JenkinsLinuxStartupScript seems to be the one that is used for this purpose. Even if you don't know anything about shell scripting, this isn't too hard... lines with a # are comments. The first line
JENKINS_USER=jenkins
sets the name of the user account jenkins will be using. Look down a littlle further, and you'll see the line
export JENKINS_BASEDIR=/home/jenkins
export CATALINA_OPTS="-DJENKINS_HOME=$JENKINS_BASEDIR/jenkins-home -Xmx512m -Djava.awt.headless=true"
This lets you set a directory to where jenkins should live, and then sets the jenkins_home directory to that /jenkins-home.
For your application, you may want to do something like this
export CATALINA_OPTS="-DJENKINS_HOME=/var/jenkinsmount/home -Xmx512m -Djava.awt.headless=true"
That would then store all of your build data (which is the part that grows!) at /var/jenkinsmount/home ... while leaving the rest of your files in their current location.
I haven't used it on redhat, but hopefully I explained enough for you to actually understand what is going on so that you can get it going!
Other INFO:
https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+as+a+Unix+daemon
I have faced the same issue and question.
Connecting some dots I could fix my Jenkins after I moved Jenkins to a new location due to the same issue -space in disk under /var/lib/jenkins.
Here is the procedures that I had to follow to get it working taking in consideration that I am pointing Jenkins to a non-default port. (I have applied this process into 2 servers)
First, move the Jenkins directory from /var/lib/jenkins to /opt/jenkins
sudo service jenkins stop
sudo mv /var/lib/jenkins /opt/
Now you can change your workspace and build directory to any other location on your machine.
Jenkins provides 3 predefined variables that can be used to specify the new location:
JENKINS_HOME — Jenkins home directory
ITEM_ROOTDIR — Root directory of a job for which the workspace is allocated
ITEM_FULLNAME — ‘/’-separated job name, like “foo/bar”
sudo su jenkins (access as Jenkins user)
JENKINS_HOME=/opt/jenkins
ITEM_ROOTDIR=/opt/jenkins
ITEM_FULLNAME=/opt/jenkins
exit (exit Jenkins user)
Now, edit the jenkins config
/opt/jenkins$ sudo nano /etc/default/jenkins
Modify the following line
#jenkins home location
#JENKINS_HOME=/var/lib/$NAME (here is the default)
JENKINS_HOME=/opt/jenkins (that is our new location)
change the home directory of a user
sudo usermod -d /opt/jenkins/ jenkins
sudo service jenkins start
If anyone is having issues with space and you have to relocate your jenkins, just wanted to reiterate that::
sudo service jenkins stop
sudo mv /var/lib/jenkins /home
sudo ln -s /home/jenkins /var/lib/jenkins
sudo service jenkins start
works great, so thanks to the person who posted that answer a few years ago!
Richard Chen's location is where I found the jenkins file on my CentOS 6.6 system.
sudo service jenkins stop
mv /var/lib/jenkins /home/mylocation/
(made sure the new location had correct ownership and group-- Jenkins)
modified the content of the file /etc/sysconfig/jenkins as follows:
JENKINS_HOME="/home/mylocation"
sudo service jenkins start
If all these don't work, then the only solution that works is to edit the following file
/lib/systemd/system/jenkins.service
then need to execute reloading of configs by running
systemctl daemon-reload
after this restarting the service would reflect the change.
Some commands work for me as below:
Step 1: Stop jenkin service and moving folder
systemctl stop jenkins
mv /var/lib/jenkins /whatever/folder
sudo chown jenkins -R /whatever/folder
Step 2: Modify jenkins home location in /etc/default/jenkins
JENKINS_HOME=/whatever/folder/$NAME
Step 3: Restart jenkins service
systemctl start jenkins
[A] Find your current jenkins home folder, if you are not aware where it is ?
Jenkins -> Manage Jenkins -> Configure System -> Check label 'Home directory'
[B] To move current Jenkins home folder to a new directory. Follow below steps :-
Stop Jenkins service - by killing the process
Follow one of below approach to set new home folder for JENKINS.
a) By default Jenkins home directory is set to ~/.jenkins
b) "JENKINS_HOME" environment variable setup in operating system.
c) "JENKINS_HOME" entry in JNDI environment.
d) "JENKINS_HOME" system property to the servlet container.
Tomcat context descriptor of the servlet, you can set below field in apache-tomcat-8.5.28/conf/context.xml :
<Context ...>
<Environment name="JENKINS_HOME" value="/path/to/jenkins_home/" type="java.lang.String"/>
</Context>
e) If jenkins.war file is deployed in a tomcat server , then even appending below content in bin/catalina.sh will setup JENKINS_HOME.
CATALINA_OPTS="-DJENKINS_HOME=/path_to/jenkins_home/"
Manually copy Jenkins home folder content from old to new home
folder. (use cp command) . Instead of moving, copy step is advised to keep one backup. Later you can delete old workspace.
Now start Jenkins, then It will pick the new home directory
from the path mentioned in JENKINS_HOME variable.
Note: - Just by setting above variable "JENKINS_HOME" to a different path will not copy
the files from current Jenkins home path to new one automatically. This copy step - you have to do it yourself, Manually.
I'm trying to install Jenkins on a Tomcat 7 container.
When I try to open the Jenkins web app I get following error:
Unable to create the home directory '/home/myuser/jenkins/work'. This is most
likely a permission problem.
To change the home directory, use JENKINS_HOME environment variable or set
the JENKINS_HOME system property. See Container-specific documentation for
more details of how to do this.
Before starting Tomcat, I did chmod uog+rwx /home/myuser/jenkins. So, I suppose that Jenkins should be able to create a subdirectory there.
But obviously it can't.
How can I fix this problem?
Update 1:
lt -lt returns
drwxrwxrwx 2 root ec2-user 4096 Jun 23 10:25 jenkins
for /home/myuser/jenkins. /home/myuser/jenkins/work doesn't exist because Jenkins is supposed to create it.
Update 2: Just tried to create the work directory and to run chmod uog+rwx on it. It didn't help.
Update 3: Additional information:
I need Jenkins in order to
run lengthy tests in the night (fast unit tests are run before every mvn install, slow tests are executed every night) and
save software quality metrics (checkstyle, PMD, FindBugs, unit test coverage etc.) over time.
I have only one machine available for that and there is a Tomcat7 container installed there already.
At the moment, I don't want to invest additional money into buying new machines.
The machine with the Tomcat7 container (and where I want Jenkins to be installed) is an Amazon EC2 microinstance (OS version is given below).
$ cat /etc/*-release
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Amazon Linux AMI release 2013.03
Update 4 (29.06.2013 13:34 MSK): The output of yum list does not contain any Jenkins/Hudson package.
If Tomcat is running as a separate user you will need to give execute permission to your home directory to that user - either by giving it to all or by creating a group especially for you and the tomcat user.
(UPDATE) More specifically: You say you already did chmod uog+rwx /home/myuser/jenkins, if Tomcat is not running asl 'myuser' it also needs execute permission on /home and on /home/myuser to be able to open /home/myuser/jenkins. If you are not picky about other users on the system opening your homedir you could allow this by: chmod a+x /home/myuser. (I'm assuming here the permissions for /home are already ok)
If you are running tomcat as 'myuser' the filsystem permissions look fine, but Tomcat's own permission system might be the problem as webapps are not allowed to touch the filesystem if the default settings of the security manager are on.
See: https://wiki.jenkins-ci.org/display/JENKINS/Tomcat
You don't specify more about your exact Tomcat/OS setup so I can't give exact details, but the fast way to find out if it's a security manager issue is to give AllPermission to you webapp. If you don't run in a safe environment it is advisable to only use that as a test, and setup only the really needed permissions later.
run these three commands
cd /usr/share/tomcat7
sudo mkdir .jenkins
sudo chown tomcat7:nogroup .jenkins
https://seleniumwithjavapython.wordpress.com/home/jenkins-installation/
It looks like the problem may be that jenkins cannot see /home/myuser, and therefore it cannot access the jenkins folder inside this (even though it has write permissions in /home/myuser/jenkins, I believe the fact it can't read /home/myuser causes a problem).
Try running the below command and then see if Jenkins works after that:
chmod +r /home/myuser
#robjohncox Yes - drwx------ 5 myuser myuser 4096 Jun 23 10:25 myuser
you must add +x to this dir to make it possible for jenkins to access it's contents, to be precise whole path has to have +x enabled for everyone.
Also, what commands have you used to move it's home dir from default - possible error is somwhere there. Cheers, Piotr