How can I make jenkins run "pip install"? - jenkins

Question

I have a git repo and would like to get jenkins to clone it then run
virtualenv venv --distribute
/bin/bash venv/source/activate
pip install -r requirements.txt
python tests.py
The console output from jenkins is:
+ virtualenv venv --distribute
New python executable in venv/bin/python
Installing distribute..........................done.
Installing pip...............done.
+ /bin/bash venv/bin/activate
+ pip install -r requirements.txt
Downloading/unpacking flask (from -r requirements.txt (line 1))
Running setup.py egg_info for package flask
SNIP
creating /usr/local/lib/python2.7/dist-packages/flask
error: could not create '/usr/local/lib/python2.7/dist-packages/flask': Permission denied
----------------------------------------
Command /usr/bin/python -c "import setuptools;__file__='/var/lib/jenkins/workspace/infatics-website/build/flask/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-hkdBAi-record/install-record.txt failed with error code 1
Storing complete log in /home/jenkins/.pip/pip.log
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I've tried adding sudo before the command, but it's doesn't work either:
+ sudo pip install -r requirements.txt
sudo: no tty present and no askpass program specified
Sorry, try again.
sudo: no tty present and no askpass program specified
Sorry, try again.
sudo: no tty present and no askpass program specified
Sorry, try again.
sudo: 3 incorrect password attempts
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Any ideas how to get around this? Also when I run pip install -r requirement.txt in a terminal as the jenkins user it doesn't need sudo permission. Can I get jenkins (the process) to run as the jenkins user?

Answer 1

The fact that you have to run use sudo to run pip is a big warning that your virtual environment isn't working. The build output shows that pip is installing the requirements in your system site-packages directory, which is not the way virtualenv works.
Your build script doesn't actually preserve the activated virtual environment. The environment variables set by the activate script are set in a child bash process and are not propagated up to the build script. You should source the activate script instead of running a separate shell:
virtualenv venv --distribute
. venv/bin/activate
pip install -r requirements.txt
python tests.py
If you're running this as one build step, that should work (and install your packages in venv). If you want to add more steps, you'll need to set the PATH environment variable in your other steps. You're probably better off providing full paths to pip and python to ensure you're not dependent on system package installations.

Answer 2

Try using
stage('test') {
agent {
docker {
image 'qnib/pytest'
}
}
steps {
sh 'virtualenv venv && . venv/bin/activate && pip install -r requirements.txt && python tests.py'
}
}

Answer 3

I totally agree what has been said.... But to make this more "Jenkins"
Create a basic Project - and in the custom steps do something like this
PROJECT="Tree"
rm -Rf ~/Builds/$PROJECT
CODE_HOME=~/Builds/$PROJECT/code
PYENV_HOME=~/Builds/$PROJECT/python
export PYENV_HOME
export PYTHONPATH=""
echo "Creating new Python env"
/usr/local/bin/python3 -m venv --clear $PYENV_HOME
source $PYENV_HOME/bin/activate
echo "Get Project"
mkdir -p $CODE_HOME
cd $CODE_HOME
git clone https://github.com/MyUsername/MyTree.git .
pip install --upgrade pip
pip install nose
pip install coverage
pip install -r requirements.txt
python setup.py build
python setup.py install
After this you can then do you nose tests etc...

Answer 4

Here is what I did to make Jenkins execute pip install requirements on Windows machine:
In "Execute Windows batch command" pipeline
REM activate venv, update pip and install package
cmd /k "cd <path to your directory like C:\WebAPI> & .\venv\Scripts\activate.bat & python -m pip install -U pip & pip install -r .\requirements.txt"
cmd /k executes windows command prompt and you can add any commands there. & is a pipeline operator. So you can have multiple commands