So I was trying to deploy a simple CD pipeline using docker by ssh’ing into my AWS Linux EC2 instance in the WSL2 terminal. The job is failing every time returning the following error:
Started by user Navdeep Singh Running as SYSTEM Building on the
built-in node in workspace /var/lib/jenkins/workspace/todo-dev
[todo-dev] $ /bin/sh -xe /tmp/jenkins6737039323529850559.sh + cd
/home/ubuntu/project/django-todo /tmp/jenkins6737039323529850559.sh:
2: cd: can’t cd to /home/ubuntu/project/django-todo Build step
‘Execute shell’ marked build as failure Finished: FAILURE
DockerFile contents:
FROM python:3 RUN pip install django==3.2
COPY . .
RUN python manage.py migrate
CMD [“python”,“manage.py”,“runserver”,“0.0.0.0:8000”]
Everything goes fine. This error cd: can’t cd to /home/ubuntu/project/django-todo Build step ‘Execute shell’ marked build as failure Finished: FAILURE is not an actual.
Your agent Node is not online.
To fix the problem, find commands on your jenkins web page after an agent setup. You need to run those commands from your terminal. See the screenshot for more details.
Make sure that your jenkins public IP and node agent public IP are the same. If an error occurs, you need to run some commands on the terminal. This is not a real error.
this issue follow this step which i give you
For Agent--->
change your ip here(44.203.138.174:8080) to your EC2 ip
1.curl -sO http://44.203.138.174:8080/jnlpJars/agent.jar
2.java -jar agent.jar -jnlpUrl http://44.203.138.174:8080/manage/computer/todo%2Dagent/jenkins-agent.jnlp -secret beb62de0f81bfd06e4cd81d1b896d85d38f82b87b21ef8baef3389e651c9f72c -workDir "/home/ubuntu"
For JOb --->
sudo vi /etc/sudoers
then add this command below root access in sudoers file
jenkins ALL=(ALL) NOPASSWD: ALL
3.then goto the ubuntu directory using cd .. then run this codes
grep ^ubuntu /etc/group
id jenkins
sudo adduser jenkins ubuntu
grep ^ubuntu /etc/group
4.restart the jenkins relogin
sudo systemctl stop jenkins
then you good to go
sshpass -p Naveen#1604 scp -r /var/lib/jenkins/workspace/fastpostclose stellar#192.168.25.154:/var/www/html/fastpostclose
Build step 'Execute shell' marked build as failure
Finished: FAILURE
im using this command for moving files from jenkins to ubuntu server but it is showing build failure
I am setting up a CI workflow with Jenkins declarative pipeline and Docker-for-Windows agents through Dockerfile.
Note: It is unfortunately currently not a solution to use a Linux-based docker daemon, since I need to run Windows binaries.
Setup: Jenkins master runs on Linux 16.04 through Docker. Jenkins build agent is
Windows 10 Enterprise 1709 (16299.551)
Docker-for-Windows 17.12.0-ce
Docker 18.x gave me headaches when trying to use Windows Containers, so I rolled back to 17.x. I still had some issues when trying to run with Jenkins and nohup not being on path, but it was solved by adding Git binaries to Windows search path (another reference). I suspect my current issue may be related.
Code: I am trying to initialize a Jenkinsfile and run a simple hello-world-printout within.
/Jenkinsfile
pipeline {
agent none
stages {
stage('Docker Test') {
agent {
dockerfile {
filename 'Dockerfile'
label 'windocker'
}
}
steps {
println 'Hello, World!'
}
}
}
}
/Dockerfile
FROM python:3.7-windowsservercore
RUN python -m pip install --upgrade pip
Basically, this should be a clean image that simply prints "Hello, World!". But it fails on Jenkins!
Output from the log:
[C:\jenkins\workspace\dockerfilecd4c215a] Running shell script
+ docker build -t cbe5e0bb1fa45f7ec37a2b15566f84aa9bd08f5d -f Dockerfile .
Sending build context to Docker daemon 337.4kB
Step 1/2 : FROM python:3.7-windowsservercore
---> 340689b75c39
Step 2/2 : RUN python -m pip install --upgrade pip
---> Using cache
---> a93f446a877f
Successfully built a93f446a877f
Successfully tagged cbe5e0bb1fa45f7ec37a2b15566f84aa9bd08f5d:latest
[C:\jenkins\workspace\dockerfilecd4c215a] Running shell script
+ docker inspect -f . cbe5e0bb1fa45f7ec37a2b15566f84aa9bd08f5d
.
Cannot run program "id": CreateProcess error=2, The system cannot find the file specified
The issue is, that windows is not supported at the moment. It is calling the linux "id" command to get the current user id.
There is an open Pull Request and JIRA Ticket at Jenkins to support Windows docker pipeline:
https://issues.jenkins-ci.org/browse/JENKINS-36776
https://github.com/jenkinsci/docker-workflow-plugin/pull/148
I am trying to build a jenkins job with below lines of code and it is failing.
cd ../../../../../
cd /home/skasturi/sdk/
cp -pr 8.0_Sprint2 8.0_Sprint3
Console output of jenkins build:
Building in workspace /var/lib/jenkins/workspace/SMGR_SDK
[SMGR_SDK] $ /bin/sh -xe /tmp/jenkins3461562069624051922.sh
+ cd ../../../../../
+ cd /home/skasturi/sdk/
/tmp/jenkins3461562069624051922.sh: line 3: cd: /home/skasturi/sdk/: Not a directory
Build step 'Execute shell' marked build as failure
Finished: FAILURE
What might be the cause of the observer behavior?
I am using a Makefile to provide consistent single commands for setting up a virtualenv, running tests, etc. I have configured my Jenkins instance to pull from a mercurial repo and then run "make virtualenv", which does this:
virtualenv --python=/usr/bin/python2.7 --no-site-packages . && . ./bin/activate && pip install -r requirements.txt
But for some reason it insists on using the system-installed pip and trying to install my package dependencies in the system site-packages rather than the virtualenv:
error: could not create '/usr/local/lib/python2.7/dist-packages/flask': Permission denied
If I add some debugging commands and explicitly point to the pip in my virtualenv, things get even more confusing:
virtualenv --python=/usr/bin/python2.7 --no-site-packages . && . ./bin/activate && ls -l bin && which pip && pwd && ./bin/pip install -r requirements.txt
Which generates the following output:
New python executable in ./bin/python2.7
Not overwriting existing python script ./bin/python (you must use ./bin/python2.7)
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python2.7
It appears Jenkins doesn't rebuild the environment from scratch for each build, which strikes me as an odd choice, but shouldn't effect my immediate issue
The output from the "ls -l bin" shows pip to be installed in the virtualenv and executable:
-rw-r--r-- 1 jenkins jenkins 2248 Apr 9 21:14 activate
-rw-r--r-- 1 jenkins jenkins 1304 Apr 9 21:14 activate.csh
-rw-r--r-- 1 jenkins jenkins 2517 Apr 9 21:14 activate.fish
-rw-r--r-- 1 jenkins jenkins 1129 Apr 9 21:14 activate_this.py
-rwxr-xr-x 1 jenkins jenkins 278 Apr 9 21:14 easy_install
-rwxr-xr-x 1 jenkins jenkins 278 Apr 9 21:14 easy_install-2.7
-rwxr-xr-x 1 jenkins jenkins 250 Apr 9 21:14 pip
-rwxr-xr-x 1 jenkins jenkins 250 Apr 9 21:14 pip2
-rwxr-xr-x 1 jenkins jenkins 250 Apr 9 21:14 pip2.7
lrwxrwxrwx 1 jenkins jenkins 9 Apr 10 19:31 python -> python2.7
lrwxrwxrwx 1 jenkins jenkins 9 Apr 10 19:31 python2 -> python2.7
-rwxr-xr-x 1 jenkins jenkins 3349512 Apr 10 19:31 python2.7
The output of "which pip" seems to want to use the correct one:
/var/lib/jenkins/jobs/Run Tests/workspace/bin/pip
My current working directory is what I expect it to be:
/var/lib/jenkins/jobs/Run Tests/workspace
But... wtf?
/bin/sh: 1: ./bin/pip: Permission denied
make: *** [virtualenv] Error 126
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I have been using python virtualenvs with Jenkins every day in the last two years, at multiple companies and for small side projects and cannot say I found "THE" answer. Still, I hope that sharing my experience will help others save time. Hopefully I will get further feedback in order to make the decision easier.
Avoid ShiningPanda - it's not well maintained, incompatible with Jenkins2 pipelines and prevents execution of jobs in parallel. Also it has the bad habit of leaving orphan environments on disk.
DIY via bash and virtualenv is my current favourite. Create it inside $WORKSPACE and, if not always cleaning, run relocatable before activating them. This is because jenkins workspace folder disk location can change between executions of job N and N+1.
If you use multiple builders that do need the same virtualenv, the easiest way is to dump your environment to a file and source it at the beginning of the new builder.
To ease the maintenance I am planning to investigate these:
direnvm
virtualenv-wrapper (mkvirtualenv)
pyenv
If you hit the shebang command line limits the best thing to do is to change your jenkins home directory to just /j.
Jenkins pipelines can be made to run with virtual environments but there are multiple things to consider.
The default shell that Jenkins uses is /bin/sh - this is configurable in Manage Jenkins -> Configure System -> Shell -> Shell executable. Setting this to /bin/bash will make source work.
An activated venv simply changes environment variables, and environment variables do not persist between stages in jenkins. See withEnv
If you are using version controlled multibranch pipelines jenkins creates a workspace with the branch name and a commit hash in the path - which can be quite long. venv scripts (e.g. pip) all start with a hashbang line which includes the full path to the python interpreter in the venv (the python interpreter itself is a symlink). E.g.,
~/workspace/ink_feature-use-jenkinsfile-VGRPYD53GGGDDSBIJDLSUDYPJ34QR63ITGMC5VJNB56W6ID244AA/env/bin$ cat pip
#!/var/jenkins_home/workspace/ink_feature-use-jenkinsfile-VGRPYD53GGGDDSBIJDLSUDYPJ34QR63ITGMC5VJNB56W6ID244AA/env/bin/python3.5
Bash only reads the first N characters of any executable file - which I found did not quite include the full venv path:
bash: ./pip: /var/jenkins_home/workspace/ink_feature-use-jenkinsfile-VGRPYD53GGGDDSBIJDLSU: bad interpreter: No such file or directory
This particular problem can be avoided by executing the script with Python instead. E.g. python3.5 ./pip
I'd recommend avoiding ShiningPanda.
I set up my virtual environments with Anaconda/Miniconda. When installing conda make sure you're running as jenkins user.
your_user#$ sudo -u jenkins sh
jenkins#$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
jenkins#$ bash Miniconda3-latest-Linux-x86_64.sh
Since Jenkins runs sh rather than bash, I added conda path to /etc/profile:
export PATH="/var/lib/jenkins/miniconda3/bin:$PATH"
Then in Jenkinsfile you can create and delete conda environments. Here's an example that creates a new environment for each build:
pipeline {
agent any
stages {
stage('Unit tests') {
steps {
sh '''
conda create --yes -n ${BUILD_TAG} python
source activate ${BUILD_TAG}
// example of unit test with nose2
pip install nose2
nose2
'''
}
}
}
post {
always {
sh 'conda remove --yes -n ${BUILD_TAG} --all'
}
}
}
I've got same problem. As I can see - you project named 'Run Tests'. So, this name contain space. That was the problem for me. I just renamed project, like RunTests - and venv working now! Attention - jenkins ask you about confirmation renaming project.
There are some issues with venv-python plugin with different OS environments.
Here is how I call python method manually. Not best practice but it work.
// Put this stage on top of pipeline
stage('Prepare venv') {
steps {
script {
if (isUnix()) {
env.ISUNIX = "TRUE" // cache isUnix() function to prevent blueocean show too many duplicate step (Checks if running on a Unix-like node) in python function below
sh 'python3 -m venv pyenv'
PYTHON_PATH = sh(script: 'echo ${WORKSPACE}/pyenv/bin/', returnStdout: true).trim()
}
else {
env.ISUNIX = "FALSE"
powershell(script:"py -3 -m venv pyenv") // windows not allow call python3.exe with venv. https://github.com/msys2/MINGW-packages/issues/5001
PYTHON_PATH = sh(script: 'echo ${WORKSPACE}/pyenv/Scripts/', returnStdout: true).trim()
}
try {
// Sometime agent with older pip version can cause error due to non compatible plugin.
Python("-m pip install --upgrade pip")
}
catch (ignore) { } // update pip always return false when already lastest version
// After this you can call Python() anywhere from pipeline
Python("-m pip install -r requirements.txt")
}
}
}
// Several plugins like WithPyenv is not working perfectly accross platform when using Virtual Env.
// Put this method outside pipeline
def Python(String command) {
if (env.ISUNIX == "TRUE") {
sh script:"source ${WORKSPACE}/pyenv/bin/activate && python ${command}", label: "python ${command}"
}
else {
powershell script:"${WORKSPACE}\\pyenv\\Scripts\\Activate.ps1 ; python ${command}", label: "python ${command}"
}
}
After acticating the virtualenv, try to run pip as a module:
python -m pip install ...
python -m pip vs pip
python -m pip: executes python interpreter binary that reads module pip.py from site packages directory
pip: executes pip binary / script picked up from $PATH
I have found that using python -m pip solved most of the pip permission problems encountered.
#hardbyte's answer
The default shell that Jenkins uses is /bin/sh - this is configurable in Manage Jenkins -> Configure System -> Shell -> Shell executable. Setting this to /bin/bash will make source work.
plus:
https://stackoverflow.com/a/70812248/1497139
pyvenv-3.4 returned non-zero exit status 1
got me working
sudo apt install python3.10-venv
and then in jenkins in the execute shell step:
python3.10 -m venv .venv
source .venv/bin/activate
...