This is what I'm getting when I run AWS terraform plan with Jenkins. Below code that we are using
Error: error: cannot delete old terraform
Is a directory
Code :
sh '''set +x
curl -L 'https://releases.hashicorp.com/terraform/0.11.10/terraform_0.11.10_linux_amd64.zip' --output terraform.zip
unzip -o terraform.zip
echo "Using $(terraform -version) from: $(which terraform)"
'''
sh "terraform init -backend-config='bucket=${bucketName}'"
Jenkins Error:
+ set +x
after terraform download
Archive: terraform.zip
error: cannot delete old terraform
Is a directory
[Pipeline] End of Pipeline
ERROR: script returned exit code 50
Finished: FAILURE
Please suggest some better solution.
Unzip refuses to overwrite the terraform/ directory that seems to be still lying around in your workspace from the previous run.
Run either a sh "rm -rf terraform/" before the unzip (or cleanWs())
unzip -f terraform.zip
Use -f instead of -o
-f freshen existing files, create none i.e unzip to replace the new files only
-n never overwrite existing files
-q quiet mode (-qq => quieter)
-o overwrite files WITHOUT prompting
Related
I am trying to build a docker image that contains all of the necessary plugins/providers that several source repos need, so that when an automated terraform validate runs, it doesn't have to download gigs of redundant data.
However, I recognize that this provides for a maintenance problem in that someone may update a plugin version, and that would needed to be downloaded, since the docker image would not contain it.
The question
How can I pre-download all providers and plugins
Tell the CLI use those predownloaded plugins AND
also tell it that, if it doesn't find what it needs locally, then it can go to the network
Below are the relevant file:
.terraformrc
plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
disable_checkpoint = true
provider_installation {
filesystem_mirror {
path = "$HOME/.terraform/providers"
}
direct {
}
}
tflint (not relevant to this question, but it shows up in the below Dockerfile)
plugin "aws" {
enabled = true
version = "0.21.1"
source = "github.com/terraform-linters/tflint-ruleset-aws"
}
plugin "azurerm" {
enabled = true
version = "0.20.0"
source = "github.com/terraform-linters/tflint-ruleset-azurerm"
}
Dockerfile
FROM ghcr.io/terraform-linters/tflint-bundle AS base
LABEL name=tflint
RUN adduser -h /home/jenkins -s /bin/sh -u 1000 -D jenkins
RUN apk fix && apk --no-cache --update add git terraform openssh
ADD .terraformrc /home/jenkins/.terraformrc
RUN mkdir -p /home/jenkins/.terraform.d/plugin-cache/registry.terraform.io
ADD .tflint.hcl /home/jenkins/.tflint.hcl
WORKDIR /home/jenkins
RUN tflint --init
FROM base AS build
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh && \
echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_ed25519 && \
chmod 400 /root/.ssh/id_ed25519 && \
touch /root/.ssh/known_hosts && \
ssh-keyscan mygitrepo >> /root/.ssh/known_hosts
RUN git clone git#mygitrepo:wrai/tools/g.git
RUN git clone git#mygitrepo:myproject/a.git && \
git clone git#mygitrepo:myproject/b.git && \
git clone git#mygitrepo:myproject/c.git && \
git clone git#mygitrepo:myproject/d.git && \
git clone git#mygitrepo:myproject/e.git && \
git clone git#mygitrepo:myproject/f.git
RUN ls -1d */ | xargs -I {} find {} -name '*.tf' | xargs -n 1 dirname | sort -u | \
xargs -I {} -n 1 -P 20 terraform -chdir={} providers mirror /home/jenkins/.terraform.d
RUN chown -R jenkins:jenkins /home/jenkins
USER jenkins
FROM base AS a
COPY --from=build /home/jenkins/a/ /home/jenkins/a
RUN cd /home/jenkins/a && terraform init
FROM base AS b
COPY --from=build /home/jenkins/b/ /home/jenkins/b
RUN cd /home/jenkins/b && terraform init
FROM base AS c
COPY --from=build /home/jenkins/c/ /home/jenkins/c
RUN cd /home/jenkins/c && terraform init
FROM base AS azure_infrastructure
COPY --from=build /home/jenkins/d/ /home/jenkins/d
RUN cd /home/jenkins/d && terraform init
FROM base AS aws_infrastructure
COPY --from=build /home/jenkins/e/ /home/jenkins/e
RUN cd /home/jenkins/e && terraform init
Staging plugins:
This is most easily accomplished with the plugin cache dir setting in the CLI. This supersedes the old usage with the -plugin-dir=PATH argument for the init command. You could also set a filesystem mirror in each terraform block within the root module config, but this would be cumbersome for your use case. In your situation, you are already configuring this in your .terraformrc, but the filesystem_mirror path conflicts with the plugin_cache_dir. You would want to resolve that conflict, or perhaps remove the mirror block entirely.
Use staged plugins:
Since the setting is captured in the CLI configuration file within the Dockerfile, this would be automatically used in future commands.
Download additional plugins if necessary:
This is default behavior of the init command, and therefore requires no further actions on your part.
Side note:
The jenkins user typically is /sbin/nologin for shell and /var/lib/jenkins for home directory. If the purpose of this Docker image is for a Jenkins build agent, then you may want the jenkins user configuration to be more aligned with the standard.
TL;DR:
Configure the terraform plugin cache directory
Create directory with a single TF file containing required_providers block
Run terraform init from there
...
I've stumbled over this question as I tried to figure out the same thing.
I first tried leveraging an implied filesystem_mirror by running terraform providers mirror /usr/local/share/terraform/plugins in a directory containing only one terraform file containing the required_providers block. This works fine as long as you only use the versions of the providers you mirrored.
However, it's not possible to use a different version of a provider than the one you mirrored, because:
Terraform will scan all of the filesystem mirror directories to see which providers are placed there and automatically exclude all of those providers from the implied direct block.
I've found it to be a better solution to use a plugin cache directory instead. EDIT: You can prefetch the plugins by setting TF_PLUGIN_CACHE_DIR to some directory and then running terraform init in a directory that only declares the required_providers.
Previously overengineered stuff below:
The only hurdle left was that terraform providers mirror downloads the providers in the packed layout:
Packed layout: HOSTNAME/NAMESPACE/TYPE/terraform-provider-TYPE_VERSION_TARGET.zip is the distribution zip file obtained from the provider's origin registry.
while Terraform expects the plugin cache directory to use the unpacked layout:
Unpacked layout: HOSTNAME/NAMESPACE/TYPE/VERSION/TARGET is a directory containing the result of extracting the provider's distribution zip file.
So I converted the packed layout to the unpacked layout with the help of find and parallel:
find path/to/plugin-dir -name index.json -exec rm {} +`
find path/to/plugin-dir -name '*.json' | parallel --will-cite 'mkdir -p {//}/{/.}/linux_amd64; unzip {//}/*.zip -d {//}/{/.}/linux_amd64; rm {}; rm {//}/*.zip'
I'm writing a jenkins pipeline jenkinsfile and within the script clause I have to ssh to a box and run some commands. I think the problem has to do with the env vars that I'm using within the quotes. I'm getting a ENDSSH command not found error and I'm at a loss. Any help would be much appreciated.
stage("Checkout my-git-repo"){
steps {
script {
sh """
ssh -o StrictHostKeyChecking=accept-new -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null -i ${JENKINS_KEY} ${JENKINS_KEY_USR}#${env.hostname} << ENDSSH
echo 'Removing current /opt/my-git-repo directory'
sudo rm -rf /opt/my-git-repo
echo 'Cloning new my-git-repo repo into /opt'
git clone ssh://${JENKINS_USR}#git.gitbox.com:30303/my-git-repo
sudo mv /home/jenkins/my-git-repo /opt
ENDSSH
"""
}
}
}
-bash: line 6: ENDSSH: command not found
I'm personally not familiar with jenkins, but I'd guess the issue is the whitespace before ENDSSH
White space in front of the delimiter is not allowed.
(https://linuxize.com/post/bash-heredoc/)
Try either removing the indentation:
stage("Checkout my-git-repo"){
steps {
script {
sh """
ssh -o StrictHostKeyChecking=accept-new -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null -i ${JENKINS_KEY} ${JENKINS_KEY_USR}#${env.hostname} << ENDSSH
echo 'Removing current /opt/my-git-repo directory'
sudo rm -rf /opt/my-git-repo
echo 'Cloning new my-git-repo repo into /opt'
git clone ssh://${JENKINS_USR}#git.gitbox.com:30303/my-git-repo
sudo mv /home/jenkins/my-git-repo /opt
ENDSSH
"""
}
}
}
OR ensure that the whitespace is only tabs and replace << with <<-:
Appending a minus sign to the redirection operator <<-, will cause all
leading tab characters to be ignored. This allows you to use
indentation when writing here-documents in shell scripts. Leading
whitespace characters are not allowed, only tab.
The file validates and I look to have the proper syntax.
script {
sh """
summon -f folder/file.yml --provider summon-aws-secrets \
sh -c 'bash folder/bin/run_me.sh' \
"""
open folder/file.yml: no such file or directory
I confirmed the existence of the file and workspace location.
Try using full path with workspace variable:
script {
sh """
summon -f ${WORKSPACE}/folder/file.yml --provider summon-aws-secrets \
sh -c 'bash folder/bin/run_me.sh' \
"""
}
so what I see happening is I wrapped the file into a script. ran git add , git commit, git push. I updated the jenkins file to ls -l the folder and I notice that file is missing. so not sure if this is a git issue or jenkins or etc
I try to execute such a scenery via Jenkins "execute shell" build step:
rm -r -f _dpatch;
mkdir _dpatch;
mkdir _dpatch/deploy;
from_revision='HEAD';
to_revision='2766920';
git diff --name-only $from_revision $to_revision > "_dpatch/deploy/files.txt";
for file in $(<"_dpatch/deploy/files.txt"); do cp --parents "$file" "_dpatch"; done;
whoami
Build ends successfully with console output:
[Deploy to production] $ /bin/sh -xe /tmp/hudson8315034696077699718.sh
+ rm -r -f _dpatch
+ mkdir _dpatch
+ mkdir _dpatch/deploy
+ from_revision=HEAD
+ to_revision=2766920
+ git diff --name-only HEAD 2766920
+
+ whoami
jenkins
Finished: SUCCESS
The problem is line "for file in" is just ignored, I do not understand why.
Content of files.txt is not empty and looks like this:
addons/tiny_mce/plugins/image/plugin.min.org.js
addons/webrtc/adapter-latest.js
templates/standard/style/review.css
More over, when I execute via ssh the same script in the same jenkins workspace folder under the same user (jenkins) - "for file in" line executes normally and creates files in "_dpatch" subfolder as it should.
My environment:
Debian 8,
Jenkins 2.45
Thanks
Possibly your /bin/sh is a POSIX bourne shell. It think that the $(< construct is a bash-ism, so it will not work with /bin/sh.
Try to replace
$(<"_dpatch/deploy/files.txt")
with
$(cat "_dpatch/deploy/files.txt")
Alternatively, prepend your build step with #!/bin/bash.
If your login shell is bash, then this also explains why everything works fine via ssh.
Try substituting for with while loop. And also add some more logging
rm -r -f _dpatch;
mkdir _dpatch;
mkdir _dpatch/deploy;
from_revision='HEAD';
to_revision='2766920';
git diff --name-only $from_revision $to_revision > "_dpatch/deploy/files.txt" && echo "git diff finished"
while IFS= read -r line; do
echo $line
cp --parent $line $_dpatch
done < _dpatch/deploy/files.txt
whoami
Following line in Dockerfile doesn't work:
RUN git archive master | tar -x -C /path
Error message:
fatal: Not a valid object name
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
How to solve this issue?
You can try a sh -c command
RUN sh -c 'git archive master | tar -x -C /path'
If not, you can include that command in a script, COPY the script and RUN it.
How about the following variation: git archive master | tar xf - -C /path?
It looks like the issue is actually with your git repository (or the directory your RUN is executing in):
fatal: Not a valid object name
This error is coming from git and suggests that there's no branch named master.