I try to build gerrit with bazel (0.13.0rc2 and 0.20) on Ubuntu. My company uses a proxy that requires authentication. The environment variables http_proxy and https_proxy are set (and other tools like wget, maven, ... work) but bazel reports "HTTP/1.0 407 Proxy Authentication Required" errors:
$ bazel build gerrit
WARNING: ignoring http_proxy in environment.
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '#io_bazel_rules_closure//closure': Error downloading [https://github.com/bazelbuild/rules_closure/archive/08039ba8ca59f64248bb3b6ae016460fe9c9914f.tar.gz] to /home/xxxxxx/.cache/bazel/_bazel_xxxxxx/5fb8c741228852f6bc57df1fc04917a8/external/io_bazel_rules_closure/08039ba8ca59f64248bb3b6ae016460fe9c9914f.tar.gz: Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Proxy Authentication Required"
Any suggestions are welcome!
If somebody else has a similar problem:
After searching a while and e. g. tried to specify the environment variables HTTP_PROXY and HTTPS_PROXY (see also bazel issue #587) I found this discussion and used gerritforge/jenkins-slave-bazel for building (proxy configured also for docker):
# use docker the docker container gerritforge/jenkins-slave-bazel to build gerrit
$ docker run --privileged -ti --entrypoint=bash --user jenkins gerritforge/jenkins-slave-bazel
$ cd && git clone --recursive https://gerrit.googlesource.com/gerrit
$ cd gerrit && bazel build gerrit
That at least solves the bazel proxy issue.
Related
Issue:
I am trying to add a bitbucket repo using an ssh key as a tool.poetry.dev.dependency in my pyproject.toml and have it installed in a docker container via docker compose. However, I encounter the following errors (see Errors) when I build the container.
Is there anything that I may be overlooking and/or suggestions to be able to pull a bitbucket repo as a dependency during poetry install in a docker container?
What I have done:
I know the bitbucket dependency is the issue because when I exchange it to a local path to a whl file instead, the container builds correctly.
I can install the dependency locally
I can clone a different repo in the docker container
REFERENCES
Errors:
HangupException
The remote server unexpectedly closed the connection.
at /usr/local/lib/python3.10/site-packages/dulwich/protocol.py:220 in read_pkt_line
216│
217│ try:
218│ sizestr = read(4)
219│ if not sizestr:
→ 220│ raise HangupException()
221│ size = int(sizestr, 16)
222│ if size == 0:
223│ if self.report_activity:
224│ self.report_activity(4, "read")
The following error occurred when trying to handle this error:
HangupException
Host key verification failed.
at /usr/local/lib/python3.10/site-packages/dulwich/client.py:1151 in fetch_pack
1147│ with proto:
1148│ try:
1149│ refs, server_capabilities = read_pkt_refs(proto.read_pkt_seq())
1150│ except HangupException as exc:
→ 1151│ raise _remote_error_from_stderr(stderr) from exc
1152│ (
1153│ negotiated_capabilities,
1154│ symrefs,
1155│ agent,
My Files/Commands:
pyproject.toml
[tool.poetry.dependencies]
# Trial 1: I am trying to use ssh key to pull to repo (see docker build command) [1]
package_name = {git = "ssh://git#bitbucket.org/tenant/repo.git", tag = "v0.0.0"}
# Trial 2: I don't really want to use http because I don't want to have to feed in credentials [1]
package_name = {git = "https://user#bitbucket.org/tenant/repo.git", tag = "v0.0.0"}
# Trial 3: I didn't know if it was just a bitbucket thing so I had also tried with git and it works locally just not in a docker container
package_name = {git="git+ssh://git#github.com/user/repo.git"}`
Dockerfile
ENV POETRY_VERSION=1.2.2
RUN pip install poetry==$POETRY_VERSION
COPY ./poetry.lock ./pyproject.toml ./
RUN poetry config installer.max-workers 4 \
&& poetry install --no-root`
Docker build command
docker build --no-cache --ssh default -t $IMAGE_NAME .
Docker compose command
docker compose build
I had the same error thrown at me after trying to install a private package in a container via git. I could get it to work using https instead of ssh.
It might help if you run the install command with -vvv to get a more verbose error.
AFAICT, this seems to be an underlying issue with dulwich.
If you really don't want to use https, I'd go report this on the corresponding poetry issue, hopefully the poetry team can find a workaround until dulwitch rolls out a fix.
I have a Jenkins CI and use it to build (mvn) and containerize (docker) my app using Jenkins scripted pipeline. Lastly, I want to deploy my container to Heroku dyno (I have already created an app).
I have followed this documentation https://devcenter.heroku.com/articles/container-registry-and-runtime and have been successfully pushed my docker image to registry.heroku.com/sunset-sailing-4049/web.
The issue is since this announcement https://devcenter.heroku.com/changelog-items/1426 I now need to explicitly execute "heroku container:release web" in order to get my docker container running from registry to app dyno. This is where I am royally stuck. See my below issues:
Heroku is not recognized by Jenkins. (My Jenkins is running on ec2, I have installed heroku toolbelt as ec2-user user. But Jenkins throws error: heroku: command not found). How do I resolve this issue?
How to do "heroku login" from Jenkins, since the login command prompts for browser login. I have added ssh key but I do not know how to use it from the command line, hence Jenkins "shell script"
The only other way I could think of is deploying via heroku pipeline using a dummy git repo onto which Jenkins will upload the source code on a successful build.
Would really appreciate your help solving the above 2 issues.
Thanks in Advance.
You need install heroku as user under which jenkins is running. Or if you installed it globally it may be not in PATH of user under which jenkins is running.
There are multiple options for setting PATH:
Set for specific command.
If your job is pipeline just wrap heroku command in withEnv closure:
withEnv(['PATH+HEROKU=/use/local/bin/']) {
your heroku command here
}
Set path for jenkins slave: go to [Manage Jenkins] -> [Manage Nodes], configure your node and set Environment variable PATH to $PATH:/use/local/bin/. This way all jobs running on slave will get environment variable injected.
For automated cli interactions heroku supports API tokens. You can either put it in ~/.netrc on build machine or supply as environment variable (see here).
(writing here incase someone is facing the same scenario)
ok I took #vladimir's suggestion and did the below:
Heroku command (for jenkins running on ec2):
The below command is needed to push a built docker image to heroku via jenkins/or other ci/cd tool; Because of a recent change (https://devcenter.heroku.com/changelog-items/1426) pushing to heroku registry isn't sufficient any longer. In order to execute the below command you need to install heroku toolbelt.
heroku container:release web
Install snap on amazon linux like below:
follow instruction to enable epel https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-repositories.html
Then Modify /etc/yum.repos.d/epel.repo. Under the section marked [epel] , change enabled=0 to enabled=1.
Then do
sudo yum install epel-release
sudo yum install yum-plugin-copr
sudo yum copr enable ngompa/snapcore-el7
sudo yum -y install snapd
sudo systemctl enable --now snapd.socket
Then install heroku toolbelt:
sudo snap install --classic heroku
Deploying to docker image to heroku:
In Jenkins scripted pipeline:
withCredentials([string(credentialsId: 'heroku-api-cred', variable: 'herokuRegistryApiCred')]) {
sh "docker login -u email#example.com -p ${herokuRegistryApiCred} registry.heroku.com"
}
// Tag docker img (in my case it was an image in dockerhub)
sh "docker tag dockerhubusername/pvtreponame:${imageTag} registry.heroku.com/your_app_name/release_type[ie>web]"
sh "docker push registry.heroku.com/your_app_name/web"
sh "/usr/local/bin/heroku container:release web --app=your_app_name"
sh "docker logout registry.heroku.com"
In order to run the app inside docker (in my case it was java) I had to add the below line (otherwise it was crashing because 1. tell app about heroku's port binding. 2. tell web process to run command. The ENTRYPOINT ["java","-jar","my_spring_boot_app-0.0.1-SNAPSHOT.jar"] does not work on heroku.):
CMD ["web", "java $JAVA_OPTS -Dserver.port=$PORT -jar /usr/app/my_spring_boot_app-0.0.1-SNAPSHOT.jar"]
I have Followed below process for installing Acumos in an Ubuntu 18 Server.
Open a shell session (bash recommended) on the host on which (for single AIO deployment) or from which (for peer-test deployment) you want to install Acumos, and clone the system-integration repo:
> git clone https://gerrit.acumos.org/r/system-integration
If you are deploying a single AIO instance, run the following command, selecting docker or kubernetes as the target environment. Further instructions for running the script are included at the top of the script.
> bash oneclick_deploy.sh
I have done it using k8s as below
> bash oneclick_deploy.sh k8s
Everything was running smoothly but at the end i am facing the below issue .
as docker API is not ready
Can anyone help me on this Please?
Note: I have checked in the kubernetes console everything is fine . A service file is created and also namespace is also created sucessfully in the name of acumos .
I'm the developer of that toolset. I'll be happy to help you thru this. Note that it's actively being developed, and will be evolving a lot. But there are some easy things you can do to provide more details so I can debug your situation.
First, start with a clean env:
$ bash clean.sh
Then reattempt the deployment, piping the console log to a file:
$ bash oneclick_deploy.sh k8s 2>&1 | tee deploy.log
Review that file to be sure that there's nothing sensitive in it (e.g. passwords or other private info about the deployment that you don't want to share), and if possible attach it here so I can review it. That will be the quickest way to debug.
Also you can let me know some more about your deployment context:
Did you ensure the Prerequisites:
Ubuntu Xenial (16.04), Bionic (18.04), or Centos 7 hosts
All hostnames specified in acumos-env.sh must be DNS-resolvable on all hosts (entries in /etc/hosts or in an actual DNS server)
Did you customize acumos-env.sh, or use the default values
Send the output of
$ kubectl get svc -n acumos
$ kubectl get pods -n acumos
$ kubectl describe pods -n acumos
I created a dockerfile which generates the docker image with my node application. My application depends on my another application which is added as dependency using git ssh.
When docker build runs npm install, it fails with error code 128. I understand it is because i do not have valid ssh token to access repo. How can i create one and have my docker build pass?
You can use ssh-keygen -t rsa to generate your local machine's key (do not provide any passcode for simplicity) that can be used for authentication. Now adding that key for git access depends on where your repository is i.e is it on hosted sites like bitbucket/github or just your another linux machine.
For repository on local server run below commands on your local machine to add your public key (id_rsa.pub) to git server.
eval "$(ssh-agent -s)"
ssh-add
ssh-copy-id user#git-server
For hosted sites you get the option to add the public-key under your profile settings.
Note: do not forget to add below in ~/.ssh/config file on your local machine to avoid the unknown host exception
Host bitbucket.org
StrictHostKeyChecking no
Host <git-server-ip>
StrictHostKeyChecking no
For more information on generating key please refer to https://confluence.atlassian.com/bitbucketserver/creating-ssh-keys-776639788.html
I would like to be able to perform linting on Jenkins pipelines and it seems that Groovy linting is not enough.
How can I do this?
HTTP without crumb.
If you want to use HTTP and don't want to use CRUMB. just add your username and password using the '-u' parameter. Replace <username> and <password> with the username and password of your user. Also Check that the url of the jenkins server is correct.
curl --user <username>:<password> -X POST -F "jenkinsfile=<Jenkinsfile" http://localhost:8080/pipeline-model-converter/validate
src
If for some reason you can't use Jenkins server linter, you can use npm-groovy-lint (works with Declarative or Scripted Jenkinsfile, and also groovy shared libraries)
https://github.com/nvuillam/npm-groovy-lint
npm install -g npm-groovy-lint
npm-groovy-lint // in the root directory of the Jenkinsfile
Looks like there are two options for linting pipeline scripts, one via the cli on the leader or an http POST call:
Linting via the CLI with SSH
# ssh (Jenkins CLI)
# JENKINS_SSHD_PORT=[sshd port on master]
# JENKINS_HOSTNAME=[Jenkins master hostname]
ssh -p $JENKINS_SSHD_PORT $JENKINS_HOSTNAME declarative-linter < Jenkinsfile
Linting via HTTP POST using curl
# curl (REST API)
# Assuming "anonymous read access" has been enabled on your Jenkins instance.
# JENKINS_URL=[root URL of Jenkins master]
# JENKINS_CRUMB is needed if your Jenkins master has CRSF protection enabled as it should
JENKINS_CRUMB=`curl "$JENKINS_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)"`
curl -X POST -H $JENKINS_CRUMB -F "jenkinsfile=<Jenkinsfile" $JENKINS_URL/pipeline-model-converter/validate
https://jenkins.io/doc/book/pipeline/development/#linter
In addition to kongkoro's answer, there is a tool to lint Jenkinsfile.
https://www.npmjs.com/package/jflint
# install
$ npm install -g jflint
# usage
# JENKINS_URL=[root URL of Jenkins master]
$ jflint -j $JENKINS_URL Jenkinsfile
What the jflint does is the same as curl in the official document, and jflint works only with declarative pipelines too. But it's easier to use.
SSH
Methods using Jenkins SSH interface to run the linter:
Enable SSH service in the Configure Global Security page and assign the port (e.g. 2222).
Add your Public SSH Key in your user's profile in Jenkins (JENKINS_URL/user/USER/configure).
Confirm the SSH access by SSHing to Jenkins and run:
ssh -l admin -p 2222 localhost help
Validate your local Jenkinsfile using the following command on Jenkins box:
ssh -l admin -p 2222 localhost declarative-linter < ./Jenkinsfile
For further details, read Pipeline Development Tools.
Furthermore, to simplify, you can add the following section to your ~/.ssh/config:
Host jenkins-cli
HostName localhost
User admin
Port 2222
ProxyJump jenkins-host.example.com
Then run: ssh jenkins-cli declarative-linter < ./Jenkinsfile.
You can also consider creating the following shell alias (e.g. to your startup files):
alias jenkins-lint="ssh jenkins-cli declarative-linter < ./Jenkinsfile"
Then just run: jenkins-lint.
POST
Validate a Jenkinsfile by using the following curl command:
curl --user username:password -X POST -F "jenkinsfile=<Jenkinsfile" http://jenkins-url:8080/pipeline-model-converter/validate
For details, please read How to validate a Jenkinsfile page.
VS Code plugin
Using VS Code IDE editor, you can install Jenkins Pipeline Linter Connector plugin and configure accordingly to the instructions, so it can post your Jenkinsfile to your Jenkins Server via POST request.
If you want to lint Jenkins pipelines which can be scripted or declarative.
Then the best solution is to lint using the jenkins-cli.jar.
I tried whatever I could possibly get my hands at but this really looks like the best and most convenient to use.
Requirements would be - java
Download the cli jar
$ curl -O https://<jenkins-server>/jnlpJars/jenkins-cli.jar
Lint the Jenkins pipeline script - either Scripted or Declarative
$ java -jar jenkins-cli.jar -s '<jenkins-server-url' -auth <username>:<password> declarative-linter < Jenkinsfile
Its always best to use the jenkins server url where it will be placed as that takes care of checking if the necessary plugins, etc are in place for the pipeline to function correctly.
Jenkins-CLI