Related
ABOUT
MINIMAL WORKING EXAMPLE: https://gitlab.com/hynek.blaha/debug-docker-poetry/-/tree/master
I have been building Docker images using Poetry with Python packages from internal PyPI registry. As our projects are in private GitLab repository and the internal packages are not top-secret, we are storing the poetry credentials directly in the poetry source URL in pyproject.toml.
On 2022-08-24, all our Docker builds started failing while installing internal package:
• Installing til-bigquery (0.3.4)
HTTPError
401 Client Error: Unauthorized for url: https://gitlab.com/api/v4/projects/38869805/packages/pypi/files/7a4731d831d4b37262481002271e359f96017570e9480ef16c89489e0b41252f/til_bigquery-0.3.4-py3-none-any.whl#sha256=7a4731d831d4b37262481002271e359f96017570e9480ef16c89489e0b41252f
at /usr/local/lib/python3.9/site-packages/requests/models.py:1021 in raise_for_status
1017│ f"{self.status_code} Server Error: {reason} for url: {self.url}"
1018│ )
1019│
1020│ if http_error_msg:
→ 1021│ raise HTTPError(http_error_msg, response=self)
1022│
1023│ def close(self):
1024│
1025│ called the underlying ``raw`` object must not be accessed again.
What I found weird:
Docker build fails even when I retry deploy job, that successfully passed a few days ago.
Considering the issue might have been caused by unpinned minor version of Docker base image python:3.7-slim or Poetry, I used older versions but got the same result.
I compared the build logs of previously successful build build_success.log (8/22/22, 3:00 PM) and the same build retry build_fail.log (8/24/22, 6:00 AM) and found both use the same poetry wheel poetry-1.1.15-py2.py3-none-any.whl.
It still works as before on my machine, but fails in Docker.
It stops working on localhost when I remove the credentials from the repository URL, so I am sure the credentials are not stored anywhere else (e.g. ~/.netrc).
How to reproduce:
Localhost - OK
git clone git#gitlab.com:hynek.blaha/debug-docker-poetry.git
poetry install
Docker - FAIL
git clone git#gitlab.com:hynek.blaha/debug-docker-poetry.git
docker build .
I am able to fix the issue by explicitly providing the credentials in Dockerfile:
RUN pip install poetry --no-cache-dir && \
poetry config virtualenvs.create false && \
poetry config repositories.my_private_repo https://gitlab.com/api/v4/projects/21870843/packages/pypi/simple && \
poetry config http-basic.my_private_repo __token__ glpat-mkEPJ4Rsy2peTCrH23pG
But it doesn't explain, why rebuilding the same image started failing.
And why it still works as expected when running on my machine (outside of Docker).
Does anyone have an idea, what might have changed? I was unable to tell what even when using diff on the build_success.log and build_fail.log
I was struggling with totally the same problem the last few days. Though I'm still not sure of the exact cause, somehow I managed to avoid the problem.
I also used a repository URL with credentials embedded in pyproject.toml until yesterday like this.
[[tool.poetry.source]]
name = 'private'
url = 'https://your_username:your_password#gitlabce.example.com/api/v4/projects/<project_id>/packages/pypi/simple'
secondary = true
Though it's basically the same as your solution, you can specify the local poetry configs for each project by creating poetry.toml at the project root. So instead of embedding credentials in the URL, you can specify them via poetry.toml as follows:
[http-basic]
[http-basic.private]
username = "your_username"
password = "your_password"
That way, you can reproduce the same behaviour with embedded credentials without the authentication error.
Why does it still work outside docker?
I guess it is due to the archive cache in your local environment. Since poetry stores downloaded archives in ~/.cache/pypoetry/artifacts/ and reuse them when execute poetry install, you didn't need to access your private PyPI server in the first place. If you manually remove archives (I'm not sure, but poetry cache clear command didn't work for my case), you'll be able to reproduce the authentication error even in your local environment.
Why did the error suddenly start to occur?
Poetry uses the embedded credentials to query the list of package links to the private PyPI server.
It works fine until there, however, when installing actual packages, Poetry uses the acquired link from the PyPI server that has no credentials embedded. That's why it fails and the URL shown in the error message doesn't have any credentials embedded.
I'm still not sure why the credential embed had been working until a few days ago though. I guess there might have been behavior changes on the GitLab side.
Locally, I am able to successfully authorize and pull modules from my private Nexus registry (.npmrc file).
However on Jenkins I get
error An unexpected error occurred: "https://myprivaterepo.com/myprivatemodule.tgz: Request failed \"401 Unauthorized\"".
When I run npm whoami on Jenkins it returns a valid user. npm config ls prints valid configuration as well.
The problem started to occur when I changed myprivaterepo url (we've migrated it). Is there something I don't know (ie. I have to logout/login again or there's some cache in Jenkins)??
Thanks in advance!
VSCode Version:
1.62.2
Local OS Version:
Windows 10.0.18363
Reproduces in: Remote - Containers
Name of Dev Container Definition with Issue:
/vscode/devcontainers/typescript-node
In our company we use a proxy which terminates the SSL connections. When I now try to start any devcontainer (the workspace is in the WSL2 filesystem), I get the following error message:
Installing VS Code Server for commit 3a6960b964327f0e3882ce18fcebd07ed191b316
[2021-11-12T17:01:44.400Z] Start: Downloading VS Code Server
[2021-11-12T17:01:44.400Z] 3a6960b964327f0e3882ce18fcebd07ed191b316 linux-x64 stable
[2021-11-12T17:01:44.481Z] Stop (81 ms): Downloading VS Code Server
[2021-11-12T17:01:44.499Z] Error: unable to get local issuer certificate
at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)
at TLSSocket.emit (events.js:315:20)
at TLSSocket._finishInit (_tls_wrap.js:932:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)
In the dockerfile I copy the company certificates and update them:
ADD ./certs /usr/local/share/ca-certificates
RUN update-ca-certificates 2>/dev/null
The proxy environment variables are also set correctly. Out of desperation I also tried to disable the certificate check for wget:
RUN su node -c "echo check_certificate=off >> ~/.wgetrc"
Even in the devcontainer configuration I have disabled the proxy and the security check for VS code via the settings:
// Set *default* container specific settings.json values on container create.
"settings": {
"http.proxy": "http://<proxy.url>:8080",
"http.proxyStrictSSL": false
},
I have tried many other things, like setting NODE_TLS_REJECT_UNAUTHORIZED=0 as env variable inside the dockerfile, unfortunately without any success. Outside the company network, without the proxy, it works wonderfully.
Maybe one of you has an idea how I can solve this problem?
A working if not so nice solution to the problem is to add HTTPS exceptions for the following domains:
https://update.code.visualstudio.com
https://az764295.vo.msecnd.net
A list of common hostnames can be found here:
https://code.visualstudio.com/docs/setup/network
I'm searching for the way to get $ go get work with private repository, after many google try.
The first try:
$ go get -v gitlab.com/secmask/awserver-go
Fetching https://gitlab.com/secmask/awserver-go?go-get=1
https fetch failed.
Fetching http://gitlab.com/secmask/awserver-go?go-get=1
Parsing meta tags from http://gitlab.com/secmask/awserver-go?go-get=1 (status code 200)
import "gitlab.com/secmask/awserver-go": parse http://gitlab.com/secmask/awserver-go?go-get=1: no go-import meta tags
package gitlab.com/secmask/awserver-go: unrecognized import path "gitlab.com/secmask/awserver-go
Yep, it did not see the meta tags because I could not know how to provide login information.
The second try:
Follow https://gist.github.com/shurcooL/6927554. Add config to .gitconfig.
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
$ go get -v gitlab.com/secmask/awserver-go --> not work
$ go get -v gitlab.com/secmask/awserver-go.git --> work but I got src/gitlab.com/secmask/awserer-go.git
Yes it work but with .git extension with my project name, I can rename it to original but do it everytime $ go get is not so good, is there an otherway?
You have one thing to configure. The example is based on GitHub but this shouldn't change the process:
$ git config --global url.git#github.com:.insteadOf https://github.com/
$ cat ~/.gitconfig
[url "git#github.com:"]
insteadOf = https://github.com/
$ go get github.com/private/repo
For Go modules to work (with Go 1.11 or newer), you'll also need to set the GOPRIVATE variable, to avoid using the public servers to fetch the code:
export GOPRIVATE=github.com/private/repo
The proper way is to manually put the repository in the right place. Once the repository is there, you can use go get -u to update the package and go install to install it. A package named
github.com/secmask/awserver-go
goes into
$GOPATH/src/github.com/secmask/awserver-go
The commands you type are:
cd $GOPATH/src/github.com/secmask
git clone git#github.com:secmask/awserver-go.git
I had a problem with go get using private repository on gitlab from our company.
I lost a few minutes trying to find a solution. And I did find this one:
You need to get a private token at:
https://gitlab.mycompany.com/profile/account
Configure you git to add extra header with your private token:
$ git config --global http.extraheader "PRIVATE-TOKEN: YOUR_PRIVATE_TOKEN"
Configure your git to convert requests from http to ssh:
$ git config --global url."git#gitlab.mycompany.com:".insteadOf "https://gitlab.mycompany.com/"
Finally you can use your go get normally:
$ go get gitlab.com/company/private_repo
For people using private GitLabs, here's a snippet that may help: https://gist.github.com/MicahParks/1ba2b19c39d1e5fccc3e892837b10e21
Also pasted below:
Problem
The go command line tool needs to be able to fetch dependencies from your private GitLab, but authenticaiton is required.
This assumes your private GitLab is hosted at privategitlab.company.com.
Environment variables
The following environment variables are recommended:
export GO111MODULE=on
export GOPRIVATE=privategitlab.company.com # this is comma delimited if using multiple private repos
The above lines might fit best in your shell startup, like a ~/.bashrc.
Explanation
GO111MODULE=on tells Golang command line tools you are using modules. I have not tested this with projects not using
Golang modules on a private GitLab.
GOPRIVATE=privategitlab.company.com tells Golang command line tools to not use public internet resources for the hostnames
listed (like the public module proxy).
Get a personal access token from your private GitLab
To future proof these instructions, please follow this guide from the GitLab docs.
I know that the read_api scope is required for Golang command line tools to work, and I may suspect read_repository as
well, but have not confirmed this.
Set up the ~/.netrc
In order for the Golang command line tools to authenticate to GitLab, a ~/.netrc file is best to use.
To create the file if it does not exist, run the following commands:
touch ~/.netrc
chmod 600 ~/.netrc
Now edit the contents of the file to match the following:
machine privategitlab.company.com login USERNAME_HERE password TOKEN_HERE
Where USERNAME_HERE is replaced with your GitLab username and TOKEN_HERE is replaced with the access token aquired in the
previous section.
Common mistakes
Do not set up a global git configuration with something along the lines of this:
git config --global url."git#privategitlab.company.com:".insteadOf "https://privategitlab.company.com"
I beleive at the time of writing this, the SSH git is not fully supported by Golang command line tools and this may cause
conflicts with the ~/.netrc.
Bonus: SSH config file
For regular use of the git tool, not the Golang command line tools, it's convient to have a ~/.ssh/config file set up.
In order to do this, run the following commands:
mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/config
chmod 600 ~/.ssh/config
Please note the permissions on the files and directory above are essentail for SSH to work in it's default configuration on
most Linux systems.
Then, edit the ~/.ssh/config file to match the following:
Host privategitlab.company.com
Hostname privategitlab.company.com
User USERNAME_HERE
IdentityFile ~/.ssh/id_rsa
Please note the spacing in the above file matters and will invalidate the file if it is incorrect.
Where USERNAME_HERE is your GitLab username and ~/.ssh/id_rsa is the path to your SSH private key in your file system.
You've already uploaded its public key to GitLab. Here are some instructions.
All of the above did not work for me. Cloning the repo was working correctly but I was still getting an unrecognized import error.
As it stands for Go v1.13, I found in the doc that we should use the GOPRIVATE env variable like so:
$ GOPRIVATE=github.com/ORGANISATION_OR_USER_NAME go get -u github.com/ORGANISATION_OR_USER_NAME/REPO_NAME
Generate a github oauth token here and export your github token as an environment variable:
export GITHUB_TOKEN=123
Set git config to use the basic auth url:
git config --global url."https://$GITHUB_TOKEN:x-oauth-basic#github.com/".insteadOf "https://github.com/"
Now you can go get your private repo.
If you've already got git using SSH, this answer by Ammar Bandukwala is a simple workaround:
$ go get uses git internally. The following one liners will make git and consequently $ go get clone your package via SSH.
Github:
$ git config --global url."git#github.com:".insteadOf "https://github.com/"
BitBucket:
$ git config --global url."git#bitbucket.org:".insteadOf "https://bitbucket.org/"
I came across .netrc and found it relevant to this.
Create a file ~/.netrc with the following content:
machine github.com
login <github username>
password <github password or Personal access tokens >
Done!
Additionally, for latest GO versions, you might need to add this to the environment variables GOPRIVATE=github.com
(I've added it to my .zshrc)
netrc also makes my development environment setup better as my personal github access for HTTPS is been configured now to be used across the machine (just like my SSH configuration).
Generate GitHub personal access tokens: https://github.com/settings/tokens
See this answer for its use with Git on Windows specifically
Ref: netrc man page
If you want to stick with the SSH authentication, then mask the request to use ssh forcefully
git config --global url."git#github.com:".insteadOf "https://github.com/"
More methods for setting up git access: https://gist.github.com/technoweenie/1072829#gistcomment-2979908
That looks like the GitLab issue 5769.
In GitLab, since the repositories always end in .git, I must specify .git at the end of the repository name to make it work, for example:
import "example.org/myuser/mygorepo.git"
And:
$ go get example.org/myuser/mygorepo.git
Looks like GitHub solves this by appending ".git".
It is supposed to be resolved in “Added support for Go's repository retrieval. #5958”, provided the right meta tags are in place.
Although there is still an issue for Go itself: “cmd/go: go get cannot discover meta tag in HTML5 documents”.
After trying multiple solutions my problem still persisted. The final solution after setting up the ~/.netrc and SSH config file, was to add the following line to my ~/.bash_profile
export GOPRIVATE="github.com/[organization]"
I have created a user specific ssh-config, so my user automatically logs in with the correct credentials and key.
First I needed to generate an key-pair
ssh-keygen -t rsa -b 4096 -C "my#email.here"
and saved it to e.g ~/.ssh/id_my_domain. Note that this is also the keypair (private and public) I've connected to my Github account, so mine is stored in~/.ssh/id_github_com.
I have then created (or altered) a file called ~/.ssh/config with an entry:
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/id_github_com
On another server, the "ssh-url" is admin#domain.com:username/private-repo.git and the entry for this server would have been:
Host domain.com
HostName domain.com
User admin
IdentityFile ~/.ssh/id_domain_com
Just to clarify that you need ensure that the User, Host and HostName is set correctly.
Now I can just browse into the go path and then go get <package>, e.g go get main where the file main/main.go includes the package (from last example above) domain.com:username/private-repo.git.
For me, the solutions offered by others still gave the following error during go get
git#gl.nimi24.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
What this solution required
As stated by others:
git config --global url."git#github.com:".insteadOf "https://github.com/"
Removing the passphrase from my ./ssh/id_rsa key which was used for authenticating the connection to the repository. This can be done by entering an empty password when prompted as a response to:
ssh-keygen -p
Why this works
This is not a pretty workaround as it is always better to have a passphrase on your private key, but it was causing issues somewhere inside OpenSSH.
go get uses internally git, which uses openssh to open the connection. OpenSSH takes the certs necessary for authentication from .ssh/id_rsa. When executing git commands from the command line an agent can take care of opening the id_rsa file for you so that you do not have to specify the passphrase every time, but when executed in the belly of go get, this did not work somewhy in my case. OpenSSH wants to prompt you then for a password but since it is not possible due to how it was called, it prints to its debug log:
read_passphrase: can't open /dev/tty: No such device or address
And just fails. If you remove the passphrase from the key file, OpenSSH will get to your key without that prompt and it works
This might be caused by Go fetching modules concurrently and opening multiple SSH connections to Github at the same time (as described in this article). This is somewhat supported by the fact that OpenSSH debug log showed the initial connection to the repository succeed, but later tried it again for some reason and this time opted to ask for a passphrase.
However the solution of using SSH connection multiplexing as put forward in the mentioned article did not work for me. For the record, the author suggested adding the collowing conf to the ssh config file for the affected host:
ControlMaster auto
ControlPersist 3600
ControlPath ~/.ssh/%r#%h:%p
But as stated, for me it did not work, maybe I did it wrong
After setting up GOPRIVATE and git config ...
People may still meeting problems like this when fetching private source:
https fetch: Get "https://private/user/repo?go-get=1": EOF
They can't use private repo without .git extension.
The reason is the go tool has no idea about the VCS protocol of this repo, git or svn or any other, unlike github.com or golang.org them are hardcoded into go's source.
Then the go tool will do a https query before fetching your private repo:
https://private/user/repo?go-get=1
If your private repo has no support for https request, please use replace to tell it directly :
require private/user/repo v1.0.0
...
replace private/user/repo => private.server/user/repo.git v1.0.0
https://golang.org/cmd/go/#hdr-Remote_import_paths
first I tried
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
but it didn't worked for my local.
I tried
ssh -t git#github.com
and it shows my ssh is fine.
finally, I fix the problem to tell the go get to consider all as private and use ssh instead of HTTPS.
adding export GOPRIVATE=*
Make sure you remove your previous gitconfigs, I had the same issue.
Previously I executed gitconfig whose token was expired, when you execute the command next time with new token make sure to delete previous one.
For standalone/final repos, an as a quick fix, why don't just to name the module within the go.mod as a package using your company's domain ... ?
module go.yourcompany.tld/the_repo
go.yourcompany.tld don't even have to exist as a valid (sub)domain...
Also, in the same go.mod you can use replacement block/lines to use private repos previously cloned the same way (within a respective folder cloned also in $GOPATH/src/go.yourcompany.tld) (why do we have to depend too much in GitHub?)
Edit
Needless to say that a private repo usually shall be a private repo, typically a standard git repo, right? With that, why not just to git clone and then go get within the cloned folder?
It's Hard Code In Go Get. Not The Git Reason. So Modify Go Source.
Reason:
repoRootForImportDynamic Will Request: https://....go-get
// RepoRootForImportPath analyzes importPath to determine the
// version control system, and code repository to use.
func RepoRootForImportPath(importPath string, mod ModuleMode, security web.SecurityMode) (*RepoRoot, error) {
rr, err := repoRootFromVCSPaths(importPath, security, vcsPaths)
if err == errUnknownSite {
rr, err = repoRootForImportDynamic(importPath, mod, security)
if err != nil {
err = importErrorf(importPath, "unrecognized import path %q: %v", importPath, err)
}
}
if err != nil {
rr1, err1 := repoRootFromVCSPaths(importPath, security, vcsPathsAfterDynamic)
if err1 == nil {
rr = rr1
err = nil
}
}
So add gitlab domain to vcsPaths will ok.
Download go source code:
vi ./src/cmd/go/internal/vcs/vcs.go
Look for code below:
var vcsPaths = []*vcsPath{
// GitHub
{
pathPrefix: "github.com",
regexp: lazyregexp.New(`^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
vcs: "git",
repo: "https://{root}",
check: noVCSSuffix,
},
add Code As Follow,XXXX Is Your Domain:
// GitLab
{
pathPrefix: "gitlab.xxxx.com",
regexp: lazyregexp.New(`^(?P<root>gitlab.xxxx\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
vcs: "git",
repo: "https://{root}",
check: noVCSSuffix,
},
compile and replace go.
I configured Jenkins in Spinnaker as follows and setup the Spinnaker pipeline.
jenkins:
# If you are integrating Jenkins, set its location here using the baseUrl
# field and provide the username/password credentials.
# You must also enable the "igor" service listed separately.
#
# If you have multiple Jenkins servers, you will need to list
# them in an igor-local.yml. See jenkins.masters in config/igor.yml.
#
# Note that Jenkins is not installed with Spinnaker so you must obtain this
# on your own if you are interested.
enabled: ${services.igor.enabled:false}
defaultMaster:
name: default
baseUrl: http://server:8080
username: spinnaker
password: password
But I am seeing the following error when trying to run the Spinnaker pipeline.
Exception ( Start Jenkins Job )
403 No valid crumb was included in the request
Finally, this post helped me to do away with the crumb problem, but still securing Jenkins from a CSRF attack.
Solution for no-valid crumb included in the request issue
Basically, we need to first request for a crumb with authentication and then issue a POST API calls with a crumb as a header along with authentication again.
This is how I did it,
curl -v -X GET http://jenkins-url:8080/crumbIssuer/api/json --user <username>:<password>
The response was,
{
"_class":"hudson.security.csrf.DefaultCrumbIssuer",
"crumb":"0db38413bd7ec9e98974f5213f7ead8b",
"crumbRequestField":"Jenkins-Crumb"
}
Then the POST API call with the above crumb information in it.
curl -X POST http://jenkins-url:8080/job/<job-name>/build --user <username>:<password> -H 'Jenkins-Crumb: 0db38413bd7ec9e98974f5213f7ead8b'
This solution is safe to use
We came along this issue when we changed Jenkins to be accessible via a reverse proxy.
There is an option in the "Configure Global Security" that "Enable proxy compatibility"
This helped with my issue.
Another solution
In the GitHub payload URL, make your URL look like this:
https://jenkins:8080/github-webhook/
Don’t forget to mention / at the end.
To resolve this issue I unchecked "Prevent Cross Site Request Forgery exploits" in jenkins.com/configureSecurity section and it started working.
I solved this by using an API token as a basic authentication password. Here is how:
curl -v -X POST http://jenkins-url:8080/job/<job-name>/buildWithParameters?param=value --user <username>:<token>
Note: To create the API token under the accounts icon → Configure → API Token → Add New token.
A crumb is nothing but an access token. Below is the API to get the crumb:
https://jenkins.xxx.xxx.xxx/crumbIssuer/api/json
// Replace it with your Jenkins URL and make a GET call in your Postman or REST API caller.
This will generate output like:
{
"_class": "hudson.security.csrf.DefaultCrumbIssuer",
"crumb": "ba4742b9d92606f4236456568a",
"crumbRequestField": "Jenkins-Crumb"
}
Below are more details and link related to same:
How to request for the crumb issuer for Jenkins
Jenkins wiki page.
If you are calling the same via REST API call, checkout the below link where it is explained how to do a REST call using jenkins-crumb.
https://blog.dahanne.net/2016/05/17/how-to-update-a-jenkins-job-posting-config-xml/
Example:
curl -X POST http://anthony:anthony#localhost:8080/jenkins/job/pof/config.xml --data-binary "#config.xml" -data ".crumb=6bbabc426436b72ec35e5ad4a4344687"
For the new release of Jenkins you should follow the solution below:
From Upgrading to Jenkins 2.176.3:
Upgrading to Jenkins 2.176.2 Improved CSRF protection
SECURITY-626
CSRF tokens (crumbs) are now only valid for the web session they were
created in to limit the impact of attackers obtaining them. Scripts
that obtain a crumb using the /crumbIssuer/api URL will now fail to
perform actions protected from CSRF unless the scripts retain the web
session ID in subsequent requests. Scripts could instead use an API
token, which has not required a CSRF token (crumb) since Jenkins 2.96.
To disable this improvement you can set the system property
hudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID to true.
Alternatively, you can install the Strict Crumb Issuer Plugin which
provides more options to customize the crumb validation. It allows
excluding the web session ID from the validation criteria, and instead
e.g. replacing it with time-based expiration for similar (or even
better) protection from CSRF
In my case, it helped for the installation of the Strict Crumb Issuer Plugin, rebooting Jenkins and applying a less strict policy for the web interface of Jenkins as it is suggested on the vendor's site.
According to Jenkins Directive
First you have to check your Jenkins version if the version is < 2.176.2 then per Jenkins guideline CSRF tokens (crumbs) are now only valid for the web session they were created in to limit the impact of attackers obtaining them. Scripts that obtain a crumb using the /crumbIssuer/api URL will now fail to perform actions protected from CSRF unless the scripts retain the web session ID in subsequent requests.
Alternatively, you can install the Strict Crumb Issuer Plugin which provides more options to customize the crumb validation. It allows excluding the web session ID from the validation criteria, and instead e.g. replacing it with time-based expiration for similar (or even better) protection from CSRF.
Steps :
you have to installed the plugin called "Strict Crumb Issuer"
Once installed restart the jenkins service
got to "Manage Jenkins" --> "Configure Global Security" --> Under CSRF Protection, select "Strict Crumb Issue" from the drop down list --> Click on Advance and uncheck everything but select "Prevent Breach Attack" option. --> Apply and save.
Now run you crumb script.
It should work now.
Check this image for your reference
You need a two-step procedure to first get a crumb from the server and then use it.
I am using this Bash script and cURL for that:
#!/bin/bash
# buildme.sh Runs a build Jenkins build job that requires a crumb
# e.g.
# $ ./buildme.sh 'builderdude:monkey123' 'awesomebuildjob' 'http://paton.example.com:8080'
# Replace with your admin credentials, build job name and Jenkins URL
#
# More background:
# https://support.cloudbees.com/hc/en-us/articles/219257077-CSRF-Protection-Explained
USERPASSWORD=$1
JOB=$2
SERVER=$3
# File where web session cookie is saved
COOKIEJAR="$(mktemp)"
CRUMB=$(curl -f -u "$USERPASSWORD" --cookie-jar "$COOKIEJAR" "$SERVER/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,%22:%22,//crumb)")
status=$?
if [[ $status -eq 0 ]] ; then
curl -f -X POST -u "$USERPASSWORD" --cookie "$COOKIEJAR" -H "$CRUMB" "$SERVER"/job/"$JOB"/build
status=$?
fi
rm "$COOKIEJAR"
exit $status
Here is an example of executing this script with the parameters you need:
./buildme.sh 'builderdude:monkey123' 'awesomebuildjob'
Output:
'http://paton.example.com:8080'
This script will return an error code if one of the cURL command fails for any reason.
More details can be found from cloudbees.
I did get the same "403 No valid crumb was included in request" error when I created a Jenkins job from a Java program using jenkins-client library, i.e., com.offbytwo.jenkins. Then I used the Jenkins API token instead of password in the following code. Now, the issue is fixed.
JenkinsServer jServer = new JenkinsServer(new URI(jenkins_url), jnkn_username, jnkn_password);
We can generate an API token from the Jenkins console. Profile → Configure → API Token (Add new token).
The same API token can also be used instead of a password with curl.
curl -v -X POST http://jenkins-url:port/job/<job-name>/buildWithParameters?param=value --user <jen_username>:<jenkins_api_token>
I lost a bunch of time trying to figure this out. At the end, I just installed the plugin Build Authorization Token Root and enabled build permissions to anonymous users.
At the end doesn't really matter, because the Jenkins instance is behind a VPN and I'm using https://smee.io to forward the webhook to the Jenkins instance.
Also the Jenkins instance is behind a reverse proxy, so the "Enable proxy compatibility" option is checked as well, and the "ignore_invalid_headers" setting set to off in the Nginx configuration at the server level. I am sharing my solution just in case someone else is struggling as well. I'm sure there are better ways to do it, but this is one option.
Note that with this plugin the build URL is set to buildByToken/build?job=JobName&token=TokenValue and the token is generated in the job settings.
This is in Jenkins 2.235.2 which doesn't have an option to disable CSRF.
Since this question is the first SO link when searching for "No valid crumb was included in the request" in Google, I think it's worth mentioning that the same error is generated if you omit/forget the Authorization HTTP header or use a blank username/password:
Relevant error messages related to the Authorization header are only generated when a value is passed:
And, yes, the crumb passed in the first screenshots is actually valid; everything works with the correct username/password:
So, not sure if that's a bug or not, but "No valid crumb was included in the request" could also mean you accidentally forgot the Authorization header.
Jenkins 2.222.3, Ubuntu Server 20.04, Java Runtime 1.8.0_252-8u252-b09-1ubuntu1-b09
For me, the below solutions work in Bitbucket:
I updated the URL to:
http://jenkinsurl:8080/bitbucket-hook/
Bitbucket Webhook:
Visiting Jenkins with https://... instead of http://... solved the problem for me.
For me the solution was to pass the X-Forwarded-Host and X-Forwarded-Port headers
as suggested in the reverse-proxy-configuration-troubleshooting chapter of the Handbook.
HaProxy config, inside the frontend section:
http-request set-header X-Forwarded-Host %[hdr(host)]
http-request set-header X-Forwarded-Port %[dst_port]
I had the same issue while using a GitLab webhook with a Jenkins Multibranch pipeline.
On the GitLab webhook page, I changed the Jenkins job URL base path word job to project, as I found on in this link:
From: http://127.0.0.1:8080/job/user-test-repo
To: http://127.0.0.1:8080/project/user-test-repo
I followed this comment: In Dashboard → Manage Jenkins → Configure Global Security. Under CSRF Protection, choose option Enable proxy compatibility. It works for me.
When I was trying to build a job in Jenkins by following options like build steps, accessing Git code, whatever the options, etc., I
faced the error
jenkins-403-no-valid-crumb-was-included-in-the-request
Seriously, I tried a number of ways to resolve it... But there wasn't any luck...!
Surprisingly, I changed my Wi-Fi network, and then it worked.
In my case, I was able to bypass the error by using Remote Desktop into the Jenkins server directly and using
a localhost-based URL instead of trying to go through the corporate proxy from my computer.
I also faced a similar problem. I was using a password instead of a token.
When updated, it solved my problem. There isn't any need to uncheck anything and make it insecure. Below are the complete steps that I followed to have Jenkins CLI working:
Step 1: Prepare environment variables
export JENKINS_URL=http://localhost:8080/
export JENKINS_USER=admin
export JENKINS_PASSWORD=b7f04f4efe5ee117912a1.....
export JENKINS_CRUMB=f360....
export FOLDER=test
Obtain a token as:
How to get the API token for Jenkins
Get the crumb as:
http://localhost:8080/crumbIssuer/api/json
Step 2: Prepare the XML file, file name creds.xml
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>TEST-CLI</id>
<username>test</username>
<password>test123</password>
<description>this secret if created confirms that jenkins-cli is working</description>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
Step 3: POST using cURL
curl -X POST -u $JENKINS_USER:$JENKINS_PASSWORD -H "Jenkins-Crumb:${JENKINS_CRUMB}" -H 'content-type:application/xml' -d #creds.xml "$JENKINS_URL/job/$FOLDER/credentials/store/folder/domain/_/createCredentials"
Here is my solution to this issue (Git hooks to launch a Jenkins job behind a reverse proxy).
Get the crumb from a first call and store the sessionid in a cookie jar:
CRUMB=$(/usr/bin/curl --cookie-jar ./cookie -sX GET https://******.net/crumbIssuer/api/json|cut -d'"' -f8)
Launch the job:
/usr/bin/curl --cookie ./cookie -X POST https://******.net/job/PROJECTNAME/build -H "Jenkins-Crumb: $CRUMB"
The guide CSRF Protection Explained explains how to generate a Jenkins crumb, save the cookies and use both the crumb and the saved cookies in the subsequent requests that require authentication. This is a must for Jenkins after version 2.176.2.
For Java code to access the Jenkins API I will let my advise out.
The answer of Santhosh does resolve the problem. That consists in changing the password for a token, but as far as I know, the token is now a legacy manner to do it.
So I tried other way, and find out a solution inside Java code.
Here how I did it.
In my Java code I use the com.offbytwo.jenkins package and the class that I use is JenkinsServer.
My problem was to create a job in Jenkins because I was getting an error: "403 No valid crumb was included in request"
Then I found a Boolean parameter called crumbFlag and passed true on it and everything worked.
My code was like this:
jenkins.createJob(job.getName(), config);
Then, I changed for this, and it worked like a charm:
jenkins.createJob(job.getName(), config, true);
This parameter is inside almost all methods of this package, by example:
createJob(String jobName, String jobXml, Boolean crumbFlag)
updateJob(String jobName, String jobXml, boolean crumbFlag)
renameJob(String oldJobName, String newJobName, Boolean crumbFlag)
Others.
The technical documentation inside the code is:
#param crumbFlag true to add crumbIssuer
* false otherwise.
I understood if you pass true for this parameter it will issue a crumb automatically.
Well, the official documentation has this information in detail. If you wish, take a look here:
Class JenkinsServer
I had the same issue when trying to set up a GitHub project with the GitHub Pull Request Builder plugin.
Here is an example of the response I was getting from my Jenkins server
Response content
The problem was happening because my payload URL was missing a forward slash at the end, /.
adding a forward slash at the end of the URL solves the problem
your payload URL should look like this: https://jenkins.host.com/ghprbhook/
Examples after adding the forward slash
I am running with a reverse proxy with nignx. I changed a Jenkins option in the "Configure Global Security", that "Enable proxy compatibility".
This fixed with my issue.
First create a user API token by going to user → API Token → Add new token.
Then use the below script for triggering.
import jenkins,requests
job_name='sleep_job'
jenkins_url = "http://10.10.10.294:8080"
auth = ("jenkins","1143e7efc9371dde2e4f312345bec")
request_url = "{0:s}/job/{1:s}/buildWithParameters".format(jenkins_url,
job_name, )
crumb_data = requests.get("{0:s}/crumbIssuer/api/json".format(jenkins_url),
auth=auth, ).json()
headers = {'Jenkins-Crumb': crumb_data['crumb']}
jenkins_job_params={}
jenkins_job_params['NODE_NAME']='10_10_1_29'
jenkins_job_params['SLEEP_TIME']='1h'
response = requests.post(request_url, data=jenkins_job_params, auth=auth, )
response.raise_for_status()
Head over to Manage Jenkins => Configure global security.
Then uncheck "Prevent Cross Site Request Forgery exploits"
I have run into the same issue. I have only refreshed my browser, logged back in to Jenkins, did the same process and everything worked.