When running npm install within circleci we fetch some node packages from our github repositories through package.json. This operation is happening when building a docker image from a Dockerfile.
This has been working great until last week when without changes in our side, we started to get errors while cloning these packages. To perform this operation, we were using Basic Authentication in the URL providing user credentials in it. For ie:
https://<username>:<password>#github.com/elektron-technogoly/<repository>.git
Now, we get the following errors:
npm ERR! Command failed: git clone ...
npm ERR! fatal: unable to access 'https://<username>:<password>#github.com/elektron-technogoly/<repository>.git':
Could not resolve host: <username>
From the error message it seems like it thinks the username is the host and thus, fails. I checked that password is still valid and it did not expire.
Has recently - around last week - something changed that could cause this error? Has Basic Authentication been disabled?
UPDATE: Playing a bit seems like when you change the base docker image (say from node:4-slim to node:4), the first time it works, subsequent times don't. Unfortunately, logs are not giving me any lead, both look exactly the same but the error appears from the first onwards.
Related
I have a Jenkins pipeline where a Docker (Podman) image is built und pushed to a private Gitea docker registry. This basically works. But I have the problem, that the first build after several hours, that means on the next day, crashes because pushing to the Gitea registry leads to:
Error: writing blob: uploading layer to https://192.168.0.5:4000/v2/myorg/myproject/blobs/uploads/ptuh7yizsrqvx5wlg9uctlzdv?digest=sha256%3A7ca0dabc572c112e5141bac7e5f29a0c1b1f727ce939ac1e7da342d3adf324a: received unexpected HTTP status: 500 Internal Server Error
When I click on the link, it shows me:
errors
0
code "UNAUTHORIZED"
message ""
Since I trigger the login from a remote host via Jenkins pipeline, I do that with a script, but I'm pretty sure that this does not matter. The content at the script is:
#!/bin/bash
podman login -u builderuser -p builderpassword 192.168.0.5:4000
I see in the logs that Login Succeeded!, but after the next step, I get the error from above after it tries to copy the blob to the registry.
I also tried to add a "logout" before login in the script via
podman logout 192.168.0.5:4000
But this does not help either.
When I trigger the same build again, the process works without problems. Maybe some caching problem anywhere? The problem appears at the first build on the next day, so I guess there is some session timeout anywhere after several hours. Any ideas?
[UPDATE]
I think this is a bug in Gitea. In log I see this:
Nov 11 08:50:40 server gitea[34985]: 2022/11/11 08:50:40 ...es/container/blob.go:66:func1() [E] [636dfed0-7] Error inserting package: pq: duplicate key value violates unique constraint »UQE_package_version_s«
And in code of Gitea in auths.go, I see a comment leading me to the assumption that they are aware of this problem:
// FIXME: if the name conflicts, it will result in 500: Error 1062: Duplicate entry 'aa' for key 'login_source.UQE_login_source_name'
What I still don't understand is, why this happens only once, at the beginning of the day, and not always.
Did you check the logs on the registry?
If you get something like
<path>/registry/docker: permission denied
it means the error happens if your user does not have the permission to write.
If on the permission side you're ok, than the error shown is quite random and has the same root cause as gitlab-org/gitlab#215715
The error has been fixed for gitlab. You'd need to check on Jenkins if they have some open issues similar to this.
ABOUT
MINIMAL WORKING EXAMPLE: https://gitlab.com/hynek.blaha/debug-docker-poetry/-/tree/master
I have been building Docker images using Poetry with Python packages from internal PyPI registry. As our projects are in private GitLab repository and the internal packages are not top-secret, we are storing the poetry credentials directly in the poetry source URL in pyproject.toml.
On 2022-08-24, all our Docker builds started failing while installing internal package:
• Installing til-bigquery (0.3.4)
HTTPError
401 Client Error: Unauthorized for url: https://gitlab.com/api/v4/projects/38869805/packages/pypi/files/7a4731d831d4b37262481002271e359f96017570e9480ef16c89489e0b41252f/til_bigquery-0.3.4-py3-none-any.whl#sha256=7a4731d831d4b37262481002271e359f96017570e9480ef16c89489e0b41252f
at /usr/local/lib/python3.9/site-packages/requests/models.py:1021 in raise_for_status
1017│ f"{self.status_code} Server Error: {reason} for url: {self.url}"
1018│ )
1019│
1020│ if http_error_msg:
→ 1021│ raise HTTPError(http_error_msg, response=self)
1022│
1023│ def close(self):
1024│
1025│ called the underlying ``raw`` object must not be accessed again.
What I found weird:
Docker build fails even when I retry deploy job, that successfully passed a few days ago.
Considering the issue might have been caused by unpinned minor version of Docker base image python:3.7-slim or Poetry, I used older versions but got the same result.
I compared the build logs of previously successful build build_success.log (8/22/22, 3:00 PM) and the same build retry build_fail.log (8/24/22, 6:00 AM) and found both use the same poetry wheel poetry-1.1.15-py2.py3-none-any.whl.
It still works as before on my machine, but fails in Docker.
It stops working on localhost when I remove the credentials from the repository URL, so I am sure the credentials are not stored anywhere else (e.g. ~/.netrc).
How to reproduce:
Localhost - OK
git clone git#gitlab.com:hynek.blaha/debug-docker-poetry.git
poetry install
Docker - FAIL
git clone git#gitlab.com:hynek.blaha/debug-docker-poetry.git
docker build .
I am able to fix the issue by explicitly providing the credentials in Dockerfile:
RUN pip install poetry --no-cache-dir && \
poetry config virtualenvs.create false && \
poetry config repositories.my_private_repo https://gitlab.com/api/v4/projects/21870843/packages/pypi/simple && \
poetry config http-basic.my_private_repo __token__ glpat-mkEPJ4Rsy2peTCrH23pG
But it doesn't explain, why rebuilding the same image started failing.
And why it still works as expected when running on my machine (outside of Docker).
Does anyone have an idea, what might have changed? I was unable to tell what even when using diff on the build_success.log and build_fail.log
I was struggling with totally the same problem the last few days. Though I'm still not sure of the exact cause, somehow I managed to avoid the problem.
I also used a repository URL with credentials embedded in pyproject.toml until yesterday like this.
[[tool.poetry.source]]
name = 'private'
url = 'https://your_username:your_password#gitlabce.example.com/api/v4/projects/<project_id>/packages/pypi/simple'
secondary = true
Though it's basically the same as your solution, you can specify the local poetry configs for each project by creating poetry.toml at the project root. So instead of embedding credentials in the URL, you can specify them via poetry.toml as follows:
[http-basic]
[http-basic.private]
username = "your_username"
password = "your_password"
That way, you can reproduce the same behaviour with embedded credentials without the authentication error.
Why does it still work outside docker?
I guess it is due to the archive cache in your local environment. Since poetry stores downloaded archives in ~/.cache/pypoetry/artifacts/ and reuse them when execute poetry install, you didn't need to access your private PyPI server in the first place. If you manually remove archives (I'm not sure, but poetry cache clear command didn't work for my case), you'll be able to reproduce the authentication error even in your local environment.
Why did the error suddenly start to occur?
Poetry uses the embedded credentials to query the list of package links to the private PyPI server.
It works fine until there, however, when installing actual packages, Poetry uses the acquired link from the PyPI server that has no credentials embedded. That's why it fails and the URL shown in the error message doesn't have any credentials embedded.
I'm still not sure why the credential embed had been working until a few days ago though. I guess there might have been behavior changes on the GitLab side.
Locally, I am able to successfully authorize and pull modules from my private Nexus registry (.npmrc file).
However on Jenkins I get
error An unexpected error occurred: "https://myprivaterepo.com/myprivatemodule.tgz: Request failed \"401 Unauthorized\"".
When I run npm whoami on Jenkins it returns a valid user. npm config ls prints valid configuration as well.
The problem started to occur when I changed myprivaterepo url (we've migrated it). Is there something I don't know (ie. I have to logout/login again or there's some cache in Jenkins)??
Thanks in advance!
I was trying to build the docker image for a project I'm working onto.
It's based on jhipster, after configuring the project it tells me to run the following maven command:
./mvnw -ntp -Pprod verify jib:dockerBuild
Unfortunately it doesn't seem to work, it returns me this errors:
[WARNING] The credential helper (docker-credential-pass) has nothing for server URL: registry.hub.docker.com
...
[WARNING] The credential helper (docker-credential-pass) has nothing for server URL: index.docker.io
[WARNING]
And finally fails with:
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.4.0:dockerBuild (default-cli) on project booking: (null exception message): NullPointerException -> [Help 1]
Recently I worked on a google cloud project, and I edited the ~/.docker/config.json configuration file. I had to remove google's configuration entries to sort out another problem. Could that be the origin of the problem I'm facing now?
I've tried to do docker logout and docker login without success.
Some considerations
I don't know if editing manually the configuration caused the error, in fact I'm pretty sure to have deleted only google-related entries, but nothing referring to docker.* or similar.
To solve this issue, avoid to edit manually the docker configuration file. In fact I think that it should be avoided whenever possible, to avoid configuration problems of any sort.
Instead, just follow what the error message is trying to tell you: docker is not able to access those urls. Excluding network problems (which you can troubleshoot with ping registry-1.docker.io for example), it should be an authentication problem.
How to fix
I've found out that running those commands fixed it:
docker login registry.hub.docker.com
docker login registry-1.docker.io
I don't know if registry-1.docker.io is just a mirror of the other first server, which the plugin tries to access after the first unsuccessful connection. You can try to loging to registry.hub.docker.com and re-launch the command to see if it sufficient. In case it's not, login to the second one and then it will work.
I ran jib via Gradle:
./gradlew jibDockerBuild
and got a similar error
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':jibDockerBuild'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Build to Docker daemon failed, perhaps you should make sure your credentials for 'registry-1.docker.io/library/openjdk' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help
What ended up solving this error for me, bizarrely enough, was to log out of Docker Desktop.
I later also tried funder7's solution while logged in to Docker Desktop, and that also worked.
Hi I have a problem that for some reason I get blocked form a private corporate repo from installing dependencies, it seems that due to the large amount of requests I generate when running bower install I get blocked... this is a known issue, and fixing it could take a couple of days.
I wonder if there is a way to stop - pause and ask for user to touch a key to continue the installation?
this way I would be able to overcome this problem that is out my scope fixing and I need to install the component to work. any hint?
I get this message
ECMDERR Failed to execute "git ls-remote --tags --heads
git#github.company.com:components/something.git", exit code of #128 client_loop:
send disconnect: Connection reset by peer fatal: Could not read from remote
repository. Please make sure you have the correct access rights and the
repository exists.
The repo exists and I do have access rights.
I have confirmed the cause of this issue is that private repo is wrongly cutting my connection due to the large amounts of requests, this triggers a brute force attack protection... the solution could take days to be completed in large company with processes that need to be initilizied with the corresponding teams.
A very tedious workaround is to install manually each package or to cut the bower.json package in parts and run them in separate batches.