Poetry install private package started suddenly failing in Docker - docker

Question

ABOUT
MINIMAL WORKING EXAMPLE: https://gitlab.com/hynek.blaha/debug-docker-poetry/-/tree/master
I have been building Docker images using Poetry with Python packages from internal PyPI registry. As our projects are in private GitLab repository and the internal packages are not top-secret, we are storing the poetry credentials directly in the poetry source URL in pyproject.toml.
On 2022-08-24, all our Docker builds started failing while installing internal package:
• Installing til-bigquery (0.3.4)
HTTPError
401 Client Error: Unauthorized for url: https://gitlab.com/api/v4/projects/38869805/packages/pypi/files/7a4731d831d4b37262481002271e359f96017570e9480ef16c89489e0b41252f/til_bigquery-0.3.4-py3-none-any.whl#sha256=7a4731d831d4b37262481002271e359f96017570e9480ef16c89489e0b41252f
at /usr/local/lib/python3.9/site-packages/requests/models.py:1021 in raise_for_status
1017│ f"{self.status_code} Server Error: {reason} for url: {self.url}"
1018│ )
1019│
1020│ if http_error_msg:
→ 1021│ raise HTTPError(http_error_msg, response=self)
1022│
1023│ def close(self):
1024│
1025│ called the underlying ``raw`` object must not be accessed again.
What I found weird:
Docker build fails even when I retry deploy job, that successfully passed a few days ago.
Considering the issue might have been caused by unpinned minor version of Docker base image python:3.7-slim or Poetry, I used older versions but got the same result.
I compared the build logs of previously successful build build_success.log (8/22/22, 3:00 PM) and the same build retry build_fail.log (8/24/22, 6:00 AM) and found both use the same poetry wheel poetry-1.1.15-py2.py3-none-any.whl.
It still works as before on my machine, but fails in Docker.
It stops working on localhost when I remove the credentials from the repository URL, so I am sure the credentials are not stored anywhere else (e.g. ~/.netrc).
How to reproduce:
Localhost - OK
git clone git#gitlab.com:hynek.blaha/debug-docker-poetry.git
poetry install
Docker - FAIL
git clone git#gitlab.com:hynek.blaha/debug-docker-poetry.git
docker build .
I am able to fix the issue by explicitly providing the credentials in Dockerfile:
RUN pip install poetry --no-cache-dir && \
poetry config virtualenvs.create false && \
poetry config repositories.my_private_repo https://gitlab.com/api/v4/projects/21870843/packages/pypi/simple && \
poetry config http-basic.my_private_repo __token__ glpat-mkEPJ4Rsy2peTCrH23pG
But it doesn't explain, why rebuilding the same image started failing.
And why it still works as expected when running on my machine (outside of Docker).
Does anyone have an idea, what might have changed? I was unable to tell what even when using diff on the build_success.log and build_fail.log

Answer 1

I was struggling with totally the same problem the last few days. Though I'm still not sure of the exact cause, somehow I managed to avoid the problem.
I also used a repository URL with credentials embedded in pyproject.toml until yesterday like this.
[[tool.poetry.source]]
name = 'private'
url = 'https://your_username:your_password#gitlabce.example.com/api/v4/projects/<project_id>/packages/pypi/simple'
secondary = true
Though it's basically the same as your solution, you can specify the local poetry configs for each project by creating poetry.toml at the project root. So instead of embedding credentials in the URL, you can specify them via poetry.toml as follows:
[http-basic]
[http-basic.private]
username = "your_username"
password = "your_password"
That way, you can reproduce the same behaviour with embedded credentials without the authentication error.
Why does it still work outside docker?
I guess it is due to the archive cache in your local environment. Since poetry stores downloaded archives in ~/.cache/pypoetry/artifacts/ and reuse them when execute poetry install, you didn't need to access your private PyPI server in the first place. If you manually remove archives (I'm not sure, but poetry cache clear command didn't work for my case), you'll be able to reproduce the authentication error even in your local environment.
Why did the error suddenly start to occur?
Poetry uses the embedded credentials to query the list of package links to the private PyPI server.
It works fine until there, however, when installing actual packages, Poetry uses the acquired link from the PyPI server that has no credentials embedded. That's why it fails and the URL shown in the error message doesn't have any credentials embedded.
I'm still not sure why the credential embed had been working until a few days ago though. I guess there might have been behavior changes on the GitLab side.