Importing github repo as dependency in poetry in docker container - docker

Issue:
I am trying to add a bitbucket repo using an ssh key as a tool.poetry.dev.dependency in my pyproject.toml and have it installed in a docker container via docker compose. However, I encounter the following errors (see Errors) when I build the container.
Is there anything that I may be overlooking and/or suggestions to be able to pull a bitbucket repo as a dependency during poetry install in a docker container?
What I have done:
I know the bitbucket dependency is the issue because when I exchange it to a local path to a whl file instead, the container builds correctly.
I can install the dependency locally
I can clone a different repo in the docker container
REFERENCES
Errors:
HangupException
The remote server unexpectedly closed the connection.
at /usr/local/lib/python3.10/site-packages/dulwich/protocol.py:220 in read_pkt_line
216│
217│ try:
218│ sizestr = read(4)
219│ if not sizestr:
→ 220│ raise HangupException()
221│ size = int(sizestr, 16)
222│ if size == 0:
223│ if self.report_activity:
224│ self.report_activity(4, "read")
The following error occurred when trying to handle this error:
HangupException
Host key verification failed.
at /usr/local/lib/python3.10/site-packages/dulwich/client.py:1151 in fetch_pack
1147│ with proto:
1148│ try:
1149│ refs, server_capabilities = read_pkt_refs(proto.read_pkt_seq())
1150│ except HangupException as exc:
→ 1151│ raise _remote_error_from_stderr(stderr) from exc
1152│ (
1153│ negotiated_capabilities,
1154│ symrefs,
1155│ agent,
My Files/Commands:
pyproject.toml
[tool.poetry.dependencies]
# Trial 1: I am trying to use ssh key to pull to repo (see docker build command) [1]
package_name = {git = "ssh://git#bitbucket.org/tenant/repo.git", tag = "v0.0.0"}
# Trial 2: I don't really want to use http because I don't want to have to feed in credentials [1]
package_name = {git = "https://user#bitbucket.org/tenant/repo.git", tag = "v0.0.0"}
# Trial 3: I didn't know if it was just a bitbucket thing so I had also tried with git and it works locally just not in a docker container
package_name = {git="git+ssh://git#github.com/user/repo.git"}`
Dockerfile
ENV POETRY_VERSION=1.2.2
RUN pip install poetry==$POETRY_VERSION
COPY ./poetry.lock ./pyproject.toml ./
RUN poetry config installer.max-workers 4 \
&& poetry install --no-root`
Docker build command
docker build --no-cache --ssh default -t $IMAGE_NAME .
Docker compose command
docker compose build

I had the same error thrown at me after trying to install a private package in a container via git. I could get it to work using https instead of ssh.
It might help if you run the install command with -vvv to get a more verbose error.
AFAICT, this seems to be an underlying issue with dulwich.
If you really don't want to use https, I'd go report this on the corresponding poetry issue, hopefully the poetry team can find a workaround until dulwitch rolls out a fix.

Related

How to change owner for docker socket in container (macOS, Intel Chip)

I have a fresh install of docker desktop on my machine and I'm attempting to create a dev environment. Using docker.io/library/wordpress:latest
However, I'm having some issues with user permissions. From what I can see the documentation doesn't mention this issue for mac users, but does mention something for ubuntu users (See Doc's). The specific issue is as follows;
// Docker error msg...
chown: invalid group: 'root:docker'
WARNING: Could not change owner for docker socket in container : exit code 1
Docker socket permission set to allow in container docker
// My setup...
macOS BigSir 11.6.5(Intel Chip)
Docker Desktop 4.8.2
VSCode Version: 1.67.1
git version 2.36.1
My Question: How do I resolve this issue? I.e. What steps do I need to take?
Any guidance would be greatly appreciated... 😅
Note: I can see other questions floating around here on stack, but from what I can see they're mostly covering ubuntu users or quite old questions and answers.
Note: Added screenshots to demonstrate what I was doing when the error occurred.
Step 1
Step 2
Step 3 -- Error
Most docker image in public registry is not designed to compatible with Docker Dev Environments (Beta).
According to documentation: https://docs.docker.com/desktop/dev-environments/specify/, specified docker image is required to have docker group and vscode user which already included to the group.
So we need to modify official docker image for your case to be working fine with docker dev environments and visual studio code.
# Dockerfile
FROM docker.io/library/wordpress:latest
# the next two commands are required based on documentation
# https://docs.docker.com/desktop/dev-environments/specify/
RUN useradd -s /bin/bash -m vscode \
&& groupadd docker \
&& usermod -aG docker vscode
USER vscode
Build our new docker image by running docker build, e.g:
docker build -t bickyeric/wordpress:dev-latest -f Dockerfile .
After that, you can update image tag from Step 2 of Question with our new image bickyeric/wordpress:dev-latest

Dockerfile FROM Insecure Registry

Is there a way to build a docker image from a Dockerfile that uses a base image from a local, insecure registry hosted in Gitlab. For example, if my Dockerfile were:
FROM insecure.registry.local:/mygroup/myproject/image:latest
When I run docker build . I get the following error:
failed to solve with frontend dockerfile.v0: failed to create LLB definition:.... http: server gave HTTP response to HTTPS client
When I've been interacting with our registry and received similar types http/https errors, I would alter the docker daemon configuration file to include:
...
"insecure-registries" : ["insecure.registry.local"]
...
and everything would work when executing a particular docker command that would trigger it. I'm able to run docker login insecure.registry.local successfully. Is there a way either in the Dockerfile itself, through the docker build command or other alternative to have it pull the image successfully in the FROM statement from an insecure registry?
Depending on your version, you may need to include the scheme in the insecure registry definition. Newer versions of buildkit should not have this issue, so an upgrade may also help.
...
"insecure-registries" : [
"insecure.registry.local",
"http://insecure.registry.local"
]
...
Unfortunately there is not that much information about the actual error.
So here are a couple of things that may fix the issue you described:
Ensure your file is called Dockerfile (only the D is supposed to be capitalized)
Reload and restart the docker daemon after your changes (sudo systemctl daemon-reload && sudo systemctl restart docker)
Don't use the docker buildkit (export DOCKER_BUILDKIT=0 && export COMPOSE_DOCKER_CLI_BUILD=0 && docker build .

Docker Build CMD fail yum not able to install the requirements

My docker build cmd is failing to create an image using Dockerfile. It shows this error
here is the screenshot of the error
Check if you can access the site on the host machine.
Check your docker networking, for a docker VM, it is usually a bridge network by default.
Check if you need to add the repository to YUM.

Docker build error: "could not connect to server" (behind proxy)

Context:
OS: Windows 10 Pro; Docker ver: 18.09.0 (build 4d60db4); Behind corporate proxy, using CNTLM to solve this issue. (currently pulling / running image works fine)
Problem:
I was trying to build the following Dockerfile:
FROM alpine:3.5
RUN apk add --update \
python3
RUN pip3 install bottle
EXPOSE 8000
COPY main.py /main.py
CMD python3 /main.py
This is what I got:
Sending build context to Docker daemon 11.26kB
Step 1/6 : FROM alpine:3.5
---> dc496f71dbb5
Step 2/6 : RUN apk add --update python3
---> Running in 7f5099b20192
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.5/main: could not connect to server (check repositories file)
WARNING: Ignoring APKINDEX.c51f8f92.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.5/community: could not connect to server (check repositories file)
WARNING: Ignoring APKINDEX.d09172fd.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
python3 (missing):
required by: world[python3]
The command '/bin/sh -c apk add --update python3' returned a non-zero code: 1
I was able to access the URL from a browser, so there is no problem with the server itself.
I suspected that it has something to do with the proxy not being propagated to the container, as explained in this question, since I also did not get the http_proxy line when running docker run alpine env. However, after entering the proxies into the config file, it finally appeared. Yet the problem still exists.
I also tried to change the DNS as instructed here, but the problem is still unsolved.
I finally managed to solve this problem, and the culprit was my setting in the CNTLM.
For a background story, please check this post.
The root cause of this problem is that the docker container could not access the internet from inside the VM due to wrong IP setting inside the CNTLM.ini.
Normally CNTLM listens to 127.0.0.1:3128 by default to forward the proxy. I followed the default, and thus set the proxy setting on Docker (for the daemon - through the GUI, and for the container - through config.json) is also set into that address and port. It turns out that this "localhost" does not apply to the VM where docker sits, since the VM has its own localhost. Long story short, the solution is to change that address into dockerNAT IP address (10.0.75.1:3128) in all of the following locations:
CNTLM.ini (on the Listen line. Actually if we use CNTLM for other purposes as well, it is possible to supply more than one Listen line)
Docker daemon's proxy (through the Docker setting GUI)
Docker container config.json (usually in C:\Users\<username>\.docker), by adding the following lines:
"proxies":
{
"default":
{
"httpProxy": "http://10.0.75.1:3128",
"httpsProxy": "http://10.0.75.1:3128",
"noProxy": <your no_proxy>
}
}
also check these related posts:
Building a docker image for a node.js app fails behind proxy
Docker client ignores HTTP_PROXY envar and build args
Beginner having trouble with docker behind company proxy
You can try to build your docker file with the following command:
docker build --build-arg http_proxy=http://your.proxy:8080 --build-arg http_proxy=http://your.proxy:8080 -t yourimage .

Unable to load the service index for source https://api.nuget.org/v3/index.json. [C:\app\aspnetapp.csproj]

https://github.com/dotnet/dotnet-docker-samples/tree/master/aspnetapp
Docker command docker build -t aspnetapp.
I am getting an error for docker build command as
C:\Program Files\dotnet\sdk\2.1.105\NuGet.targets(104,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [C:\app\aspnetapp.csproj]
[1]PS C:\Users\pssharma\dotnet-docker-samples\aspnetapp> docker build -t aspnetapp .
Sending build context to Docker daemon 2.444MB
Step 1/10 : FROM microsoft/aspnetcore-build:2.0 AS build-env
---> eb21d939e0d8
Step 2/10 : WORKDIR /app
---> Using cache
---> d4ec30216ed7
Step 3/10 : COPY *.csproj ./
---> Using cache
---> 2ff39b5e6cb4
Step 4/10 : RUN dotnet restore
---> Running in 776764a35311
Restoring packages for C:\app\aspnetapp.csproj...
Restoring packages for C:\app\aspnetapp.csproj...
C:\Program Files\dotnet\sdk\2.1.105\NuGet.targets(104,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [C:\app\asp
C:\Program Files\dotnet\sdk\2.1.105\NuGet.targets(104,5): error : An error occurred while sending the request. [C:\app\aspnetapp.csproj]
C:\Program Files\dotnet\sdk\2.1.105\NuGet.targets(104,5): error : The operation timed out [C:\app\aspnetapp.csproj]
The following command returned a non-zero code: 1
powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; dotnet restore
This appears to happen when there are multiple network adaptors present for the host and the priority of said adaptor is misconfigured. Run the following to display a table of your network adaptors:
Get-NetIPInterface -AddressFamily IPv4 | Sort-Object -Property InterfaceMetric -Descending
You want your primary adaptor (in my case wifi) to have the lowest InterfaceMetric.
Set-NetIPInterface -InterfaceAlias 'Wi-Fi' -InterfaceMetric 1
My network adaptor table for reference:
See this github thread for more information:
https://github.com/docker/for-win/issues/2760#issuecomment-430889666
For me, this issue appeared randomly (along with multiple similar issues, such as "permission denied for nuget" etc. To fix that, restart Visual Studio)
For windows, it was fixed by restarting docker service and restarting Visual Studio (2017).
Shut down Docker and Visual Studio
Winkey + R, type in services.msc
Find Docker for Windows Service and restart it (restarting Docker normally is not enough)
Start Docker and Visual Studio
Just add --network=host option to docker build
docker build -t aspnetapp . --network=host
I got this working by passing in docker build environment variables from the command line like so:
docker build --build-arg HTTP_PROXY=http://{un}:{pw}#{proxyhost}:{port} --build-arg HTTPS_PROXY=http://{un}:{pw}#{proxyhost}:{port} -t aspnetapp .
*note that your username must be url encoded (ei. DOMAIN\name => DOMAIN%5Cname)
Had the same issue, the problem was a missing dependency used by NuGet.
I am using the microsoft/dotnet:sdk image, and it works after installing libcurl3 to my image.
Try adding
RUN apt-get update && apt-get install -y libcurl3
to your Dockerfile before restoring.
if it doesn't work, try restarting your Docker instance.
edit: if that doesn't help you, there is this open issue on the NuGet github where some people are trying to solve the same problem.
edit2:added Ignas solution: restarting docker instance.
I'm suprised no one said this already. This was caused by not having the latest sdk installed. In my case docker was looking for the file
C:\Program Files\dotnet\sdk\3.1.302\NuGet.targets
However this file did not exist on my machine instead I had a different minor version.
C:\Program Files\dotnet\sdk\3.1.301\NuGet.targets
I'm sure they're away to get it working using an older version, but in my case I just updated the downloaded the latest sdk
If the issue is caused by your organisation's self-signed certificates, you can fix it by:
Get the certificate file (*.cer) provided by your IT department.
Copy it into the SSL certificate location: COPY ["./combined.cer", "/etc/ssl/certs/"]
Update SSL cert list: RUN update-ca-certificates
These steps work for Debian, need to be adapted for other distros.
configuring the dns in the docker engine helped me
you go to docker setting then to docker engine you can see a json formatted doc
you add this :
"dns": ["10.0.0.2","8.8.8.8"]
then restart docker
i wish that helps you
Open CMD or Powershell and run: dotnet nuget locals http-cache --clear
https://status.nuget.org/
In my case, in Dockerfile, I have
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build
and
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
But I only have .net 2.1 installed (view by dotnet --list-sdks). So I have to change 2.2 to 2.1 to make it work.
After one hour fighting with proxy, finally following solution worked just fine for me:
docker build --build-arg HTTP_PROXY=http://xxxx.com:8080 --build-arg HTTPS_PROXY=http://yyyy:8080 -t kdekarin/aspnet .
See #patrickbadley's comment for more info!
One more reason why this happens. If your docker file targeting windows container and your docker is running linux mode or vice versa.
I had the same issue on Debian 9. I fudge the error with disconnecting from local network. It works for me because I did not change anything on codebase and I do not need to get packages from nuget.
I got this resolved on my Windows 10 Machine by Disabling Avast Antivirus and unInstalling McAffee Antivirus.
Right click the Avast Icon in the Notification Tab
Select Avast Shields Control
Click on "Disable Until Computer is Restarted".
And you can now run the Docker Build Successfully.
I can give Further Assistance if needed.

Resources