Adding a dir to root in docker still wont allow git access - docker

I have a private git repo on bitbucket that I'm using to pip install a library. During my docker build, I copy the dir with the keys and config file into root. Then it pulls down the requirements and pip installs them. (It pip installs fine when I'm just using my local terminal, so I know it's not the pip install.) However, I keep getting a
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
If I remove the line in the Dockerfile that install the pip install that causes the image to fail, and then shell into the instance and pip install it works fine.
My directory looks like this:
app/
requirements.txt
docker_keys/
.ssh/
id_rsa
id_rsa.pub
config
My Dockerfile looks like this:
FROM python:3.5
RUN apt-get update && apt-get dist-upgrade -qqy && apt-get clean && rm -rf /var/lib/apt/lists/*
ENV PYTHONUNBUFFERED 1
RUN pip install --upgrade pip uwsgi
RUN mkdir /app
ADD . /app
WORKDIR /app
COPY docker_keys/.ssh /root/.ssh
RUN pip install -r requirements.txt
I'm assuming it has something to do with how I'm copying the key dir into root. Any help would be greatly appreciated.

Docker won't necessarily read from your identity file. Check /etc/ssh/ssh_config to ensure something like this exists:
IdentityFile ~/.ssh/id_rsa"
It's worth noting however that this is really insecure, and you shouldn't leave SSH private keys inside docker files.

Related

got pip error while trying to convert an existing docker file to use distroless image

I have a dockerfile in which i am using python:3.9.2-slim-buster as base image and i am doing the following stuff.
FROM lab.com:5000/python:3.9.2-slim-buster
ENV PYTHONPATH=base_platform_update
RUN apt-get update && apt-get install -y curl && apt-get clean
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin
WORKDIR /script
RUN pip install SomePackage
COPY base_platform_update ./base_platform_update
ENTRYPOINT ["python3", "base_platform_update/core/main.py"]
I want to convert this to use distroless image. I tried but its not working. I found these resources
https://github.com/GoogleContainerTools/distroless/blob/main/examples/python3/Dockerfile
https://www.abhaybhargav.com/stories-of-my-experiments-with-distroless-containers/
I know this is not correct but this is what i came up with after following these resources
# first stage
FROM lab.com:5000/python:3.9.2-slim-buster AS build-env
WORKDIR /script
COPY base_platform_update ./base_platform_update
RUN apt-get update && apt-get install -y curl && apt-get clean
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
RUN mv ./kubectl /usr/local/bin
# second stage
FROM gcr.io/distroless/python3
WORKDIR /script
COPY --from=build-env /script/base_platform_update ./base_platform_update
COPY --from=build-env /usr/local/bin/kubectl /usr/local/bin/kubectl
COPY --from=build-env /bin/chmod /bin/chmod
COPY --from=build-env /usr/local/bin/pip /usr/local/bin/pip
RUN chmod +x /usr/local/bin/kubectl
ENV PYTHONPATH=base_platform_update
RUN pip install SomePackage
ENTRYPOINT ["python3", "base_platform_update/core/main.py"]
it gives the following error:
/bin/sh: 1: pip: not found
The command '/bin/sh -c pip install SomePackage' returned a non-zero code: 127
I also thought of moving RUN pip install SomePackage to first stage but the couldn't figure out how to do that.
Any help would be appreciated. Thanks
EDIT:
docker images output
gcr.io/distroless/python3 latest 7f711ebcfe29 51 years ago 52.2MB
gcr.io/distroless/python3 debug 7c587fbe3d02 51 years ago 53.3MB
It could be that you need to add that dir to the PATH.
ENV PATH="/usr/local/bin:$PATH"
consider though the final image size difference after adding all those dependencies, it might not be worth all the hassle.
the latest image tagged as python:3.8.5-alpine is 42.7MB while gcr.io/distroless/python3 as of writing this is 52.2MB, after adding the binaries, the script, and nonetheless the package you want to install you may surpass that figure at the end. If pull time is important and network bandwidth usage is expensive that might be a thought to have, otherwise for the current use case seems like too much.
Distroless images are meant only for runtime, as a result, you can't (by default) use the python package manager to install packages, see Google GitHub project readme
"Distroless" images contain only your application and its runtime
dependencies. They do not contain package managers, shells or any
other programs you would expect to find in a standard Linux
distribution.
you could install the packages in a second new stage and copy the installed packages from it to the third but that's not bound to work cause of target OS the package was meant for, incompatibility between the second and third stage etc`.
Here's an exame Dockerfile for that:
# first stage
FROM python:3.8 AS builder
COPY requirements.txt .
# install dependencies to the local user directory (eg. /root/.local)
RUN pip install --user -r requirements.txt
# second unnamed stage
FROM python:3.8-slim
WORKDIR /code
# copy only the dependencies installation from the 1st stage image
COPY --from=builder /root/.local /root/.local
COPY ./src .
# update PATH environment variable
ENV PATH=/root/.local:$PATH
CMD [ "python", "./server.py" ]
Dockerfile credits
You could package your application to a binary using any number of python libs but that depends on how much you need it. You can do that with packages like pyinstaller though it mainly packages the project rather than turning it to a single binary, nuitka which is a rising option and very popular along with cx_Freeze.
Here's a relevant thread on the topic if you're interested.
There's also this article.

Permissions problem in Docker container built in Ubuntu VM composed of files created on Windows host

I work on a project that has a large number of Java SpringBoot services (and other types) running in k8s clusters. Each service has a small start script that executes a more complex script that is provided in a configmap. This all works fine in builds and at runtime.
I need to make some changes to that complex script. I've already made the changes and tested the concept in an isolated script. I still need to do more testing of it. I am attempting to take some of the command lines that run in our Linux build system and run them on my VirtualBox Ubuntu VM that runs on my Windows 10 laptop. Although I am running this on the VM, most of the files were created and written on the host Windows 10 laptop that I get to using a VirtualBox Shared Folder.
When I look at the "ls -l" output of "startService.sh", I just get this:
-rwxrwx--- 1 root vboxsf 634 Aug 24 15:07 startService.sh*
Note that I am running docker with my own uid, and I have that uid in the "vboxsf" group.
It seems like when the file gets copied into the image, either the owner or the perms get changed in a way that make it inaccessible from within the container.
I tried adding a "RUN chmod 777 startService.sh" in the Dockerfile, just before the ENTRYPOINT, but that fails at build time with this:
Step 23/26 : RUN chmod 777 startService.sh
---> Running in 6dbb89c930c1
chmod: startService.sh: Operation not permitted
The command '/bin/sh -c chmod 777 startService.sh' returned a non-zero code: 1
I don't know why this is happening, or whether this is something that might mitigate this.
My "docker build" command looks like it went fine. I saw it execute all the steps that the normal build shows. The "docker run" step seemed to go fine, but it finished very quickly. When I looked at the "docker log" for the container, it just said entirely:
/bin/sh: ./startService.sh: Permission denied
Note that everything here is done the same way it is on the build server. There seems to be something funny with the fact that I'm running an Ubuntu
You have to write chmod +x startService.sh before docker run or docker-compose up -d --build
And example Dockerfile for django. Look at actions with wait-for, you must make same
###########
# BUILDER #
###########
# pull official base image
FROM python:3.8.3-slim as builder
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install psycopg2 dependencies
RUN apt-get update \
&& apt-get -y install libpq-dev gcc \
python3-dev musl-dev libffi-dev\
&& pip install psycopg2
# lint
RUN pip install --upgrade pip
COPY . .
# install dependencies
COPY ./requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt
# copy project
COPY . .
#########
# FINAL #
#########
# pull official base image
FROM python:3.8.3-slim
# create directory for the app user
RUN mkdir -p /home/app
# create the app user
RUN addgroup --system app && adduser --system --group app
# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
RUN mkdir $APP_HOME/static
RUN mkdir $APP_HOME/media
RUN mkdir $APP_HOME/currencies
WORKDIR $APP_HOME
# install dependencies
RUN apt-get update && apt-get install -y libpq-dev bash netcat rabbitmq-server
COPY --from=builder /usr/src/app/wheels /wheels
COPY --from=builder /usr/src/app/requirements.txt .
COPY wait-for /bin/wait-for
COPY /log /var/log
COPY /run /var/run
RUN pip install --no-cache /wheels/*
# copy project
COPY . $APP_HOME
# chown all the files to the app user
RUN chown -R app:app $APP_HOME
RUN chown -R app:app /var/log/
RUN chown -R app:app /var/run/
EXPOSE 3000
# change to the app user
USER app
# only for dgango
CMD ["gunicorn", "Config.asgi:application", "--bind", "0.0.0.0:8000", "--workers", "3", "-k","uvicorn.workers.UvicornWorker","--log-file","-"]

Creating a dockerfile for a .deb file

I want to create a dockerfile for a debian file extension which runs on ubuntu 18.04. So far I've written this
FROM ubuntu:18.04 AS ubuntu
RUN apt-get update
WORKDIR /Downloads/invisily
RUN apt-get install ./invisily.deb
All phases run fine except the last one. It shows this error:
E: Unsupported file ./invisily.deb given on commandline
The command '/bin/sh -c apt-get install ./invisily.deb' returned a non-zero code: 100
I'm new to docker and cloud so any help would be appreciated thanks!
Edit:
I solved it by putting the dockerfile and the debian file in the same directory and using COPY . ./
This is what my dockerfile looks like now:
FROM ubuntu:18.04 AS ubuntu
RUN apt-get update
WORKDIR /invisily
COPY . ./
USER root
RUN chmod +x a.deb && \
apt-get install a.deb
A few things,
WORKDIR is the working directory inside of your container.
You will need to copy the file invisily.deb from locally to your container when building your Docker image.
You can pass multiple bash commands in the RUN field combining them with multilines.
Try something like this
FROM ubuntu:18.04 AS ubuntu
WORKDIR /opt/invisily
#Drop the invisily.deb in to the same directory as your Dockerfile
#This will copy it from local to your container, inside of /opt/invisily dir
COPY invisily.deb .
RUN apt-get update && \
chmod +x invisily.deb && \
apt-get install invisily.deb
in your WORKDIR there isn't any invisly.deb file, so if you have it you can copy it the container like this:
FROM ubuntu ...
WORKDIR /Downloads/invisily
RUN apt-get update
COPY ./your invisly file path ./
RUN chmod +x ./invisily
RUN apt-get install ./invisily.deb

Host key verification failed on dockerfile build

I have created a ssh key using ssh-keygen and added id-rsa.pub content to my github>settings>SSH & GPG keys.
I am able to clone the repo from my terminal with git clone git#github:myname/myrepo.git
But the same is giving the following error while building the docker file.
Cloning into 'Project-Jenkins'...
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
This is how I added the command in dockerfile
RUN git clone git#github.com:myname/myrepo.git
What went wrong here?
here is my dockerfile
FROM ubuntu
COPY script.sh /script.sh
CMD ["/script.sh"]
FROM python:2.7
RUN apt-get update
RUN apt-get install libmysqlclient-dev
RUN apt-get install -y cssmin
RUN apt-get install -y python-psycopg2
RUN pip install --upgrade setuptools
RUN pip install ez_setup
RUN apt install -y libpq-dev python-dev
RUN apt install -y postgresql-server-dev-all
COPY requirements.txt ./
CMD ["apt-get","install","pip"]
RUN apt-get install -y git
RUN git clone git#github.com:myname/myrepo.git
WORKDIR ./myrepo/LIMA
RUN pip install -r requirements.txt
CMD ["python","manage.py","migrate"]
CMD ["python","manage.py","collectstatic","--noinput"]
CMD ["python","manage.py","runserver"]
EXPOSE 8000
The syntax you have used finally ends up using SSH to clone, and inside the docker container, your github private key is not available which leads to error that you are getting. So instead try using,
RUN git clone https://{myusername}:{mypassword}#github.com/{myusername}/myrepo.git
Also remember if your password has '#' symbol use '%40' instead.
If you want to still go with private key approach, refer this question, How to access GIT repo with my private key from Dockerfile

How do I add a package to an already existing image?

I have a RoR app that uses imagemagick specified in the Gemfile. I am using Docker's official rails image to build my image with the following Dockerfile:
FROM rails:onbuild
RUN apt-get install imagemagick
and get the following error:
Cant install RMagick 2.13.2. Cant find Magick-config in /usr/local/bundle/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Now, that's probably because the imagemagic package is missing on the OS, even though I specified it in my Dockerfile. So I guess the bundle install command is issued before my RUN apt-get command is issued.
My question - using this base image, is there a way to ensure imagemagic is installed prior to bundling?
Do I need to fork and change the base image Dockerfile to achieve that?
you are right, the ONBUILD instructions from the rails:onbuild image will be executed just after the FROM instruction of your Dockerfile.
What I suggest is to change your Dockerfile as follow:
FROM ruby:2.2.0
RUN apt-get install imagemagick
# throw errors if Gemfile has been modified since Gemfile.lock
RUN bundle config --global frozen 1
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN apt-get update && apt-get install -y nodejs --no-install-recommends && rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y mysql-client postgresql-client sqlite3 --no-install-recommends && rm -rf /var/lib/apt/lists/*
COPY Gemfile /usr/src/app/
COPY Gemfile.lock /usr/src/app/
RUN bundle install
COPY . /usr/src/app
EXPOSE 3000
CMD ["rails", "server"]
which I made based on the rails:onbuild Dockerfile moving down the ONBUILD instructions and removing the ONBUILD flavor.
Most packages clean out the cache to save on size. Try this:
apt-get update && apt-get install imagemagick
Or spool up a copy of the container and look for yourself
docker run -it --remove <mycontainernameorid> /bin/bash
The --remove will ensure that the container is removed after you exit the shell. Once in the shell look for the package binary (or dpkg --list)

Resources