I have the following dockerfile:
FROM python:3.10-alpine
LABEL Name=my_app
WORKDIR /app
RUN addgroup --gid 1000 -S my_app && \
adduser --uid 1000 -D -S my_app -G my_app -s /sbin/nologin
COPY --chown=1000:1000 pyproject.toml README.rst src ./
RUN apk add --no-cache --virtual=.build-deps build-base libffi-dev curl openssl-dev && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && \
source $HOME/.cargo/env && \
pip install --upgrade pip && pip install --no-cache-dir ./ && \
apk del .build-deps && \
rustup self uninstall -y
RUN chown -R 1000:1000 /app
USER my_app
It is working and I am capable to run inside of this container the following command: my_app run --checks all path_name. Because some reason I need to change my_app key phrase to my_app_2 key phrase. So I would able to run the following command: my_app_2 run --checks all path_name. I cannot change underling files outside of dockerfile because of some server configuration. What changes in dockerfile could I make?
I tried to add:
RUN ln -s /usr/local/bin/my_app /usr/local/bin/my_app_2
But getting an error during build prosses::
=> ERROR [7/7] RUN ln -s /usr/local/bin/my_app /usr/local/bin/my_app 0.3s
------
> [7/7] RUN ln -s /usr/local/bin/my_app /usr/local/bin/my_app_2:
#0 0.256 ln: /usr/local/bin/my_app_2: Permission denied
------
failed to solve: executor failed running [/bin/sh -c ln -s /usr/local/bin/my_app /usr/local/bin/my_app_2]: exit code: 1
Also I tried to swap my_app_2 and my_app:
RUN ln -s /usr/local/bin/my_app_2 /usr/local/bin/my_app
But getting another error during build prosses:
=> ERROR [7/7] RUN ln -s /usr/local/bin/my_app_2 /usr/local/bin/my_app 0.5s
------
> [7/7] RUN ln -s /usr/local/bin/my_app_2 /usr/local/bin/my_app:
#0 0.516 ln: /usr/local/bin/my_app: File exists
------
failed to solve: executor failed running [/bin/sh -c ln -s /usr/local/bin/my_app_2 /usr/local/bin/my_app]: exit code: 1
Third attempt. I added:
RUN echo "alias my_app_2='my_app'" >> ~/.bashrc
Built was successful but when I run
my_app_2 run --checks all my_path_here
I am getting:
sh: my_app_2: not found
Run the command while you are still root. Unprivileged users can't modify arbitrary files on the filesystem:
FROM python:3.10-alpine
LABEL Name=my_app
WORKDIR /app
RUN addgroup --gid 1000 -S my_app && \
adduser --uid 1000 -D -S my_app -G my_app -s /sbin/nologin
COPY --chown=1000:1000 pyproject.toml README.rst src ./
RUN apk add --no-cache --virtual=.build-deps build-base libffi-dev curl openssl-dev && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && \
source $HOME/.cargo/env && \
pip install --upgrade pip && pip install --no-cache-dir ./ && \
apk del .build-deps && \
rustup self uninstall -y
# run other steps as root here
RUN ln -s /usr/local/bin/my_app /usr/local/bin/my_app_2
RUN chown -R 1000:1000 /app
USER my_app
# you are no longer root here
Related
My project has the following structure:
docker-compose.yml - docker-compose file I created
schemathesis - a folder with source code and dockerfile
Where docker-compose.yml is:
version: '3'
services:
schemathesis:
build: ./schemathesis
ports:
- "8180:80"
networks:
- default
Dockerfile in schemathesis folder looks like:
FROM python:3.10-alpine
LABEL Name=Schemathesis
WORKDIR /app
RUN addgroup --gid 1000 -S schemathesis && \
adduser --uid 1000 -D -S schemathesis -G schemathesis -s /sbin/nologin
COPY --chown=1000:1000 pyproject.toml README.rst src ./
RUN apk add --no-cache --virtual=.build-deps build-base libffi-dev curl openssl-dev && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && \
source $HOME/.cargo/env && \
pip install --upgrade pip && pip install --no-cache-dir ./ && \
apk del .build-deps && \
rustup self uninstall -y
# Needed for the `.hypothesis` directory
RUN chown -R 1000:1000 /app
USER schemathesis
ENTRYPOINT ["schemathesis"]
When I run: sudo docker-compose up --build I am getting an error ERROR: Service 'schemathesis' failed to build:
Building schemathesis
Step 1/9 : FROM python:3.10-alpine
---> 8a3a8409a638
Step 2/9 : LABEL Name=Schemathesis
---> Using cache
---> f108325765ba
Step 3/9 : WORKDIR /app
---> Using cache
---> 9fc24fc24456
Step 4/9 : RUN addgroup --gid 1000 -S schemathesis && adduser --uid 1000 -D -S schemathesis -G schemathesis -s /sbin/nologin
---> Using cache
---> 64881e35ad07
Step 5/9 : COPY --chown=1000:1000 pyproject.toml README.rst src ./
---> Using cache
---> 8b9dfeb0ed09
Step 6/9 : RUN apk add --no-cache --virtual=.build-deps build-base libffi-dev curl openssl-dev && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && source $HOME/.cargo/env && pip install --upgrade pip && pip install --no-cache-dir ./ && apk del .build-deps && rustup self uninstall -y
---> Running in fafc492c1f99
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/main: temporary error (try again later)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.17/community: temporary error (try again later)
ERROR: unable to select packages:
.build-deps-20230120.185651:
masked in: cache
satisfies: world[.build-deps=20230120.185651]
build-base (no such package):
required by: .build-deps-20230120.185651[build-base]
libffi-dev (no such package):
required by: .build-deps-20230120.185651[libffi-dev]
curl (no such package):
required by: .build-deps-20230120.185651[curl]
openssl-dev (no such package):
required by: .build-deps-20230120.185651[openssl-dev]
ERROR: Service 'schemathesis' failed to build: The command '/bin/sh -c apk add --no-cache --virtual=.build-deps build-base libffi-dev curl openssl-dev && curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && source $HOME/.cargo/env && pip install --upgrade pip && pip install --no-cache-dir ./ && apk del .build-deps && rustup self uninstall -y' returned a non-zero code: 5
Any help would be greatly appreciated.
No idea what was the problem but the following set of command solve it:
sudo service firewalld stop
sudo service docker restart
sudo systemctl daemon-reload; systemctl restart docker
I am trying to build in Bamboo and got this error,
Failed to build pyarrow
21-Sep-2022 06:24:14 ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
21-Sep-2022 06:24:15 The command '/bin/sh -c pip install --upgrade pip && pip install pyarrow' returned a non-zero code: 1
21-Sep-2022 06:24:15 =An error occurred when executing task 'DockerBuild'.
This error occurs only when I add pyarrow or fastparquet in requirements.txt.
This is my requirements.txt file:
requests
urllib3
fastapi
uvicorn[standard]
gunicorn
pytest-cov
prometheus-fastapi-instrumentator
prometheus_client
fastapi-health
python-decouple
ecs-logging
fastapi_health
psycopg2
arrow
anyio
asgiref
certifi
charset-normalizer
click
colorama
h11
idna
python-dotenv
pydantic
sniffio
starlette
typing_extensions
datetime
fastapi_resource_server
sendgrid
PyJWT==2.4.0
bcrypt==3.2.
cryptography==37.0.2
passlib
jose
jira
adal==1.2.7
aiohttp==3.8.1
aiosignal==1.2.0
async-timeout==4.0.2
azure-core==1.25.0
azure-identity==1.10.0
azure-storage-blob==12.13.1
pandas==1.4.4
multidict==6.0.2
numpy==1.23.2
ordered-set==4.1.0
oauthlib==3.2.0
packaging==21.3
python-dateutil==2.8.2
pytz==2022.2.1
requests-oauthlib==1.3.1
six==1.16.0
yarl==1.8.1
Below is my dockerfile:
FROM python:3.10.4-alpine3.15
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.12/main' >> /etc/apk/repositories
RUN apk upgrade && apk add make gcc g++
RUN apk update
RUN apk add libffi-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN apk add --no-cache musl-dev linux-headers g++
RUN pip install --upgrade pip && pip install arrow && pip install pyarrow
RUN pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "1000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]
Is this error because of the python image?
I am still learning docker so not sure what went wrong here. Can anyone please help me in understanding this?
I have changed the docker file and built it from source since I came to know that pyarrow wheels are not provided for alpine.
FROM python:3.9-alpine
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
libressl-dev \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "5000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0","-m 3000m"]
I am getting docker image build error "Operation not permitted". chmod 777 R /PredictionIO-0.14.0/lib/spark/ did not solve it.
I am trying to create PredictionIO docker image through git CICD to run in K8t
pio build --verbose
[INFO] DOCKER> [91mls: cannot access '/PredictionIO-0.14.0/lib/spark/pio-data-elasticsearch-assembly-0.14.0.jar': Operation not permitted ls: cannot access '/PredictionIO-0.14.0/lib/spark/pio-data-hbase-assembly-0.14.0.jar': Operation not permitted
[INFO] DOCKER> [91mls: cannot access '/PredictionIO-0.14.0/lib/spark/pio-data-hdfs-assembly-0.14.0.jar': Operation not permitted
ls: cannot access '/PredictionIO-0.14.0/lib/spark/pio-data-jdbc-assembly-0.14.0.jar': Operation not permitted
ls: cannot access '/PredictionIO-0.14.0/lib/spark/pio-data-localfs-assembly-0.14.0.jar': Operation not permitted
ls: cannot access '/PredictionIO-0.14.0/lib/spark/pio-data-s3-assembly-0.14.0.jar': Operation not permitted
[INFO] DOCKER> [91mError: Could not find or load main class org.apache.predictionio.tools.console.Console
[INFO] DOCKER> Removing intermediate container 8c164b439d1b
[ERROR] DOCKER> Unable to build image [myimage/bellmedia-pio-0.14.0:latest] : "The command '/bin/sh -c pio build --verbose' returned a non-zero code: 1" ["The command '/bin/sh -c pio build --verbose' returned a non-zero code: 1" ]
Dockerfile
FROM openjdk:8
MAINTAINER Jhon
ARG HTTP_PROXY=http://myproxyServer:8083/
ARG HTTPS_PROXY=http://myproxyServer:8083/
ENV DEBIAN_FRONTEND noninteractive
RUN export http_proxy=${HTTP_PROXY} \
&& export https_proxy=${HTTPS_PROXY} \
&& apt-get update && \
apt-get install -y net-tools curl patch gawk g++ gcc make libc6-dev vim telnet bash patch libreadline6-dev zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 autoconf libgdbm-dev libncurses5-dev automake libtool bison pkg-config libffi-dev libffi-dev krb5-user libpam-krb5 libpam-ccreds python3-pip tini libc6 libpam-modules libnss3 cron
ENV PIO_VERSION 0.14.0
ENV SPARK_VERSION 2.4.4
ENV ELK_VERSION 5.6.9
ENV SCALA_VERSION 2.11.12
ENV HBASE_VERSION 1.0.0
ENV HADOOP_VERSION 2.7
ENV PIO_GIT_URL http://archive.apache.org/dist/predictionio/
ENV PIO_HOME /PredictionIO-${PIO_VERSION}
ENV PATH=${PIO_HOME}/bin:/PredictionIO-0.14.0/vendors/spark-2.4.4-bin-hadoop2.7/bin:$PATH
ENV JAVA_HOME /usr/local/openjdk-8
EXPOSE 9200 9300
RUN export http_proxy=${HTTP_PROXY} \
&& export https_proxy=${HTTPS_PROXY} \
&& curl -O https://archive.apache.org/dist/predictionio/${PIO_VERSION}/apache-predictionio-${PIO_VERSION}.tar.gz
RUN mkdir predictionio
RUN tar -zxf apache-predictionio-${PIO_VERSION}.tar.gz -C /predictionio \
&& rm apache-predictionio-${PIO_VERSION}.tar.gz
COPY sbt-launch.jar /root/.sbt/launchers/1.2.3/
WORKDIR /predictionio
RUN export http_proxy=${HTTP_PROXY} \
&& export https_proxy=${HTTPS_PROXY} \
&& ./make-distribution.sh -Dscala.version=${SCALA_VERSION} -Dspark.version=${SPARK_VERSION} -Delasticsearch.version=${ELK_VERSION}
WORKDIR /
RUN tar -zxvf predictionio/PredictionIO-${PIO_VERSION}.tar.gz \
&& rm predictionio/PredictionIO-${PIO_VERSION}.tar.gz
COPY pio-env.sh ${PIO_HOME}/conf/
COPY log4j.properties ${PIO_HOME}/conf/
RUN mkdir ${PIO_HOME}/vendors
WORKDIR ${PIO_HOME}/vendors
RUN export http_proxy=${HTTP_PROXY} \
&& export https_proxy=${HTTPS_PROXY} \
&& curl -O https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz
RUN tar -zxvf spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz \
&& rm spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz
COPY spark-defaults.conf spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}/conf/
WORKDIR ${PIO_HOME}
RUN export http_proxy=${HTTP_PROXY} \
&& export https_proxy=${HTTPS_PROXY} \
&& mkdir URM \
&& cd URM \
&& git clone https://github.com/actionml/universal-recommender.git
COPY build.sbt /PredictionIO-0.14.0/URM/universal-recommender/
COPY engine.json /PredictionIO-0.14.0/URM/universal-recommender/
RUN rm -rf target
RUN ln -s /PredictionIO-0.14.0/vendors/spark-2.4.4-bin-hadoop2.7/ /opt/spark
WORKDIR ${PIO_HOME}/URM/universal-recommender/
RUN mkdir lib
COPY sbt-launch.jar /root/.sbt/launchers/0.13.13/
RUN export SBT_OPTS="-Dhttp.proxyHost=myproxy.ca -Dhttp.proxyPort=8083 -Dhttps.proxyHost=myproxy.ca -Dhttps.proxyPort=8083"
RUN chmod 777 -R /PredictionIO-0.14.0/lib/spark/
########################### Command below is failing ##################
RUN export http_proxy=${HTTP_PROXY} \
&& export https_proxy=${HTTPS_PROXY} \
&& pio build --verbose
CMD ["sh", "-c", "tail -f /dev/null"]
Any suggestion is highly applicated.
Thanks
Jhon
I have to execute few OpenSSL commands inside my Kubernetes pod. However, I get OpenSSL to command not found.
Below is my docker file.
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y build-essential cmake zlib1g-dev libcppunit-dev git subversion wget && rm -rf /var/lib/apt/lists/*
RUN wget https://www.openssl.org/source/openssl-1.0.2g.tar.gz -O - | tar -xz
WORKDIR /openssl-1.0.2g
RUN ./config --prefix=/usr/local/openssl --openssldir=/usr/local/openssl && make && make install
FROM openjdk:8-jdk-alpine
RUN addgroup -g 1000 -S spring && adduser -u 1000 -S spring -G spring
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
Can anyone please help me here?
The following docker file helped me
FROM openjdk:8-jdk-alpine
RUN apk add --update openssl && \
rm -rf /var/cache/apk/*
RUN addgroup -g 1000 -S spring && adduser -u 1000 -S spring -G spring
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
I'm trying to run a Flask app with Celery (worker + beat) on Docker Alpine using docker-compose.
I want it to run with a non-root user celery in my Docker container.
The flask app is building ok and works, but my celery containers are failing with this error:
File "/usr/lib/python3.6/site-packages/celery/platforms.py", line 543, in maybe_drop_privileges
_setuid(uid, gid)
File "/usr/lib/python3.6/site-packages/celery/platforms.py", line 564, in _setuid
initgroups(uid, gid)
File "/usr/lib/python3.6/site-packages/celery/platforms.py", line 507, in initgroups
return os.initgroups(username, gid)
PermissionError: [Errno 1] Operation not permitted
My Dockerfile:
I tried to add RUN chown celery:celery /etc/group thinking that was the issue but it's still failing
FROM alpine:3.8
RUN apk update && \
apk add build-base python3 python3-dev libffi-dev libressl-dev && \
cd /usr/bin && \
ln -sf python3 python && \
ln -sf pip3 pip && \
pip install --upgrade pip
COPY requirements.txt .
RUN pip install -r requirements.txt
RUN addgroup celery
RUN adduser celery -G celery -s /bin/sh -D
RUN mkdir -p /var/log/celery/ && chown celery:celery /var/log/celery/
RUN mkdir -p /var/run/celery/ && chown celery:celery /var/run/celery/
RUN chown celery:celery /etc/group # added to try fixing the issue
USER celery
ENV FLASK_APP=flask_app
WORKDIR app/
COPY flask_app flask_app
My docker-compose:
(...)
celeryworker:
build: .
command: celery -A flask_app.tasks worker --loglevel=INFO --uid=celery --pidfile=/tmp/celeryworker-shhh.pid
celerybeat:
build: .
command: celery -A flask_app.tasks beat --loglevel=INFO --uid=celery --pidfile=/tmp/celerybeat-shhh.pid
You should do like that
RUN mkdir -p /var/log/celery/ /var/run/celery/
RUN useradd -G root celery && \
chgrp -Rf root /var/log/celery/ /var/run/celery/ && \
chmod -Rf g+w /var/log/celery/ /var/run/celery/c && \
chmod g+w /etc/passwd
...
RUN chmod a+x /start.sh
USER celery
ENTRYPOINT ["/start.sh"]
You should create user celery firsts. Then, add this user into group root. After that you need set write permission for this folder you need to put logs and /etc/passwd.
You also need to have one script to add your user into /etc/passwd
#!/bin/bash
#
if [ `id -u` -ge 10000 ]; then
echo "celery:x:`id -u`:`id -g`:,,,:/home/web:/bin/bash" >> /etc/passwd
fi
Both answers from #Shashank V and #Kine were really relevant and helpful but still had some issues afterward.
After doing some research, I finally made it works with the following configuration
Dockerfile
FROM alpine:3.11.0
RUN apk update && \
apk add build-base python3 python3-dev libffi-dev libressl-dev && \
ln -sf /usr/bin/python3 /usr/bin/python && \
ln -sf /usr/bin/pip3 usr/bin/pip && \
pip install --upgrade pip
RUN mkdir -p /var/log/celery/ /var/run/celery/
RUN addgroup app && \
adduser --disabled-password --gecos "" --ingroup app --no-create-home app && \
chown app:app /var/run/celery/ && \
chown app:app /var/log/celery/
USER app
ENV PATH="/home/app/.local/bin:${PATH}"
WORKDIR app/
COPY requirements.txt .
RUN pip install --user -r requirements.txt\
COPY flask_app flask_app
ENV FLASK_APP=flask_app
docker-compose
(...)
celeryworker:
build: .
command: >
celery -A shhh.tasks worker
--loglevel=INFO
--logfile=/var/log/celery/celeryworker-shhh.log
--pidfile=/var/run/celery/celeryworker-shhh.pid
celerybeat:
build: .
command: >
celery -A shhh.tasks beat
--loglevel=INFO
--logfile=/var/log/celery/celerybeat-shhh.log
--pidfile=/var/run/celery/celerybeat-shhh.pid
--schedule=/var/run/celery/celerybeat-schedule # specify schedule db in a loc where app has read/write access
You have to be root user if you want to use --uid or --gid argument. Try removing these arguments.