I have the following image
FROM jupyter/pyspark-notebook:python-3.9.12
ARG HADOOP_VERSION=3.3.1
ARG AWS_SDK_VERSION=1.11.901
USER root
RUN wget -q "https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar" -P /usr/local/jars/ && \
wget -q "https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar" -P /usr/local/jars/ && \
wget -q "https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/12.2.0.1/ojdbc8-12.2.0.1.jar" -P /usr/local/jars/
RUN echo 'spark.driver.extraClassPath /usr/local/jars/*' >> "${SPARK_HOME}/conf/spark-defaults.conf" && \
echo 'spark.serializer org.apache.spark.serializer.KryoSerializer' >> "${SPARK_HOME}/conf/spark-defaults.conf" && \
echo 'spark.hadoop.fs.s3a.fast.upload True' >> "${SPARK_HOME}/conf/spark-defaults.conf" && \
echo 'spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem' >> "${SPARK_HOME}/conf/spark-defaults.conf"
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt && \
pip install "jupyterlab>=3" "ipywidgets>=7.6"
When I try to build this image on WSL with "docker build ." I got the following error:
"The command '/bin/bash -o pipefail -c wget -q "https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar" -P /usr/local/jars/ && wget -q "https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar" -P /usr/local/jars/ && wget -q "https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/12.2.0.1/ojdbc8-12.2.0.1.jar" -P /usr/local/jars/' returned a non-zero code: 8"
When I build this image on Docker using Windows the build works ok! But when I run in WSL terminal I got this error. The image is the same on windows (Docker desktop) and WSL.
Why Am I getting this error since the image file is the same?
The error was in ARG HADOOP_VERSION=3.3.1, debugging I found a non Unix char in the line. I used dos2unix to parse the characters to Unix standard.
Related
I am a newbie to Docker and I am trying to install csvtk via Docker using debian:stretch-slim.
This below is my Dockerfile
FROM debian:stretch-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
jq \
perl \
python3 \
wget \
&& rm -rf /var/lib/apt/lists/*
RUN wget -qO- https://github.com/shenwei356/csvtk/releases/download/v0.23.0/csvtk_linux_amd64.tar.gz | tar -xz \
&& cp csvtk /usr/local/bin/
It fails at the csvtk step with the below error message:
Step 3/3 : RUN wget -qO- https://github.com/shenwei356/csvtk/releases/download/v0.23.0/csvtk_linux_amd64.tar.gz | tar -xz && cp csvtk /usr/local/bin/
---> Running in 0f3a0e75a5de
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c wget -qO- https://github.com/shenwei356/csvtk/releases/download/v0.23.0/csvtk_linux_amd64.tar.gz | tar -xz && cp csvtk /usr/local/bin/' returned a non-zero code: 2
I would appreciate any help/suggestions.
Thanks in advance.
wget was exiting with error code meaning 5 SSL verification failed on wget. From this answer, you just needed to install ca-certificates before wget.
This Dockerfile should build successfully:
FROM debian:stretch-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
jq \
perl \
python3 \
wget \
# added this package to help with ssl certs in Docker
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
RUN wget -qO- https://github.com/shenwei356/csvtk/releases/download/v0.23.0/csvtk_linux_amd64.tar.gz | tar -xz \
&& cp csvtk /usr/local/bin/
As a general tip when debugging issues like these, it's likely easiest to remove the offending RUN line from your Dockerfile and then try building and running the container in a shell and manually executing the commands you want. Like this:
docker build -t test:v1 .
docker run --rm -it test:v1 /bin/bash
# run commands manually and check the full error output
While combining different RUN instructions with && is best practice to reduce the number of image layers, it's difficult to debug when building.
I am asking for a massive favor. I was stuck below the issue for the last couple of days. If someone helps then that would be great. Going back to the issue. I have installed a docker and docker container using the following code (Docker-Apache spark).
Docker File:-
FROM debian:stretch
MAINTAINER Getty Images "https://github.com/gettyimages"
RUN apt-get update \
&& apt-get install -y locales \
&& dpkg-reconfigure -f noninteractive locales \
&& locale-gen C.UTF-8 \
&& /usr/sbin/update-locale LANG=C.UTF-8 \
&& echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
&& locale-gen \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Users with other locales should set this in their derivative image
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN apt-get update \
&& apt-get install -y curl unzip \
python3 python3-setuptools \
&& ln -s /usr/bin/python3 /usr/bin/python \
&& easy_install3 pip py4j \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# http://blog.stuart.axelbrooke.com/python-3-on-spark-return-of-the-pythonhashseed
ENV PYTHONHASHSEED 0
ENV PYTHONIOENCODING UTF-8
ENV PIP_DISABLE_PIP_VERSION_CHECK 1
# JAVA
RUN apt-get update \
&& apt-get install -y openjdk-8-jre \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# HADOOP
ENV HADOOP_VERSION 3.0.0
ENV HADOOP_HOME /usr/hadoop-$HADOOP_VERSION
ENV HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
ENV PATH $PATH:$HADOOP_HOME/bin
RUN curl -sL --retry 3 \
"http://archive.apache.org/dist/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz" \
| gunzip \
| tar -x -C /usr/ \
&& rm -rf $HADOOP_HOME/share/doc \
&& chown -R root:root $HADOOP_HOME
# SPARK
ENV SPARK_VERSION 2.4.1
ENV SPARK_PACKAGE spark-${SPARK_VERSION}-bin-without-hadoop
ENV SPARK_HOME /usr/spark-${SPARK_VERSION}
ENV SPARK_DIST_CLASSPATH="$HADOOP_HOME/etc/hadoop/*:$HADOOP_HOME/share/hadoop/common/lib/*:$HADOOP_HOME/share/hadoop/common/*:$HADOOP_HOME/share/hadoop/hdfs/*:$HADOOP_HOME/share/hadoop/hdfs/lib/*:$HADOOP_HOME/share/hadoop/hdfs/*:$HADOOP_HOME/share/hadoop/yarn/lib/*:$HADOOP_HOME/share/hadoop/yarn/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*:$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/tools/lib/*"
ENV PATH $PATH:${SPARK_HOME}/bin
RUN curl -sL --retry 3 \
"https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_PACKAGE}.tgz" \
| gunzip \
| tar x -C /usr/ \
&& mv /usr/$SPARK_PACKAGE $SPARK_HOME \
&& chown -R root:root $SPARK_HOME
WORKDIR $SPARK_HOME
CMD ["bin/spark-class", "org.apache.spark.deploy.master.Master"]
Command:
ubuntu#ip-123.43.11.136:~$ sudo docker run -it --rm -v $(pwd):/home/ubuntu sparkimage /home/ubuntu bin/spark-submit ./count.py
Got Error below
Error :- Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/home/ubuntu\": permission denied": unknown.
Can some help me what could be the issue? I have gone through several links but no luck still not able to resolve the issue.
ERRO[0001] error waiting for the container: context cancelled
Everything passes after image sparkimage are considered as an argument to the Docker Entrypoint.
For example
Entrypoint ["node"]
so when you start
docker run -it my_image app.js
Now here app.js will be the argument for the node which will start app.js and docker will treat them like node app.js.
So you are passing invalid option to the image in your docker run command as there is no entrypoint in your Dockefile and command become
CMD ["/home/ubuntu bin/spark-submit ./count.py"]
That's is its throw error for /home/ubuntu permission denied.
You can try these two combinations.
Etnrypoint ["bin/spark-class", "org.apache.spark.deploy.master.Master"]
with the run command.
sudo docker run -it --rm -v $(pwd):/home/ubuntu sparkimage /home/ubuntu bin/spark-submit ./count.py
OR
CMD ["bin/spark-class", "org.apache.spark.deploy.master.Master","/home/ubuntu bin/spark-submit ./count.py"]
with docker run command
sudo docker run -it --rm -v $(pwd):/home/ubuntu sparkimage
The issue has been resolved. the corrected right mount path and executed and it was working fine without any issue.
I am setting up Kafka and zookeeper through docker; however, my whenever I build my image I keep getting a code 8 error when it gets to:
wget -q https://www.apache.org/dist/zookeeper/zookeeper-${ZOOKEEPER_VERSION}/zookeeper-${ZOOKEEPER_VERSION}.tar.gz.asc .
I have tried to change the file format in the download-kafka.sh to unix already.
Below is my dockerfile:
FROM Wurstmeister/base
MAINTAINER Wurstmeister
ENV ZOOKEEPER_VERSION 3.4.13
#Download Zookeeper
RUN wget -q http://mirror.vorboss.net/apache/zookeeper/zookeeper-${ZOOKEEPER_VERSION}/zookeeper-${ZOOKEEPER_VERSION}.tar.gz && \
wget -q https://www.apache.org/dist/zookeeper/KEYS && \
wget -q https://www.apache.org/dist/zookeeper/zookeeper-${ZOOKEEPER_VERSION}/zookeeper-${ZOOKEEPER_VERSION}.tar.gz.asc && \
wget -q https://www.apache.org/dist/zookeeper/zookeeper-${ZOOKEEPER_VERSION}/zookeeper-${ZOOKEEPER_VERSION}.tar.gz.md5
#Verify download
RUN md5sum -c zookeeper-${ZOOKEEPER_VERSION}.tar.gz.md5 && \
gpg --import KEYS && \
gpg --verify zookeeper-${ZOOKEEPER_VERSION}.tar.gz.asc
#Install
RUN tar -xzf zookeeper-${ZOOKEEPER_VERSION}.tar.gz -C /opt
#Configure
RUN mv /opt/zookeeper-${ZOOKEEPER_VERSION}/conf/zoo_sample.cfg /opt/zookeeper-${ZOOKEEPER_VERSION}/conf/zoo.cfg
ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64
ENV ZK_HOME /opt/zookeeper-${ZOOKEEPER_VERSION}
RUN sed -i "s|/tmp/zookeeper|$ZK_HOME/data|g" $ZK_HOME/conf/zoo.cfg; mkdir $ZK_HOME/data
ADD start-zk.sh /usr/bin/start-zk.sh
EXPOSE 2181 2888 3888
WORKDIR /opt/zookeeper-${ZOOKEEPER_VERSION}
VOLUME ["/opt/zookeeper-${ZOOKEEPER_VERSION}/conf", "/opt/zookeeper-${ZOOKEEPER_VERSION}/data"]
CMD /usr/sbin/sshd && bash /usr/bin/start-zk.sh
If you go to this link, then 3.4.13 doesn't exist anymore
https://www.apache.org/dist/zookeeper/
You can change to ENV ZOOKEEPER_VERSION 3.4.14, or just use an existing Zookeeper Docker image
I have a Dockerfile which was originally pulling from ubuntu and I recently came across alpine which is more lightweight so would like to pull from that instead. Part of the code I'm trying to build is called Healpix which depends on cfitsio. When I originally built the ubuntu version I found this Dockerfile https://github.com/MilesCranmer/dockers/blob/master/dockerfiles/healpix.
Essentially the problem is the following works in ubuntu but not with alpine:
RUN echo "3\ngfortran\n\nY\n\n\ngcc\n\n\n\n\nN\n1\nY\nN\nN\n0\n" |
./configure && make
The error I get is
Something went wrong ...
Quitting configuration script !
./configure: exit: line 162: Illegal number: -1
The command '/bin/sh -c echo "3\ngfortran\n\nY\n\n\ngcc\n\n\n\n\nN\n1\nY\nN\nN\n0\n" | ./configure && make' returned a non-zero code: 2
somewhat confusingly the configure script in question isn't 162 lines long https://sourceforge.net/p/healpix/code/HEAD/tree/branches/branch_v350r1006/configure. I have tried installing bash and changing script to that but that didn't work.
ubuntu Dockerfile
FROM ubuntu
RUN apt-get update && apt-get install -y gcc g++ gfortran make wget
WORKDIR /home
RUN wget \
http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio_latest.tar.gz \
&& tar xzf cfitsio_latest.tar.gz
WORKDIR cfitsio
RUN ./configure --prefix=/usr && make && make install
WORKDIR /home
RUN wget \
https://sourceforge.net/projects/healpix/files/Healpix_3.50/Healpix_3.50_2018Dec10.tar.gz \
&& tar xzf Healpix*.tar.gz
WORKDIR Healpix_3.50
RUN echo \
"3\ngfortran\n\nY\n\n\ngcc\n\n\n\n\nN\n1\nY\nN\nN\n0\n" | ./configure \
&& make
alpine Dockerfile
FROM alpine
RUN apk --no-cache add gcc g++ gfortran make wget
WORKDIR /home
RUN wget \
http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio_latest.tar.gz \
&& tar xzf cfitsio_latest.tar.gz
WORKDIR cfitsio
RUN ./configure --prefix=/usr && make && make install
WORKDIR /home
RUN wget \
https://sourceforge.net/projects/healpix/files/Healpix_3.50/Healpix_3.50_2018Dec10.tar.gz \
&& tar xzf Healpix*.tar.gz
WORKDIR Healpix_3.50
RUN echo \
"3\ngfortran\n\nY\n\n\ngcc\n\n\n\n\nN\n1\nY\nN\nN\n0\n" | ./configure \
&& make
TL;DR
In your Dockerfile, use :
RUN /bin/echo -e "3\ngfortran\n[...]" | ./configure && make
to have the same behavior on Ubuntu and Alpine.
Explanations
The ./configure script is executed with /bin/sh (see the shebang). On Ubuntu, /bin/sh is a link to /bin/dash, while on Alpine, /bin/sh is a link to /bin/busybox.
The following small example reproduces your problem.
Consider the following ./configure script :
#!/bin/sh
read -p "1st prompt : " first
read -p "2nd prompt : " second
echo "$first-$second"
On Ubuntu :
docker run --rm -v $PWD/configure:/configure ubuntu:18.04 \
/bin/sh -c 'echo "a\nb" | ./configure'
prints :
a-b
While, on Alpine :
docker run --rm -v $PWD/configure:/configure alpine:3.8 \
/bin/sh -c 'echo "a\nb" | ./configure'
prints :
anb-
On Alpine (busybox), the echoed string (a\nb) is interpreted as a single argument, while on Ubuntu (dash), the \n is used to separate both arguments.
To have the same behavior as Ubuntu on Alpine, you can run :
docker run --rm -v $PWD/configure:/configure alpine:3.8 /bin/sh -c 'echo "a
b
" | ./configure'
or :
docker run --rm -v $PWD/configure:/configure alpine:3.8 /bin/sh -c \
'echo -e "a\nb" | ./configure'
(see the -e parameter of echo)
These 2 commands print :
a-b
As for your Dockerfile, you should write something like :
RUN /bin/echo -e "3\ngfortran\n[...]" | ./configure && make
/bin/echo is used instead of echo because on Ubuntu, echo -e "3\ngfortran\n[...]" will print -e 3\nngfortran\n[...].
This is because echo is parsed a shell built-in, while /bin/echo is explicitly not (source : https://github.com/moby/moby/issues/8949#issuecomment-61682684).
I'm trying to build a Docker image for my Gitlab CI pipeline containing docker client + gcloud along with the following gcloud components:
kubectl
docker-credential-gcr
This is my dockerfile:
FROM docker:git
RUN mkdir /opt \
&& cd /opt \
&& wget -q https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-152.0.0-linux-x86_64.tar.gz \
&& tar -xzf google-cloud-sdk-152.0.0-linux-x86_64.tar.gz \
&& rm google-cloud-sdk-152.0.0-linux-x86_64.tar.gz \
&& ln -s /opt/google-cloud-sdk/bin/gcloud /usr/bin/gcloud \
&& apk -q update \
&& apk -q add python \
&& apk add --update libintl \
&& apk add --virtual build_deps gettext \
&& cp /usr/bin/envsubst /usr/local/bin/envsubst \
&& apk del build_deps \
&& rm -rf /var/cache/apk/* \
&& echo "y" | gcloud components install kubectl docker-credential-gcr \
&& ln -s /opt/google-cloud-sdk/bin/kubectl /usr/bin/kubectl \
&& ln -s /opt/google-cloud-sdk/bin/docker-credential-gcr /usr/bin/docker-credential-gcr
Inside my CI flow, I need to run docker-credential-gcr (because of this issue).
The docker-credential-gcr executable is correctly installed inside /opt/google-cloud-sdk/bin like shown by running docker run -i -t gitlabci-test ls /opt/google-cloud-sdk/bin
It is also correctly simlinked inside /usr/bin as shown by docker run -i -t gitlabci-test ls -la /usr/bin
And yet, trying to call it with any of the methods below fails miserably
docker run -i -t gitlabci-test docker-credential-gcr
docker run -i -t gitlabci-test /usr/bin/docker-credential-gcr
docker run -i -t gitlabci-test /opt/google-cloud-sdk/bin/docker-credential-gcr
Error message:
/usr/local/bin/docker-entrypoint.sh: exec: line 20: docker-credential-gcr: not found
On the other hand, running the kubectl component works fine
docker run -i -t gitlabci-test kubectl version
Any idea how I can fix this issue to be able to run docker-credential-gcr with the container ?