My dockerfile on fedora 22
FROM java:latest
ENV HBASE_VERSION=1.1.0.1
RUN groupadd -r hbase && useradd -m -r -g hbase hbase
USER hbase
ENV HOME=/home/hbase
# Download'n extract hbase
RUN cd /home/hbase && \
wget -O - -q \
http://apache.mesi.com.ar/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz \
| tar --strip-components=1 -zxf -
# Upload local configuration
ADD ./conf/ /home/hbase/conf/
USER root
RUN chown -R hbase:hbase /home/hbase/conf
USER hbase
# Prepare data volumes
RUN mkdir /home/hbase/data
RUN mkdir /home/hbase/logs
VOLUME /home/hbase/data
VOLUME /home/hbase/logs
# zookeeper
EXPOSE 2181
# HBase Master API port
EXPOSE 60000
# HBase Master Web UI
EXPOSE 60010
# Regionserver API port
EXPOSE 60020
# HBase Regionserver web UI
EXPOSE 60030
WORKDIR /home/hbase
CMD /home/hbase/bin/hbase master start
As I understand when I set "FROM java:latest" my current dockerfile overlays on that one, so JAVA_HOME must be setted as it is in java:latest? Am I right? This Dockerfile is builded, but when I "docker run" image, It shows "JAVA_HOME not found" error. How can I properly set it up?
use the ENV directive, something like ENV JAVA_HOME /abc/def the doc https://docs.docker.com/reference/builder/#env
add to ~./bashrc (or for global /etc/bashrc:
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
Related
I want an image with elasticsearch and zipkin in it but i dont want to download it from docker hub instead I have downloaded the tar.gz file of those and then creating those images. I am able to run both of them individually but not simultaneously (by docker run command).
Please see below Dockerfile
FROM openjdk:11
RUN groupadd -g 1000 elk-zipkin && useradd elk-zipkin -u 1000 -g 1000
RUN mkdir /usr/share/elasticsearch/
RUN mkdir /usr/share/zipkin
#RUN mkdir /usr/share/kibana
COPY /artifacts/elasticsearch-7.17.6.tar.gz /usr/share/elasticsearch
COPY artifacts/zipkin.jar /usr/share/zipkin
#COPY /artifacts/kibana-7.17.6.tar.gz /usr/share/kibana
COPY script.sh /usr/share/zipkin
WORKDIR /usr/share/elasticsearch
RUN tar xvf elasticsearch-7.17.6.tar.gz
#RUN tar xvf kibana-7.17.6.tar.gz
WORKDIR /usr/share/elasticsearch/elasticsearch-7.17.6
RUN set -ex && for path in data logs config config/scripts; do \
mkdir -p "$path"; \
chown -R elk-zipkin:elk-zipkin "$path"; \
done
USER elk-zipkin
ENV PATH=$PATH:/usr/share/elasticsearch/elasticsearch-7.17.6/bin
WORKDIR /usr/share/elasticsearch/elasticsearch-7.17.6/config
#RUN sed -i "s|#network.host: 192.168.0.1|network.host: 0.0.0.0|g" elasticsearch.yml
#RUN sed -i "s|#discovery.seed_hosts: ["host1", "host2"]|discovery.type: single-node|g" elasticsearch.yml
COPY /artifacts/elasticsearch.yml /usr/share/elasticsearch/elasticsearch-7.17.6/config
#CMD ["elasticsearch"]
#EXPOSE 9200 9300
#WORKDIR /usr/share/zipkin
#CMD ["java","-jar","zipkin.jar"]
#EXPOSE 9411
WORKDIR /usr/share/zipkin
CMD ["sh","script.sh"]
script.sh:
java -jar zipkin.jar elasticsearch
Run command for them:
for zipkin -
docker run -d --name=zipkin \ -p=9411:9411 \ --env=STORAGE_TYPE="elasticsearch" \ --env=ES_HOSTS="someurl" IMAGEID
for elasticsearch -
docker run -d --name=elasticsearch1 -p=9200:9200 -p=9300:9300 IMAGEID
I have tried to run both of the service i.e. elasticsearch and zipkin individually and simultaneously.
I am expecting that both should be in one image and by only single docker run command both of the services should get run.
Somehow I made it, one can create a Dockerfile like mentioned in the question and then have to pass some sleep time into the script file to give some extra time for getting up the previous services.
Example:
nohup elasticsearch &
sleep 10
nohup java -jar zipkin.jar
Note: As per comments and the basics of container, one should not create multiple services inside the same container.
if someone can help me with this, i would be very grateful, i have a docker image in which a kafka is displayed where i pretend to have 3 brokers and i would like that nothing more be created when the docker container is created, the script that i have to raise kafka will be executed, i have tried in many ways using CMD and ENTRYPOINT commands but i am not successful, the container is created for me but the script is not executed i have to enter the container to start it
Dockerfile
FROM ubuntu
RUN apt-get update
RUN apt-get install -y openjdk-8-jdk
RUN apt-get install -y wget \
&& wget http://apache.rediris.es/kafka/2.4.0/kafka_2.12-2.4.0.tgz \
&& tar -xzf kafka_2.12-2.4.0.tgz \
&& rm -R kafka_2.12-2.4.0.tgz
#WORKDIR /home
RUN chmod +x /kafka_2.12-2.4.0
### COPY ###
COPY server-1.properties /kafka_2.12-2.4.0/config/
COPY server-2.properties /kafka_2.12-2.4.0/config/
#ADD runzk-kf.sh .
COPY runzk-kf.sh /usr/local/bin/runzk-kf.sh
#COPY runzk-kf.sh .
RUN chmod +x /usr/local/bin/runzk-kf.sh
EXPOSE 2181
EXPOSE 9092
EXPOSE 9093
EXPOSE 9094
CMD ./bin/bash
script
#!/bin/sh
# turn on bash's job control
set -m
### RUN Zookeper
./kafka_2.12-2.4.0/bin/zookeeper-server-start.sh /kafka_2.12-2.4.0/config/zookeeper.properties &
### RUN Kafka brokers ###
./kafka_2.12-2.4.0/bin/kafka-server-start.sh /kafka_2.12-2.4.0/config/server.properties &
./kafka_2.12-2.4.0/bin/kafka-server-start.sh /kafka_2.12-2.4.0/config/server-1.properties &
./kafka_2.12-2.4.0/bin/kafka-server-start.sh /kafka_2.12-2.4.0/config/server-2.properties &
View all code
Sorry, but please don't do this.
Docker images should be one service, not 4. Use Compose or MiniKube + Helm Charts to orchestrate multiple.
It's not clear what property files you changed for that to work properly.
JDK 8 is end of life, use 11 or 13, which Kafka supports.
Just use existing Docker images. If you want something minimal, personally I use bitnami/kafka. If you want something more fully featured, take a look over at Confluent's repo on running 3 Brokers via Docker Compose.
while dockerizing mlflow , only .trash is getting created
beacuse of that in mlflow ui , getting error as "no experiments exists"
dockerfile
FROM python:3.7.0
RUN pip install mlflow==1.0.0
WORKDIR /data
EXPOSE 5000
CMD mlflow server \
--backend-store-uri /data/ \
--default-artifact-root /data/ \
--host 0.0.0.0
docker compose :
mlflow:
# builds track_ml Dockerfile
build:
context: ./mlflow_dockerfile
expose:
- "5000"
ports:
- "5000:5000"
volumes:
- ./data:/data
You can use this Dockerfile, Taken from mlflow-workshop which is more generic and support different ENV to debug and working with different version.
By default it will store the artifacts and files inside /opt/mlflow. It's possible to define the following variables:
MLFLOW_HOME (/opt/mlflow)
MLFLOW_VERSION (0.7.0)
SERVER_PORT (5000)
SERVER_HOST (0.0.0.0)
FILE_STORE (${MLFLOW_HOME}/fileStore)
ARTIFACT_STORE (${MLFLOW_HOME}/artifactStore)
Dockerfile
FROM python:3.7.0
LABEL maintainer="Albert Franzi"
ENV MLFLOW_HOME /opt/mlflow
ENV MLFLOW_VERSION 0.7.0
ENV SERVER_PORT 5000
ENV SERVER_HOST 0.0.0.0
ENV FILE_STORE ${MLFLOW_HOME}/fileStore
ENV ARTIFACT_STORE ${MLFLOW_HOME}/artifactStore
RUN pip install mlflow==${MLFLOW_VERSION} && \
mkdir -p ${MLFLOW_HOME}/scripts && \
mkdir -p ${FILE_STORE} && \
mkdir -p ${ARTIFACT_STORE}
COPY scripts/run.sh ${MLFLOW_HOME}/scripts/run.sh
RUN chmod +x ${MLFLOW_HOME}/scripts/run.sh
EXPOSE ${SERVER_PORT}/tcp
VOLUME ["${MLFLOW_HOME}/scripts/", "${FILE_STORE}", "${ARTIFACT_STORE}"]
WORKDIR ${MLFLOW_HOME}
ENTRYPOINT ["./scripts/run.sh"]
scripts/run.sh
#!/bin/sh
mlflow server \
--file-store $FILE_STORE \
--default-artifact-root $ARTIFACT_STORE \
--host $SERVER_HOST \
--port $SERVER_PORT
Launch MLFlow Tracking Docker
docker build -t my_mflow_image .
docker run -d -p 5000:5000 --name mlflow-tracking my_mflow_image
Run trainings
Since we have our MLflow Tracking docker exposed at 5000, we can log
executions by setting the env variable MLFLOW_TRACKING_URI.
MLFLOW_TRACKING_URI=http://localhost:5000 python example.py
Also, better to remove - ./data:/data on first run, debug with out mount, and the suggest dockerfile you might need to mount different path that is mentioned in ENV based on your need.
Here is a link to Github where I put MLflow in a docker that uses azurite in the background to also pull the models later from it.
As a short notification, you need to give your script how ever you execute it the address where it should save the artifacts. You can do this with .env files or set these things manually.
set MLFLOW_TRACKING_URI=http://localhost:5000
Important is to also give these information not only your docker but also the script for the model training ;)
Here you can find a complete tutorial how to use MLflow and SKlearn together in different theoretical szenarios since it is also a bit tricky later on.
I hope you get enough inspiration how to use it.
I have a Ubuntu based droplet on Digital Ocean with Docker installed, and where I uploaded my docker image.tar file from my desktop. I uploaded this image.tar file into /home/newuser/app directory. Next, I loaded the image.tar using following command:
sudo docker load -i image.tar
The image has been loaded. I checked.
When I run these following lines, I can't see my image app on public IP connected to my droplet instance:
sudo docker run image
or
sudo docker run -p 80:80 image
How do you guys go about this?
Here is the dockerfile:
FROM r-base:3.5.0
# Install Ubuntu packages
RUN apt-get update && apt-get install -y \
sudo \
gdebi-core \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev/unstable \
libxt-dev \
libssl-dev
# Add shiny user
RUN groupadd shiny \
&& useradd --gid shiny --shell /bin/bash --create-home shiny
# Download and install ShinyServer
RUN wget --no-verbose https://download3.rstudio.org/ubuntu-14.04/x86_64/shiny-server-1.5.7.907-amd64.deb && \
gdebi shiny-server-1.5.7.907-amd64.deb
# Install R packages that are required
RUN R -e "install.packages(c('Benchmarking', 'plotly', 'DT'), repos='http://cran.rstudio.com/')"
RUN R -e "install.packages('shiny', repos='https://cloud.r-project.org/')"
# Copy configuration files into the Docker image
COPY shiny-server.conf /etc/shiny-server/shiny-server.conf
COPY /app /srv/shiny-server/
# Make the ShinyApp available at port 80
EXPOSE 80
# Copy further configuration files into the Docker image
COPY shiny-server.sh /usr/bin/shiny-server.sh
CMD ["/usr/bin/shiny-server.sh"]
The code for shiny-server.conf :
# Define the user we should use when spawning R Shiny processes
run_as shiny;
# Define a top-level server which will listen on a port
server {
# Instruct this server to listen on port 80. The app at dokku-alt need expose PORT 80, or 500 e etc. See the docs
listen 80;
# Define the location available at the base URL
location / {
# Run this location in 'site_dir' mode, which hosts the entire directory
# tree at '/srv/shiny-server'
site_dir /srv/shiny-server;
# Define where we should put the log files for this location
log_dir /var/log/shiny-server;
# Should we list the contents of a (non-Shiny-App) directory when the user
# visits the corresponding URL?
directory_index on;
}
}
And code for shiny-server.sh :
# Make sure the directory for individual app logs exists
mkdir -p /var/log/shiny-server
chown shiny.shiny /var/log/shiny-server
exec shiny-server >> /var/log/shiny-server.log 2>&1
There's really no need to EXPOSE port 80 in the docker file when you run the container with -p 80:80, except maybe as a hint to others: https://forums.docker.com/t/what-is-the-use-of-expose-in-docker-file/37726/2
You should probably post your shiny-server.conf, but I betcha that you either specified no port (in which case shiny-server starts on port 3838) or a port other than 80. Make sure you modify this line in the config file:
listen 3838
I am trying to create an image with fedora which has karaf inside it.I am not understanding how to do this using the Dockerfile.I am using the following docker file
FROM fedora
FROM openjdk:8-jre-slim
ENV RESOURCE_DIR /var/resources
ENV VAROPT_DIR /var/opt
ENV LOG_DIR /var/opt/log
ENV KARAF_VERSION 3.0.8
ENV OSGI_BASE /usr/local/osgi
ENV INSTANCE_NAME cus
ENV OSGI_DEPLOY_DIR /deploy
# Install Karaf
RUN mkdir -p ${OSGI_BASE}
ADD http://www.apache.org/dyn/closer.lua/karaf/3.0.8/apache-karaf-3.0.8.tar.gz ${OSGI_BASE}/
WORKDIR ${OSGI_BASE}
RUN gunzip apache-karaf-${KARAF_VERSION}.tar.gz
RUN tar -xvf apache-karaf-${KARAF_VERSION}.tar
RUN ln -s apache-karaf-${KARAF_VERSION} latest
# Cutomize to enable mount volumes
WORKDIR ${OSGI_BASE}/latest/etc
RUN sed -i 's/karaf.framework=felix/karaf.framework=equinox/' config.properties
RUN sed -i 's/^\(felix\.fileinstall\.dir\s*=\s*\).*$/\1\/deploy/' org.apache.felix.fileinstall-deploy.cfg
RUN sed -i 's%${karaf.data}%/var/opt%' org.ops4j.pax.logging.cfg
# Include war as part of base feature
RUN sed -i '/^featuresBoot=/s/$/,http,war/' org.apache.karaf.features.cfg
ENV PATH ${OSGI_BASE}/latest/bin:$PATH
# Expose ports
EXPOSE 8101 8181 2181
# Define Mountable volumes
RUN mkdir -p ${LOG_DIR}
VOLUME ${VAROPT_DIR}
RUN mkdir ${OSGI_DEPLOY_DIR}
VOLUME ${OSGI_DEPLOY_DIR}
WORKDIR ${RESOURCE_DIR}
RUN mkdir -p ${RESOURCE_DIR}
The dockerfile runs and works fine.The only thing i dont understand is weather it has served the purpose. ie whether the karaf got copied on to the fedora os properly or did it just create a image with karaf on it.How to check it.Please help.