So I have a Dockerfile, using which I create an image. The instruction in Dockerfile are:
#This is a docker file
FROM ubuntu:14.04
MAINTAINER amit
# Install python-pip
RUN apt-get update && apt-get install -y python-pip
# Install virtual-env
RUN mkdir ~/.virtualenvs
RUN pip install virtualenv
RUN pip install virtualenvwrapper
RUN touch ~/.bashrc
RUN echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.bashrc
RUN echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
RUN /bin/bash -c "source /usr/local/bin/virtualenvwrapper.sh && mkvirtualenv be"
# INSTALL REQUIRED PACKAGES
RUN apt-get update && apt-get install -y \
xclip \
python-dev \
libffi-dev \
libpam0g-dev \
sqlite3 \
libsqlite3-dev \
subversion \
g++ \
libxslt1-dev \
libxml2-dev \
zlib1g-dev \
swig \
node \
git \
libssl-dev
# Expose port
EXPOSE 5000
# Get the source ideally one should do a get on source release
COPY src /src
WORKDIR /src
RUN touch installer.sh
RUN echo "#!/bin/bash" >> installer.sh
RUN echo "source `which virtualenvwrapper.sh`" >> installer.sh
RUN echo "workon be" >> installer.sh
RUN echo "./tools/install_dependencies" >> installer.sh
RUN echo "deactivate be" >> installer.sh
RUN chmod +x installer.sh
Now I build an image from this. I execute the installer.sh by running the docker container and everything works great.
But when some one pulls this very image from the repository , and runs the file installer.sh, there is an error stating that virtualenv be is not present and then one has to do "mkvirtualenv be" once again.
What is wrong here? Shouldn't the "virtualenv be" automatically be present in the container?
Related
I'm trying to use asdf-direnv in docker. Following the README of asdf-direnv, I make this Dockerfile
FROM nvidia/cuda:10.2-devel-ubuntu18.04
RUN chsh -s /bin/bash
SHELL ["/bin/bash", "-ic", "-l"]
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
# Python
RUN apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
# Utils
RUN apt-get install -y git
RUN apt-get clean
WORKDIR /venv
RUN git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.8.1
RUN echo ". $HOME/.asdf/asdf.sh" >> ~/.bashrc
RUN echo ". $HOME/.asdf/completions/asdf.bash" >> ~/.bashrc
RUN asdf plugin add direnv
RUN asdf install direnv 2.28.0
RUN asdf local direnv 2.28.0
RUN echo "eval \"\$(asdf exec direnv hook bash)\"" >> ~/.bashrc
RUN echo "direnv() { asdf exec direnv \"\$#\"; }" >> ~/.bashrc
RUN mkdir -p ~/.config/direnv/
RUN echo "source \"\$(asdf direnv hook asdf)\"" >> ~/.config/direnv/direnvrc
RUN echo "export DIRENV_LOG_FORMAT=" >> ~/.config/direnv/direnvrc
RUN asdf plugin add python
RUN asdf install python 3.8.7
RUN asdf local python 3.8.7
RUN echo "use asdf" >> .envrc
RUN echo "layout python" >> .envrc
RUN direnv allow
RUN echo $(which python)
CMD ["/bin/bash"]
The issue is, the line RUN echo $(which python) works properly when I run but not when I build the image. I got
/root/.asdf/shims/python when building docker build . -t venv-gpu -f docker-gpu/Dockerfile
/venv/.direnv/python-3.8.7/bin/python when docker run --gpus all -it venv-gpu
How can I fixed this?
I am trying to convert a bash script to a Dockerfile since we are going the containerization route with AWS Batch
Basically I install CPLEX (an optimization library) and Anaconda, install some related packages, check if my environment it good to go, and then kick off a shell script to run the batch job.
Here is a snippet of my Dockerfile:
FROM amazonlinux:latest
# Download packages for container
RUN yum update -y
RUN yum -y install which unzip aws-cli \
RUN yum install -y tar.x86_64
RUN yum install gzip -y
RUN yum install ncompress -y
RUN yum -y install wget
RUN yum install -y nano
# Set working directory
WORKDIR /setup
#: Copy CPLEX installer binary and installation script.
COPY cplex_odee1210.linux-x86-64.bin /setup/
COPY cplex_installer_input.sh /setup/
#: Install CPLEX and update .bashrc
RUN chmod +x /setup/cplex_odee1210.linux-x86-64.bin
RUN chmod +x cplex_installer_input.sh
RUN ./cplex_installer_input.sh | bash cplex_odee1210.linux-x86-64.bin
RUN echo 'export PATH=$PATH:/opt/ibm/ILOG/CPLEX_Optimizer1210/cplex/bin/x86-64_linux' >>/root/.bashrc \
&& /bin/bash -c "source ~/.bashrc"
ENV PATH $PATH:/opt/ibm/ILOG/CPLEX_Optimizer1210/cplex/bin/x86-64_linux
#: Download Anaconda
COPY Anaconda3-2019.10-Linux-x86_64.sh /setup/
RUN bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /home/ec2-user/anaconda3
RUN echo 'export PATH=$PATH:/home/ec2-user/anaconda3/bin' >>/root/.bashrc \
&& /bin/bash -c "source ~/.bashrc"
ENV PATH $PATH:/home/ec2-user/anaconda3/bin
RUN conda install pandas -y \
&& conda install numpy -y \
&& conda install ujson -y \
&& pip install docplex \
&& pip install boto3 \
&& pip install grpcio \
&& pip install grpcio-tools
RUN python3 -m docplex.mp.environment
ADD fetch_and_run.sh /usr/local/bin/fetch_and_run.sh
ENTRYPOINT ["/usr/local/bin/fetch_and_run.sh"]
From there, I kick off a bash script
#!/bin/bash
date
echo "Args: $#"
env
echo "script_path: $1"
echo "script_name: $2"
echo "path_prefix: $3"
echo "jobID: $AWS_BATCH_JOB_ID"
echo "jobQueue: $AWS_BATCH_JQ_NAME"
echo "computeEnvironment: $AWS_BATCH_CE_NAME"
echo "current directory: $(pwd)"
mkdir /tmp/scripts/
aws s3 cp $1 /tmp/scripts/$2
python3 /tmp/scripts/${#:2}
But for some reason, I keep getting
/tmp/tmp.hQlWYBEFs/batch-file-temp: line 20: python3: command not found
Do I need to change some PATH variables? Why isn't Docker picking up my Python 3 version?
The image needs to have python3 installed. Building images works off of files and programs that exist in the container. The python3 you have installed on your own system is not available.
I am trying to switch from Cmake to gradle. I want to configure gradle to work as follow
$ cd myapp && ls myapp
$ Dockerfile build.gradle src
$ gradle build
Build the docker image from docker file
start container
build the application
The docker image contains complete environment for my app.
FROM debian:stretch
RUN apt-get update -y && apt install -y git \
python3-dev libncurses5-dev libxml2-dev \
libedit-dev swig doxygen graphviz xz-utils ninja-build
RUN echo "deb http://ftp.de.debian.org/debian stretch main" >> /etc/apt/source.list
RUN apt-get update && apt-get install -y openjdk-8-jre openjdk-8-jdk
# Clang 8 as a compiler
RUN apt-get update && apt-get install -y \
xz-utils \
build-essential \
curl \
&& rm -rf /var/lib/apt/lists/* \
&& curl -SL http://releases.llvm.org/8.0.0/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz \
| tar -xJC . && \
mv clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04 clang_8.0.0 && \
echo 'export PATH=/clang_8.0.0/bin:$PATH' >> ~/.bashrc && \
echo 'export LD_LIBRARY_PATH=/clang_8.0.0/lib:LD_LIBRARY_PATH' >> ~/.bashrc
#
RUN apt-get update
#install sdkman
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get -qq -y install curl wget unzip zip
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
#install gradle
RUN yes | /bin/bash -l -c 'sdk install gradle 6.1'
PS: This is cpp project
You can build a docker image from Gradle tasks by using **com.bmuschko:gradle-docker-plugin:3.1.0 plugin**
buildscript {
repositories {
jcenter()
mavenCentral()
}
dependencies {
classpath 'com.bmuschko:gradle-docker-plugin:3.1.0'
}
}
apply plugin: 'com.bmuschko.docker-remote-api'
import com.bmuschko.gradle.docker.tasks.image.Dockerfile
import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
import com.bmuschko.gradle.docker.tasks.image.*
task buildImage(type: DockerBuildImage) {
group = ''
inputDir = file('.')
tag = 'image name:'+tag
}
read the documentation for more details https://bmuschko.github.io/gradle-docker-plugin/
Build an image from Gradle task - ./gradlew taskname
To start container and run the command inside it you can use CMD or ENTRYPOINT and specify the command in Dockerfile
CMD ["start.sh"]
in start.sh you can specify your command to be executed after running the container
Let me slightly clean up that Dockerfile first:
FROM debian:stretch
RUN echo "deb http://ftp.de.debian.org/debian stretch main" >> /etc/apt/source.list
RUN apt-get update -y && apt install -qq -y \
python3-dev libncurses5-dev libxml2-dev \
libedit-dev swig doxygen graphviz xz-utils ninja-build \
openjdk-8-jre openjdk-8-jdk \
xz-utils curl git build-essential wget unzip zip
# Clang 8 as a compiler
RUN curl -SL http://releases.llvm.org/8.0.0/clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz \
| tar -xJC . && \
mv clang+llvm-8.0.0-x86_64-linux-gnu-ubuntu-18.04 clang_8.0.0 && \
echo 'export PATH=/clang_8.0.0/bin:$PATH' >> ~/.bashrc && \
echo 'export LD_LIBRARY_PATH=/clang_8.0.0/lib:LD_LIBRARY_PATH' >> ~/.bashrc
#install sdkman
RUN ln -fs /bin/bash /bin/sh
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
RUN yes | /bin/bash -l -c 'sdk install gradle 6.1'
RUN mkdir /src /work
WORKDIR /src
ENTRYPOINT gradle build -p /src
The important bits are at the bottom: it creates a /src directory and executes gradle build there. All that remains for you is to make that directory available when you build.
Assuming you built the container once with docker build -t my-build-container ., you can run it as follows:
docker run -v $(pwd):/src my-build-container
Depending on your build system, this might pollute your source tree with various build artifacts owned by root. If so, consider switching to out-of-tree builds by changing the default working directory to /work instead. All build results will go to /work, and you can extract them from the container afterwards.
Add docker plugin first
buildscript {
dependencies {
classpath("se.transmode.gradle:gradle-docker:1.2")
}
}
create simple task like this in build.gradle file
task buildDocker(type: Docker, dependsOn: build) {
push = false
project.group = 'testProject'
project.archivesBaseName = jar.baseName
applicationName = jar.baseName
dockerfile = file('src/main/docker/Dockerfile')
doFirst {
copy {
from jar
into stageDir
}
}
}
I am hitting issues running a start script (eg npm run gulp-dist) for my container as specified in my docker compose file. I traced the issue down to a node version compatibility issue which has led me to some confusion.
If I enter the container with docker-compose run workspace bash and then run node -v I get back v10.5.0 as expected (and what my script requires).
Yet if in docker-compose I set command: node -v it prints v4.2.6 when bringing up the container with docker-compose up workspace.
So I'm wondering where are the commands run that I specify in docker-compose (I thought they were run in the container once it had started). And how do I run a command in the container - I want to specify it in docker-compose as I run a different command in two different docker-compose files (one for dev env, one for production).
Note: My dev machine has node version 11, so I have no idea where four is.
Also, if run docker-compose run workspace bash and then run the original script, it works fine - it is just failing when run as a docker-compose command.
Here's my dockerfile (sorry, it's big):
# FROM laradock/workspace:1.8-71
# copied the contents of the above laradock workspace
# dockerfile and replaced put here directly.
FROM phusion/baseimage:latest
MAINTAINER Mahmoud Zalt <mahmoud#zalt.me>
RUN DEBIAN_FRONTEND=noninteractive
RUN locale-gen en_US.UTF-8
ENV LANGUAGE=en_US.UTF-8
ENV LC_ALL=en_US.UTF-8
ENV LC_CTYPE=en_US.UTF-8
ENV LANG=en_US.UTF-8
ENV TERM xterm
# Add the "PHP 7" ppa
RUN apt-get install -y software-properties-common && \
add-apt-repository -y ppa:ondrej/php
#
#--------------------------------------------------------------------------
# Software's Installation
#--------------------------------------------------------------------------
#
# Install "PHP Extentions", "libraries", "Software's"
RUN apt-get update && \
apt-get install -y --allow-downgrades --allow-remove-essential \
--allow-change-held-packages \
php7.1-cli \
php7.1-common \
php7.1-curl \
php7.1-intl \
php7.1-json \
php7.1-xml \
php7.1-mbstring \
php7.1-mcrypt \
php7.1-mysql \
php7.1-pgsql \
php7.1-sqlite \
php7.1-sqlite3 \
php7.1-zip \
php7.1-bcmath \
php7.1-memcached \
php7.1-gd \
php7.1-dev \
pkg-config \
libcurl4-openssl-dev \
libedit-dev \
libssl-dev \
libxml2-dev \
xz-utils \
libsqlite3-dev \
sqlite3 \
git \
curl \
vim \
nano \
postgresql-client \
&& apt-get clean
#####################################
# Composer:
#####################################
# Install composer and add its bin to the PATH.
RUN curl -s http://getcomposer.org/installer | php && \
echo "export PATH=${PATH}:/var/www/vendor/bin" >> ~/.bashrc && \
mv composer.phar /usr/local/bin/composer
# Source the bash
RUN . ~/.bashrc
#
# other - workspace specific config
#
RUN apt-get -y update && \
apt-get install pkg-config libmagickwand-dev -y && \
pecl install imagick
#####################################
# Non-Root User:
#####################################
# Add a non-root user to prevent files being created with root permissions on host machine.
ENV PUID 1000
ENV PGID 1000
RUN groupadd -g ${PGID} laradock && \
useradd -u ${PUID} -g laradock -m laradock && \
apt-get update -yqq
#####################################
# Set Timezone
#####################################
ARG TZ=UTC
ENV TZ ${TZ}
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#####################################
# Composer:
#####################################
# Add the composer.json
COPY ./composer.json /home/laradock/.composer/composer.json
# Make sure that ~/.composer belongs to laradock
RUN chown -R laradock:laradock /home/laradock/.composer
USER laradock
# Check if global install need to be ran
ARG COMPOSER_GLOBAL_INSTALL=false
ENV COMPOSER_GLOBAL_INSTALL ${COMPOSER_GLOBAL_INSTALL}
RUN if [ ${COMPOSER_GLOBAL_INSTALL} = true ]; then \
# run the install
composer global install \
;fi
USER root
#####################################
# Node / NVM:
#####################################
# Check if NVM needs to be installed
ARG NODE_VERSION=10.5.0
ENV NODE_VERSION 10.5.0
ENV NVM_DIR /home/laradock/.nvm
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash && \
. $NVM_DIR/nvm.sh && \
nvm install ${NODE_VERSION} && \
nvm use ${NODE_VERSION} && \
npm install -g gulp bower vue-cli \
;fi
# link node and nodejs
RUN ln -s /usr/bin/nodejs /usr/bin/node
# Wouldn't execute when added to the RUN statement in the above block
# Source NVM when loading bash since ~/.profile isn't loaded on non-login shell
RUN echo "" >> ~/.bashrc && \
echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc && \
echo '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm' >> ~/.bashrc \
;fi
# install required things
RUN apt-get update && apt-get install apt-transport-https && \
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
apt-get update && apt-get install -y --allow-unauthenticated yarn mysql-client
# Add NVM binaries to root's .bashrc
USER root
RUN apt-get install npm -y
# set npm registry address
RUN npm config set registry http://registry.npmjs.org/
#
#--------------------------------------------------------------------------
# Final Touch
#--------------------------------------------------------------------------
#
# Clean up
USER root
RUN apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Set default work directory
WORKDIR /var/www
# # copy in our code, so as not to rely on a volume in prod
COPY . /var/www
# ensure directories we need are writable
RUN chmod -R o+w /var/www/user-api-laravel/storage
RUN chmod -R o+w /var/www/user-api-laravel/bootstrap/cache
RUN chmod -R o+w /var/www/auto/storage
RUN chmod -R o+w /var/www/auto/bootstrap/cache
# install php project dependencies
RUN cd /var/www/user-api-laravel && composer install
RUN cd /var/www/auto && composer install
WORKDIR /var/www
USER root
# install auto-scalar deps
RUN cd /var/www/auto-scaler && npm i
# php.ini for cli
ADD ./php-cli.ini /etc/php/7.1/cli/php.ini
And relevant part of docker-compose:
workspace:
build:
context: ./www-workspace
args:
- TZ=${WORKSPACE_TIMEZONE}
- NODE_VERSION=${WORKSPACE_NODE_VERSION}
command: [bash, -c, "cd /var/www/spa && npm run dist-prod"]
Still don't know what context the commands run in, but made mine work. It was due to node being installed via NVM. Or at least when I installed, as #Noogen suggested, via curl -sL https://deb.nodesource.com/setup_10.x | sudo bash - I could then run commands against my container and they would have access to the correct node version. I had to settled for a lower node version (not 10.5.0 as I could specify with NVM) but in the end it worked so no worries.
I need to build the dockerfile that downloads jenkins.war and through it jenkins-cli.jar need to be downloaded.
I have conf.xml also to configure it.**
Then I need that image to run in the bash, which needs to run that jar file commands.
Here is the code:
FROM ubuntu:14.04
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y software-properties-common && \
add-apt-repository ppa:webupd8team/java -y && \
apt-get update && \
echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \
apt-get install -f -y oracle-java8-installer && \
apt install -y default-jre curl wget git nano; \
apt-get clean
# Install dependencies
RUN apt-get -y update && \
apt-get -yqq --no-install-recommends install git bzip2 curl unzip && \
apt-get update
ENV JAVA_HOME /usr
ENV PATH $JAVA_HOME/bin:$PATH
# copy jenkins war file to the container
ADD http://mirrors.jenkins.io/war-stable/2.107.1/jenkins.war /opt/jenkins.war
RUN chmod 644 /opt/jenkins.war
ENV JENKINS_HOME /jenkins
# configure the container to run jenkins, mapping container port 8080 to that host port
RUN mkdir /jenkins/
RUN echo 2.107.1 > /jenkins/jenkins.install.UpgradeWizard.state
RUN echo 2.107.1 > /jenkins/jenkins.install.InstallUtil.lastExecVersion
CMD ["nohup","java", "-jar", "/opt/jenkins.war"]
EXPOSE 8080
VOLUME /jenkins
#COPY jenkins-cli.jar /jenkins/jenkins-cli.jar
#jenkins-cli installation
ENV JENKINS_URL "http://192.168.99.100:8080"
RUN curl --insecure http://192.168.99.100:8080/jnlpJars/jenkins-cli.jar \
--output /jenkins/jenkins-cli.jar
CMD ["java","-jar","/jenkins/jenkins-cli.jar","-noCertificateCheck","-noKeyAuth"]
Here is what im getting.
MY ASSUMPTION
Do I need to run along congf.xml?If yes , then HOW?
Do I need to be running jenkins.war instance in background??? HOW?
Thank you in advance
If you see the reference , I can find those comments.
There can only be one CMD instruction in a Dockerfile. If you list more than one CMD then only the last CMD will take effect.
in your dockerfile, there are multiple CMD commands. only the last one will be executed.
If you want to run multiple commands at once. try bash scripts. here is the example
#!/bin/bash
echo "Starting sshd"
exec /usr/sbin/sshd -D &
if [ -z "$1" ];
then
tail -f $HADOOP_INSTALL/logs/*
fi