Can't compile Rust project in conda environment in Docker - docker

I'm trying to build the following sightglass benchmarking suite/ Dockerfile:
FROM ubuntu:22.04
RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/00-docker
RUN echo 'APT::Install-Recommends "0";' >> /etc/apt/apt.conf.d/00-docker
RUN DEBIAN_FRONTEND=noninteractive \
apt-get update \
&& apt-get install -y python3 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /usr/src
ADD rust-benchmark rust-benchmark
WORKDIR /usr/src/rust-benchmark
RUN apt update --yes
RUN apt install clang lldb lld wget curl git xz-utils bzip2 --yes
RUN apt-get install --reinstall ca-certificates --yes
RUN apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6 -y
RUN mkdir /usr/local/share/ca-certificates/cacert.org
RUN wget -P /usr/local/share/ca-certificates/cacert.org http://www.cacert.org/certs/root.crt http://www.cacert.org/certs/class3.crt
RUN update-ca-certificates
RUN git config --global http.sslCAinfo /etc/ssl/certs/ca-certificates.crt
RUN wget https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh --no-check-certificate
RUN cd / && find . -name cargo
RUN chmod +x Anaconda3-2022.10-Linux-x86_64.sh
RUN yes yes | ./Anaconda3-2022.10-Linux-x86_64.sh
RUN rm Anaconda3-2022.10-Linux-x86_64.sh
RUN echo "export PATH=./yes/bin:$PATH" >> ~/.bashrc
ENV CONDA ./yes/bin/
ENV PATH="${CONDA}:${PATH}"
RUN ln -s ./yes/bin/conda /usr/local/bin/conda
RUN eval $(conda shell.bash hook)
RUN conda init bash
RUN conda update --all
RUN cd / && find . -name cargo
RUN conda create -c conda-forge -n rustenv rust
RUN activate rustenv
SHELL ["./yes/bin/conda", "run", "-n", "rustenv", "/bin/bash", "-c"]
RUN rustc --version
ENV GIT_SSL_NO_VERIFY=1
RUN git clone https://github.com/emscripten-core/emsdk.git
RUN cd emsdk && git pull
RUN chmod +x ./emsdk/emsdk
RUN ./emsdk/emsdk install latest
RUN ./emsdk/emsdk activate latest
RUN chmod +x ./emsdk/emsdk_env.sh
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN cd emsdk && source ./emsdk_env.sh
RUN ./emsdk/emsdk_env.sh
ENV EMSDK ./emsdk
ENV EMSCRIPTEN=${EMSDK}/emscripten/sdk
ENV EM_DATA ${EMSDK}/.data
ENV EM_CONFIG ${EMSDK}/.emscripten
ENV EM_CACHE ${EM_DATA}/cache
ENV EM_PORTS ${EM_DATA}/ports
ENV PATH="${EMSDK}:${EMSDK}/emscripten/sdk:${EMSDK}/llvm/clang/bin:${EMSDK}/node/current/bin:${EMSDK}/binaryen/bin:${PATH}"
RUN curl https://sh.rustup.rs -ksSf | sh -s -- -y
RUN chmod +x $HOME/.cargo/env
RUN $HOME/.cargo/env
ENV RUST ~/.cargo/bin
ENV PATH="${RUST}:${PATH}"
RUN rustup default nightly
RUN rustup target add wasm32-wasi --toolchain nightly
RUN ./yes/envs/rustenv/bin/cargo build --release --target wasm32-wasi
RUN cp target/wasm32-wasi/release/bls-381-wasm-benchmark.wasm /benchmark.wasm
The build process always aborts on the compile step with the following error:
error[E0463]: can't find crate for `core`
|
= note: the `wasm32-wasi` target may not be installed
= help: consider downloading the target with `rustup target add wasm32-wasi`
error[E0463]: can't find crate for `compiler_builtins`
My full setup can be found here: https://github.com/achimcc/arkworks-wasmtime-benchmarks/tree/main/benchmarks/bls12-381

It seems you are compiling something whose target is wasm32-wasi.
Rust can compile source codes for different "targets", but only few of them was enabled by default.
To install the wasm32-wasi target, you can run this command:
rustup target add wasm32-wasi
Any other questions about compling or environments, feel easy to comment here.

Related

Why is my container when starting as root seem to be empty?

When I get into my container nothing seems to have ebeen installed?
docker pull brandojazz/iit-term-synthesis:test
then
docker run -u root -ti brandojazz/iit-term-synthesis:test_arm bash
see:
(base) root#897a4007076f:/home/bot# opam switch
[WARNING] Running as root is not recommended
[ERROR] Opam has not been initialised, please run `opam init'
it should have been initialized.
FROM continuumio/miniconda3
# FROM --platform=linux/amd64 continuumio/miniconda3
MAINTAINER Brando Miranda "me#gmail.com"
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ssh \
git \
m4 \
libgmp-dev \
opam \
wget \
ca-certificates \
rsync \
strace \
gcc
# rlwrap \
# sudo
# https://github.com/giampaolo/psutil/pull/2103
RUN useradd -m bot
# format for chpasswd user_name:password
# RUN echo "bot:bot" | chpasswd
# RUN && adduser docker sudo
WORKDIR /home/bot
USER bot
ADD https://api.github.com/repos/IBM/pycoq/git/refs/heads/main version.json
# -- setup opam like VP's PyCoq
RUN opam init --disable-sandboxing
# compiler + '_' + coq_serapi + '.' + coq_serapi_pin
RUN opam switch create ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1 ocaml-variants.4.07.1+flambda
RUN opam switch ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1
RUN eval $(opam env)
RUN opam repo add coq-released https://coq.inria.fr/opam/released
# RUN opam pin add -y coq 8.11.0
# ['opam', 'repo', '--all-switches', 'add', '--set-default', 'coq-released', 'https://coq.inria.fr/opam/released']
RUN opam repo --all-switches add --set-default coq-released https://coq.inria.fr/opam/released
RUN opam update --all
RUN opam pin add -y coq 8.11.0
#RUN opam install -y --switch ocaml-variants.4.07.1+flambda_coq-serapi_coq-serapi_8.11.0+0.11.1 coq-serapi 8.11.0+0.11.1
RUN opam install -y coq-serapi
#RUN eval $(opam env)
#
## makes sure depedencies for pycoq are installed once already in the docker image
#RUN pip install https://github.com/ddelange/psutil/releases/download/release-5.9.1/psutil-5.9.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
#ENV WANDB_API_KEY="SECRET"
#RUN pip install wandb --upgrade
#
#RUN pip install ultimate-utils
## RUN pip install pycoq # do not uncomment on arm, unless serlib is removed from setup.py in the pypi pycoq version.
## RUN pip install ~/iit-term-synthesis # likely won't work cuz we don't have iit or have pused it to pypi
#
## then make sure editable mode is done to be able to use changing pycoq from system
#RUN echo "pip install -e /home/bot/ultimate-utils" >> ~/.bashrc
#RUN echo "pip install -e /home/bot/pycoq" >> ~/.bashrc
#RUN echo "pip install -e /home/bot/iit-term-synthesis" >> ~/.bashrc
#RUN echo "pip install wandb --upgrade" >> ~/.bashrc
#
#RUN echo "eval $(opam env)" >> ~/.bashrc
## - set env variable for bash terminal prompt p1 to be nicely colored
#ENV force_color_prompt=yes
#
#RUN mkdir -p /home/bot/data/
# RUN pytest --pyargs pycoq
#CMD /bin/bash
NB: This may not be your only problem (I have no idea what opam is or how it works), but one thing jumps out:
This...
RUN eval $(opam env)
...doesn't do anything. Each RUN invocation happens in a new subshell; environment variables set in one RUN command aren't going to be visible in a subsequent RUN command.
Rather than a list of single-command RUN commands, chain everything together in a single command:
RUN eval $(opam env) && \
opam repo add coq-released https://coq.inria.fr/opam/released && \
opam repo --all-switches add --set-default coq-released https://coq.inria.fr/opam/released && \
opam update --all && \
opam pin add -y coq 8.11.0 && \
opam install -y coq-serapi
Because the above runs in a single shell, the environment set by eval $(opam env) will be available to all the following commands.

Tmux as Entrypoint deactivates unicode characters

I'm building a Docker image including a ready to use terminal with all my usual tools.
I'm running a 2020 Macbook Air M1 running Monterey 12.5.1.
I'd like to start the container directly in a tmux session, but the characters display behavior is inconsistent.
When ENTRYPOINT is ["zsh"] and I execute tmux in the interactive container, the characters are as expected :
and when executing tmux :
but when changing the ENTRYPOINT to ["zsh", "-c", "tmux"] :
Here is my Dockerfile :
FROM ubuntu:22.04
ARG USER=ben
ENV GROUP=${USER}
ENV HOME=/home/${USER}
ENV TMUX_SESSION_NAME=devops
RUN groupadd ${GROUP}
RUN useradd -m -g ${GROUP} ${USER}
RUN apt-get update -y && apt-get upgrade -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata
RUN apt-get install -y \
ca-certificates \
curl \
git \
wget \
docker \
vim \
fzf \
zsh \
fd-find \
zsh-syntax-highlighting \
tmux \
locales \
locales-all
RUN usermod -s /bin/zsh ${USER}
# Configuring locales
RUN ln -fs /usr/share/zoneinfo/Europe/Paris /etc/localtime \
&& dpkg-reconfigure --frontend noninteractive tzdata
USER ${USER}
WORKDIR /home/${USER}
# Oh-My-Zsh configuration
RUN wget https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O - | zsh || true
# ZSH plugins
RUN git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k
RUN git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-${HOME}/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting
RUN git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-${HOME}/.oh-my-zsh/custom}/plugins/zsh-autosuggestions
COPY --chown=${USER}:${GROUP} zshrc ${HOME}/.zshrc
COPY --chown=${USER}:${GROUP} tmux.conf ${HOME}/.tmux.conf
COPY --chown=${USER}:${GROUP} p10k.zsh ${HOME}/.p10k.zsh
# ENTRYPOINT ["zsh", "-c", "tmux"]
ENTRYPOINT ["zsh"]
I couldn't find the reason for this behavior, but I investigated starting tmux directly from zsh and not in the ENTRYPOINT, and the solution that solved my issue was to set the environment variables ZSH_TMUX_AUTOSTART=true.
Thank you all for your help !

CD command is not working inside a docker container in ubuntu box :18.04(vagrant) [duplicate]

the command :
docker build -t nginx-ubuntu .
whith the Dockerfile below :
FROM ubuntu:12.10
RUN apt-get update
RUN apt-get -y install libpcre3 libssl-dev
RUN apt-get -y install libpcre3-dev
RUN apt-get -y install wget zip gcc
RUN wget http://nginx.org/download/nginx-1.4.1.tar.gz
RUN gunzip nginx-1.4.1.tar.gz
RUN tar -xf nginx-1.4.1.tar
RUN wget --no-check-certificate https://github.com/max-l/nginx_accept_language_module/archive/master.zip
RUN unzip master
RUN cd nginx-1.4.1
RUN ./configure --add-module=../nginx_accept_language_module-master --with-http_ssl_module --with-pcre=/lib/x86_64-linux-gnu --with-openssl=/usr/lib/x86_64-linux-gnu
Fails at the last line (./configure ...)
If I remove the last line and run a bash in the container, and
execute the last line manually, it works.
I would expect that whatever command runs successfully within a container should work
when the command is appended in the Dockerfile (prefixed by RUN)
am I missing something ?
The pwd is not persistent across RUN commands. You need to cd and configure within the same RUN.
This Dockerfile works fine:
FROM ubuntu:12.10
RUN apt-get update
RUN apt-get -y install libpcre3 libssl-dev
RUN apt-get -y install libpcre3-dev
RUN apt-get -y install wget zip gcc
RUN wget http://nginx.org/download/nginx-1.4.1.tar.gz
RUN gunzip nginx-1.4.1.tar.gz
RUN tar -xf nginx-1.4.1.tar
RUN wget --no-check-certificate https://github.com/max-l/nginx_accept_language_module/archive/master.zip
RUN unzip master
RUN cd nginx-1.4.1 && ./configure --add-module=../nginx_accept_language_module-master --with-http_ssl_module --with-pcre=/lib/x86_64-linux-gnu --with-openssl=/usr/lib/x86_64-linux-gnu
As an alternative to #creak's answer, you can change the default working directory in a Dockerfile with the WORKDIR command:
FROM ubuntu:12.10
# Run update & install together so that the docker cache doesn't
# contain an out-of-date APT cache (leading to 404's when installing
# packages)
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y install libpcre3 libssl-dev libpcre3-dev wget zip gcc
ADD http://nginx.org/download/nginx-1.4.1.tar.gz nginx-1.4.1.tar.gz
RUN tar -zxf nginx-1.4.1.tar.gz
RUN wget --no-check-certificate https://github.com/max-l/nginx_accept_language_module/archive/master.zip
RUN unzip master
WORKDIR nginx-1.4.1
RUN ./configure --add-module=../nginx_accept_language_module-master --with-http_ssl_module --with-pcre=/lib/x86_64-linux-gnu --with-openssl=/usr/lib/x86_64-linux-gnu
This also affects the default directory when you use docker run (overridden by the -w switch).
When I called wget or tar with RUN I needed to specify a path, it looks like ADD is the correct approach if you want to use WORKDIR as it's path instead. So either of the below resolved my issue.
RUN
RUN wget http://nginx.org/download/nginx-1.4.1.tar.gz -P ~/directory
RUN tar -zxf ~/directory/nginx-1.4.1.tar.gz -C ~/directory
or
ADD
WORKDIR ~/directory
ADD http://nginx.org/download/nginx-1.4.1.tar.gz nginx-1.4.1.tar.gz
RUN tar -zxf nginx-1.4.1.tar.gz
The second approach prevented me from needing to install wget in the container.
Another way of doing this using the \ operator to start a new line and proceed with an additional command
Example
RUN cd /Desktop \
cd Work \
pwd

Dockerfile: Python3 not found

I am trying to convert a bash script to a Dockerfile since we are going the containerization route with AWS Batch
Basically I install CPLEX (an optimization library) and Anaconda, install some related packages, check if my environment it good to go, and then kick off a shell script to run the batch job.
Here is a snippet of my Dockerfile:
FROM amazonlinux:latest
# Download packages for container
RUN yum update -y
RUN yum -y install which unzip aws-cli \
RUN yum install -y tar.x86_64
RUN yum install gzip -y
RUN yum install ncompress -y
RUN yum -y install wget
RUN yum install -y nano
# Set working directory
WORKDIR /setup
#: Copy CPLEX installer binary and installation script.
COPY cplex_odee1210.linux-x86-64.bin /setup/
COPY cplex_installer_input.sh /setup/
#: Install CPLEX and update .bashrc
RUN chmod +x /setup/cplex_odee1210.linux-x86-64.bin
RUN chmod +x cplex_installer_input.sh
RUN ./cplex_installer_input.sh | bash cplex_odee1210.linux-x86-64.bin
RUN echo 'export PATH=$PATH:/opt/ibm/ILOG/CPLEX_Optimizer1210/cplex/bin/x86-64_linux' >>/root/.bashrc \
&& /bin/bash -c "source ~/.bashrc"
ENV PATH $PATH:/opt/ibm/ILOG/CPLEX_Optimizer1210/cplex/bin/x86-64_linux
#: Download Anaconda
COPY Anaconda3-2019.10-Linux-x86_64.sh /setup/
RUN bash Anaconda3-2019.10-Linux-x86_64.sh -b -p /home/ec2-user/anaconda3
RUN echo 'export PATH=$PATH:/home/ec2-user/anaconda3/bin' >>/root/.bashrc \
&& /bin/bash -c "source ~/.bashrc"
ENV PATH $PATH:/home/ec2-user/anaconda3/bin
RUN conda install pandas -y \
&& conda install numpy -y \
&& conda install ujson -y \
&& pip install docplex \
&& pip install boto3 \
&& pip install grpcio \
&& pip install grpcio-tools
RUN python3 -m docplex.mp.environment
ADD fetch_and_run.sh /usr/local/bin/fetch_and_run.sh
ENTRYPOINT ["/usr/local/bin/fetch_and_run.sh"]
From there, I kick off a bash script
#!/bin/bash
date
echo "Args: $#"
env
echo "script_path: $1"
echo "script_name: $2"
echo "path_prefix: $3"
echo "jobID: $AWS_BATCH_JOB_ID"
echo "jobQueue: $AWS_BATCH_JQ_NAME"
echo "computeEnvironment: $AWS_BATCH_CE_NAME"
echo "current directory: $(pwd)"
mkdir /tmp/scripts/
aws s3 cp $1 /tmp/scripts/$2
python3 /tmp/scripts/${#:2}
But for some reason, I keep getting
/tmp/tmp.hQlWYBEFs/batch-file-temp: line 20: python3: command not found
Do I need to change some PATH variables? Why isn't Docker picking up my Python 3 version?
The image needs to have python3 installed. Building images works off of files and programs that exist in the container. The python3 you have installed on your own system is not available.

Setting up our Rasa/NLU container, error?

I have this file Dockerfile.nlu
FROM chatbot/spacy:latest
WORKDIR /app
COPY nlu ./agent_nlu
RUN python –m rasa_nlu.train --config agent_nlu/config.yml --data agent_nlu/data/ --path agent_nlu/agent --fixed_model_name default
and I get the error below:
]$ sudo docker build -t nlu:latest -f docker/Dockerfile.nlu .
Sending build context to Docker daemon 9.216kB
Step 1/4 : FROM chatbot/spacy:latest
---> 496dc6a38abb
Step 2/4 : WORKDIR /app
---> Using cache
---> 7f02012c8452
Step 3/4 : COPY nlu ./agent_nlu
COPY failed: stat /var/lib/docker/tmp/docker-builder363868051/nlu: no such file or directory
It doesn't look like Docker can find the nlu directory. Are you sure it exists? Are you sure that you are executing the command from the correct directory?
But you also aren't installing Rasa at all or any of it's requirements. Is there a reason you aren't using the pre-built Rasa images? available here with docs here.
Here is a fully functional Docker file pulled from their repo.
FROM python:3.6-slim
ENV RASA_NLU_DOCKER="YES" \
RASA_NLU_HOME=/app \
RASA_NLU_PYTHON_PACKAGES=/usr/local/lib/python3.6/dist-packages
# Run updates, install basics and cleanup
# - build-essential: Compile specific dependencies
# - git-core: Checkout git repos
RUN apt-get update -qq \
&& apt-get install -y --no-install-recommends build-essential git-core openssl libssl-dev libffi6 libffi-dev curl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
WORKDIR ${RASA_NLU_HOME}
COPY . ${RASA_NLU_HOME}
# use bash always
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN pip install -r alt_requirements/requirements_spacy_sklearn.txt
RUN pip install -e .
RUN pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.0.0/en_core_web_md-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link en_core_web_md en \
&& pip install https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz --no-cache-dir > /dev/null \
&& python -m spacy link de_core_news_sm de
COPY sample_configs/config_spacy.yml ${RASA_NLU_HOME}/config.yml
VOLUME ["/app/projects", "/app/logs", "/app/data"]
EXPOSE 5000
ENTRYPOINT ["./entrypoint.sh"]
CMD ["start", "-c", "config.yml", "--path", "/app/projects"]

Resources