I'm trying to run protoc command into a docker container.
I've tried using the gRPC image but protoc command is not found:
/bin/sh: 1: protoc: not found
So I assume I have to install manually using RUN instructions, but is there a better solution? An official precompiled image with protoc installed?
Also, I've tried to install via Dockerfile but I'm getting again protoc: not found.
This is my Dockerfile
#I'm not using "FROM grpc/node" because that image can't unzip
FROM node:12
...
# Download proto zip
ENV PROTOC_ZIP=protoc-3.14.0-linux-x86_32.zip
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/${PROTOC_ZIP}
RUN unzip -o ${PROTOC_ZIP} -d ./proto
RUN chmod 755 -R ./proto/bin
ENV BASE=/usr/local
# Copy into path
RUN cp ./proto/bin/protoc ${BASE}/bin
RUN cp -R ./proto/include/* ${BASE}/include
RUN protoc -I=...
I've done RUN echo $PATH to ensure the folder is in path and is ok:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Also RUN ls -la /usr/local/bin to check protoc file is into the folder and it shows:
-rwxr-xr-x 1 root root 4849692 Jan 2 11:16 protoc
So the file is in /bin folder and the folder is in the path.
Have I missed something?
Also, is there a simple way to get the image with protoc installed? or the best option is generate my own image and pull from my repository?
Thanks in advance.
Edit: Solved downloading linux-x86_64 zip file instead of x86_32. I downloaded the lower architecture requirements thinking a x86_64 machine can run a x86_32 file but not in the other way. I don't know if I'm missing something about architecture requirements (It's probably) or is a bug.
Anyway in case it helps someone I found the solution and I've added an answer with the neccessary Dockerfile to run protoc and protoc-gen-grpc-web.
The easiest way to get non-default tools like this is to install them through the underlying Linux distribution's package manager.
First, look at the Docker Hub page for the node image. (For "library" images like node, construct the URL https://hub.docker.com/_/node.) You'll notice there that there are several variations named "alpine", "buster", or "stretch"; plain node:12 is the same as node:12-stretch and node:12.20.0-stretch. The "alpine" images are based on Alpine Linux; the "buster" and "stretch" ones are different versions of Debian GNU/Linux.
For Debian-based packages, you can then look up the package on https://packages.debian.org/ (type protoc into the "Search the contents of packages" form at the bottom of the page). That leads you to the protobuf-compiler package. Knowing that contains the protoc binary, you can install it in your Dockerfile with:
FROM node:12 # Debian-based
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
protobuf-compiler
# The rest of your Dockerfile as above
COPY ...
RUN protoc ...
You generally must run apt-get update and apt-get install in the same RUN command, lest a subsequent rebuild get an old version of the package cache from the Docker build cache. I generally have only a single apt-get install command if I can manage it, with the packages list alphabetically one to a line for maintainability.
If the image is Alpine-based, you can do a similar search on https://pkgs.alpinelinux.org/contents to find protoc, and similarly install it:
FROM node:12-alpine
RUN apk add --no-cache protoc
# The rest of your Dockerfile as above
Finally I solved my own issue.
The problem was the arch version: I was using linux-x86_32.zip but works using linux-x86_64.zip
Even #David Maze answer is incredible and so complete, it didn't solve my problem because using apt-get install version 3.0.0 and I wanted 3.14.0.
So, the Dockerfile I have used to run protoc into a docker container is like this:
FROM node:12
...
# Download proto zip
ENV PROTOC_ZIP=protoc-3.14.0-linux-x86_64.zip
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/${PROTOC_ZIP}
RUN unzip -o ${PROTOC_ZIP} -d ./proto
RUN chmod 755 -R ./proto/bin
ENV BASE=/usr
# Copy into path
RUN cp ./proto/bin/protoc ${BASE}/bin/
RUN cp -R ./proto/include/* ${BASE}/include/
# Download protoc-gen-grpc-web
ENV GRPC_WEB=protoc-gen-grpc-web-1.2.1-linux-x86_64
ENV GRPC_WEB_PATH=/usr/bin/protoc-gen-grpc-web
RUN curl -OL https://github.com/grpc/grpc-web/releases/download/1.2.1/${GRPC_WEB}
# Copy into path
RUN mv ${GRPC_WEB} ${GRPC_WEB_PATH}
RUN chmod +x ${GRPC_WEB_PATH}
RUN protoc -I=...
Because this is currently the highest ranked result on Google and the above instructions above won't work, if you want to use docker/dind for e.g. gitlab, this is the way how you can get the glibc-dependency working for protoc there:
#!/bin/bash
# install gcompat, because protoc needs a real glibc or compatible layer
apk add gcompat
# install a recent protoc (use a version that fits your needs)
export PB_REL="https://github.com/protocolbuffers/protobuf/releases"
curl -LO $PB_REL/download/v3.20.0/protoc-3.20.0-linux-x86_64.zip
unzip protoc-3.20.0-linux-x86_64.zip -d $HOME/.local
export PATH="$PATH:$HOME/.local/bin"
Related
I'm using clusters of my corporation by ICM. It provides a convenient way to configure the remote by docker:
So, I want to build a docker image of my developing environment (python packages, cuda, some utility scripts like screen and rsync, also some necessary data) to deploy on the remote machine. Here is my Dockerfile:
# syntax=docker/dockerfile:1
FROM pytorch/pytorch:1.7.1-cuda11.0-cudnn8-runtime
WORKDIR /app
RUN sudo apt-get install rsync
RUN sudo apt-get install screen
RUN conda create --prefix /data/vxxx/nn python=3.8
RUN conda init
# shotcut for activating my environment
RUN echo 'alias nn="conda activate /data/xxx/nn"' >> ~/.bashrc
RUN source ~/.bashrc
RUN nn
RUN pip3 install torchtext==0.8.1 pandas scipy scikit-learn transformers tensroboard -f https://download.pytorch.org/whl/torch_stable.html
# copy files from windows
COPY /mnt/c/test .
RUN cd /data
RUN mkdir xxx
RUN cd xxx
RUN mkdir Data
RUN mkdir Code
RUN cd Code
RUN git clone https://github.com/namespace-Pt/Document-Reduction.git
RUN git config --global user.name 'xxx'
RUN git config --global user.email 'xxx#1.com'
CMD [ "sleep", "infinity"]
I'm new to docker and I followed the official python image tutorial, I have the following questions:
what is WORKDIR, does it mean to create a new directory where all files will be stored?
why my COPY command is not working?
how to publish my image to make it usable for the cluster?
Beginner here, from what i understood.
WORKDIR Is work directory for other commands as RUN, CMD, COPY or ADD.
Don't know. I will double check directories paths.
Don't know. Normally i'am using dockerhub
I have a Go app. Some of its dependencies are in a private Github repo and another part of dependencies are local packages in my app folder. The app compiles and works on my computer without a problem (when I simply compile it without docker). I am using the below Dockerfile.
FROM ubuntu as intermediate
# install git
RUN apt-get update
RUN apt-get install -y git
RUN mkdir /root/.ssh/
COPY github_rsa.ppk /root/.ssh/github_rsa.ppk
RUN chmod 700 /root/.ssh/github_rsa.ppk
RUN eval $(ssh-agent) && \
ssh-add /root/.ssh/github_rsa.ppk && \
ssh-keyscan -H github.com >> /etc/ssh/ssh_known_hosts && \
git clone git#github.myusername/shared.git
FROM golang:latest
ENV GOPATH=/go
RUN echo $GOPATH
ADD . /go/src/SCMicroServer
WORKDIR /go/src/SCMicroServer
COPY --from=intermediate /shared /go/src/github.com/myusername/shared
RUN go get /go/src/SCMicroServer
RUN go install SCMicroServer
ENTRYPOINT /go/src/SCMicroServer
EXPOSE 8080
First build section related to Git is working fine, It works until this line: RUN go get /go/src/SCMicroServer in second section. I receive this error in mentioned step:
package SCMicroServer/controllers/package1: unrecognized import path "SCMicroServer/controllers/package1" (import path does not begin with hostname)
The command '/bin/sh -c go get /go/src/SCMicroServer' returned a non-zero code: 1
"SCMicroServer/controllers/package1" is one of the local packages in my app folder (or its subfolders) and I have many more in my local folder. I am setting GOPATH env variable in my Dockerfile, so I am not sure what I am missing.
I found the answer, it was not really Dockerfile problem, I referenced my package 2 times in 2 different way in my main file:
package1 "SCMicroServer/controllers/package1"
"SCMicroServer/controllers/package1"
After I removed the second one, I stopped receiving the error.
I am trying to use a Python wrapper for a Java library called Tabula. I need both Python and Java images within my Docker container. I am using the openjdk:8 and python:3.5.3 images. I am trying to build the file using Docker-compose, but it returns the following message:
/bin/sh: 1: java: not found
when it reaches the line RUN java -version within the Dockerfile. The line RUN find / -name "java" also doesn't return anything, so I can't even find where Java is being installed in the Docker environment.
Here is my Dockerfile:
FROM python:3.5.3
FROM openjdk:8
FROM tailordev/pandas
RUN apt-get update && apt-get install -y \
python3-pip
# Create code directory
ENV APP_HOME /usr/src/app
RUN mkdir -p $APP_HOME/temp
WORKDIR /$APP_HOME
# Install app dependencies
ADD requirements.txt $APP_HOME
RUN pip3 install -r requirements.txt
# Copy source code
COPY *.py $APP_HOME/
RUN find / -name "java"
RUN java -version
ENTRYPOINT [ "python3", "runner.py" ]
How do I install Java within the Docker container so that the Python wrapper class can invoke Java methods?
This Dockerfile can not work because the multiple FROM statements at the beginning don't mean what you think it means. It doesn't mean that all the contents of the Images you're referring to in the FROM statements will end up in the Images you're building somehow, it actually meant two different concepts throughout the history of docker:
In the newer Versions of Docker multi stage builds, which is a very different thing from what you're trying to achieve (but very interesting nontheless).
In earlier Versions of Docker, it gave you the ability to simply build multiple images in one Dockerfile.
The behavior you are describing makes me assume you are using such an earlier Version. Let me explain what's actually happening when you run docker build on this Dockerfile:
FROM python:3.5.3
# Docker: "The User wants me to build an
Image that is based on python:3.5.3. No Problem!"
# Docker: "Ah, the next FROM Statement is coming up,
which means that the User is done with building this image"
FROM openjdk:8
# Docker: "The User wants me to build an Image that is based on openjdk:8. No Problem!"
# Docker: "Ah, the next FROM Statement is coming up,
which means that the User is done with building this image"
FROM tailordev/pandas
# Docker: "The User wants me to build an Image that is based on python:3.5.3. No Problem!"
# Docker: "A RUN Statement is coming up. I'll put this as a layer in the Image the user is asking me to build"
RUN apt-get update && apt-get install -y \
python3-pip
...
# Docker: "EOF Reached, nothing more to do!"
As you can see, this is not what you want.
What you should do instead is build a single image where you will first install your runtimes (python, java, ..), and then your application specific dependencies. The last two parts you're already doing, here's how you could go about installing your general dependencies:
# Let's start from the Alpine Java Image
FROM openjdk:8-jre-alpine
# Install Python runtime
RUN apk add --update \
python \
python-dev \
py-pip \
build-base \
&& pip install virtualenv \
&& rm -rf /var/cache/apk/*
# Install your framework dependencies
RUN pip install numpy scipy pandas
... do the rest ...
Note that I haven't tested the above snippet, you may have to adapt a few things.
I'm trying to bundle my Jekyll blog as a docker container.
I found this Dockerfile which seems to suit my use case but wanted to be more hands on so I copied it directly into my repo:
FROM ruby:latest
MAINTAINER Peter Etelej <peter#etelej.com>
RUN apt-get -qq update && \
apt-get -qq install nodejs -y && \
gem install -q bundler
RUN mkdir -p /etc/jekyll && \
printf 'source "https://rubygems.org"\ngem "github-pages"\ngem "execjs"\ngem "rouge"' > /etc/jekyll/Gemfile && \
printf "\nBuilding required Ruby gems. Please wait..." && \
bundle install --gemfile /etc/jekyll/Gemfile --clean --quiet
RUN apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
ENV BUNDLE_GEMFILE /etc/jekyll/Gemfile
EXPOSE 4000
ENTRYPOINT ["bundle", "exec"]
CMD ["jekyll", "serve","--host=0.0.0.0"]
When I run it I get an error
jekyll 3.4.3 | Error: No such file or directory # rb_sysopen - /etc/modules-load.d/modules.conf
The host system has this file but my assumption was that the container didn't have access to it so I tried to add it into the Dockerfile
ADD /etc/modules-load.d/modules.conf /etc/modules-load.d/modules.conf
I then docker build and get the error
lstat etc/modules-load.d/: no such file or directory
I don't understand why the container is looking for this file in the first place but I'm even more confused by the fact that I can't add a file which is clearly there.
Docker builds run on the docker host, not necessarily the client where you run the command, and so all the files needed to run the build are sent in the build context to the host. That context is most often the current directory, or ., that you pass at the end of the docker build -t $image_name . command.
Everything that you try to include in the image with a COPY or ADD is done in reference to that build context, not the filesystem on your client or host machine. So if you need a modules.conf, you'll need to first copy that into your directory with the Dockerfile, and then COPY the file from there.
As for why jekyll is looking for the file, I'm not familiar with jekyll, but it doesn't look promising for something running inside of a container. The modules are kernel specific and containers are designed to be moved to different hosts with potentially different kernels.
I am using an alpine linux container and specifically python:3.4-alpine and openjdk:8-jdk-alpine. When I try to execute any script or executable that I have placed in the executable I get Not Found error.
For example. When in the python:3.4-alpine container I want to install jfrog I follow the command here (after I install curl via apk). This command downloads a shell script and pipes it to sh which downloads and creates a jfrog executable with the correct permissions. When I am trying to run this executable I am getting
bin/sh: ./jfrog: not found
update
I discovered that the root user is using bin/ash by default, which I have no idea what it is. So I invoked bin/sh jfrog manually and I get
/ # bin/sh jfrog
jfrog: line 1: ELF: not found
jfrog: line 1: syntax error: unterminated quoted string
Any idea what I am doing wrong? I suspect that it has to do with only root user existing in the container.
I'm not sure but the jfrog executable is dynamically linked, and with ldd jfrog you get :
ldd jfrog
/lib64/ld-linux-x86-64.so.2 (0x55ffb4c8d000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x55ffb4c8d000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x55ffb4c8d000)
As you can see you have libc dependencies, and alpine come with musl.
You can try to add apk add libc6-compat but I'm not sure it will work
the problem is, that jfrog cli was compiled against glibc and alpine linux only provides uclibc. To make it run under alpine its not trivial, you have to install a sandbox that is bigger than then alpine env. https://wiki.alpinelinux.org/wiki/Running_glibc_programs
Another possibility is to compile the jfrog binary yourself in alpine. This Dockerfile worked for me.
FROM golang:alpine
WORKDIR /app/
RUN apk update && apk add git
# checkout the latest tag of jfrog cli
RUN mkdir -p /go/src/github.com/jfrogdev/jfrog-cli-go \
&& git clone https://github.com/JFrogDev/jfrog-cli-go /go/src/github.com/jfrogdev/jfrog-cli-go\
&& cd /go/src/github.com/jfrogdev/jfrog-cli-go \
&& git checkout $(git describe --tags `git rev-list --tags --max-count=1`)
RUN GOOS=linux go get github.com/jfrogdev/jfrog-cli-go/jfrog
FROM alpine
COPY --from=0 /go/bin/jfrog /usr/bin/
ENTRYPOINT ["jfrog"]
The script you are running begins with:
#!/bin/bash
Bash is not included with alpine by default. You can install it with:
apk update && apk add bash
Note that alpine is fairly stripped down by design, so there may be other missing dependencies that you'll need to add to make this script work.
May be too late, but this probably might help someone else.
RUN curl -Lo /usr/bin/jfrog https://api.bintray.com/content/jfrog/jfrog-cli-go/\$latest/jfrog-cli-linux-386/jfrog?bt_package=jfrog-cli-linux-386 \
&& chmod a+x /usr/bin/jfrog
(Click Here for Reference Link)