I am running docker (via docker-compose) and can't run varnishadm from within the container. The error produced is:
Cannot open /var/lib/varnish/4f0dab1efca3/_.vsm: No such file or directory
Could not open shared memory
I have tried searching on the 'shared memory' issue and _.vsm with no luck. It seems that the _.vsm is not created at all and /var/lib/varnish/ inside the container is empty.
I have tried a variety of -T settings without any luck.
Why run varnishadm?
The root of why I need to run varnishadm is to reload varnish while saving the cache. My backup backup backup option is to set up varnish as a service. We are on an old version of Varnish for the time being.
How am I starting docker?
CMD varnishd -F -f /etc/varnish/varnish.vcl \
-s malloc,1G \
-a :80
Full Dockerfile
FROM ubuntu:12.04
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install wget dtrx varnish -y \
&& apt-get install pkg-config autoconf autoconf-archive automake libtool python-docutils libpcre3 libpcre3-dev xsltproc make -y \ && rm -rf /var/lib/apt/lists/*
RUN export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
RUN wget https://github.com/varnishcache/varnish-cache/archive/varnish-
3.0.2.tar.gz --no-check-certificate \
&& dtrx -n varnish-3.0.2.tar.gz
WORKDIR /varnish-3.0.2/varnish-cache-varnish-3.0.2/
RUN cd /varnish-3.0.2/varnish-cache-varnish-3.0.2/ && ./autogen.sh &&
cd /varnish-3.0.2/varnish-cache-varnish-3.0.2/ && ./configure && make install
RUN cd / && wget --no-check-certificate https://github.com/Dridi/libvmod-querystring/archive/v0.3.tar.gz && dtrx -n ./v0.3.tar.gz
WORKDIR /v0.3/libvmod-querystring-0.3
RUN ./autogen.sh && ./configure VARNISHSRC=/varnish-3.0.2/varnish-cache-varnish-3.0.2/ && make install
RUN cp /usr/local/lib/varnish/vmods/* /usr/lib/varnish/vmods/
WORKDIR /etc/varnish/
CMD varnishd -F -f /etc/varnish/varnish.vcl \
-s malloc,1G \
-a :80
EXPOSE 80
Full docker-compose
version: "3"
services:
varnish:
build: ./
ports:
- "8000:80"
volumes:
- ./default.vcl:/etc/varnish/varnish.vcl
- ./devicedetect.vcl:/etc/varnish/devicedetect.vcl
restart: unless-stopped
Related
I am a docker newbie and i can't rly figure out how the changes that will be made to my working directory will be continuously copied to the docker container. Is there a command that copies all my changes to the docker container all the time ?
Edit : i added docker file and docker compose
My docker file
FROM scratch
ADD centos-7-x86_64-docker.tar.xz /
LABEL \
org.label-schema.schema-version="1.0" \
org.label-schema.name="CentOS Base Image" \
org.label-schema.vendor="CentOS" \
org.label-schema.license="GPLv2" \
org.label-schema.build-date="20201113" \
org.opencontainers.image.title="CentOS Base Image" \
org.opencontainers.image.vendor="CentOS" \
org.opencontainers.image.licenses="GPL-2.0-only" \
org.opencontainers.image.created="2020-11-13 00:00:00+00:00"
RUN yum clean all && yum update -y && yum -y upgrade
RUN yum groupinstall "Development Tools" -y
RUN yum install -y wget gettext-devel curl-devel openssl-devel perl-devel perl-CPAN zlib-devel && wget https://github.com/git/git/archive/v2.26.2.tar.gz\
&& tar -xvzf v2.26.2.tar.gz && cd git-2.26.2 && make configure && ./configure --prefix=/usr/local && make install
# RUN mkdir -p /root/.ssh && \
# chmod 0700 /root/.ssh && \
# ssh-keyscan github.com > /root/.ssh/known_hosts
# RUN ssh-keygen -q -t rsa -N '' -f /id_rsa
# RUN echo "$ssh_prv_key" > /root/.ssh/id_rsa && \
# echo "$ssh_pub_key" > /root/.ssh/id_rsa.pub && \
# chmod 600 /root/.ssh/id_rsa && \
# chmod 600 /root/.ssh/id_rsa.pub
RUN ls
RUN cd / && git clone https://github.com/odoo/odoo.git \
&& cd odoo \
&& git fetch \
&& git checkout 9.0
RUN yum install python-devel libxml2-devel libxslt-dev openldap-devel libtiff-devel libjpeg-devel libzip-devel freetype-devel lcms2-devel \
libwebp-devel tcl-devel tk-devel python-pip nodejs
RUN pip install setuptools==1.4.1 beautifulsoup4==4.9.3 pillow openpyxl==2.6.4 luhn gmp-devel paramiko==1.7.7.2 python2-secrets cffi pysftp==0.2.8
RUN pip install -r requirements.txt
RUN npm install -g less
CMD ["/bin/bash","git"]
My docker-compose
version: '3.3'
services:
app: &app
build:
context: .
dockerfile: ./docker/app/Dockerfile
container_name: app
tty: true
db:
image: postgres:9.2.18
environment:
- POSTGRES_DB=test
ports:
- 5432:5432
volumes:
- ./docker/db/pg-data:/var/lib/postgresql/data
odoo:
<<: *app
command: python odoo.py -w odoo -r odoo
ports:
- '8069:8069'
depends_on:
- db
If I understand correctly you want to mount a path from the host into a container which can be done using volumes. Something like this would keep the folders in sync which can be useful for development
docker run -v /path/to/local/folder:/path/in/container busybox
I'm trying to build a gRPC PHP Client and gRPC NodeJs Server in docker. But the problem is I can't install protoc-gen-php-grpc to my docker server. When I try to run this run this makefile:
proto_from_within_container:
# PHP
protoc /var/www/protos/smellycat.proto \
--php_out=/var/www/php-client/src \
$(: 👇 generate server interface) \
--php-grpc_out=/var/www/php-client/src \
$(: 👇 generates the client code) \
--grpc_out=/var/www/php-client/src \
--plugin=protoc-gen-grpc=/protobuf/grpc/bins/opt/grpc_php_plugin \
--proto_path /var/www/protos
proto:
powershell rm -r -fo php-client/src -ErrorAction SilentlyContinue
powershell New-Item -ItemType Directory -Path php-client/src -Force -ErrorAction SilentlyContinue
docker-compose run grpc-server make proto_from_within_container
With this command: make proto
Getting this error message after docker containers builded:
protoc /var/www/protos/smellycat.proto \
--php_out=/var/www/php-client/src \
\
--php-grpc_out=/var/www/php-client/src \
\
--grpc_out=/var/www/php-client/src \
--plugin=protoc-gen-grpc=/protobuf/grpc/bins/opt/grpc_php_plugin \
--proto_path /var/www/protos
protoc-gen-php-grpc: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--php-grpc_out: protoc-gen-php-grpc: Plugin failed with status code 1.
Makefile:4: recipe for target 'proto_from_within_container' failed
make: *** [proto_from_within_container] Error 1
This is my docker-compose file
version: "3"
services:
grpc-server:
container_name: grpc-server
build:
context: .
dockerfile: Dockerfile-server
working_dir: /var/www
volumes:
- .:/var/www
grpc-client:
image: php:7.4-cli
container_name: grpc-client
build:
context: .
dockerfile: Dockerfile-client
working_dir: /var/www
volumes:
- .:/var/www
command: bash -c [php php_client.php && composer install]
And this is my grpc-server docker file:
FROM node:latest
ENV DEBIAN_FRONTEND=noninteractive
#Versions
ARG PROTOBUF_VERSION=3.14.0
ARG PHP_GRPC_VERSION=1.34.0
# Utils
RUN apt-get update -yqq \
&& apt-get install -yqq wget unzip zlib1g-dev git autoconf libtool automake build-essential software-properties-common curl zip \
&& rm -rf /var/lib/apt/lists/*
# Protobuf
RUN mkdir -p /protobuf
RUN cd /protobuf \
&& wget https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-linux-x86_64.zip -O protobuf.zip \
&& unzip protobuf.zip && rm protobuf.zip
# grpc PHP (generate client)
RUN apt-get update -yqq && apt-get upgrade -yqq
RUN apt-get install php php-dev php-pear phpunit zlib1g-dev -yqq
RUN pecl install grpc-${PHP_GRPC_VERSION}
RUN cd /protobuf && git clone -b v${PHP_GRPC_VERSION} https://github.com/grpc/grpc \
&& cd /protobuf/grpc && git submodule update --init
RUN cd /protobuf/grpc && make grpc_php_plugin
ENV PATH "/protobuf/bin:${PATH}"
ENV PATH "/protobuf/grpc/bins/opt:${PATH}"
# NPM Installation
WORKDIR /var/www
COPY . /var/www
RUN npm install
CMD ["node", "server.js"]
Do you have any advice?
After a lot of search and readings, I finally managed to build a full application that communicates with each other.
The problem was at the Makefile, at this step:
--plugin=protoc-gen-grpc=/protobuf/grpc/bins/opt/grpc_php_plugin
I was assigning the wrong path for grpc_php_plugin.
There is my new dockerfile:
FROM php:7.4-cli
# Environment variables
ENV DEBIAN_FRONTEND=noninteractive
# Utils
RUN apt-get update -yqq && \
apt-get upgrade -yqq && \
apt-get install -y unzip build-essential git software-properties-common curl pkg-config zip zlib1g-dev
# Composer installation
COPY --from=composer:latest /usr/bin/composer /usr/local/bin/composer
# Install grpc and probuf with pecl
RUN pecl install grpc && pecl install protobuf
# Enable grpc and protobuf extensions in php.ini file
RUN echo starting && \
docker-php-ext-enable grpc && \
docker-php-ext-enable protobuf
# Install cmake
RUN apt-get update -yqq && apt-get -y install cmake
# Install grpc_php_plugin and protoc
RUN git clone -b v1.36.2 https://github.com/grpc/grpc && \
cd grpc && git submodule update --init && \
mkdir cmake/build && cd cmake/build && \
cmake ../.. && make protoc grpc_php_plugin
# Setting node, protoc and grpc_php_plugin paths
ENV PATH "/grpc/cmake/build:${PATH}"
ENV PATH "/grpc/cmake/build/third_party/protobuf:${PATH}"
# Moving client folder to vm
WORKDIR /var/www
COPY ./client /var/www
# Packages
RUN composer install
# Generate php libraries from proto file
RUN make proto
CMD [ "php", "./handler.php" ]
For my full application, click.
I installed Elasticsearch in my image based on ubuntu:16.04.
And start the service using
RUN service elasticsearch start
but, it was not started.
If I go into the container and run it, it starts.
I want to run the service and dump the index when I create the image, below is a part of my Dockerfile.
How do I start Elasticsearch in the Dockerfile?
#install OpenJDK-8
RUN apt-get update && apt-get install -y openjdk-8-jdk && apt-get install -y ant && apt-get clean
RUN apt-get update && apt-get install -y ca-certificates-java && apt-get clean
RUN update-ca-certificates -f
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN export JAVA_HOME
#download ES
RUN wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
RUN apt-get install -y apt-transport-https
RUN echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
RUN apt-get update && apt-get install -y elasticsearch
RUN service elasticsearch start
The RUN command executes only during the build phase. It stops after the build is completed. You should use CMD (or ENTRYPOINT) instead:
CMD service elasticsearch start && /bin/bash
It's better wrapping the starting command in your own file and then only execute the file:
CMD /start_elastic.sh
I don't know why not take official oss image, but, this Docker file based on Debian work:
FROM java:8-jre
ENV ES_NAME=elasticsearch \
ELASTICSEARCH_VERSION=6.6.1
ENV ELASTICSEARCH_URL=https://artifacts.elastic.co/downloads/$ES_NAME/$ES_NAME-$ELASTICSEARCH_VERSION.tar.gz
RUN apt-get update && apt-get install -y --assume-yes openssl bash curl wget \
&& mkdir -p /opt \
&& echo '[i] Start create elasticsearch' \
&& wget -T 15 -O /tmp/$ES_NAME-$ELASTICSEARCH_VERSION.tar.gz $ELASTICSEARCH_URL \
&& tar -xzf /tmp/$ES_NAME-$ELASTICSEARCH_VERSION.tar.gz -C /opt/ \
&& ln -s /opt/$ES_NAME-$ELASTICSEARCH_VERSION /opt/$ES_NAME \
&& useradd elastic \
&& mkdir -p /var/lib/elasticsearch /opt/$ES_NAME/plugins /opt/$ES_NAME/config/scripts \
&& chown -R elastic /opt/$ES_NAME-$ELASTICSEARCH_VERSION/
ENV PATH=/opt/elasticsearch/bin:$PATH
USER elastic
CMD [ "/bin/sh", "-c", "/opt/elasticsearch/bin/elasticsearch --E cluster.name=test --E network.host=0 $ELASTIC_CMD_OPTIONS" ]
I believe most of the commands you'll be able to use on Ubuntu.
Don't forget to run sudo sysctl -w vm.max_map_count=262144 on your host
I am very new to Docker.
I am getting an error saying "cannot find module 'express'" while trying to containerize simple node-red application. The details are as follows:
Base machine
OS -Debian 9 (stretch) 64-bit
RAM -8 gb
GNOME - 3.22.2
Env - Oracle Virtual Box
Node-Red source
https://github.com/node-red/node-red.git
Docker Version
17.12.0-ce, build c97c6d6
docker-compose -v
1.20.1, build 5d8c71b
Docker File
FROM debian:stretch-slim
RUN useradd -c 'Node-Red user' -m -d /home/nodered -s /bin/bash nodered
RUN chown -R nodered.nodered /home/nodered
RUN echo "Acquire::http::Proxy \"http://xxxx:yyyy";" > /etc/apt/apt.conf.d/01turnkey \
&& echo "Acquire::https::Proxy \"http://xxxx.yyyy";" >> /etc/apt/apt.conf.d/01turnkey
ENV http_proxy="http://xxxx:yyyy \
https_proxy="http://xxxx:yyyy"
USER root
RUN apt-get update && apt-get -y install --no-install-recommends \
ca-certificates \
apt-utils \
curl \
sudo \
git \
python \
make \
g++ \
gnupg2
RUN mkdir -p /home/nodered/shaan-node-red && chown -R nodered.nodered /home/nodered/shaan-node-red
ENV HOME /home/nodered/shaan-node-red
WORKDIR /home/nodered/shaan-node-red
RUN ls -la
RUN env
USER root
RUN echo "nodered ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/nodered && \
chmod 0440 /etc/sudoers.d/nodered
RUN curl -sL https://deb.nodesource.com/setup_9.x | bash -
RUN apt-get -y install nodejs
RUN rm -rf node-v9.x
RUN node -v (v9.9.0) && npm -v (5.6.0)
RUN npm config set proxy "http://xxxx:yyyy" \
npm config set http-proxy "http://xxxx:yyyy"
COPY . /home/nodered/shaan-node-red
RUN cd /home/nodered/shaan-node-red && ls -la && npm install
RUN npm run build && ls -la
RUN cd /home/nodered/shaan-node-red/node_modules/ && git clone https://github.com/netsmarttech/node-red-contrib-s7.git && ls -la | grep s7 && cd ./node-red-contrib-s7 && npm install
RUN ls -la /home/nodered/shaan-node-red/node_modules
ENTRYPOINT ["sh","entrypoint.sh"]
entrypoint.sh
node /home/nodered/shaan-node-red/red.js
Docker-compose.yml
version: '2.0'
services:
web:
image: shaan-node-red
build: .
volumes:
- .:/home/nodered/shaan-node-red
ports:
- "1880:1880"
- "5858:5858"
network_mode: host
Building with command:
docker-compose up
Error description
Note
Not getting any error while building same node-red at the base machine.
I want to mount s3fs inside of docker container.
I made docker image with s3fs, and did like this:
host$ docker run -it --rm docker/s3fs bash
[ root#container:~ ]$ s3fs s3bucket /mnt/s3bucket -o allow_other -o allow_other,default_acl=public-read -ouse_cache=/tmp
fuse: failed to open /dev/fuse: Operation not permitted
Showing "Operation not permitted" error.
So I googled, and did like this (adding --privileged=true) again:
host$ docker run -it --rm --privileged=true docker/s3fs bash
[ root#container:~ ]$ s3fs s3bucket /mnt/s3bucket -o allow_other -o allow_other,default_acl=public-read -ouse_cache=/tmp
[ root#container:~ ]$ ls /mnt/s3bucket
ls: cannot access /mnt/s3bucket: Transport endpoint is not connected
[ root#container:~ ]$ fusermount -u /mnt/s3bucket
[ root#container:~ ]$ s3fs s3bucket /mnt/s3bucket -o allow_other -o allow_other,default_acl=public-read -ouse_cache=/tmp
[ root#container:~ ]$ ls /mnt/s3bucket
ls: cannot access /mnt/s3bucket: Transport endpoint is not connected
Then, mounting not shows error, but if run ls command, "Transport endpoint is not connected" error is occured.
How can I mount s3fs inside of docker container?
Is it impossible?
[UPDATED]
Add Dockerfile configuration.
Dockerfile:
FROM dockerfile/ubuntu
RUN apt-get update
RUN apt-get install -y build-essential
RUN apt-get install -y libfuse-dev
RUN apt-get install -y fuse
RUN apt-get install -y libcurl4-openssl-dev
RUN apt-get install -y libxml2-dev
RUN apt-get install -y mime-support
RUN \
cd /usr/src && \
wget http://s3fs.googlecode.com/files/s3fs-1.74.tar.gz && \
tar xvzf s3fs-1.74.tar.gz && \
cd s3fs-1.74/ && \
./configure --prefix=/usr && \
make && make install
ADD passwd/passwd-s3fs /etc/passwd-s3fs
ADD rules.d/99-fuse.rules /etc/udev/rules.d/99-fuse.rules
RUN chmod 640 /etc/passwd-s3fs
RUN mkdir /mnt/s3bucket
rules.d/99-fuse.rules:
KERNEL==fuse, MODE=0777
I'm not sure what you did that did not work, but I was able to get this to work like this:
Dockerfile:
FROM ubuntu:12.04
RUN apt-get update -qq
RUN apt-get install -y build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support automake libtool wget tar
RUN wget https://github.com/s3fs-fuse/s3fs-fuse/archive/v1.77.tar.gz -O /usr/src/v1.77.tar.gz
RUN tar xvz -C /usr/src -f /usr/src/v1.77.tar.gz
RUN cd /usr/src/s3fs-fuse-1.77 && ./autogen.sh && ./configure --prefix=/usr && make && make install
RUN mkdir /s3bucket
After building with:
docker build --rm -t ubuntu/s3fs:latest .
I ran the container with:
docker run -it -e AWSACCESSKEYID=obscured -e AWSSECRETACCESSKEY=obscured --privileged ubuntu/s3fs:latest bash
and then inside the container:
root#efa2689dca96:/# s3fs s3bucket /s3bucket
root#efa2689dca96:/# ls /s3bucket
testing.this.out work.please working
root#efa2689dca96:/#
which successfully listed the files in my s3bucket.
You do need to make sure the kernel on your host machine supports fuse, but it would seem you have already done so?
Note: Your S3 mountpoint will not show/work from inside other containers when using Docker's --volume or --volumes-from directives. For example:
docker run -t --detach --name testmount -v /s3bucket -e AWSACCESSKEYID=obscured -e AWSSECRETACCESSKEY=obscured --privileged --entrypoint /usr/bin/s3fs ubuntu/s3fs:latest -f s3bucket /s3bucket
docker run -it --volumes-from testmount --entrypoint /bin/ls ubuntu:12.04 -ahl /s3bucket
total 8.0K
drwxr-xr-x 2 root root 4.0K Aug 21 21:32 .
drwxr-xr-x 51 root root 4.0K Aug 21 21:33 ..
returns no files even though there are files in the bucket.
Adding another solution.
Dockerfile:
FROM ubuntu:16.04
# Update and install packages
RUN DEBIAN_FRONTEND=noninteractive apt-get -y update --fix-missing && \
apt-get install -y automake autotools-dev g++ git libcurl4-gnutls-dev wget libfuse-dev libssl-dev libxml2-dev make pkg-config
# Clone and run s3fs-fuse
RUN git clone https://github.com/s3fs-fuse/s3fs-fuse.git /tmp/s3fs-fuse && \
cd /tmp/s3fs-fuse && ./autogen.sh && ./configure && make && make install && ldconfig && /usr/local/bin/s3fs --version
# Remove packages
RUN DEBIAN_FRONTEND=noninteractive apt-get purge -y wget automake autotools-dev g++ git make && \
apt-get -y autoremove --purge && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Set user and group
ENV USER='appuser'
ENV GROUP='appuser'
ENV UID='1000'
ENV GID='1000'
RUN groupadd -g $GID $GROUP && \
useradd -u $UID -g $GROUP -s /bin/sh -m $USER
# Install fuse
RUN apt-get update && \
apt install fuse && \
chown ${USER}.${GROUP} /usr/local/bin/s3fs
# Config fuse
RUN chmod a+r /etc/fuse.conf && \
perl -i -pe 's/#user_allow_other/user_allow_other/g' /etc/fuse.conf
# Copy credentials
ENV SECRET_FILE_PATH=/home/${USER}/passwd-s3fs
COPY ./passwd-s3fs $SECRET_FILE_PATH
RUN chmod 600 $SECRET_FILE_PATH && \
chown ${USER}.${GROUP} $SECRET_FILE_PATH
# Switch to user
USER ${UID}:${GID}
# Create mnt point
ENV MNT_POINT_PATH=/home/${USER}/data
RUN mkdir -p $MNT_POINT_PATH && \
chmod g+w $MNT_POINT_PATH
# Execute
ENV S3_BUCKET = ''
WORKDIR /home/${USER}
CMD exec sleep 100000 && /usr/local/bin/s3fs $S3_BUCKET $MNT_POINT_PATH -o passwd_file=passwd-s3fs -o allow_other
docker-compose-yaml:
version: '3.8'
services:
s3fs:
privileged: true
image: <image-name:tag>
##Debug
#stdin_open: true # docker run -i
#tty: true # docker run -t
environment:
- S3_BUCKET=my-bucket-name
devices:
- "/dev/fuse"
cap_add:
- SYS_ADMIN
- DAC_READ_SEARCH
cap_drop:
- NET_ADMIN
Build image with docker build -t <image-name:tag> .
Run with: docker-compose -d up
If you would prefer to use docker-compose for testing on your localhost use the following. Note you don't need to use --privileged flag as we are passing --cap-add SYS_ADMIN --device /dev/fuse flags in the docker-compose.yml
create file .env
AWS_ACCESS_KEY_ID=xxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxx
AWS_BUCKET_NAME=xxxxxx
create file docker-compose.yml
version: "3"
services:
s3-fuse:
image: debian-aws-s3-mount
restart: always
build:
context: .
dockerfile: Dockerfile
environment:
- AWSACCESSKEYID=${AWS_ACCESS_KEY_ID}
- AWSSECRETACCESSKEY=${AWS_SECRET_ACCESS_KEY}
- AWS_BUCKET_NAME=${AWS_BUCKET_NAME}
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse
create file Dockerfile. i.e You can use any docker image you prefer but first, check if your distro is supported here
FROM node:16-bullseye
RUN apt-get update -qq
RUN apt-get install -y s3fs
RUN mkdir /s3_mnt
To run container execute:
$ docker-compose run --rm -t s3-fuse /bin/bash
Once inside the container. You can mount your s3 Bucket by running the command:
# s3fs ${AWS_BUCKET_NAME} s3_mnt/
Note: For this setup to work .env, Dockerfile and docker-compose.yml must be created in the same directory. Don't forget to update your .env file with the correct credentials to the s3 bucket.