Running Puppeter inside Docker Container - docker

I'm trying to run my application in container, this is my Dockerfile:
FROM node:12-slim
WORKDIR /app
# Install Chrome
RUN apt-get update \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN yarn add puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& chown -R pptruser:pptruser /node_modules
USER pptruser
CMD ["google-chrome-stable"]
COPY package.json ./
RUN yarn install
COPY . .
EXPOSE 3000
CMD ["yarn","start"]
I wrap my puppeter function with try catch, and when I try to run the application the catch is trigged:
try{
...
}catch(error){
Running as root without --no-sandbox is not supported. See https://crbug.com/638180.\n\n\nTROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md\n"}]
}
I add this arg const browser = await puppeteer.launch({ args: ['--no-sandbox', '--disable-setuid-sandbox'] });
My docker-compose:
version: '3.7'
services:
api:
build: .
ports:
- "3000:3000"
cap_add:
- SYS_ADMIN
init: true
container_name: web_scraping
I'm running the docker-compose, none erro appers while it is running, only when I call the api the catch is trigged

follow this Node.js + Puppeteer on Docker, No usable sandbox answer, I got this
docker-compose:
version: '3.7'
services:
api:
build: .
ports:
- "3000:3000"
container_name: web_scraping
Dockerfile:
FROM node:12
RUN apt-get update \
&& apt-get install -y wget gnupg ca-certificates \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
# We install Chrome to get all the OS level dependencies, but Chrome itself
# is not actually used as it's packaged in the node puppeteer library.
# Alternatively, we could could include the entire dep list ourselves
# (https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix)
# but that seems too easy to get out of date.
&& apt-get install -y google-chrome-stable \
&& rm -rf /var/lib/apt/lists/* \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
WORKDIR /app
COPY package.json ./
RUN yarn install
COPY . .
EXPOSE 3000
CMD ["yarn","start"]
And Add this args in launch instance:
const browser = await puppeteer.launch({
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
],
});

Related

Enable/Disable xDebug through Docker-Compose, is it possible?

I have the following Dockerfile:
FROM php:7.2-apache
RUN apt-get update && apt-get install -y \
zlib1g-dev \
git \
unzip \
&& rm -rf /var/lib/apt/lists/ \
&& docker-php-ext-install zip pdo_mysql bcmath \
&& a2enmod rewrite \
&& pecl install xdebug-2.9.0 redis \
&& docker-php-ext-enable xdebug redis \
&& mv "$PHP_INI_DIR/php.ini-development" "$PHP_INI_DIR/php.ini" \
&& mv /var/www/html /var/www \
&& chown -R www-data:www-data /var/www/ \
&& ln -s /usr/local/bin/php /usr/bin/php
COPY --from=composer:1.9 /usr/bin/composer /usr/bin/composer
WORKDIR /var/www/drm-case
CMD ["docker/apache/bootstrap.sh"]
And this is the docker-compose.yml file that uses the previous Dockerfile to build the containers (not all the time though):
version: "2.4"
services:
case-v2-apache:
container_name: local-dev
image: local-dev:1.6 # increase this number when changing the Dockerfile
depends_on:
mysql-server:
condition: service_healthy
volumes:
- ${LOCAL_PATH}:/var/www:delegated
- ${LOCAL_PATH}/docker/apache/conf.d:/usr/local/etc/php/conf.d
- ${LOCAL_PATH}/docker/apache/conf-enabled/servername.conf:/etc/apache2/conf-enabled/servername.conf
- ${LOCAL_PATH}/docker/apache/sites-available/000-default.conf:/etc/apache2/sites-available/000-default.conf
ports:
- "8009:80"
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
build:
context: ${LOCAL_PATH:-./local-dev}
dockerfile: docker/apache/Dockerfile
networks:
- main
environment:
- VIRTUAL_HOST=local-dev.localhost
- COMPOSER_MEMORY_LIMIT=-1
- COMPOSER_AUTH=${COMPOSER_AUTH}
- COMPOSER_ALLOW_SUPERUSER=1
- PHP_IDE_CONFIG=serverName=local-dev
One of the files being copied in this line: ${LOCAL_PATH}/docker/apache/conf.d:/usr/local/etc/php/conf.d is xdebug related meaning is the one enabling and setting up the extension.
Wonder if there is a way to tell Docker by using ARG or ENV variables to enable/disable xDebug while starting the container? Has anyone tried such thing before? If so can you help me with some ideas?
You could simply move/rename the xdebug conf file based on a build arg…
FROM php:7.2-apache
RUN apt-get update && apt-get install -y \
zlib1g-dev \
git \
unzip \
&& rm -rf /var/lib/apt/lists/ \
&& docker-php-ext-install zip pdo_mysql bcmath \
&& a2enmod rewrite \
&& pecl install xdebug-2.9.0 redis \
&& docker-php-ext-enable xdebug redis \
&& mv "$PHP_INI_DIR/php.ini-development" "$PHP_INI_DIR/php.ini" \
&& mv /var/www/html /var/www \
&& chown -R www-data:www-data /var/www/ \
&& ln -s /usr/local/bin/php /usr/bin/php
COPY --from=composer:1.9 /usr/bin/composer /usr/bin/composer
RUN if [[ "$DISABLE_XDEBUG" == "1" ]] ; then mv "$PHP_INI_DIR/conf.d/xdebug.ini" "$PHP_INI_DIR/conf.d/xdebug.ini.disabled"
WORKDIR /var/www/drm-case
CMD ["docker/apache/bootstrap.sh"]
… or something like that?

composer install with prestissimo in docker container

I can try composer install in docker container on ubuntu16.04 image with hirak/prestissimo plugin.
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y software-properties-common \
&& LC_ALL=C.UTF-8 add-apt-repository ppa:ondrej/php \
&& apt-get update \
&& apt-get install -y php7.1 git vim supervisor curl unzip mysql-client wget \
&& apt-get install -y --no-install-recommends php7.1-cli php7.1-dev \
php7.1-pgsql php7.1-sqlite3 php7.1-gd \
php7.1-curl php7.1-memcached \
php7.1-imap php7.1-mysql php7.1-mbstring \
php7.1-xml php7.1-imagick php7.1-zip php7.1-bcmath php7.1-soap \
php7.1-intl php7.1-readline php7.1-mcrypt php7.1-fpm
RUN curl -sS https://getcomposer.org/installer | php && mv composer.phar /usr/local/bin/composer
RUN composer --no-interaction global require 'hirak/prestissimo'
COPY docker/config/ /config/
RUN cp /config/supervisord.conf /etc/supervisor/supervisord.conf
WORKDIR /app
EXPOSE 9000
ENTRYPOINT ["/config/entrypoint.sh"]
And my docker-compose.yml file
version: '3'
services:
php71:
restart: unless-stopped
image: mylogin/myimage
volumes:
- ./:/app
networks:
- my-network
But when i run composer-install in my docker container parallel installation with hirak/prestissimo plugin doesn't work.

How to access remote debugging page for dockerized Chromium launch by Puppeteer?

When the chromium succeed to launch, its Debugging WebSocket URL should be like ws://127.0.0.1:9222/devtools/browser/ec261e61-0e52-4016-a5d7-d541e82ecb0a.
127.0.0.1:9222 should be able to browse by Chrome to inspect the headless Chromium. However, I cannot access the remote debugger URL by Chrome after I dockerize my application.
launchOption for launching chromium by Puppeteer:
{
"args": [
"--remote-debugging-port=9222",
"--window-size=1920,1080",
"--mute-audio",
"--disable-notifications",
"--force-device-scale-factor=0.8",
"--no-sandbox",
"--disable-setuid-sandbox"
],
"defaultViewport": {
"height": 1080,
"width": 1920
},
"headless": true
}
Dockerfile:
FROM node:10.16.3-slim
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/* \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
WORKDIR /usr/app
COPY ./ ./
VOLUME ["......." ]
RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& chown -R pptruser:pptruser /usr/app \
&& npm install
USER pptruser
CMD npm run start
EXPOSE 3000 9222
Run the new container by :
docker run \
-p 3000:3000 \
-p 9222:9222 \
pptr
Port 9222 should be accessible in my host machine. But Chrome shows the error ERR_EMPTY_RESPONSE when I browse 127.0.0.1:9222 and DOCKER-INTERNAL-IP:9222 will timeout.
I managed to make this work with puppeteer using the following Dockerfile, docker run and puppeteer config:
FROM ubuntu:18.04
RUN apt update \
&& apt install -y \
curl \
wget \
gnupg \
gcc \
g++ \
make \
&& curl -sL https://deb.nodesource.com/setup_10.x | bash - \
&& apt install -y nodejs \
&& rm -rf /var/lib/apt/lists/*
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser
ADD . /app
WORKDIR /app
RUN chown -R pptruser:pptruser /app
RUN rm -rf node_modules
RUN rm -rf build/*
USER pptruser
RUN npm install --dev
RUN chmod +x /app/entrypoint.sh
ENTRYPOINT /app/entrypoint.sh
Docker run:
docker run -p 9223:9222 -it myimage
Puppeteer launch:
this.browser = await puppeteer.launch(
{
headless: true,
args: [
'--remote-debugging-port=9222',
'--remote-debugging-address=0.0.0.0',
'--no-sandbox'
]
}
);
The entrypoint just launches the platform like: node build/main.js
After that I just had to connect to localhost:9223 on Chrome to see the browser. Hope it helps!
I know there is already an accepted answer, but let me add onto this in hopes to greatly reduce your image size. One shouldn't add too many extras into the Dockerfile if one can help it. But ultimately, adding --remote-debugging-port=9222 and --remote-debugging-address=0.0.0.0 will allow you to access it.
Dockerfile
FROM ubuntu:latest
LABEL Full Name <email#email.com> https://yourwebsite.com
WORKDIR /home/
COPY wrapper-script.sh wrapper-script.sh
# install chromium-browser and cleanup.
RUN apt update && apt install chromium-browser --no-install-recommends -y && apt autoremove && apt clean && apt autoclean && rm -rf /var/lib/apt/lists/*
# Run your commands and add environment variables from your compose file.
CMD ["sh", "wrapper-script.sh"]
I use a wrapper script so that I can include environment variables here. You can see URL and USERNAME set so that I can configure them from the compose file. Of course, i'm sure there is a better way to do this, but I do this so that I can scale my containers horizontally with ease.
wrapper-script.sh
#!/bin/bash
# Start the process
chromium-browser --headless --disable-gpu --no-sandbox --remote-debugging-port=9222 --remote-debugging-address=0.0.0.0 ${URL}${USERNAME}
status=$?
if [ $status -ne 0 ]; then
echo "Failed to start chromium-browser: $status"
exit $status
fi
# Naive check runs checks once a minute to see if either of the processes exited.
# This illustrates part of the heavy lifting you need to do if you want to run
# more than one service in a container. The container exits with an error
# if it detects that either of the processes has exited.
# Otherwise it loops forever, waking up every 60 seconds
while sleep 60; do
ps aux |grep chromium-browser | grep -q -v grep
PROCESS_1_STATUS=$?
# If the greps above find anything, they exit with 0 status
# If they are not both 0, then something is wrong
if [ $PROCESS_1_STATUS -ne 0 ]; then
echo "One of the processes has already exited."
exit 1
fi
done
Lastly, I have the docker-compose file. This is where I define all my settings so that I can configure my wrapper-script.sh with what I need and scale horizontally. Notice the environment section of the docker-compose file. USERNAME and URL are environment variables, and they can be called from the wrapper script.
docker-compose.yml
version: '3.7'
services:
chrome:
command: [ 'sh', 'wrapper-script.sh' ]
image: headless-chrome
build:
context: .
dockerfile: Dockerfile
environment:
- USERNAME=eaglejs
- URL=https://teamtreehouse.com/
ports:
- 9222:9222
If you are wondering what my folder structure looks like. all three files are at the root of the folder. For example:
My_Docker_Repo:
Dockerfile
docker-compose.yml
wrapper-script.sh
After that is all said and done, I simply run docker-compose up and I have one container running. Right now, using the ports section, you'll have to do something to scale that as well. if you were to run docker-compose up --scale chrome=5 your ports will clash, but let me know if you want to try that and i'll see what I can do for scaling, but other than that, if it is for testing, this should work well the way it is. :) Happy coding!
eaglejs

Ionic serve hot reload with docker not working

I had an ionic app which I have dockerized. The build and up commands are successful and I can access the app at http://localhost:8100. However, hot reload doesn't work. Whenever I edit some file, those changes are nor reflected until redo an ionic serve, but if I rename some file inside the container, the ionic rebuild the app automatically.
Dockerfile
FROM ubuntu:18.04
ENV DEBIAN_FRONTEND=noninteractiv \
TERM=xterm \
# https://cordova.apache.org/docs/en/8.x/guide/platforms/android/#installing-the-requirements
JAVA_VERSION=8 \
# https://github.com/gradle/gradle/releases
GRADLE_VERSION=5.2.1 \
# https://developer.android.com/studio/releases/sdk-tools V26.1.1
ANDROID_SDK_URL="https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip" \
# https://developer.android.com/studio/releases/build-tools
ANDROID_BUILD_TOOLS_VERSION=28.0.3 \
# https://source.android.com/setup/start/build-numbers
ANDROID_APIS="android-28" \
ANDROID_HOME="/opt/android" \
# https://nodejs.org/en/
NODE_VERSION=11.10.0 \
# https://www.npmjs.com/package/cordova
CORDOVA_VERSION=8.1.2 \
# https://www.npmjs.com/package/ionic
IONIC_VERSION=4.10.3
# dependences
WORKDIR /tmp
RUN mkdir /app && \
buildDeps='software-properties-common'; \
set -x && \
dpkg --add-architecture i386 && \
apt-get -qq update && \
apt-get -qq install -y $buildDeps curl git ca-certificates bzip2 openssh-client unzip wget libncurses5:i386 libstdc++6:i386 zlib1g:i386 --no-install-recommends
# java # use WebUpd8 PPA
RUN add-apt-repository ppa:webupd8team/java -y && \
apt-get -qq update -y && \
# automatically accept the Oracle license
echo oracle-java"$JAVA_VERSION"-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \
apt-get -qq install -y oracle-java"$JAVA_VERSION"-installer oracle-java"$JAVA_VERSION"-set-default
ENV JAVA_HOME=/usr/lib/jvm/java-"$JAVA_VERSION"-oracle
# gradle
RUN wget https://services.gradle.org/distributions/gradle-"$GRADLE_VERSION"-bin.zip && \
mkdir /opt/gradle && \
unzip -d /opt/gradle gradle-"$GRADLE_VERSION"-bin.zip
ENV PATH=$PATH:/opt/gradle/gradle-"$GRADLE_VERSION"/bin
# android
WORKDIR /opt/android
ENV PATH $PATH:"$ANDROID_HOME"/platform-tools:"$ANDROID_HOME"/tools:"$ANDROID_HOME"/build-tools/"$ANDROID_BUILD_TOOLS_VERSION"
RUN wget -O tools.zip "$ANDROID_SDK_URL" && \
unzip tools.zip && \
rm tools.zip && \
# echo y | android update sdk -a -u -t platform-tools,"$ANDROID_APIS",build-tools-"$ANDROID_BUILD_TOOLS_VERSION" && \
yes Y | "$ANDROID_HOME"/tools/bin/sdkmanager "build-tools;$ANDROID_BUILD_TOOLS_VERSION" "platforms;$ANDROID_APIS" "platform-tools" && \
chmod a+x -R "$ANDROID_HOME" && \
chown -R root:root "$ANDROID_HOME"
# node
WORKDIR /opt/node
RUN curl -sL https://nodejs.org/dist/v"$NODE_VERSION"/node-v"$NODE_VERSION"-linux-x64.tar.gz | tar xz --strip-components=1
ENV PATH=$PATH:/opt/node/bin
# ionic & cordova
WORKDIR /tmp
RUN npm i -g ionic#"$IONIC_VERSION" cordova#"$CORDOVA_VERSION" && \
ionic config set -g daemon.updates false && \
cordova telemetry off
# clean up
RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
apt-get purge -y --auto-remove $buildDeps && \
apt-get autoremove -y && \
apt-get clean
# app
WORKDIR /app
EXPOSE 8100 35729
docker-compose
version: '3'
networks:
default:
driver: bridge
ipam:
config:
- subnet: 172.18.19.0/24
services:
ionic:
container_name: Ionic
image: ionic
restart: 'no'
networks:
default:
ports:
- "8100:8100"
- "35729:35729"
volumes:
- w:\projects\test:/app
command: "ionic serve"
Is the problem something I did? or will it be a docker / ionic bug?
I found this on github. you can try to put this in /etc/sysctl.conf
fs.inotify.max_user_watches=524288
and then reboot or
sudo sysctl --system

Why are environment variables not being set inside of my docker container?

I have the following in my docker-compose.yml:
php:
build:
args:
http_proxy:
https_proxy:
no_proxy:
context: ./vm-images/php
environment:
http_proxy:
https_proxy:
no_proxy:
CURRENT_ENVIRONMENT: DOCKER
container_name: php
ports:
- "9000:9000"
depends_on:
- mysql
links:
- mysql:mysql
logging:
driver: "json-file"
volumes:
- /var/www/:/var/www/
My DOCKERFILE has the following:
FROM ubuntu:latest
RUN locale-gen en_US.UTF-8 && \
export LANG=en_US.UTF-8 && \
export LC_ALL=en_US.UTF-8
RUN apt-get update && \
apt-get install -y --reinstall ca-certificates
RUN apt-get update \
&& apt-get install -y --allow-unauthenticated --no-install-recommends \
software-properties-common python-software-properties
RUN add-apt-repository ppa:ondrej/php
RUN apt-get update \
&& apt-get install -y --allow-unauthenticated --no-install-recommends \
git-core \
libjpeg-progs \
mysql-client \
optipng \
php5.6 \
php5.6-curl \
php5.6-intl \
php5.6-fpm \
php5.6-gd \
php5.6-mcrypt \
php5.6-mysqli \
php5.6-pdo \
php5.6-xdebug \
php5.6-xml \
php5.6-zip
RUN rm -r /var/lib/apt/lists/*
RUN rm -r /var/cache/apt/*
RUN php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" && \
php -r "if (hash_file('SHA384', 'composer-setup.php') === 'e115a8dc7871f15d853148a7fbac7da27d6c0030b848d9b3dc09e2a0388afed865e6a3d6b3c0fad45c48e2b5fc1196ae') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" && \
php composer-setup.php --install-dir=/usr/local/bin --filename=composer && \
php -r "unlink('composer-setup.php');"
RUN mkdir -p /run/php && chown www-data:www-data /run/php
COPY files/etc/php/fpm/php-fpm.conf /etc/php/5.6/fpm/php-fpm.conf
COPY files/etc/php/fpm/pool.d/www.conf /etc/php/5.6/fpm/pool.d/www.conf
COPY files/etc/php/fpm/php.ini /etc/php/5.6/fpm/php.ini
COPY files/etc/php/cli/php.ini /etc/php/5.6/cli/php.ini
COPY files/etc/php/mods-available/xdebug.ini /etc/php/5.6/mods-available/xdebug.ini
RUN phpenmod curl intl gd mcrypt mysqli pdo xdebug xml zip
CMD ["php-fpm5.6", "--nodaemonize", "--fpm-config", "/etc/php/5.6/fpm/php-fpm.conf"]
When I run docker-compose --verbose up -d I notice that there are no env vars passed in for my php container. I've verified that those env vars are set on my host, and they are passed in for all of my other containers without fail. Anyone have any ideas?
EDIT: Strangely enough, I've also noticed that /proc/$PID/environ (where $PID is the PID of the php container [obtained by running docker inspect --format "{{.State.Pid}}" php] on the host machine is empty.

Resources