GitLab CI: UI Testing with Selenium - docker

I have built some UI tests for my application using Selenium and PyTest. I want to automate these tests by integrating the in the GitLab CI. As the latter is docker-based and I am building by UI using a docker-compose script, I need docker in docker (docker:dind) capabilities besides Firefox, Selenium and Python.
My current gitlab-ci.yml looks like this:
ui_test:
stage: Build_and_Test
image:
name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker/compose
entrypoint: [ '/bin/sh', '-c' ]
services:
- name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:dind
alias: docker
- selenium/standalone-firefox:latest
variables:
GET_SOURCES_ATTEMPTS: 1
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ''
PROXY_REGISTRY_PREFIX: '${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/'
before_script:
- apk add --update --no-cache bash git openssh gcc musl-dev g++ make build-base libffi-dev python3-dev libressl-dev python3 wget tar npm
- apk --no-cache add --repository http://dl-cdn.alpinelinux.org/alpine/edge/community firefox
- echo which firefox
- python3 -m ensurepip
- wget https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-linux64.tar.gz
- tar -xvf geckodriver-v0.31.0-linux64.tar.gz
- chmod +x geckodriver
- export PATH=$PATH:/builds/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
script:
- docker login -u "$CI_DEPENDENCY_PROXY_USER" -p "$CI_DEPENDENCY_PROXY_PASSWORD" "$CI_DEPENDENCY_PROXY_SERVER"
- alias python=python3
- echo 'alias python=python3' >> ~/.bashrc
- source ~/.bashrc
- python3 -m pip install --upgrade pip setuptools wheel
- python3 -m pip install --no-cache-dir --upgrade cryptography requests webdriver-manager
- python3 -m pip install --no-cache-dir --upgrade pytest
- python3 -m pip install --no-cache-dir --upgrade pytest-asyncio
- python3 -m pip install --no-cache-dir selenium==4.1.5
- mkdir -p ./test_output
- ./docker/build_hdssl_mock.sh
- ./docker/start_hdssl_mock.sh & cd ui & npm run serve & python3 -m pytest hdssl_control/tests/acceptance_tests
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_PIPELINE_SOURCE == "merge_request_event" || $CI_COMMIT_BRANCH == "develop"
The problem here is that Firefox requires all kinds of dependencies that I do not know how to install in Alpine:
ERROR: unsatisfiable constraints:
so:libcjson.so.1 (missing):
required by: librist-0.2.6-r1[so:libcjson.so.1]
so:libmbedcrypto.so.7 (missing):
required by: librist-0.2.6-r1[so:libmbedcrypto.so.7]
so:libffi.so.8 (missing):
required by: firefox-101.0.1-r0[so:libffi.so.8]
so:libicui18n.so.71 (missing):
required by: firefox-101.0.1-r0[so:libicui18n.so.71]
so:libicuuc.so.71 (missing):
required by: firefox-101.0.1-r0[so:libicuuc.so.71]
Any help is greatly appreciated.

Related

Generating PHP library with Dockerized gRPC

I'm trying to build a gRPC PHP Client and gRPC NodeJs Server in docker. But the problem is I can't install protoc-gen-php-grpc to my docker server. When I try to run this run this makefile:
proto_from_within_container:
# PHP
protoc /var/www/protos/smellycat.proto \
--php_out=/var/www/php-client/src \
$(: 👇 generate server interface) \
--php-grpc_out=/var/www/php-client/src \
$(: 👇 generates the client code) \
--grpc_out=/var/www/php-client/src \
--plugin=protoc-gen-grpc=/protobuf/grpc/bins/opt/grpc_php_plugin \
--proto_path /var/www/protos
proto:
powershell rm -r -fo php-client/src -ErrorAction SilentlyContinue
powershell New-Item -ItemType Directory -Path php-client/src -Force -ErrorAction SilentlyContinue
docker-compose run grpc-server make proto_from_within_container
With this command: make proto
Getting this error message after docker containers builded:
protoc /var/www/protos/smellycat.proto \
--php_out=/var/www/php-client/src \
\
--php-grpc_out=/var/www/php-client/src \
\
--grpc_out=/var/www/php-client/src \
--plugin=protoc-gen-grpc=/protobuf/grpc/bins/opt/grpc_php_plugin \
--proto_path /var/www/protos
protoc-gen-php-grpc: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--php-grpc_out: protoc-gen-php-grpc: Plugin failed with status code 1.
Makefile:4: recipe for target 'proto_from_within_container' failed
make: *** [proto_from_within_container] Error 1
This is my docker-compose file
version: "3"
services:
grpc-server:
container_name: grpc-server
build:
context: .
dockerfile: Dockerfile-server
working_dir: /var/www
volumes:
- .:/var/www
grpc-client:
image: php:7.4-cli
container_name: grpc-client
build:
context: .
dockerfile: Dockerfile-client
working_dir: /var/www
volumes:
- .:/var/www
command: bash -c [php php_client.php && composer install]
And this is my grpc-server docker file:
FROM node:latest
ENV DEBIAN_FRONTEND=noninteractive
#Versions
ARG PROTOBUF_VERSION=3.14.0
ARG PHP_GRPC_VERSION=1.34.0
# Utils
RUN apt-get update -yqq \
&& apt-get install -yqq wget unzip zlib1g-dev git autoconf libtool automake build-essential software-properties-common curl zip \
&& rm -rf /var/lib/apt/lists/*
# Protobuf
RUN mkdir -p /protobuf
RUN cd /protobuf \
&& wget https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-linux-x86_64.zip -O protobuf.zip \
&& unzip protobuf.zip && rm protobuf.zip
# grpc PHP (generate client)
RUN apt-get update -yqq && apt-get upgrade -yqq
RUN apt-get install php php-dev php-pear phpunit zlib1g-dev -yqq
RUN pecl install grpc-${PHP_GRPC_VERSION}
RUN cd /protobuf && git clone -b v${PHP_GRPC_VERSION} https://github.com/grpc/grpc \
&& cd /protobuf/grpc && git submodule update --init
RUN cd /protobuf/grpc && make grpc_php_plugin
ENV PATH "/protobuf/bin:${PATH}"
ENV PATH "/protobuf/grpc/bins/opt:${PATH}"
# NPM Installation
WORKDIR /var/www
COPY . /var/www
RUN npm install
CMD ["node", "server.js"]
Do you have any advice?
After a lot of search and readings, I finally managed to build a full application that communicates with each other.
The problem was at the Makefile, at this step:
--plugin=protoc-gen-grpc=/protobuf/grpc/bins/opt/grpc_php_plugin
I was assigning the wrong path for grpc_php_plugin.
There is my new dockerfile:
FROM php:7.4-cli
# Environment variables
ENV DEBIAN_FRONTEND=noninteractive
# Utils
RUN apt-get update -yqq && \
apt-get upgrade -yqq && \
apt-get install -y unzip build-essential git software-properties-common curl pkg-config zip zlib1g-dev
# Composer installation
COPY --from=composer:latest /usr/bin/composer /usr/local/bin/composer
# Install grpc and probuf with pecl
RUN pecl install grpc && pecl install protobuf
# Enable grpc and protobuf extensions in php.ini file
RUN echo starting && \
docker-php-ext-enable grpc && \
docker-php-ext-enable protobuf
# Install cmake
RUN apt-get update -yqq && apt-get -y install cmake
# Install grpc_php_plugin and protoc
RUN git clone -b v1.36.2 https://github.com/grpc/grpc && \
cd grpc && git submodule update --init && \
mkdir cmake/build && cd cmake/build && \
cmake ../.. && make protoc grpc_php_plugin
# Setting node, protoc and grpc_php_plugin paths
ENV PATH "/grpc/cmake/build:${PATH}"
ENV PATH "/grpc/cmake/build/third_party/protobuf:${PATH}"
# Moving client folder to vm
WORKDIR /var/www
COPY ./client /var/www
# Packages
RUN composer install
# Generate php libraries from proto file
RUN make proto
CMD [ "php", "./handler.php" ]
For my full application, click.

Change Xdebug Mode PHP Docker on Gitlab CI

I want run PHPUnit code coverage using GitLab CI but when i run the command vendor/bin/phpunit --coverage-text --colors=never is only show code test result and not generating code coverage.
When I test it on local using XAMPP with Xdebug enable the code coverage generated with using xdebug.mode=coverage. How to change the xdebug.mode on php.ini on docker setup script ?
I also override it via PHPUnit XML configuration like this also not work.
<ini name="xdebug.mode" value="coverage" />
Here is my .gitlab-ci.yml
image: php:7.4.14
cache:
key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
paths:
- vendor
before_script:
# Install git
- apt-get update -yqq
- apt-get install git -yqq
# Install packages
- apt-get install libcurl4-openssl-dev
# Install required php extension
- docker-php-ext-install curl
# Install Xdebug
- pecl install xdebug
- docker-php-ext-enable xdebug
# Install composer
- curl -sS https://getcomposer.org/installer | php
# Install dependencies
- php composer.phar update --prefer-stable --prefer-dist --no-interaction --no-progress
run-test:
script:
- vendor/bin/phpunit --coverage-text --colors=never
After searching some article finally i can do it like this.
Here my example .gitlab-ci.yml
image: php:7.4.14
cache:
key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
paths:
- vendor
before_script:
# Install git
- apt-get update -yqq
- apt-get install git -yqq
# Install packages
- apt-get install libcurl4-openssl-dev
# Install required php extension
- docker-php-ext-install curl
# Install Xdebug
- pecl install xdebug
- docker-php-ext-enable xdebug
# Install composer
- curl -sS https://getcomposer.org/installer | php
# Install dependencies
- php composer.phar update --prefer-stable --prefer-dist --no-interaction --no-progress
# Here to change xdebug mode
- echo xdebug.mode=coverage > /usr/local/etc/php/conf.d/xdebug.ini
run-test:
script:
- vendor/bin/phpunit --coverage-text --colors=never
In my dockerfile I added this lines :
RUN pecl install xdebug \
&& docker-php-ext-enable xdebug \
&& echo "xdebug.mode=debug" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& echo "xdebug.client_host=host.docker.internal" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
&& echo "xdebug.mode=coverage" >> /usr/local/etc/php/conf.d/xdebug.ini

Stenciljs e2e tests on Google Cloud Build

TL;DR: does anyone know how to make a valid stencil.js docker image to run stencil build & test?
Long form:
To run stencil.js e2e tests on Google Cloud Build you need a custom build step as a docker image.
Here's a sample Dockerfile:
# THESE STEPS GET STENCIL BUILD WORKING & SHOULD HAVE GOT TESTING WORKING
FROM node:10-jessie-slim
WORKDIR /
RUN npm init stencil app stencil
WORKDIR /stencil
COPY package*.json ./
RUN npm install
WORKDIR /stencil/node_modules/puppeteer
RUN npm install
WORKDIR /stencil
# STEPS ADDED BASED ON https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#running-puppeteer-in-docker
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl wget && rm -rf /var/lib/apt/lists/*
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN npm i puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /stencil/home/pptruser/Downloads \
&& chown -R pptruser:pptruser /stencil/home/pptruser \
&& chown -R pptruser:pptruser /stencil/node_modules
ENTRYPOINT ["npm"]
Now you insert this into a cloud build.yaml file:
steps:
#1 Build stencil project
- name: 'gcr.io/$PROJECT_ID/stencil'
args: ['run','build']
#2 Test stencil project
- name: 'gcr.io/$PROJECT_ID/stencil'
args: ['test']
In this build file step #1 works, validating the stencil install. However, step #2 fails with error message:
[ ERROR ] Chromium revision is not downloaded. Run "npm install" or "yarn
Step #2: install" Error: Chromium revision is not downloaded. Run "npm
Step #2: install" or "yarn install" at Launcher.launch
Step #2: (/workspace/node_modules/puppeteer/lib/Launcher.js:120:15)
The error above is about puppeteer not finding chromium (even though a local version is installed) and I have already ran the npm install on puppeteer and validated that the local chromium is installed. However Puppeteer documentation does mention running puppeteer in Docker is tricky and gives a solution, but their solution is for a docker container dedicated just to Puppeteer.
Does anyone have an idea how I can create a valid stencil.js docker image with a valid puppeteer setup?
I finally managed to get stencil working in a container suitable for Google Cloud Build.
The main problem was that puppeteer, which is needed for e2e test, does not work as installed because it does not have a chrome install with all necessary dependencies.
To fix you have to do three things:
Install Chrome separately
Point puppeteer to the installed Chrome
Modify stencil config to invoke testing without a sandbox
1 & 2 are addressed with the following Dockerfile:
# Need jessie to install dependencies
FROM node:10-jessie-slim
# Copy files from stencil project
WORKDIR /
COPY package*.json ./
COPY node_modules/ ./node_modules
# Install wget & dependencies needed to install Chrome (next step)
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl wget && rm -rf /var/lib/apt/lists/*
# Install Chromium dev & dependencies
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
# Point puppeteer to the Chromium just installed
ENV PUPPETEER_EXECUTABLE_PATH '/usr/bin/google-chrome-unstable'
# Set entry point
ENTRYPOINT ["npm"]
Note that this Dockerfile must be placed in the same directory as your stencil project --i.e. it's in the same location as your stencil.config.ts and package.json.
Also note that this will only work if you have already run your tests in your local environment. Doing so ensures that the necessary testing dependencies are installed.
3 is fixed by modifying your stencil.config.ts as shown in stencil documentation so chrome runs without a sandbox:
export const config: Config = {
namespace: 'Foo',
testing: {
// run chrome with no sandbox to have it work in a container
browserArgs: ['--no-sandbox', '--disable-setuid-sandbox'],
},
outputTargets: [
{ type: 'dist' },
{
type: 'www',
},
],
};

Docker Debian nc command not found

When I build my Debian image from docker-compose, with the command $ docker-compose -f docker-compose-dev.yml build web, like so:
docker-compose-fev.yml
services:
web:
build:
context: ./services/web
dockerfile: Dockerfile-dev
volumes:
- './services/web:/usr/src/app'
ports:
- 5001:5000
environment:
- FLASK_ENV=development
- APP_SETTINGS=project.config.DevelopmentConfig
- DATABASE_URL=postgres://postgres:postgres#web-db:5432/web_dev
- DATABASE_TEST_URL=postgres://postgres:postgres#web-db:5432/web_test
- SECRET_KEY=my_precious
depends_on:
- web-db
- redis
As though it appears to build all packages successfully, I'm getting:
web_1| /usr/src/app/entrypoint.sh: 5: /usr/src/app/entrypoint.sh: nc: not found
If I change #!/bin/sh to #!/bin/bash, error log changes:
web_1| /usr/src/app/entrypoint.sh: line 5: nc: command not found
Dockerfile:
FROM python:3.7-slim-buster
RUN apt-get update && apt-get -y dist-upgrade
RUN apt-get -y install build-essential libssl-dev libffi-dev libblas3 libc6 liblapack3 gcc python3-dev python3-pip cython3
RUN apt-get -y install python3-numpy python3-scipy
# set working directory
WORKDIR /usr/src/app
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip3 install -r requirements.txt
# add entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# add app
COPY . /usr/src/app
# run server
CMD ["/usr/src/app/entrypoint.sh"]
entrypoint.sh
#!/bin/sh
echo "Waiting for postgres..."
while ! nc -z web-db 5432; do
sleep 0.1
done
rm -rf celery_logs/*
echo "PostgreSQL started"
python manage.py run -h 0.0.0.0
Note: this entrypoint configuration used to work with Alpine, and now has changed to Debian.
what am I missing?
Update the Dockerfile and append,
RUN apt install -y netcat
It should be like,
FROM python:3.7-slim-buster
RUN apt-get update && apt-get -y dist-upgrade
RUN apt-get -y install build-essential libssl-dev libffi-dev libblas3 libc6 liblapack3 gcc python3-dev python3-pip cython3
RUN apt-get -y install python3-numpy python3-scipy
RUN apt install -y netcat

Docker-compose does not reflect changes in requirements.txt

Changes in my requirements.txt are not being reflected when I run:
docker-compose -f docker-compose-dev.yml up -d
docker-compose-dev.yml
version: '3.6'
services:
web:
build:
context: ./services/web
dockerfile: Dockerfile-dev
volumes:
- './services/web:/usr/src/app'
ports:
- 5001:5000
environment:
- FLASK_ENV=development
- APP_SETTINGS=project.config.DevelopmentConfig
- DATABASE_URL=postgres://postgres:postgres#web-db:5432/web_dev
- DATABASE_TEST_URL=postgres://postgres:postgres#web-db:5432/web_test
depends_on:
- web-db
web-db:
build:
context: ./services/web/project/db
dockerfile: Dockerfile
ports:
- 5435:5432
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
nginx:
build:
context: ./services/nginx
dockerfile: Dockerfile-dev
restart: always
ports:
- 80:80
depends_on:
- web
- client
client:
build:
context: ./services/client
dockerfile: Dockerfile-dev
volumes:
- './services/client:/usr/src/app'
- '/usr/src/app/node_modules'
ports:
- 3007:3000
environment:
- NODE_ENV=development
- REACT_APP_WEB_SERVICE_URL=${REACT_APP_WEB_SERVICE_URL}
depends_on:
- web
Dockerfile-dev
# base image
FROM python:3.6-alpine
# install dependencies
RUN apk update && \
apk add --virtual build-deps gcc python-dev musl-dev && \
apk add libffi-dev && \
apk add postgresql-dev && \
apk add netcat-openbsd && \
apk add bind-tools && \
apk add --update --no-cache g++ libxslt-dev && \
apk add jpeg-dev zlib-dev
ENV PACKAGES="\
dumb-init \
musl \
libc6-compat \
linux-headers \
build-base \
bash \
git \
ca-certificates \
freetype \
libgfortran \
libgcc \
libstdc++ \
openblas \
tcl \
tk \
libssl1.0 \
"
ENV PYTHON_PACKAGES="\
numpy \
matplotlib \
scipy \
scikit-learn \
nltk \
"
RUN apk add --no-cache --virtual build-dependencies python3 \
&& apk add --virtual build-runtime \
build-base python3-dev openblas-dev freetype-dev pkgconfig gfortran \
&& ln -s /usr/include/locale.h /usr/include/xlocale.h \
&& python3 -m ensurepip \
&& rm -r /usr/lib/python*/ensurepip \
&& pip3 install --upgrade pip setuptools \
&& ln -sf /usr/bin/python3 /usr/bin/python \
&& ln -sf pip3 /usr/bin/pip \
&& rm -r /root/.cache \
&& pip install --no-cache-dir $PYTHON_PACKAGES \
&& pip3 install 'pandas<0.21.0' \
&& apk del build-runtime \
&& apk add --no-cache --virtual build-dependencies $PACKAGES \
&& rm -rf /var/cache/apk/*
# set working directory
WORKDIR /usr/src/app
# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt # <--- refer to EDIT
RUN pip install -r requirements.txt
# add entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# add app
COPY . /usr/src/app
# run server
CMD ["/usr/src/app/entrypoint.sh"]
what am I missing?
EDIT
Like the accepted answer in [Docker how to run pip requirements.txt only if there was a change?, I'm already copying the requirements.txt file in a separate build step before adding the entire application into the image, but it does not seem to work.
I think the problem likely is that $ docker-compose up alone will not rebuild your images if you make changes. In order to get docker-compose to include your changes to your requirements.txt you will need to pass the --build flag to docker-compose.
I.e instead run:
docker-compose -f docker-compose-dev.yml up --build -d
Which will force a docker-compose rebuild the image. However this will rebuild all images in the docker-compose file which may or may not be desired.
If you only want to rebuild the image of a single service you can first run docker-compose -f docker-compose-dev.yml build web, then afterwards just run your original docker-compose command.
More info on the build command here.
Try to install requirements from the copied file
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
It is an example of their Dockerfile
COPY requirements.txt /tmp/
RUN pip install --requirement /tmp/requirements.txt
This is what you have
RUN pip install -r requirements.txt
Then after you have changed your docker file, you have to stop your container, remove your image, build a new one, and run container from it.
Stop container and remove the image.
docker-compose down
docker-compose --rmi all
--rmi all - removes all images. You might want to use --rmi IMAGE_NAME
And to start it (if you use not default parameters, change these commands with your arguments).
docker-compose up
Update
In case you have running docker and you do not want to stop it and rebuild an image (if you just want to install a package or run some commands or even start a new application), you can connect the container from your local machine and run command line commands.
docker exec -it [CONTAINER_ID] bash
To get [CONTAINER_ID], run
docker ps
Note docker-compose ps will give you containers names, but you need container id to ssh the container.

Resources