I have a spring boot application I want to test via .gitlab-ci.yml.
It's set up already like this:
image: openjdk:12
# services:
# - docker:dind
stages:
- build
before_script:
# - apk add --update python py-pip python-dev && pip install docker-compose
# - docker version
# - docker-compose version
- chmod +x mvnw
build:
stage: build
script:
# - docker-compose up -d
- ./mvnw package
artifacts:
paths:
- target/rest-SNAPSHOT.jar
The commented out portions are from the answer to Run docker-compose build in .gitlab-ci.yml which I noticed has a fully distinct docker image.
Obviously I need java installed to run my spring boot application, so does that mean docker is just not an option?
Related
I'm trying to get GitLab CI runner to build my project off the Docker image and install NPM package during the build. My .gitlab-ci.yml file was inspired by this topic Gitlab CI with Docker and NPM where the PO was dealing with identical problem:
image: docker:stable
services:
- docker:dind
stages:
- build
cache:
paths:
- node_modules/
before_script:
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
compile:
image: node:8
stage: build
script:
- apk add --no-cache py-pip python-dev libffi-dev openssl-dev gcc libc-dev make
- pip install docker-compose
- docker-compose up -d
- docker-compose exec -T users python manage.py recreate_db
- docker-compose exec -T users python manage.py seed_db
- npm install
- bash test.sh
after_script:
- docker-compose down
Sadly, that solution didn't work well but I feel like I'm little bit closer to actual solution now. I'm getting two errors during the build:
/bin/bash: line 89: apk: command not found
Running after script...
$ docker-compose down
/bin/bash: line 88: docker-compose: command not found
How can I troubleshoot this ?
Edit:
image: docker:stable
services:
- docker:dind
stages:
- build
- test
before_script:
- export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
compile:
stage: build
script:
- apk add --no-cache py-pip python-dev libffi-dev openssl-dev gcc libc-dev make
- pip install docker-compose
- docker-compose up -d
- docker-compose exec -T users python manage.py recreate_db
- docker-compose exec -T users python manage.py seed_db
testing:
image: node:alpine
stage: test
script:
- npm install
- bash test.sh
after_script:
- docker-compose down
I moved tests into separate stage testing which I should've done anyway and I figured I'd defined the image there to separate it from the build stage. No change. Docker can't be found and bash test also can't be ran:
$ bash test.sh
/bin/sh: eval: line 87: bash: not found
Running after script...
$ docker-compose down
/bin/sh: eval: line 84: docker-compose: not found
image: node:8 this image is not based on alpine so as a result, you got error
apk: command not found
node:<version>
These are the suite code names for releases of Debian and indicate
which release the image is based on. If your image needs to install
any additional packages beyond what comes with the image, you'll
likely want to specify one of these explicitly to minimize breakage
when there are new releases of Debian.
just replace image to
node:alpine
and it should work.
the second error is because docker-compose is not installed.
You can check this answer for more details about composer.
I would like to configure a CI such as TravisCI to build my application from Docker. My application has two part: Javascript and Python.
I thought to use docker-compose to do this:
version: '3'
services:
node:
image: node:12.8.0-buster
volumes:
- .:/srv
python:
image: python:3.7.4-buster
volumes:
- .:/src
I would like to have a Makefile such as:
all: foo bar
foo:
docker-compose exec node /bin/bash -c ' \
cd /workdir; \
npm install; \
npm run build'
bar:
docker-compose exec python /bin/bash -c ' \
cd /workdir; \
pip install sphinx; \
make html'
Is this correct to use docker compose like this? And what should I change to make it work?
docker compose not only support container run, but also image build, see this.
So, for your scenario, you should add your package build in Dockerfile and exeucte it with docker-compose up -d --build which will first build out a docker image then start the service base on the new docker image.
A simple fake code is as next, note next is just to explain the main idea, not a fully workable example, you need to add your stuff base on your real situation.
Dockerfile.node:
FROM node:12.8.0-buster
# Add related to build
ADD . /srv
# Add all package install
RUN cd /workdir && npm install && npm run build
# Others
......
Dockerfile.python:
FROM python:3.7.4-buster
# Add related to build
ADD . /srv
# Add all package install
RUN cd /workdir && pip install sphinx && make html
# Others
......
docker-compose.yaml:
version: '3'
services:
node:
build:
context: .
dockerfile: Dockerfile.node
volumes:
- .:/srv
python:
build:
context: .
dockerfile: Dockerfile.python
volumes:
- .:/src
I have a Dockerfile for a Django and Vue.js app that I use along with Gitlab.
The problem that I'm about to describe only happens when deploying via Gitlab CI and the corresponding .gitlab-ci.yml file. When running the docker-compose up command in my local machine, this doesn happen.
So I run docker-compose up and all the instructions in the Dockerfile run apparently fine. But when I check the production server, the dist folder (where the bundle.js and bundle.css should be stored) doesn't exist.
The logs that are spit out while running the Dockerfile confirm that the npm install and npm run build commands are run, and it even confirms that the dist/bundle.js and dist/bundle.css files have been generated. But for some reason they seem to be deleted.
This is my Dockerfile:
FROM python:3.7-alpine
MAINTAINER My Name
ENV PYTHONUNBUFFERED 1
RUN mkdir /app
# make the 'app' folder the current working directory
WORKDIR /app
# copy project files and folders to the current working directory (i.e. 'app' folder)
COPY ./app .
COPY ./requirements.txt /requirements.txt
RUN apk add --update --no-cache postgresql-client
RUN apk add --update --no-cache --virtual .tmp-build-deps \
gcc libc-dev linux-headers postgresql-dev
RUN pip install -r /requirements.txt
RUN apk del .tmp-build-deps
# copy both 'package.json' and 'package-lock.json' (if available)
COPY app/frontend/package*.json ./frontend/
# Install npm
RUN apk add --update nodejs && apk add --update nodejs-npm
# install project dependencies
WORKDIR /app/frontend
RUN npm install
# build app for production with minification
RUN npm run build
RUN adduser -D user
USER user
CMD ["sh ../scripts/entrypoint.sh"]
This is the .gitlab-ci.yml file:
image: docker:latest
services:
- docker:dind
before_script:
- echo "Runnig before_script"
- sudo apt-get install -y python-pip
- sudo apt-get install -y nodejs
- pip install docker-compose
stages:
- test
- build
- deploy
test:
stage: test
script:
- echo "Testing the app"
- docker-compose run app sh -c "python /app/manage.py test && flake8"
build:
stage: build
only:
- develop
- production
- feature/gitlab_ci
script:
- echo "Building the app"
- docker-compose build
deploy:
stage: deploy
only:
- master
- develop
- feature/gitlab_ci
script:
- echo "Deploying the app"
- docker-compose up --build -d
This is the content of the docker-compose.yml file:
version: "3"
services:
app:
build:
context: .
ports:
- "8000:8000"
volumes:
- ./app:/app
command: >
sh -c "python /app/manage.py runserver 0.0.0.0:8000"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=postgres
depends_on:
- db
db:
image: postgres:10-alpine
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
This is the content of the entrypoint.sh file:
#!/bin/bash
(cd .. && ./manage.py collectstatic --noinput)
# Migration files are commited to git. Makemigrations is not needed.
# ./manage.py makemigrations app_name
(cd .. && ./manage.py migrate)
I would like to know why the dist/ folder disappears and how to keep it.
When your docker-compose.yml file says
volumes:
- ./app:/app
that hides everything that your Dockerfile builds in the /app directory and replaces it with whatever's in your local system. If your host doesn't have a ./app/frontend/dist then your container won't have that path either, regardless of whatever the Dockerfile does.
I would generally recommend just deleting this volumes: block entirely. It introduces an awkward live-development path (where all of your tooling needs to know that the actual service runs in Docker) and simultaneously isn't what you'd run in development (you want the image to be self-contained and not to need to copy the application separately from the image).
In your compose file, you set a volume which is going to replace your local environment with the one in your container even after npm run build
volumes:
- ./app:/app
You can either build it in your local or remove volumes
We had a similar issue with a nestjs build. Lately we noticed, that we had excluded the src file in the .dockerignore.
Issue is not with docker file. It issue with your dependency. please check package.json file in root folder.
I use gitlab ci to build docker image and I want to install python. When I build, the following is my gitlab-ci.yml:
image: docker:stable
stages:
- test
- build
before-script:
- apt install -y python-dev python pip
test1:
stage: test
script:
...
- pytest
build:
stage: build
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
but i got a Job failed
/bin/sh: eval: line : apt: not found
ERROR: Job failed: exit code 127
I also tried to apt-get install but the result is the same.
How do I install python??
It's actually not a problem but you can say it featured by package-manager with Alpine you are using image: docker:stable or any such images like tomcat or Django they are on Alpine Linux. with minimal in the size .
image: docker:stable
stages:
- test
- build
before-script:
- apk add python python-dev python pip
test1:
stage: test
script:
...
- pytest
build:
stage: build
- docker build -t $IMAGE_TAG .
- docker push $IMAGE_TAG
apk is Alpine Linux package management
The image that you are using docker: stable is based on Alpine Linux which uses apk as its package manager. installation with apk will look like that: apk add python
The error you see is because apt doesn’t exist in alpine docker.
This line solved the problem for me:
apk update && apk add python
My Docker container builds fine on OSX:
Docker version 17.12.0-ce, build c97c6d6
docker-compose version 1.18.0, build 8dd22a9
But doesn't build on Amazon Linux:
Docker version 17.12.0-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
docker-compose version 1.20.1, build 5d8c71b
Full Dockerfile:
# Specify base image
FROM andreptb/oracle-java:8-alpine
# Specify author / maintainer
MAINTAINER Douglas Duhaime <douglas.duhaime#gmail.com>
# Add source to a directory and use that directory
# NB: /app is a reserved directory in tomcat container
ENV APP_PATH="/lts-app"
RUN mkdir "$APP_PATH"
ADD . "$APP_PATH"
WORKDIR "$APP_PATH"
##
# Build BlackLab
##
RUN apk add --update --no-cache \
wget \
tar \
git
# Store the path to the maven home
ENV MAVEN_HOME="/usr/lib/maven"
# Add maven and java to the path
ENV PATH="$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH"
# Install Maven
RUN MAVEN_VERSION="3.3.9" && \
cd "/tmp" && \
wget "http://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz" -O - | tar xzf - && \
mv "/tmp/apache-maven-$MAVEN_VERSION" "$MAVEN_HOME" && \
ln -s "$MAVEN_HOME/bin/mvn" "/usr/bin/mvn" && \
rm -rf "/tmp/*"
# Get the BlackLab source
RUN git clone "git://github.com/INL/BlackLab.git"
# Build BlackLab with Maven
RUN cd "BlackLab" && \
mvn clean install
##
# Build Python + Node dependencies
##
# Install system deps with Alpine Linux package manager
RUN apk add --update --no-cache \
g++ \
gcc \
make \
openssl-dev \
python3-dev \
python \
py-pip \
nodejs
# Install Python dependencies
RUN pip install -r "requirements.txt" && \
npm install --no-optional && \
npm run build
# Store Mongo service name as mongo host
ENV MONGO_HOST=mongo_service
ENV TOMCAT_HOST=tomcat_service
ENV TOMCAT_WEBAPPS=/tomcat_webapps/
# Make ports available
EXPOSE 7082
# Seed the db
CMD npm run seed && \
gunicorn -b 0.0.0.0:7082 --access-logfile - --reload server.app:app
Full docker-compose.yml
version: '2'
services:
tomcat_service:
image: 'bitnami/tomcat:latest'
ports:
- '8080:8080'
volumes:
- docker-data-tomcat:/bitnami/tomcat/data/
- docker-data-blacklab:/lts-app/lts/
mongo_service:
image: 'mongo'
command: mongod
ports:
- '27017:27017'
web:
# gain access to linked containers
links:
- mongo_service
- tomcat_service
# explicitly declare service dependencies
depends_on:
- mongo_service
- tomcat_service
# set environment variables
environment:
PYTHONUNBUFFERED: 'true'
# use the image from the Dockerfile in the cwd
build: .
ports:
- '7082:7082'
volumes:
- docker-data-tomcat:/tomcat_webapps
- docker-data-blacklab:/lts-app/lts/
volumes:
docker-data-tomcat:
docker-data-blacklab:
The command I'm running is: docker-compose up --build
The result on Amazon Linux is:
Running setup.py install for pymongo: started
Running setup.py install for pymongo: finished with status 'done'
Running setup.py install for pluggy: started
Running setup.py install for pluggy: finished with status 'done'
Running setup.py install for coverage: started
Running setup.py install for coverage: finished with status 'done'
Successfully installed Faker-0.8.12 Flask-0.12.2 Flask-Cors-3.0.3 Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 astroid-1.6.2 attrs-17.4.0 backports.functools-lru-cache-1.5 beautifulsoup4-4.5.1 click-6.7 configparser-3.5.0 coverage-4.5.1 enum34-1.1.6 funcsigs-1.0.2 futures-3.2.0 gunicorn-19.7.1 ipaddress-1.0.19 isort-4.3.4 itsdangerous-0.24 lazy-object-proxy-1.3.1 mccabe-0.6.1 more-itertools-4.1.0 pluggy-0.6.0 py-1.5.3 py4j-0.10.6 pylint-1.8.3 pymongo-3.6.1 pytest-3.5.0 pytest-cov-2.5.1 python-dateutil-2.7.2 singledispatch-3.4.0.3 six-1.11.0 text-unidecode-1.2 wrapt-1.10.11
You are using pip version 8.1.2, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
npm WARN deprecated redux-mock-store#1.5.1: breaking changes in minor version
> base62#1.2.7 postinstall /lts-app/node_modules/base62
> node scripts/install-stats.js || exit 0
ERROR: Service 'web' failed to build: The command '/bin/sh -c pip install -r "requirements.txt" && npm install --no-optional && npm run build' returned a non-zero code: 1
Does anyone know what might be causing this discrepancy? The error message from Docker doesn't give many clues. I'd be very grateful for any ideas others can offer!
To solve this problem, I followed #MazelTov's advice and built the containers on my local OSX development machine, then published the images to Docker Cloud, then pulled those images down onto and ran the images from my production server (AWS EC2).
Install Dependencies
I'll try and outline the steps I followed below in case they help others. Please note these steps require you to have docker and docker-compose installed on your development and production machines. I used the gui installer to install Docker for Mac.
Build Images
After writing a Dockerfile and docker-compose.yml file, you can build your images with docker-compose up --build.
Upload Images to Docker Cloud
Once the images are built, you can upload them to Docker Cloud with the following steps. First, create an account on Docker Cloud.
Then store your Docker Cloud username in an environment variable (so your ~/.bash_profile should contain export DOCKER_ID_USER='yaledhlab' (use your username though).
Next login to your account from your developer machine:
docker login
Once you're logged in, list your docker images:
docker ps
This will display something like:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
89478c386661 yaledhlab/let-them-speak-web "/bin/sh -c 'npm run…" About an hour ago Up About an hour 0.0.0.0:7082->7082/tcp letthemspeak_web_1
5e9c75d29051 training/webapp:latest "python app.py" 4 hours ago Up 4 hours 0.0.0.0:5000->5000/tcp heuristic_mirzakhani
890f7f1dc777 bitnami/tomcat:latest "/app-entrypoint.sh …" 4 hours ago Up About an hour 0.0.0.0:8080->8080/tcp letthemspeak_tomcat_service_1
09d74e36584d mongo "docker-entrypoint.s…" 4 hours ago Up About an hour 0.0.0.0:27017->27017/tcp letthemspeak_mongo_service_1
For each of the images you want to publish to Docker Cloud, run:
docker tag image_name $DOCKER_ID_USER/my-uploaded-image-name
docker push $DOCKER_ID_USER/my-uploaded-image-name
For example, to upload mywebapp_web to your user's account on Docker cloud, you can run:
docker tag mywebapp_web $DOCKER_ID_USER/web
docker push $DOCKER_ID_USER/web
You can then run open https://cloud.docker.com/swarm/$DOCKER_ID_USER/repository/list to see your uploaded images.
Deploy Images
Finally, you can deploy your images on EC2 with the following steps. First, install Docker and Docker-Compose on the Amazon-flavored EC2 instance:
# install docker
sudo yum install docker -y
# start docker
sudo service docker start
# allow ec2-user to run docker
sudo usermod -a -G docker ec2-user
# get the docker-compose binaries
sudo curl -L https://github.com/docker/compose/releases/download/1.20.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
# change the permissions on the source
sudo chmod +x /usr/local/bin/docker-compose
Log out, then log back in to update your user's groups. Then start a screen and run the server: screen. Once the screen starts, you should be able to add a new docker-compose config file that specifies the path to your deployed images. For example, I needed to fetch the let-them-speak-web container housed within yaledhlab's Docker Cloud account, so I changed the docker-compose.yml file above to the file below, which I named production.yml:
version: '2'
services:
tomcat_service:
image: 'bitnami/tomcat:latest'
ports:
- '8080:8080'
volumes:
- docker-data-tomcat:/bitnami/tomcat/data/
- docker-data-blacklab:/lts-app/lts/
mongo_service:
image: 'mongo'
command: mongod
ports:
- '27017:27017'
web:
image: 'yaledhlab/let-them-speak-web'
# gain access to linked containers
links:
- mongo_service
- tomcat_service
# explicitly declare service dependencies
depends_on:
- mongo_service
- tomcat_service
# set environment variables
environment:
PYTHONUNBUFFERED: 'true'
ports:
- '7082:7082'
volumes:
- docker-data-tomcat:/tomcat_webapps
- docker-data-blacklab:/lts-app/lts/
volumes:
docker-data-tomcat:
docker-data-blacklab:
Then the production compose file can be run with: docker-compose -f production.yml up. Finally, ssh in with another terminal, and detach the screen with screen -D.