Tl;dr I'm using Docker to run my Postman/Newman tests and my API tests hang when ran in Travis-CI but not when ran locally. Why am I encountering tests that run infinitely?
Howdy guys! I've recently started to learn Docker, Travis-CI and Newman for a full stack application. I started with developing the API and I'm taking a TDD approach. As such, I'm testing my API first. I setup my .travis.yml file to download a specific version of Docker-Compose and then use Docker-Compose to run my tests in a container I name api-test. The container has an image, dannydainton/htmlextra, which is built from the official postman/newman:alpine image like so:
language: node_js
node_js:
- "14.3.0"
env:
global:
- DOCKER_COMPOSE_VERSION: 1.26.2
- PGHOST: db
- PGDATABASE: battle_academia
- secure: "xDZHJ9ZVe3WPXr6WetERMjFnTlMowyEoeckzLcRvWyEIe2qbnWrJRo7cIRxA0FsyJ7ao4QLVv4XhOIeqJupwW3nfnljo35WGcuRBLh76CW6JSuTIbpV1dndOpATW+aY3r6GSwpojnN4/yUVS53pvIeIn03PzQWmnbiJ0xfjStrJzYNpSVIHLn0arujDUMyze8+4ptS1qfekOy2KRifG5+viFarUbWUXaUiJfZCn14S4Wy5N/T+ycltNjX/qPAVZYV3fxY1ZyNX7wzJA+oV71MyApp5PgNW2SBlePkeZTnkbI7FW100MUnE4bvy00Jr/aCoWZYTySz86KT+8HSGzy6d+THO8zjOXKJV5Vn93+XWmxtp/yjBsg+dtFlZUWkN99EBkEjjwJc1Oy5zrOQNjsptNGpl1kid5+bAT4XcP4xn7X5pc7QB8ZE3igbfKTM11LABYN1adcIwgGIjUz1eQnFuibtkVM4oqE92JShUF/6gbwGJsWjQGBNBCOBBueYNB86sk0TiAfS08z2VW9L3pcljA2IwdXclw3f1ON6YelBTJmc88EmxI4TS0hRC5KgMCkegW1ndcTZwqIQGFm+NFbe1hKMmqTfgOg5M8OQZBtUkF60Lox09ECg59IrYj+BIa9J303+bo+IMgZ1JVYlL7FA2qc0bE8J/9A1C2wCRjDLLE="
- secure: "F/Ru7QZvA+zWjQ7K7vhA3M2ZrYKXVIlkIF1H7v2dPv/lsc18eWGpOQep4uAjX4IMyLY/6n7uYRLnSlbvOWulVUW8U52zWiQkYFF9OwosuTdIlVTAQGp3B0CAA+RCxMtDQay6fN9H6e2bL3KwjT//VUHd1E6BPu+O1/RyX+0+0KvTmExmMSuioSpDPcI20Mym2vRCgNPb1gfajr5QfWKPJlrPjfyNhDxWMhM94nwTuLYIVZwZPTZ0Ro5D6hhXFVZOFIlHr5VDbbFa+Xo0TIdP/ZudxZ7p3Mn7ncA8seLx2Q5/zH6tJ4DSUpEm67l5IqUrvd9qp0CNCjlTcl3kOJK4qIB1WtLm6oW2rBqDyvthhuprPpqEcs7C9z2604VLybdOmJ0+Y/7uIo6po388avGN4ZwZbWQ1xiiW+Ja8kkHZYEKo4m0AbKdX9pn8otcNO+1xlDtUU7CZey2QA8WrFlfHWqRapIgNfT5tTSTAul3yWAFCRw09PHYELuO7oQCqFZi7zu3HKWknbkzjf+Cz3TfIFTX/3saiqyquhieOPbnGC5xgTmTrA2ShfNxQ6nkDJPU0/qmaCNJt9CwpNS2ArqcK3xYijiNi+SHaKwEsYh0VqiUqSCWn05eYKNAe3MUQDsyKFEkykJW60yEkN7JsvO1WpI53VKmOnZlRHLzJyc5WkZw="
- PGPORT: 5432
services:
- docker
before_install:
- npm rebuild
- sudo rm /usr/local/bin/docker-compose
- curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- sudo mv docker-compose /usr/local/bin
jobs:
include:
- stage: api tests
script:
- docker --version
- docker-compose --version
- >
docker-compose run api-test
newman run battle-academia_placement-exam_api-test.postman-collection.json
-e battle-academia_placement-exam_docker.postman-environment.json
-r htmlextra,cli
And, my docker-compose.yml file has 4 containers:
client is the React front end,
api is the NodeJs/Express back end,
db is the database that the API pulls data from in the test environment,
api-test is the container with Newman/Postman and some reporters which I believe is built from NodeJs.
I hardcode in the environment variables when running locally, but the file is as follows:
version: '3.8'
services:
client:
build: ./client
ports:
- "80:80"
depends_on:
- api
api:
build: ./server
environment:
- PGHOST=${PGHOST}
- PGDATABASE=${PGDATABASE}
- PGUSER=${PGUSER}
- PGPASSWORD=${PGPASSWORD}
- PGPORT=${PGPORT}
ports:
- "3000:3000"
depends_on:
- db
db:
image: postgres:12.3-alpine
restart: always
environment:
- POSTGRES_DB=${PGDATABASE}
- POSTGRES_USER=${PGUSER}
- POSTGRES_PASSWORD=${PGPASSWORD}
ports:
- "5432:5432"
volumes:
- ./server/db/scripts:/docker-entrypoint-initdb.d
api-test:
image: dannydainton/htmlextra
entrypoint: [""]
command: newman run -v
volumes:
- ./server/api/postman-collections:/etc/newman
depends_on:
- api
Now that the setup is out of the way, my issue is that this config works locally when I cut out .travis.yml and run the commands myself, however, putting Travis-CI in the mix stirs up an issue where my first test just... runs.
I appreciate any advice or insight towards this issue that anyone provides. Thanks in advance!
The issue did not come from where I had expected. After debugging, I thought that the issue originally came from permission errors since I discovered that the /docker-entrypoint-initdb.d directory got ignored during container startup. After looking at the Postgres Dockerfile, I learned that the files are given permission for a user called postgres. The actual issue stemmed from me foolishly adding the database initialization scripts to my .gitignore.
Edit
Also the Newman tests were hanging because they were trying to access database tables that did not exist.
Related
I want Travis CI to build my app for test when I push my app to GitHub.
I think there is cooperation between Travis CI and GitHub.
But it didn't work.
docker-compose.yml here.
version: '3'
volumes:
db-data:
services:
web:
build: .
ports:
- '3000:3000'
volumes:
- '.:/product-register'
environment:
- 'DATABASE_PASSWORD=postgres'
tty: true
stdin_open: true
depends_on:
- db
links:
- db
db:
image: postgres
volumes:
- 'db-data:/var/lib/postgresql/data'
environment:
- 'POSTGRES_HOST_AUTH_METHOD=trust'
- 'POSTGRES_USER=postgres'
- 'POSTGRES_PASSWORD=postgres'
.travis.yml here.
sudo: required
services: docker
before_install:
- docker login -u polymetisoutis -p 5fb47200-dd19-4772-a9ad-c98913ef1cb9
- docker-compose up --build -d
script:
- docker-compose exec --env 'RAILS_ENV=test' web rails db:create
- docker-compose exec --env 'RAILS_ENV=test' web rails db:migrate
- docker-compose exec --env 'RAILS_ENV=test' web rails test
Then repository I pushed to GitHub here.⇒https://github.com/PolymetisOutis/product-register
After the next command executed,
git push origin master
I think Travis CI should build the app on the travis-ci.com page for test.
But Travis CI didn't work.
Why?
Is there anyone who have an idea and clue about this?
Your Travis CI Build Request Page [1] always gives the idea of why the build wasn't triggered actually. I can see that at a time that you have triggered some requests your account wasn't confirmed and your build requests were rejected, however, now seems like you are able to trigger builds.
[1] https://app.travis-ci.com/github/PolymetisOutis/product-register/requests
I'm currently trying to run our app that's in Grails 2.3.11 through docker-compose with the database. I have the database up and running without issue, and also the app container sets up grails and starts the compilation process, but it goes on to downloading all the packages every time I stop and restart the package. This becomes an issue because we have to download so many packages (And there's a bunch of errors we have to work around because Grails 2). I've tried to mount my local grails folders into the container to have it run off of those but it seems to not be having any success. Is there something obvious I'm doing wrong, or some way I can easily check where the issue might be?
I'm also attempting to map all local database information into the mysql container with issue. But I haven't looked into it much yet, if you see an obvious issue there that would be helpful.
docker-compose.yml:
version: '2'
services:
grails:
image: ibbrussell/grails:2.3.11
command: run-app
volumes:
- ~/.m2:/home/developer/.m2
- ~/.gradle:/home/developer/.gradle
- ~/.grails:/home/developer/.grails
- ./:/app
ports:
- "8080:8080" #Grails default port
- "5005:5005" #Grails debug port
links:
- db
deploy:
resources:
limits:
memory: 4G
reservations:
memory: 4G
db:
image: mysql:5.6
container_name: grails_mysql
ports:
- "3306:3306"
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: 1
MYSQL_DATABASE: grails
volumes:
- "/usr/local/mysql/data:/var/lib/mysql"
Dockerfile:
FROM java:8
# Set customizable env vars defaults.
ENV GRAILS_VERSION 2.3.11
# Install Grails
WORKDIR /usr/lib/jvm
RUN wget https://github.com/grails/grails-core/releases/download/v$GRAILS_VERSION/grails-$GRAILS_VERSION.zip && \
unzip grails-$GRAILS_VERSION.zip && \
rm -rf grails-$GRAILS_VERSION.zip && \
ln -s grails-$GRAILS_VERSION grails
# Setup Grails path.
ENV GRAILS_HOME /usr/lib/jvm/grails
ENV PATH $GRAILS_HOME/bin:$PATH
ENV GRAILS_OPTS="-XX:MaxPermSize=4g -Xms4g -Xmx4g"
# Create App Directory
RUN mkdir /app
# Set Workdir
WORKDIR /app
# Set Default Behavior
ENTRYPOINT ["grails"]
So the mapping I was using ended up not being correct. I was going off a file mapping from 1 article and ended up working after trying another working mapping. I made the switch below:
original:
volumes:
- ~/.m2:/home/developer/.m2
- ~/.gradle:/home/developer/.gradle
- ~/.grails:/home/developer/.grails
- ./:/app
new:
volumes:
- ~/.m2:/root/.m2
- ~/.gradle:/root/.gradle
- ~/.grails:/root/.grails
- ./:/app
I've created a simple Sonatype API client in Elixir that returns the repositories and the components of the repositories.
I now need to create tests in Elixir so that I can verify the repo. I am using docker-compose to start the sonatype container. I need the tests to start with a fresh Docker(sonatype) repo to work with, via docker-compose up, then verify that it doesn't have any containers in it. Then from there add one or more images, then validate that the images I added are present. As cleanup, I could delete those images. It must be an automated set of tests that can run in CI or a user can run on their local machine.
My question is how would I be able to do that by either a .exs test file or bash script file?
You can build a docker-compose.yml file with something similar to this:
version: "2.2"
services:
my_app:
build:
context: .
ports:
- 4000:4000
command: >
bash -c 'wait-for-it -t 60 sonatype:1234
&& _build/prod/rel/my_app/bin/my_app start'
tests:
extends:
service: my_app
environment:
MIX_ENV: test
LOG_LEVEL: "warn"
working_dir: /my_app
depends_on:
- sonatype
command:
bash -c 'mix test'
sonatype:
image: sonatype/nexus3:3.19.1
ports:
- "1234:1234"
Then you have a bash script like test.sh:
docker-compose build tests
docker-compose run tests
EXIT=$?
docker-compose down --volumes
exit $EXIT
I'm not familiar with Sonatype, so this might not make sense, and you need to adapt.
I'm having an issue with my travis-ci before_script while trying to connect to my docker postgres container:
Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
I've seen this problem raised but never fully addressed around SO and github issues, and i'm not clear whether it is specific to docker or travis. One linked issue (below) works around it by using 5433 as the host postgres address but i'd like to know for sure what is going on before i jump into something.
my travis.yml:
sudo: required
services:
- docker
env:
DOCKER_COMPOSE_VERSION: 1.7.1
DOCKER_VERSION: 1.11.1-0~trusty
before_install:
# list docker-engine versions
- apt-cache madison docker-engine
# upgrade docker-engine to specific version
- sudo apt-get -o Dpkg::Options::="--force-confnew" install -y docker-engine=${DOCKER_VERSION}
# upgrade docker-compose
- sudo rm /usr/local/bin/docker-compose
- curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- sudo mv docker-compose /usr/local/bin
before_script:
- echo "Before Script:"
- docker-compose -f docker-compose.ci.yml build
- docker-compose -f docker-compose.ci.yml run app rake db:setup
- docker-compose -f docker-compose.ci.yml run app /bin/sh
script:
- echo "Running Specs:"
- rake spec
my docker-compose.yml for ci:
postgres:
image: postgres:9.4.5
environment:
POSTGRES_USER: web
POSTGRES_PASSWORD: yourpassword
expose:
- '5432' # added this as an attempt to open the port
ports:
- '5432:5432'
volumes:
- web-postgres:/var/lib/postgresql/data
redis:
image: redis:3.0.5
ports:
- '6379:6379'
volumes:
- web-redis:/var/lib/redis/data
web:
build: .
links:
- postgres
- redis
volumes:
- ./code:/app
ports:
- '8000:8000'
# env_file: # setting these directly in the environment
# - .docker.env # (they work fine locally)
sidekiq:
build: .
command: bundle exec sidekiq -C code/config/sidekiq.yml
links:
- postgres
- redis
volumes:
- ./code:/app
Docker & Postgres: Failed to bind tcp 0.0.0.0:5432 address already in use
How to get Docker host IP on Travis CI?
It seems that Postgres service is enabled by default in Travis CI.
So you could :
Try to disable the Postgres service in your Travis config. See How to stop services on Travis CI running by default?. See also https://docs.travis-ci.com/user/database-setup/#PostgreSQL .
Or
Map your postgres container to another host port (!= 5432). Like -p 5455:5432.
It could also be useful to check if the service is already running : Check If a Particular Service Is Running on Ubuntu
Do you use Travis' Postgres?
services:
- postgresql
Would be easier if you provide travis.yml
I'm using docker as my dev environment for my rails app with the following docker-compose.yml :
app:
build: .
ports:
- "3000:3000"
links:
- db
- mail
volumes:
- .:/usr/src/app
- gemrc:/etc/gemrc
db:
image: mdillon/postgis
ports:
- "5432:5432"
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=database
volumes:
- ./docker/pgdata:/var/lib/postgresql/data
mail:
image: djfarrelly/maildev
ports:
- "1080:80"
And my Dockerfile :
FROM rails:onbuild
When I need to add a new gem to my Gemfile, I have to first generate my Gemfile.lock :
docker run --rm -v gemrc:/etc/gemrc -v /home/user/project:/usr/src/app -w /usr/src/app ruby bundle install
And the rebuild the docker image:
docker-compose build
docker-compose up
Because of this I have to run bundle install twice without being able to add the --without development test flag. In order to do it quicker I added this to my gemrc file:
gem: --no-document
But is there a way to avoid the double bundle install ?
Perhaps you might want to try the following docker-compose workflow for development environment.
Similar to database.yml our docker-compose.yml is not included in our CVS (git), providing the similar benefits for developer custom config.
Build your image locally before starting your app container and tag it something like foo_app:latest. It makes sense because you're in dev. Just execute docker build . in your app's root directory, assuming your Dockefile is in that directory.
Define a data volume container for bundle and mount it in your app container. Your docker-compose.yml might look something like:
app:
image: foo_app
ports:
- "3000:3000"
links:
- db
- mail
volumes:
- .:/usr/src/app
volumes_from:
- bundle
bundle:
image: foo_app:latest
volumes:
- /home/app/bundle
db:
image: mdillon/postgis
ports:
- "5432:5432"
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=database
volumes:
- ./docker/pgdata:/var/lib/postgresql/data
mail:
image: djfarrelly/maildev
ports:
- "1080:80"
Every time you need to add a new gem, just add it to your Gemfile and execute bundle install inside your app container. For example if your app container's name is foo_app_1:
docker exec foo_app_1 bundle install
The data volume container will always have the latest/edge snapshot of your app's gems.
Tag your releases and build the "official release image" in a central repository accessible for your staging/production/team.
With this approach every time you start/recreate your app container, all of your gems be just as they were the last time you updated them. You can also use this approach for other kind of data you want to be persisted across containers life cycles, adding "components" to manage state in your stateless applications.
See https://docs.docker.com/engine/userguide/containers/dockervolumes/ for more information