Storing secret env variables as separate data container in Docker - docker

I want to use 'secret' (passwords etc) env variables from host OS inside docker-compose.
I thought it is possible to achieve this using data container. But it is not working - 'No file present'. Please advice what is wrong and if this doable this way.
docker-compose.yml
version: '2'
services:
web:
build: .
command: bundle exec puma
env_file: .env
environment:
- RACK_ENV=production
- RAILS_ENV=production
volumes_from:
- config
ports:
- "3000:3000"
links:
- db
config:
image: busybox
volumes:
- /myapp/config/.env:.env
I also tried to use /myapp/config:/config as volume and /config/.env as env_file but with same result.
Dockerfile doesnt mention or reference 'config' or '.env' in any way.
Thanks for responses.

Compose env file
The env_file file is read locally by docker-compose, in the directory you are running it from. The variables are then passed to the containers at run time (like docker run -e).
web:
build: .
command: bundle exec puma
env_file: ./.env
environment:
- RACK_ENV=production
- RAILS_ENV=production
ports:
- "3000:3000"
links:
- db
Substitution
Compose can also substitute environment variables into the config
If you have SECRET_HOST_ENV_VARIABLE set on your host when you run docker-compose with an environment definition like below
web:
build: .
command: bundle exec puma
environment:
- RACK_ENV=production
- RAILS_ENV=production
- SECRET_ENV_VARIABLE=${SECRET_HOST_ENV_VARIABLE}
ports:
- "3000:3000"
links:
- db
SECRET_ENV_VARIABLE will become available in the containers environment.

Related

Issue in docker compose with volume undefined

I get the below error when I run docker-compose up, any pointers why I am getting this error
service "mysqldb-docker" refers to undefined volume mysqldb: invalid compose project
Also, is there a way to pass the $ENV value in CLI to docker-compose up , currently I have a ENV variable that specified dev, uat or prod that I use to specify the db name. Are there better alternatives to do this other than create a .env file explicitly for this
version: '3.8'
services:
mysqldb-docker:
image: '8.0.27'
restart: 'unless-stopped'
ports:
- "3309:3306"
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_PASSWORD=root
- MYSQL_DATABASE=reco-tracker-$ENV
volumes:
- mysqldb:/var/lib/mysql
reco-tracker-docker:
image: 'reco-tracker-docker:v1'
ports:
- "8083:8083"
environment:
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=root
- SPRING_DATASOURCE_URL="jdbc:mysql://mysqldb-docker:3309/reco-tracker-$ENV"
depends_on: [mysqldb-docker]
You must define volumes at the top level like this:
version: '3.8'
services:
mysqldb-docker:
# ...
volumes:
- mysqldb:/var/lib/mysql
volumes:
mysqldb:
You can pass environment variables from your shell straight through to a service’s containers with the ‘environment’ key by not giving them a value
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers
web:
environment:
- ENV
but from my tests you cant write $ENV in the compose file and expect it to read your env
for this you need to call docker-compose that way :
docker-compose run -e ENV web python console.py
see this : https://docs.docker.com/compose/environment-variables/#set-environment-variables-with-docker-compose-run

Push Rails 6 application to Docker Hub

I'm trying to get my head around docker. I got it working with my Rails 6 application, it builds and runs succesful. Now I want to push the application into my docker hub repository.
I'm not quite sure how to do this, because I got 3 containers but in every tutorial I read the people just push one.
That's the output of docker ps -a:
That's my docker-compose.yml:
version: '3'
services:
db:
image: postgres
volumes:
- postgres:/var/lib/postgresql/data
env_file:
- .env
ports:
- "5432"
web:
build: .
command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
volumes:
- .:/webmenue
- bundler_gems:/bundle
- ./docker/database.yml:/webmenue/config/database.yml
- cache:/cache
ports:
- "3000:3000"
env_file:
- .env
environment:
- RAILS_ENV=development
- WEBPACKER_DEV_SERVER_HOST=webpacker
- SPROCKETS_CACHE=/cache
depends_on:
- db
webpacker:
build: .
command: ./bin/webpack-dev-server
volumes:
- .:/webmenue
ports:
- '3035:3035'
environment:
- NODE_ENV=development
- RAILS_ENV=development
- WEBPACKER_DEV_SERVER_HOST=0.0.0.0
volumes:
postgres:
bundler_gems:
cache:
I read about the --link flag, but this seems to be deprecated.
So: Do I need to push all containers or is there a way to get them into one container?
That is easy in short as simple as: if you have multiple images, you need to push multiple times ;)
Now the helpful answer, you have 2 images: postgress and your current working directory. Postgress is already an image you download from docker hub, so no need to push it.
As for your other 2 apps, they are currently in one docker file, so only one push is needed.
In your compose file they are both using the build: ., therefore they share the docker image. Let's say you pushed it with: docker push max-kirsch/my-cool-app:1.0.0, you would change the docker compose file to look like this:
version: '3'
services:
db:
image: postgres
volumes:
- postgres:/var/lib/postgresql/data
env_file:
- .env
ports:
- "5432"
web:
image: max-kirsch/my-cool-app:1.0.0
command: bash -c "rm -f tmp/pids/server.pid && bundle exec rails s -p 3000 -b '0.0.0.0'"
volumes:
- .:/webmenue
- bundler_gems:/bundle
- ./docker/database.yml:/webmenue/config/database.yml
- cache:/cache
ports:
- "3000:3000"
env_file:
- .env
environment:
- RAILS_ENV=development
- WEBPACKER_DEV_SERVER_HOST=webpacker
- SPROCKETS_CACHE=/cache
depends_on:
- db
webpacker:
image: max-kirsch/my-cool-app:1.0.0
command: ./bin/webpack-dev-server
volumes:
- .:/webmenue
ports:
- '3035:3035'
environment:
- NODE_ENV=development
- RAILS_ENV=development
- WEBPACKER_DEV_SERVER_HOST=0.0.0.0
volumes:
postgres:
bundler_gems:
cache:
I think you are confused because you have 2 apps in your compose file that shares the same docker file. And it might be that this confusing is correct. In my personal opinion, you should create them both there ow docker file where only the components are installed that you need. It is hard to be sure without seeing the docker file, but as an example, it does not look like your webpacker needs ruby installed and can start with a simple node image as the base instead of installing node in your ruby image.

Database not persisting in docker

I know I am missing something very basic here. I have see some of the older questions on persisting data using docker, but I think I am following the most recent documentation found here.
I have a rails app that I am trying to run in docker. It runs fine but every time I start it up i get ActiveRecord::NoDatabaseError. After I create the database and migrate it, the app runs fine, until I shut it down and restart it.
here is my docker file:
FROM ruby:2.3.0
RUN apt-get update -qq && apt-get install -y build-essential libpq-dev nodejs
ENV RAILS_ROOT /ourlatitude
RUN mkdir -p $RAILS_ROOT/tmp/pids
WORKDIR $RAILS_ROOT
COPY Gemfile Gemfile
COPY Gemfile.lock Gemfile.lock
RUN gem install bundler
RUN bundle install
COPY . .
and here is my docker-compose.yml file
version: '2'
services:
db:
image: postgres:9.4.5
app:
build: .
environment:
RAILS_ENV: $RAILS_ENV
ports:
- "3000:3000"
command: bundle exec rails s -b 0.0.0.0
volumes:
- .:/ourlatitude/database
depends_on:
- db
the basic flow I am following is this:
export RAILS_ENV=development
docker-compose build
docker-compose up
docker-compose run app rake db:create
docker-compose run app rake db:migrate
at this point the app will be running fine
but then I do this
docker-compose down
docker-compose up
and then I am back to the ActiveRecord::NoDatabaseError
So as I said, I think I am missing something very basic.
It doesn't look like you put your postgres on a volume, you may be missing other persistent data sources in your app container, and it appears you missed some indentation on your app container definition.
version: '2'
services:
db:
image: postgres:9.4.5
volumes:
- postgres-data:/var/lib/postgresql/data
app:
build: .
environment:
RAILS_ENV: $RAILS_ENV
ports:
- "3000:3000"
command: bundle exec rails s -b 0.0.0.0
volumes:
- .:/ourlatitude/database
depends_on:
- db
volumes:
postgres-data:
driver: local
In the example above, the postgres data is stored in a named volume. See the advice on docker hub for more details on persisting data for that application. If you are still losing data, check the output of docker diff $container_id on a container to see what files are changing outside of your volumes that would be lost on a down/up.
I managed to get this to work properly using the following docker-compose.yml file.
version: '2'
volumes:
postgres-data:
services:
db:
image: postgres:9.4.5
volumes:
- postgres-data:/var/lib/postgresql/data
app:
build: .
environment:
RAILS_ENV: $RAILS_ENV
ports:
- "3000:3000"
command: bundle exec rails s -b 0.0.0.0
depends_on:
- db
The key was to add the
volumes:
postgres-data:
which creates the named volume and then the
volumes:
- postgres-data:/var/lib/postgresql/data
under the db section which maps the named volume to the expected location in the container of /var/lib/postgresql/data

Adding sphinx container docker-compose shows an error

i have an Ruby on Rails project, which i want to place into the containers( there are database, redis and web(includes rails project) containers). I want to add search feature, so i added a sphinx container in my compose file
docker-compose.yml
web:
dockerfile: Dockerfile-rails
build: .
command: bundle exec rails s -p 3000 -b '0.0.0.0'
ports:
- "3000:3000"
links:
- redis
- db
**- sphinx**
environment:
- REDISTOGO_URL=redis://user#redis:6379/
redis:
image: redis
**sphinx:
image: centurylink/sphinx**
db:
dockerfile: Dockerfile-db
build: .
env_file: .env_db
docker-compose build works fine but when i run docke-compose up i get
ERROR: Cannot start container 096410dafc86666dcf1ffd5f60ecc858760fb7a2b8f2352750f615957072d961: Cannot link to a non running container: /metartaf_sphinx_1 AS /metartaf_web_1/sphinx_1
How can i fix this ?
According to https://hub.docker.com/r/centurylink/sphinx/ the Sphinx container runs needs some amount of configuration files to run properly. See the *Daemonized usage (2). You need data source files and a configuration.
In my test, it fails to start as is with error:
FATAL: no readable config file (looked in /usr/local/etc/sphinx.conf, ./sphinx.conf)
Your docker-compose.yml shouldn't have these * in it.
If you want sphinx latest version you can do this:
web:
dockerfile: Dockerfile-rails
build: .
command: bundle exec rails s -p 3000 -b '0.0.0.0'
ports:
- "3000:3000"
links:
- redis
- db
- sphinx
environment:
- REDISTOGO_URL=redis://user#redis:6379/
redis:
image: redis
sphinx:
image: centurylink/sphinx:latest
db:
dockerfile: Dockerfile-db
build: .
env_file: .env_db
If you want a specific version you write this way : centurylink/sphinx:2.1.8

Docker container is not able to populate SECRET_TOKEN env variable for rails app

I am following https://semaphoreci.com/community/tutorials/dockerizing-a-ruby-on-rails-application to create a sample rails app from rails docker image. Idea is to dockerize a rail application. I have creates a .drkiq.env file in rails app root directory in docker's recommended format of KEY=value as given below
SECRET_TOKEN=asecuretokenwouldnormallygohere
WORKER_PROCESSES=1
LISTEN_ON=0.0.0.0:8000
DATABASE_URL=postgresql://drkiq:yourpassword#postgres:5432/drkiq?encoding=utf8&pool=5&timeout=5000
CACHE_URL=redis://redis:6379/0
JOB_WORKER_URL=redis://redis:6379/0
I am reading the environment file from my docker=compose.yml file (also residing in app root directory)
postgres:
image: postgres:9.4.5
environment:
POSTGRES_USER: drkiq
POSTGRES_PASSWORD: yourpassword
ports:
- '5432:5432'
volumes:
- drkiq-postgres:/var/lib/postgresql/data
redis:
image: redis:3.0.5
ports:
- '6379:6379'
volumes:
- drkiq-redis:/var/lib/redis/data
drkiq:
build: .
links:
- postgres
- redis
volumes:
- .:/drkiq
ports:
- '8000:8000'
env_file:
- .drkiq.env
sidekiq:
build: .
command: bundle exec sidekiq -C config/sidekiq.yml
links:
- postgres
- redis
volumes:
- .:/drkiq
env_file:
- .drkiq.env
Inside my Dockerfile (residing in app root directory), I am running unicorn server
CMD bundle exec unicorn -c config/unicorn.rb
But when I run command
docker-compose up
and access http://my-host:8000/ It gives me "RuntimeError at /
Missing secret_token and secret_key_base for 'development' environment, set these values in config/secrets.yml" error. I am not sure what I am missing here.
My bad. in my rails app I was actually looking for SECRET_KEY_BASE variable but in .drkiq.env file (as pasted above) I was setting SECRET_TOKEN. I replaced SECRET_TOKEN with SECRET_KEY_BASE and restarted docker and every thing was shiny and warm.

Resources