use ssh keyfile as environment variable in docker-compose [duplicate] - docker

I have setup Jenkins within a Docker container and I am trying to access that my private Bitbucket repo with that server. I need to copy my SSH key into that container so that Bitbucket recognizes it and I can have my Jenkins server access the repo then.
I have in my docker-compose.yml file the following:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:
However, echo $SSH_PRIVATE_KEY gives /.ssh/id_rsa literally instead of the value stored inside. I have heard the problem with doing this inside the Dockerfile instead would be that it still can be viewed in one of the layers of the image that will be pushed.
My question is how can I set the value of SSH_PRIVATE_KEY to the value of the contents of my file?
I believe this could be a duplicate of How to set environment variable into docker container using docker-compose however that solution does not appear to change anything for me.

You could create an Environment variable in your shell from which you are running your compose :
export SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
and then use it in your compose like :
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY
ports:
- "8080:8080"
- "50000:50000"
It should pick up the value for your environment variable for container from shell environment as specified in the docs :
The value of the variable in the container is taken from the value for the same variable in the shell in which Compose is run.

Possible solution:
environment:
- SSH_PRIVATE_KEY
and call the docker-compose like this:
SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa) docker-compose build
Unfortunately, it's currently not possible to use multiline variables in .env.
Another possibility would be:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
- "/home/user/.ssh/id_rsa:/home/user/.ssh/id_rsa:ro"
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:

Related

Issue in docker compose with volume undefined

I get the below error when I run docker-compose up, any pointers why I am getting this error
service "mysqldb-docker" refers to undefined volume mysqldb: invalid compose project
Also, is there a way to pass the $ENV value in CLI to docker-compose up , currently I have a ENV variable that specified dev, uat or prod that I use to specify the db name. Are there better alternatives to do this other than create a .env file explicitly for this
version: '3.8'
services:
mysqldb-docker:
image: '8.0.27'
restart: 'unless-stopped'
ports:
- "3309:3306"
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_PASSWORD=root
- MYSQL_DATABASE=reco-tracker-$ENV
volumes:
- mysqldb:/var/lib/mysql
reco-tracker-docker:
image: 'reco-tracker-docker:v1'
ports:
- "8083:8083"
environment:
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=root
- SPRING_DATASOURCE_URL="jdbc:mysql://mysqldb-docker:3309/reco-tracker-$ENV"
depends_on: [mysqldb-docker]
You must define volumes at the top level like this:
version: '3.8'
services:
mysqldb-docker:
# ...
volumes:
- mysqldb:/var/lib/mysql
volumes:
mysqldb:
You can pass environment variables from your shell straight through to a service’s containers with the ‘environment’ key by not giving them a value
https://docs.docker.com/compose/environment-variables/#pass-environment-variables-to-containers
web:
environment:
- ENV
but from my tests you cant write $ENV in the compose file and expect it to read your env
for this you need to call docker-compose that way :
docker-compose run -e ENV web python console.py
see this : https://docs.docker.com/compose/environment-variables/#set-environment-variables-with-docker-compose-run

Docker-compose - setting environment variables that are not literals

I have setup Jenkins within a Docker container and I am trying to access that my private Bitbucket repo with that server. I need to copy my SSH key into that container so that Bitbucket recognizes it and I can have my Jenkins server access the repo then.
I have in my docker-compose.yml file the following:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:
However, echo $SSH_PRIVATE_KEY gives /.ssh/id_rsa literally instead of the value stored inside. I have heard the problem with doing this inside the Dockerfile instead would be that it still can be viewed in one of the layers of the image that will be pushed.
My question is how can I set the value of SSH_PRIVATE_KEY to the value of the contents of my file?
I believe this could be a duplicate of How to set environment variable into docker container using docker-compose however that solution does not appear to change anything for me.
You could create an Environment variable in your shell from which you are running your compose :
export SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa)
and then use it in your compose like :
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
environment:
- SSH_PRIVATE_KEY
ports:
- "8080:8080"
- "50000:50000"
It should pick up the value for your environment variable for container from shell environment as specified in the docs :
The value of the variable in the container is taken from the value for the same variable in the shell in which Compose is run.
Possible solution:
environment:
- SSH_PRIVATE_KEY
and call the docker-compose like this:
SSH_PRIVATE_KEY=$(cat ~/.ssh/id_rsa) docker-compose build
Unfortunately, it's currently not possible to use multiline variables in .env.
Another possibility would be:
services:
jenkins:
build: .
volumes:
- jenkins-data:/var/jenkins_home
- "/home/user/.ssh/id_rsa:/home/user/.ssh/id_rsa:ro"
ports:
- "8080:8080"
- "50000:50000"
volumes:
jenkins-data:

Docker: Why does my project have a .env file?

I'm working on a group project involving Docker that has a .env file, which looks like this:
DATABASE_URL=xxx
DJANGO_SETTINGS_MODULE=xxx
SECRET_KEY=xxx
Couldn't this just be declared inside the Dockerfile? If so, what is the advantage of making a .env file?
Not sure if I'm going in the right direction with this, but this Docker Docs page says (emphasis my own):
Your configuration options can contain environment variables. Compose
uses the variable values from the shell environment in which
docker-compose is run. For example, suppose the shell contains
POSTGRES_VERSION=9.3 and you supply this configuration:
db:
`image: "postgres:${POSTGRES_VERSION}"`
When you run docker-compose up with this configuration, Compose looks for the POSTGRES_VERSION environment variable in the shell and substitutes its value in. For this example, Compose resolves the image to postgres:9.3 before running the configuration.
If an environment variable is not set, Compose substitutes with an empty string. In the example above, if POSTGRES_VERSION is not set, the value for the image option is postgres:.
You can set default values for environment variables using a .env file, which Compose automatically looks for. Values set in the shell environment override those set in the .env file.
If we're using a .env file, then wouldn't I see some ${...} syntax in our docker-compose.yml file? I don't see anything like that, though.
Here's our docker-compose.yml file:
version: '3'
services:
server:
build:
context: ./server
dockerfile: Dockerfile
env_file: .env.dev
command: python3 manage.py runserver 0.0.0.0:8000
volumes:
- ./server:/app
ports:
- "8500:8000"
depends_on:
- db
stdin_open: true
tty: true
db:
image: postgres
client:
build:
context: ./client
dockerfile: Dockerfile
command: bash -c "npm install; npm run start"
volumes:
- ./client:/app
- /app/node_modules
ports:
- "3000:3000"
depends_on:
- server
Idea there is probably to have a place to keep secrets separated from docker-compose.yml, which you then can keep in VCS and/or share.

Use files on host system with running containers using docker compose

I want to use excel file and some folders with my container .
I am using volumes but , dont know what the problem is with my compose .
seleniumhub:
image: selenium/hub
ports:
- "4444:4444"
firefoxnode:
image: selenium/node-firefox-debug
ports:
- "5901:5900"
links:
- "seleniumhub:hub"
shm_size: '2gb'
environment:
- "NODE_MAX_SESSION=2"
- "NODE_MAX_INSTANCES=2"
chromenode2:
image: selenium/node-chrome-debug
ports:
- "5902:5900"
links:
- "seleniumhub:hub"
shm_size: '2gb'
environment:
- "NODE_MAX_SESSION=2"
- "NODE_MAX_INSTANCES=2"
test:
image: raveena1/dilsel
ports:
- 4579
links:
- "seleniumhub:hub"
container_name: mywebcontainer
**volumes:
- /$$(pwd)/Newfolder/Config/framework-config.properties:/var/lib/docker/**
I want to use the above property file in my container , how can i achieve this ?
I don't think docker-compose can interpret bash command inside a compose file. However, what you can do is use environment variable. In your case, you might want to use $PWD.
[...]
volumes:
- $PWD/Newfolder/Config/framework-config.properties:/var/lib/docker/
[...]
This will interpret the environment variable $PWD (which resolves to your current working directy) and mount this to /var/lib/docker.
Below is an example of using environment variable in docker-compose :
docker-compose.yml:
test:
image: debian:stretch-slim
ports:
- 4579
container_name: mywebcontainer
volumes:
- $PWD/:/current_directory_of_host
entrypoint: "ls -l /current_directory_of_host"
Start this container with docker-compose up. You should see a list of file that is in your current working directory.
You can also use custom environment variable : CUSTOM_ENV=$(pwd) docker-compose up. This will forward CUSTOM_ENV to docker-compose which can be used in your docker-compose.yml.

Set extra host in environment variables

I'm using docker compose to run my application. And for do that I need to set the hosts inside container (it's depends on the environment i'm running).
My approach was:
Create an environment file and set the variable:
#application.env
SERVER_IP=10.10.9.134
My docker compose file looks like:
version: '2'
services:
api:
container_name: myApplication
env_file:
- application.env
build: ./myApplication/
entrypoint: ./docker/api-startup.sh
ports:
- "8080:8080"
depends_on:
- redis
extra_hosts: &extra_hosts
myip: $SERVER_IP
But my problem is that the variable SERVER_IP is never replaced.
When I run docker-compose config I see:
services:
api:
build:
context: /...../myApplication
container_name: myApplication
depends_on:
- redis
entrypoint: ./docker/api-startup.sh
environment:
SERVER_IP: 10.10.9.134
extra_hosts:
myip: ''
ports:
- 8080:8080
I've tried to replace the variable reference using $SERVER_IP or ${SERVER_IP} but it didn't work.
I created a file .env, added single line HOST=test.example.com, then did this in docker-compose:
extra_hosts:
- myip:${HOST}
docker-compose config then shows
extra_hosts:
myip: test.example.com
To do this I followed the documentation from Docker-compose environment variables the section about .env file
UPDATE
According to the Docker documentation,
Note: If your service specifies a build option, variables defined in
environment files will not be automatically visible during the build.
Use the args sub-option of build to define build-time environment
variables.
It basically means if you place your variables in .env file, you can use them for substitution in docker-compose.yml, but if you use env_file option for the particular container, you can only see the variables inside the Docker container, not during the build. It is also logical, env_file replaces docker run --env-file=FILE ... and nothing else.
So, you can only place your values into .env. Alternatively, as William described, you can use host's environment variables.
EDIT
Try the following:
version: '2'
services:
api:
container_name: myApplication
env_file:
- application.env
build: ./myApplication/
entrypoint: ./docker/api-startup.sh
ports:
- "8080:8080"
depends_on:
- redis
extra_hosts:
- "myip:${SERVER_IP}"
Ensure curly bracers and that the environment variable exists on the host os.

Resources