In docker-compose there is a .env file which can hold all the properties of used in docker-compose.yml
Is there an equivalent of that in docker run command? I have exhausted the docs and forums but couldn't find any answers.
Here is what I am looking for:
Rather than
docker run -v /dir1:/dir1 -v /dir2:dir2 -p 80:80 repo/image
run docker run -config config.yml repo/image' with config.yml file holding all the property mappings
One option could be to have the parameters stored in a file and just get the string of the file using cat:
docker run $(cat config.file) repo/image
Where config.file content should be something like:
-v /dir1:/dir1 -v /dir2:dir2 -p 80:80
This does seem to be an unfortunate gap. The best workaround I've found is to use docker-compose with a docker-compose.yml file to define the container and all it's flags and then use
docker-compose run your-service-name-here
To just run a single one-off container.
Unfortunatelly, it is no way to do this as you describe.
However you can add several env config files and merge them to one .env like describe this answer
$ awk -F= '!a[$1]++' first.env second.env > .env
Related
I am relatively new to using docker-compose and am running a stack with the following command
docker-compose \
--project-name version-12 \
-f installation/docker-compose-common.yml \
-f installation/docker-compose-erpnext.yml \
--project-directory installation \
up -d
now, with the non-default docker-compose.yml files I can't manage to have docker-compose stop, docker-compose ps to work. I have tried to use the -f, or --project-name flags but couldn't make it happen.
Can anyone kindly advise how to make this work in such a scenario?
You need to repeat all of the docker-compose options for every command you need to run.
There are two ways around this. One is to write a shell script wrapper that invokes this command:
#!/bin/sh
# I am `docker-compose-erpnext.sh`
# Run me with any normal `docker-compose` options
exec docker-compose \
--project-name version-12 \
-f installation/docker-compose-common.yml \
-f installation/docker-compose-erpnext.yml \
--project-directory installation \
"$#"
Docker Compose also supports environment variables for most of its settings; many of these in turn can also be included in a .env file. You can't specify --project-directory this way, but it's documented to default to the directory of the Compose file.
export COMPOSE_PROJECT_NAME=version-12
export COMPOSE_FILE=installation/docker-compose-common.yml:installation/docker-compose-erpnext.yml
docker-compose up -d
docker-compose ps
You can put these two settings in a file name .env in the directory from which you're running docker-compose (not the installation subdirectory); but if you have multiple deployments you're trying to manage, you can't specify an alternate name for the file (there is neither a CLI option nor an environment variable setting for it).
I am trying to set up a docker run configuration in Pycharm, i am pretty new to this functionality in pycharm, and i can't get it working.
In docker I would run the container with the following command
docker build -t test-container . && docker run --name container-pycharm -t -i --env-file .env -v $(pwd):/srv/app -p 8080:8080 --rm test-container ./serve-app
I set up this in pycharm, by adding the following line
--rm --env-file .env -i -t -p 8080:8080 -v $(pwd):/srv/app
to command line options section in the relevant docker Run/Debug Configuration Pycharm window. Unfortunately I get
Failed to deploy 'container-pycharm Dockerfile: Dockerfile': com.github.dockerjava.api.exception.BadRequestException: {"message":"create $(pwd): \"$(pwd)\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path"}
Clearly, I cant use $(pwd) in my command line options, any idea on how to solve this in pycharm?
Pycharm doesn't invoke docker directly via the command you see in the command preview, it goes through its custom parser, currently they haven't implemented the feature to read envs. Thus "If you intended to pass a host directory, use absolute path"
And -v is not officially supported as command line options in the current version. Ref
Use Bind mounts instead
i am a totally docker newb, so sorry for that
i have stand-alone docker image (some node app),
that i want to run in different environments.
i want to set up the env file with run RUN --env-file <path>
How ever, i want to use the env files that inside the image (so i can use different files per env),
and not on server.
so would be the path inside image.
is there any way to do so?
perhaps like "cp" (docker cp [OPTIONS] CONTAINER:<path>)
but doesn't seem to work.
what the best practice here?
am i making sense?
Thanks!!
Docker bind mounts are a fairly effective way to inject configuration files like this into a running container. I would not try to describe every possible configuration in your built image; instead, let that be configuration that's pushed in from the host.
Pick some single specific file to hold the configuration. For the sake of argument, let's say it's /usr/src/app/env. Set up your application however it's built to read that file at startup time. Either make sure the application can still start up if the file is missing, or build your image with some file there with reasonable default settings.
Now when you run your container, it will always read settings from that known file; but, you can specify a host file that will be there:
docker run -v $PWD/env.development:/usr/src/app/env myimage
Now you can locally have an env.development that specifies extended logging and a local database, and an env.production with minimal logging and pointing at your production database. If you set up a third environment (say a shared test database with some known data in it) you can just run the container with this new configuration, without rebuilding it.
Following is the command to run docker
docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
Example
docker run --name test -it debian
focus on following switch
--env , -e Set environment variables
--env-file You can pass environment variables to your containers with the -e flag.
An example from a startup script:
sudo docker run -d -t -i -e REDIS_NAMESPACE='staging' \
-e POSTGRES_ENV_POSTGRES_PASSWORD='foo' \
-e POSTGRES_ENV_POSTGRES_USER='bar' \
-e POSTGRES_ENV_DB_NAME='mysite_staging' \
-e POSTGRES_PORT_5432_TCP_ADDR='docker-db-1.hidden.us-east-1.rds.amazonaws.com' \
-e SITE_URL='staging.mysite.com' \
-p 80:80 \
--link redis:redis \
--name container_name dockerhub_id/image_name
In case, you have many environment variables and especially if they're meant to be secret, you can use an env-file:
$ docker run --env-file ./env.list ubuntu bash
The --env-file flag takes a filename as an argument and expects each
line to be in the VAR=VAL format, mimicking the argument passed to
--env. Comment lines need only be prefixed with #
My current setup for running a docker container is on the lines of this:
I've got a main.env file:
# Main
export PRIVATE_IP=\`echo localhost\`
export MONGODB_HOST="$PRIVATE_IP"
export MONGODB_URL="mongodb://$MONGODB_HOST:27017/development"
In my service file (upstart), I source this file . /path/to/main.env
I then call docker run with multiple -e for each of the environment variables I want inside of the container. In this case I would call something like: docker run -e MONGODB_URL=$MONGODB_URL ubuntu bash
I would then expect MONGODB_URL inside of the container to equal mongodb://localhost:27017/development. Notice that in reality echo localhost is replaced by a curl to amazon's api for an actual PRIVATE_IP.
This becomes a bit unwieldy when you start having more and more environment variables you need to give your container. There is a fine point to see here which is that the environment variables need to be resolved at run time, such as with a call to curl or by referring to other env variables.
The solution I was hoping to use is:
calling docker run with an --env-file parameter such as this:
# Main
PRIVATE_IP=\`echo localhost\`
MONGODB_HOST="$PRIVATE_IP"
MONGODB_URL="mongodb://$MONGODB_HOST:27017/development"
Then my docker run command would be significantly shortened to docker run --env-file=/path/to/main.env ubuntu bash (keep in mind usually I've got around 12-15 environment variables.
This is where I hit my problem which is that inside the container none of the variables resolve as expected. Instead I end up with:
PRIVATE_IP=`echo localhost`
MONGODB_HOST="$PRIVATE_IP"
MONGODB_URL="mongodb://$MONGODB_HOST:27017/development"
I could circumvent this by doing the following:
Sourcing the main.env file.
Creating a file containing just the names of the variables I want (meaning docker would search for them in the environment).
Then calling docker run with this file as an argument to --env-file. This would work but would mean I would need to maintain two files instead of one, and really wouldn't be that big of an improvement of the current situation.
What I would prefer is to have the variables resolve as expected.
The closest question to mine that I could find is:
12factor config approach with Docker
Ceate a .env file
example: test=123 val=Guru
Execute command
docker run -it --env-file=.env bash
Inside the bash verify using
echo $test (should print 123)
Both --env and --env-file setup variables as is and do not replace nested variables.
Solomon Hykes talks about configuring containers at run time and the the various approaches. The one that should work for you is to volume mounting the main.env from host into the container and sourcing it.
So I just faced this issue as well, what solved it for me was I specified the --env-file or -e KEY=VAL before the name of the container image. For example
Broken:
docker run my-image --env-file .env
Fixed:
docker run --env-file .env my-image
creating an ENV file that is nothing more than key/value pairs can be processed in normal shell commands and appended to the environment. Look at the bash -a pragma.
What you can do is create a startup script that can be run when the container starts. So if your current docker file looks something like this
From ...
...
CMD command
Change it to
From ...
...
ADD start.sh start.sh
CMD ["start.sh"]
In your start.sh script do the following:
export PRIVATE_IP=\`echo localhost\`
export MONGODB_HOST="$PRIVATE_IP"
export MONGODB_URL="mongodb://$MONGODB_HOST:27017/development"
command
I had a very similar problem to this. If I passed the contents of the env file to docker as separate -e directives then everything ran fine however if I passed the file using --env-file the container failed to run properly.
Turns out there were some spurious line endings in the file (I had copied from windows and ran docker in Ubuntu). When I removed them the container ran the same with --env or --env-file.
I had this issue when using docker run in a separate run script run.sh file, since I wanted the credentials ADMIN_USER and ADMIN_PASSWORD to be accessible in the container, but not show up in the command.
Following the other answers and passing a separate environment file with --env or --env-file didn't work for my image (though it worked for the Bash image). What worked was creating a separate env file...
# env.list
ADMIN_USER='username'
ADMIN_PASSWORD='password'
...and sourcing it in the run script when launching the container:
# run.sh
source env.list
docker run -d \
-e ADMIN_USER=$INFLUXDB_ADMIN_USER \
-e ADMIN_PASSWORD=$INFLUXDB_ADMIN_PASSWORD \
image_repo/name:tag
Now that I found a way to expose host files to the container (-v option) I would like to do kind of the opposite:
How can I edit files from a running container with a host editor?
sshfs could probably do the job but since a running container is already some kind of host directory I wonder if there is a portable (between aufs, btrfs and device mapper) way to do that?
The best way is:
$ docker cp CONTAINER:FILEPATH LOCALFILEPATH
$ vi LOCALFILEPATH
$ docker cp LOCALFILEPATH CONTAINER:FILEPATH
Limitations with $ docker exec: it can only attach to a running container.
Limitations with $ docker run: it will create a new container.
Whilst it is possible, and the other answers explain how, you should avoid editing files in the Union File System if you can.
Your definition of volumes isn't quite right - it's more about bypassing the Union File System than exposing files on the host. For example, if I do:
$ docker run --name="test" -v /volume-test debian echo "test"
The directory /volume-test inside the container will not be part of the Union File System and instead will exist somewhere on the host. I haven't specified where on the host, as I may not care - I'm not exposing host files, just creating a directory that is shareable between containers and the host. You can find out exactly where it is on the host with:
$ docker inspect -f "{{.Volumes}}" test
map[/volume_test:/var/lib/docker/vfs/dir/b7fff1922e25f0df949e650dfa885dbc304d9d213f703250cf5857446d104895]
If you really need to just make a quick edit to a file to test something, either use docker exec to get a shell in the container and edit directly, or use docker cp to copy the file out, edit on the host and copy back in.
We can use another way to edit files inside working containers (this won't work if container is stoped).
Logic is to:
-)copy file from container to host
-)edit file on host using its host editor
-)copy file back to container
We can do all this steps manualy, but i have written simple bash script to make this easy by one call.
/bin/dmcedit:
#!/bin/sh
set -e
CONTAINER=$1
FILEPATH=$2
BASE=$(basename $FILEPATH)
DIR=$(dirname $FILEPATH)
TMPDIR=/tmp/m_docker_$(date +%s)/
mkdir $TMPDIR
cd $TMPDIR
docker cp $CONTAINER:$FILEPATH ./$DIR
mcedit ./$FILEPATH
docker cp ./$FILEPATH $CONTAINER:$FILEPATH
rm -rf $TMPDIR
echo 'END'
exit 1;
Usage example:
dmcedit CONTAINERNAME /path/to/file/in/container
The script is very easy, but it's working fine for me.
Any suggestions are appreciated.
There are two ways to mount files into your container. It looks like you want a bind mount.
Bind Mounts
This mounts local files directly into the container's filesystem. The containerside path and the hostside path both point to the same file. Edits made from either side will show up on both sides.
mount the file:
❯ echo foo > ./foo
❯ docker run --mount type=bind,source=$(pwd)/foo,target=/foo -it debian:latest
# cat /foo
foo # local file shows up in container
in a separate shell, edit the file:
❯ echo 'bar' > ./foo # make a hostside change
back in the container:
# cat /foo
bar # the hostside change shows up
# echo baz > /foo # make a containerside change
# exit
❯ cat foo
baz # the containerside change shows up
Volume Mounts
mount the volume
❯ docker run --mount type=volume,source=foovolume,target=/foo -it debian:latest
root#containerB# echo 'this is in a volume' > /foo/data
the local filesystem is unchanged
docker sees a new volume:
❯ docker volume ls
DRIVER VOLUME NAME
local foovolume
create a new container with the same volume
❯ docker run --mount type=volume,source=foovolume,target=/foo -it debian:latest
root#containerC:/# cat /foo/data
this is in a volume # data is still available
syntax: -v vs --mount
These do the same thing. -v is more concise, --mount is more explicit.
bind mounts
-v /hostside/path:/containerside/path
--mount type=bind,source=/hostside/path,target=/containerside/path
volume mounts
-v /containerside/path
-v volumename:/containerside/path
--mount type=volume,source=volumename,target=/containerside/path
(If a volume name is not specified, a random one is chosen.)
The documentaion tries to convince you to use one thing in favor of another instead of just telling you how it works, which is confusing.
Here's the script I use:
#!/bin/bash
IFS=$'\n\t'
set -euox pipefail
CNAME="$1"
FILE_PATH="$2"
TMPFILE="$(mktemp)"
docker exec "$CNAME" cat "$FILE_PATH" > "$TMPFILE"
$EDITOR "$TMPFILE"
cat "$TMPFILE" | docker exec -i "$CNAME" sh -c 'cat > '"$FILE_PATH"
rm "$TMPFILE"
and the gist for when I fix it but forget to update this answer:
https://gist.github.com/dmohs/b50ea4302b62ebfc4f308a20d3de4213
If you think your volume is a "network drive", it will be easier.
To edit the file located in this drive, you just need to turn on another machine and connect to this network drive, then edit the file like normal.
How to do that purely with docker (without FTP/SSH ...)?
Run a container that has an editor (VI, Emacs). Search Docker hub for "alpine vim"
Example:
docker run -d --name shared_vim_editor \
-v <your_volume>:/home/developer/workspace \
jare/vim-bundle:latest
Run the interactive command:
docker exec -it -u root shared_vim_editor /bin/bash
Hope this helps.
I use sftp plugin from my IDE.
Install ssh server for your container and allow root access.
Run your docker container with -p localport:22
Install from your IDE a sftp plugin
Example using sublime sftp plugin:
https://www.youtube.com/watch?v=HMfjt_YMru0
The way I am doing is using Emacs with docker package installed. I would recommend Spacemacs version of Emacs. I would follow the following steps:
1) Install Emacs (Instruction) and install Spacemacs (Instruction)
2) Add docker in your .spacemacs file
3) Start Emacs
4) Find file (SPC+f+f) and type /docker:<container-id>:/<path of dir/file in the container>
5) Now your emacs will use the container environment to edit the files
docker run -it -name YOUR_NAME IMAGE_ID /bin/bash
$>vi path_to_file
The following worked for me
docker run -it IMAGE_NAME /bin/bash
eg. my image was called ipython/notebook
docker run -it ipython/notebook /bin/bash