Unable to use variable substitution in docker-compose - docker

I use a mac to run docker and hence I am dependent on boot2docker or docker-machine . My application running in the container needs the docker host's ip.
I saw that the variable substitution feature of docker-compose is exactly what I need . When I try something like the below it works fine
extra_hosts:
- "somehost:162.242.195.82"
BUT when I try something like:
export DOCKER_HOST="64.88.225.66"
extra_hosts:
- "host:${DOCKER_HOST}"
I get :
$ docker-compose build --no-cache
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "/code/compose/cli/main.py", line 54, in main
File "/code/compose/cli/docopt_command.py", line 23, in sys_dispatch
File "/code/compose/cli/docopt_command.py", line 26, in dispatch
File "/code/compose/cli/main.py", line 169, in perform_command
File "/code/compose/cli/command.py", line 53, in project_from_options
File "/code/compose/cli/command.py", line 89, in get_project
File "/code/compose/cli/command.py", line 70, in get_client
File "/code/compose/cli/docker_client.py", line 28, in docker_client
File "/code/.tox/py27/lib/python2.7/site-packages/docker/client.py", line 58, in __init__
File "/code/.tox/py27/lib/python2.7/site-packages/docker/utils/utils.py", line 362, in parse_host
docker.errors.DockerException: Bind address needs a port: 64.88.225.66
docker-compose returned -1
I'm not sure why it wants a port. It wasn't happy when I gave it a port number too.
So, how can I get my docker host ip to my container? It would be nice if i can pass commands like boot2docker ip or docker-machine ip default in the extra_hosts section or maybe if docker-compose could execute a script to get this value it would be great !

The problem is that DOCKER_HOST is a variable used by docker-compose itself (https://docs.docker.com/compose/reference/overview/#docker-host) and the format is wrong (it needs a tcp:// scheme and a port I believe).
You'll have to choose a different variable to include it in extra_hosts:.
Otherwise you could just pass it in as an environment variable (environment: [DOCKER_HOST=]) and parse the ip address out of it in the container.

Related

how to use environment variable in docker-compose child Dockerfile FROM clause

I'm using docker-compose to build some dockerfiles. I'm using an environment variable $IMAGE_TAG to tag images
game-client:
image: 09809809.dkr.ecr.us-east-1.amazonaws.com/crashgiants-web:${IMAGE_TAG}
build:
context: ..
dockerfile: devops/client.Dockerfile
args:
- IMAGE_TAG=${IMAGE_TAG}
- SIGNALHUB_HOST=crashgiants.io
- SOCKETIO_HOST=crashgiants.io
- TURNSERVER_HOST=crashgiants.io
- ASSET_HOST=crashgiants.io
ports:
- "80:80"
I have a dockerfile client.Dockerfile with a FROM clause like:
FROM 0809809708.dkr.ecr.us-east-1.amazonaws.com/crashgiants:$IMAGE_TAG as game-base
I can get the environment variable to the docker-compose ok but not sure how to use it within the child client.Dockerfile
I am invoking the compose file with: IMAGE_TAG=local_tester docker-compose -f ./devops/docker-compose.yml build
This currently throws an error
Building game-client
Traceback (most recent call last):
File "compose/cli/main.py", line 67, in main
File "compose/cli/main.py", line 126, in perform_command
File "compose/cli/main.py", line 302, in build
File "compose/project.py", line 468, in build
File "compose/project.py", line 450, in build_service
File "compose/service.py", line 1147, in build
compose.service.BuildError: (<Service: game-client>, {'message': 'No build stage in current context'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose/cli/main.py", line 78, in main
TypeError: can only concatenate str (not "dict") to str
[95080] Failed to execute script docker-compose
something about can only concatenate str (not "dict") to str
Modify your dockerfile as follows:
ARG IMAGE_TAG=latest
FROM 0809809708.dkr.ecr.us-east-1.amazonaws.com/crashgiants:$IMAGE_TAG as game-base
ARG IMAGE_TAG
This will expose the IMAGE_TAG as argument to the image. The second ARG IMAGE_TAG is if you want to use the argument inside the image build, since the arguments before FROM will be in a different context.

How to call non-interactively docker-compose run command - SSH?

Description of the situation : I have a Ubuntu VM running on remote computer B. I can access this computer B via SSH executed on my host (computer A).
When I enter an interactive shell via SSH on computer B, I can execute all my docker commands without any trouble, the steps are the following :
ssh user#remote_IP (enters interactively in the remote shell)
docker-compose run -d --name wfirst --user 1000:1000 container (compose a docker container and starts it)
This works perfectly, my container is mounted, up and running.
However, if I try to run that command in a non-interactive way by writing in my host terminal :
ssh user#remoteIP "cd /path_to_docker_file;docker-compose run -d --name wfirst --user 1000:1000 container"
The command does not succeed, my container is not mounted and does not run. I was able to get more information by using the "--verbose" flag on the docker-compose command.
Here the interesting part of the output of the successful method :
compose.config.config.find: Using configuration files: ./docker-compose.yml
WARNING: compose.config.environment.__getitem__: The DISPLAY variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The NO_PROXY variable is not set. Defaulting to a blank string.
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
urllib3.connectionpool._new_conn: Starting new HTTP connection (1): localhost:2375
urllib3.connectionpool._make_request: http://localhost:2375 "GET /v1.25/version HTTP/1.1" 200 758
compose.cli.command.get_client: docker-compose version 1.21.0, build unknown
docker-py version: 3.4.1
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1c 28 May 2019
compose.cli.command.get_client: Docker base_url: http://localhost:2375
...
We can see that a HTTP connection to docker is successfully established. The command then continues its execution and is able to create the container from the docker image.
Here the output of the failing method :
compose.config.config.find: Using configuration files: ./docker-compose.yml
compose.config.environment.__getitem__: The DISPLAY variable is not set. Defaulting to a blank string.
compose.config.environment.__getitem__: The NO_PROXY variable is not set. Defaulting to a blank string.
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 11, in <module>
load_entry_point('docker-compose==1.21.0', 'console_scripts', 'docker-compose')()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 71, in main
command()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 124, in perform_command
project = project_from_options('.', options)
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 41, in project_from_options
compatibility=options.get('--compatibility'),
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 121, in get_project
host=host, environment=environment
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 95, in get_client
version_info = six.iteritems(client.version())
File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 198, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
We can see that the HTTP connection cannot be established.
Why do I need to send my commands in a non-interactive way ? Originally, I want to send those commands using Jenkins (I added a SSH plugin in Jenkins), and I noticed that the docker commands were not working (same output as shown in this post). After a couple of tests, I realised that when Jenkins uses SSH, he sends the commands in a non-interactive way :
- ssh user#remote_IP "commands_to_execute"
This non-interactive way is not an issue for simple commands, but it appears to be an issue for some docker commands which require to be executed in an interactive shell I guess ?
Has someone found a work-around to successfully execute docker commands in a non-interactive shell ? Any help, hint, or alternative solutions will be greatly appreciated, as I tried a lot of things without any success so far..
Did you check if it is docker-engine running on tcp or by unix socket?
Maybe isn't reading right environment vars when you login by ssh u # h "command".
If it's running on unix socket (by default) try...
ssh user # remoteIP "cd /path_to_docker_file;DOCKER_HOST=unix:///var/run/docker.sock docker-compose run -d --name wfirst --user 1000: 1000 container"
or if is running on TCP, you can try ...
ssh user # remoteIP "cd /path_to_docker_file;DOCKER_HOST=tcp://127.0.0.1:2375 docker-compose run -d --name wfirst --user 1000: 1000 container"

Docker compose returns error : UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

I am trying to create a MySQL container inside Jenkins pipeline using Docker Compose . I run the following command after
installing docker compose version 1.9.0
docker-compose -f ./jenkins/docker-compose.yml run -rm redis
and my compose file looks like
version: '2.1'
services:
redis:
image: "redis:alpine"
When running this I am getting the error as follows :
docker-compose $'\342\200\223f' ./jenkins/docker-compose.yml run $'\342\200\223rm' redis
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "compose/cli/main.py", line 62, in main
File "compose/cli/main.py", line 93, in dispatch
File "compose/cli/docopt_command.py", line 31, in parse
File "compose/cli/docopt_command.py", line 42, in get_handler
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
docker-compose returned -1
How to fix this ?
Did you copy and paste your Jenkins config by chance? \342\200\223 is the octal representation of an "en dash" which is being used in places where you want a hyphen. Try adjusting your Jenkins config to use hyphens instead.

docker-compose to Bluemix Failed. “TypeError: string indices must be integers”

I am trying to this tutorial. Tutorial: Creating a multi-container deployment with Docker Compose and IBM Bluemix Container Service
However, an error occurred and I cannot understand what does it mean. Please tell me what is wrong.
Versions:
OS: macOS Sierra
bx version 0.5.3+e1de255-2017-05-17T04:07:46+00:00
Docker version 17.03.1-ce, build c6d412e
docker-compose version 1.11.2, build dfed245
Environment Variables:
DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443
DOCKER_CERT_PATH=/Users/{myname}/.ice/certs/containers-api.ng.bluemix.net/******
DOCKER_TLS_VERIFY=1
docker-compose.yml
lets-chat:
image: registry.ng.bluemix.net/hiroga/lets-chat
ports:
- "8080"
links:
- mongo
mongo:
image: registry.ng.bluemix.net/hiroga/mongo
expose:
- "27017"
Error
$ docker-compose --verbose up
compose.config.config.find: Using configuration files: ./docker-compose.yml
~~~~~~~~~~
compose.parallel.feed_queue: Pending: set([])
compose.parallel.parallel_execute_iter: Failed: <Service: lets-chat>
compose.parallel.feed_queue: Pending: set([])
ERROR: for lets-chat string indices must be integers
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose/cli/main.py", line 64, in main
File "compose/cli/main.py", line 116, in perform_command
File "compose/cli/main.py", line 876, in up
File "compose/project.py", line 416, in up
File "compose/parallel.py", line 66, in parallel_execute
TypeError: string indices must be integers
Failed to execute script docker-compose
Any solution or hint is welcome. Thank you.
Is there perhaps something earlier in the output that might hint as to the actual error? (before the last string of compose.parallel.feed_queue: Pending: set([]) messages, maybe?)
Double check the image names (compare vs bx ic images, including the version if it isn't latest).
Other possibilities are - if the last messages above that aren't the feed_queue are dealing with docker attach or docker start, then the deploy may have actually been successful, but docker-compose is very optimistic in how long it defaults to waiting. You can try bumping the timeout with export COMPOSE_HTTP_TIMEOUT=300 or similar to give it a bit more patience to deal with remote servers.
If you check with docker ps -a and docker-compose ps, do those show if the actual containers got started?

docker compose, vagrant and insecure Repository

I have setup docker-compose to pull my image from a custom repository.
Here is how the yaml file looks like
my_service:
image: d-myrepo:5000/mycompany/my_service:latest
ports:
- "8079:8079"
Now if I run vagrant up, it gets errors
==> default: File "/code/build/docker-compose/out00-PYZ.pyz/docker.auth.auth", line 46, in expand_registry_url
==> default: docker.errors
==> default: .
==> default: DockerException
==> default: :
==> default: HTTPS endpoint unresponsive and insecure mode isn't enabled.
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
/usr/local/bin/docker-compose -f "/vagrant/docker-compose.yml" up -d
Stdout from the command:
Stderr from the command:
stdin: is not a tty
Creating vagrant_y2y_1...
Pulling image d-myrepo:5000/mycompany/my_service:latest...
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.main", line 31, in main
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.docopt_command", line 21, in sys_dispatch
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.command", line 27, in dispatch
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.docopt_command", line 24, in dispatch
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.command", line 59, in perform_command
File "/code/build/docker-compose/out00-PYZ.pyz/compose.cli.main", line 464, in up
File "/code/build/docker-compose/out00-PYZ.pyz/compose.project", line 208, in up
File "/code/build/docker-compose/out00-PYZ.pyz/compose.service", line 214, in recreate_containers
File "/code/build/docker-compose/out00-PYZ.pyz/compose.service", line 197, in create_container
File "/code/build/docker-compose/out00-PYZ.pyz/docker.client", line 710, in pull
File "/code/build/docker-compose/out00-PYZ.pyz/docker.auth.auth", line 67, in resolve_repository_name
File "/code/build/docker-compose/out00-PYZ.pyz/docker.auth.auth", line 46, in expand_registry_url
docker.errors.DockerException: HTTPS endpoint unresponsive and insecure mode isn't enabled.
I read about it on the internet, that it has to do with having an insecure repo.
It only works, only if I edit the file
/etc/default/docker
with content
DOCKER_OPTS="-r=true --insecure-registry d-myrepo:5000 ${DOCKER_OPTS}"
restart the docker service and manually pull the image. i.e.
docker pull d-myrepo:5000/mycompany/my_service:latest
Is there a way to avoid this error? and having the provisioning running smoothly? maybe I am missing an option inside the docker-composer.yml file?
thanks for your feedack, the best way to achieve this is to set the vagrant provisioning the following way
config.vm.provision :docker
config.vm.provision :docker_compose
config.vm.provision "shell", path: "provision.sh", privileged: false
while the shell script provision.sh would include the following relevant lines.
sudo echo "DOCKER_OPTS=\"-r=true --insecure-registry my_repo:5000 \${DOCKER_OPTS}\"" | sudo tee /etc/default/docker
sudo service docker restart
sudo /usr/local/bin/docker-compose -f /vagrant/docker-compose.yml up -d --allow-insecure-ssl

Resources