how to use environment variable in docker-compose child Dockerfile FROM clause - docker

I'm using docker-compose to build some dockerfiles. I'm using an environment variable $IMAGE_TAG to tag images
game-client:
image: 09809809.dkr.ecr.us-east-1.amazonaws.com/crashgiants-web:${IMAGE_TAG}
build:
context: ..
dockerfile: devops/client.Dockerfile
args:
- IMAGE_TAG=${IMAGE_TAG}
- SIGNALHUB_HOST=crashgiants.io
- SOCKETIO_HOST=crashgiants.io
- TURNSERVER_HOST=crashgiants.io
- ASSET_HOST=crashgiants.io
ports:
- "80:80"
I have a dockerfile client.Dockerfile with a FROM clause like:
FROM 0809809708.dkr.ecr.us-east-1.amazonaws.com/crashgiants:$IMAGE_TAG as game-base
I can get the environment variable to the docker-compose ok but not sure how to use it within the child client.Dockerfile
I am invoking the compose file with: IMAGE_TAG=local_tester docker-compose -f ./devops/docker-compose.yml build
This currently throws an error
Building game-client
Traceback (most recent call last):
File "compose/cli/main.py", line 67, in main
File "compose/cli/main.py", line 126, in perform_command
File "compose/cli/main.py", line 302, in build
File "compose/project.py", line 468, in build
File "compose/project.py", line 450, in build_service
File "compose/service.py", line 1147, in build
compose.service.BuildError: (<Service: game-client>, {'message': 'No build stage in current context'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose/cli/main.py", line 78, in main
TypeError: can only concatenate str (not "dict") to str
[95080] Failed to execute script docker-compose
something about can only concatenate str (not "dict") to str

Modify your dockerfile as follows:
ARG IMAGE_TAG=latest
FROM 0809809708.dkr.ecr.us-east-1.amazonaws.com/crashgiants:$IMAGE_TAG as game-base
ARG IMAGE_TAG
This will expose the IMAGE_TAG as argument to the image. The second ARG IMAGE_TAG is if you want to use the argument inside the image build, since the arguments before FROM will be in a different context.

Related

Docker compose deploy in a remote server

I'm trying to deploy a docker image to a remote server using docker-compose but I'm getting authentication issue:
Command:
docker-compose -H "ssh://root#host_ip" --tlscacert "~/.ssh/private_key" ps
Output:
sudo docker-compose -H "ssh://root#jenkins.evercam.io" --tlscacert "~/.ssh/id_jenkins" -f evercam_camera_server/docker-compose.yml ps
Traceback (most recent call last):
File "bin/docker-compose", line 3, in <module>
File "compose/cli/main.py", line 67, in main
File "compose/cli/main.py", line 123, in perform_command
File "compose/cli/command.py", line 69, in project_from_options
File "compose/cli/command.py", line 132, in get_project
File "compose/cli/docker_client.py", line 43, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 164, in __init__
File "docker/transport/sshconn.py", line 113, in __init__
File "docker/transport/sshconn.py", line 121, in _connect
File "paramiko/client.py", line 446, in connect
File "paramiko/client.py", line 764, in _auth
File "paramiko/client.py", line 740, in _auth
File "paramiko/transport.py", line 1580, in auth_publickey
File "paramiko/auth_handler.py", line 250, in wait_for_response
paramiko.ssh_exception.AuthenticationException: Authentication failed.
[28861] Failed to execute script docker-compose
docker-compose version 1.27.4, build 40524192
docker version 20.10.5, build 55c4c88
ubuntu version 18.04
I tried with a recent version of docker compose which is 1.28.5 it's able to access the given host, but strangely it's asking the password five times!

How to call non-interactively docker-compose run command - SSH?

Description of the situation : I have a Ubuntu VM running on remote computer B. I can access this computer B via SSH executed on my host (computer A).
When I enter an interactive shell via SSH on computer B, I can execute all my docker commands without any trouble, the steps are the following :
ssh user#remote_IP (enters interactively in the remote shell)
docker-compose run -d --name wfirst --user 1000:1000 container (compose a docker container and starts it)
This works perfectly, my container is mounted, up and running.
However, if I try to run that command in a non-interactive way by writing in my host terminal :
ssh user#remoteIP "cd /path_to_docker_file;docker-compose run -d --name wfirst --user 1000:1000 container"
The command does not succeed, my container is not mounted and does not run. I was able to get more information by using the "--verbose" flag on the docker-compose command.
Here the interesting part of the output of the successful method :
compose.config.config.find: Using configuration files: ./docker-compose.yml
WARNING: compose.config.environment.__getitem__: The DISPLAY variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The NO_PROXY variable is not set. Defaulting to a blank string.
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
urllib3.connectionpool._new_conn: Starting new HTTP connection (1): localhost:2375
urllib3.connectionpool._make_request: http://localhost:2375 "GET /v1.25/version HTTP/1.1" 200 758
compose.cli.command.get_client: docker-compose version 1.21.0, build unknown
docker-py version: 3.4.1
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1c 28 May 2019
compose.cli.command.get_client: Docker base_url: http://localhost:2375
...
We can see that a HTTP connection to docker is successfully established. The command then continues its execution and is able to create the container from the docker image.
Here the output of the failing method :
compose.config.config.find: Using configuration files: ./docker-compose.yml
compose.config.environment.__getitem__: The DISPLAY variable is not set. Defaulting to a blank string.
compose.config.environment.__getitem__: The NO_PROXY variable is not set. Defaulting to a blank string.
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/user/.docker/config.json', '/home/user/.dockercfg']
docker.utils.config.find_config_file: No config file found
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse
response.begin()
File "/usr/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 11, in <module>
load_entry_point('docker-compose==1.21.0', 'console_scripts', 'docker-compose')()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 71, in main
command()
File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 124, in perform_command
project = project_from_options('.', options)
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 41, in project_from_options
compatibility=options.get('--compatibility'),
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 121, in get_project
host=host, environment=environment
File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 95, in get_client
version_info = six.iteritems(client.version())
File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/docker/api/client.py", line 198, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
We can see that the HTTP connection cannot be established.
Why do I need to send my commands in a non-interactive way ? Originally, I want to send those commands using Jenkins (I added a SSH plugin in Jenkins), and I noticed that the docker commands were not working (same output as shown in this post). After a couple of tests, I realised that when Jenkins uses SSH, he sends the commands in a non-interactive way :
- ssh user#remote_IP "commands_to_execute"
This non-interactive way is not an issue for simple commands, but it appears to be an issue for some docker commands which require to be executed in an interactive shell I guess ?
Has someone found a work-around to successfully execute docker commands in a non-interactive shell ? Any help, hint, or alternative solutions will be greatly appreciated, as I tried a lot of things without any success so far..
Did you check if it is docker-engine running on tcp or by unix socket?
Maybe isn't reading right environment vars when you login by ssh u # h "command".
If it's running on unix socket (by default) try...
ssh user # remoteIP "cd /path_to_docker_file;DOCKER_HOST=unix:///var/run/docker.sock docker-compose run -d --name wfirst --user 1000: 1000 container"
or if is running on TCP, you can try ...
ssh user # remoteIP "cd /path_to_docker_file;DOCKER_HOST=tcp://127.0.0.1:2375 docker-compose run -d --name wfirst --user 1000: 1000 container"

Docker compose returns error : UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

I am trying to create a MySQL container inside Jenkins pipeline using Docker Compose . I run the following command after
installing docker compose version 1.9.0
docker-compose -f ./jenkins/docker-compose.yml run -rm redis
and my compose file looks like
version: '2.1'
services:
redis:
image: "redis:alpine"
When running this I am getting the error as follows :
docker-compose $'\342\200\223f' ./jenkins/docker-compose.yml run $'\342\200\223rm' redis
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "compose/cli/main.py", line 62, in main
File "compose/cli/main.py", line 93, in dispatch
File "compose/cli/docopt_command.py", line 31, in parse
File "compose/cli/docopt_command.py", line 42, in get_handler
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
docker-compose returned -1
How to fix this ?
Did you copy and paste your Jenkins config by chance? \342\200\223 is the octal representation of an "en dash" which is being used in places where you want a hyphen. Try adjusting your Jenkins config to use hyphens instead.

nvidia-docker can't talk to: http://localhost:3476/docker/cli/json

nvidia-docker can't talk to http://localhost:3476/docker/cli/json
Traceback (most recent call last):
File "/usr/local/bin/nvidia-docker-compose", line 43, in <module>
resp = request.urlopen('http://{0}/docker/cli/json'.format(args.nvidia_docker_host)).read().decode()
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1228, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>
A fresh install of nvidia-docker-compose fixed this:
wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.1/nvidia-docker_1.0.1-1_amd64.deb
sudo dpkg -i /tmp/nvidia-docker*.deb && rm /tmp/nvidia-docker*.deb
Then to test it:
Test nvidia-smi
nvidia-docker run --rm nvidia/cuda nvidia-smi
Encountered this also, a customer wasn't managing to run nvidia-docker-compose. Turns out even after reinstallations of docker and nvidia-docker, the query made by nvidia-docker to docker on localhost:3476 was not getting any response (see nvidia-docker-compose code here)
I managed to solve this by generating a hand-made docker-compose file as they turn out to be quite simple, follow this example, replace 375.66 with your nvidia driver version and put as many /dev/nvidia[n] lines as you have graphic cards (did not try to put services on separate GPUs but go for it !):
services:
exampleservice0:
devices:
- /dev/nvidia0
- /dev/nvidia1
- /dev/nvidiactl
- /dev/nvidia-uvm
- /dev/nvidia-uvm-tools
environment:
- EXAMPLE_ENV_VARIABLE=example
image: company/image
volumes:
- ./disk:/disk
- nvidia_driver_375.66:/usr/local/nvidia:ro
version: '2'
volumes:
media: null
nvidia_driver_375.66:
external: true
Then just run this hand-made docker-compose file with a classic docker-compose command.

Unable to use variable substitution in docker-compose

I use a mac to run docker and hence I am dependent on boot2docker or docker-machine . My application running in the container needs the docker host's ip.
I saw that the variable substitution feature of docker-compose is exactly what I need . When I try something like the below it works fine
extra_hosts:
- "somehost:162.242.195.82"
BUT when I try something like:
export DOCKER_HOST="64.88.225.66"
extra_hosts:
- "host:${DOCKER_HOST}"
I get :
$ docker-compose build --no-cache
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "/code/compose/cli/main.py", line 54, in main
File "/code/compose/cli/docopt_command.py", line 23, in sys_dispatch
File "/code/compose/cli/docopt_command.py", line 26, in dispatch
File "/code/compose/cli/main.py", line 169, in perform_command
File "/code/compose/cli/command.py", line 53, in project_from_options
File "/code/compose/cli/command.py", line 89, in get_project
File "/code/compose/cli/command.py", line 70, in get_client
File "/code/compose/cli/docker_client.py", line 28, in docker_client
File "/code/.tox/py27/lib/python2.7/site-packages/docker/client.py", line 58, in __init__
File "/code/.tox/py27/lib/python2.7/site-packages/docker/utils/utils.py", line 362, in parse_host
docker.errors.DockerException: Bind address needs a port: 64.88.225.66
docker-compose returned -1
I'm not sure why it wants a port. It wasn't happy when I gave it a port number too.
So, how can I get my docker host ip to my container? It would be nice if i can pass commands like boot2docker ip or docker-machine ip default in the extra_hosts section or maybe if docker-compose could execute a script to get this value it would be great !
The problem is that DOCKER_HOST is a variable used by docker-compose itself (https://docs.docker.com/compose/reference/overview/#docker-host) and the format is wrong (it needs a tcp:// scheme and a port I believe).
You'll have to choose a different variable to include it in extra_hosts:.
Otherwise you could just pass it in as an environment variable (environment: [DOCKER_HOST=]) and parse the ip address out of it in the container.

Resources