I'm using weave to launch some containers which form a database cluster. I have gotten this working manually on two hosts in EC2 by doing the following:
$HOST1> weave launch
$HOST2> weave launch $HOST1
$HOST1> eval $(weave env)
$HOST2> eval $(weave env)
$HOST1> docker run --name neo-1 -d -P ... my/neo4j-cluster
$HOST2> docker run --name neo-2 -d -P ... my/neo4j-cluster
$HOST3> docker run --name neo-1 -d -P -e ARBITER=true ... my/neo4j-cluster
I can check the logs and everthing starts up ok.
When using ansible I can get the above to work using the command: ... module and an environment variable:
- name: Start Neo Arbiter
command: 'docker run --name neo-2 -d -P ... my/neo4j-cluster'
environment:
DOCKER_HOST: 'unix:///var/run/weave/weave.sock'
As that's basically all eval $(weave env) does.
But when I use the docker module for ansible, even with the docker_url parameter set to the same thing you see above with DOCKER_HOST, DNS does not resolve between hosts. Here's what that looks like:
- name: Start Neo Arbiter
docker:
image: "my/neo4j-cluster:{{neo4j_version}}"
docker_url: unix:///var/run/weave/weave.sock
name: neo-3
pull: missing
state: reloaded
detach: True
publish_all_ports: True
OR
- name: Start Neo Arbiter
docker:
image: "my/neo4j-cluster:{{neo4j_version}}"
docker_url: unix:///var/run/weave/weave.sock
name: neo-3
pull: missing
state: reloaded
detach: True
publish_all_ports: True
environment:
DOCKER_HOST: 'unix:///var/run/weave/weave.sock'
Neither of those work. The DNS does not resolve so the servers never start. I do have other server options (like SERVER_ID for neo4j, etc set just not shown here for simplicity).
Anyone run into this? I know the docker module for ansible uses docker-py and stuff. I wonder if there's some type of incompatibility with weave?
EDIT
I should mention that when the containers launch they actually show up in WeaveDNS and appear to have been added to the system. I can ping the local hostname of each container as long as its on the host. When I go to the other host though, it cannot ping the ones on the other host. This despite them registering in WeaveDNS (weave status dns) and weave status showing correct # of peers and established connections.
This could be caused by the client sending a HostConfig struct in the Docker start request, which is not really how you're supposed to do it but is supported by Docker "for backwards compatibility".
Weave has been fixed to cope, but the fix is not in a released version yet. You could try the latest snapshot version if you're brave.
You can probably kludge it by explicitly setting the DNS resolver to the docker bridge IP in your containers' config - weave has an undocumented helper weave docker-bridge-ip to find this address, and it generally won't change.
Related
I'm trying to run a continuous integration in GitLab CI consisting of:
build the docker image
run tests
push the docker image to a registry
Those are running inside one job. I can do it without any problem until come up some test that needs to communicate with database. My container can't communicate with Postgres services defined.
I've reproduce it in a public repository with simple ping script
image: docker:stable
services:
- docker:dind
- postgres:latest
job1:
script:
- ping postgres -c 5
- docker run --rm --network="host" alpine:latest sh -c "ping postgres -c 5"
The first script could run without any problem, but the second one failed with error
ping: bad address 'postgres'
How can I access the service?
Or should I run the test in a different job?
The solution is to use --add-host=postgres:$POSTGRES_IP to pass over the ip address present in job container.
To find out postgres ip linked to the outer container you can use for example getent hosts postgres | awk '{ print $1 }'
So the yml would look like
image: docker:stable
services:
- docker:dind
- postgres:latest
job1:
script:
- ping postgres -c 5
- docker run --rm --add-host=postgres:$(getent hosts postgres | awk '{ print $1 }') alpine:latest sh -c "ping postgres -c 5"
To understand why the other more common ways to connect containers wont work in this case, we have to remember we are trying to link a nested container with a service linked to its "parent". Something like this:
gitlab ci runner --> docker -> my-container (alpine)
-> docker:dind
-> postgres
So we are trying to connect a container with its "uncle". Or connecting nested containers
As noted by #tbo, using --network host will not work. This is probably because gitlab ci use --link (as explained here) to connect containers instead of the newer --network. The way --link works makes that the services containers are connected to the job container, but not connected with one another. So using host network wont make the nested container inherit postgres hostname.
One could also think that using --link postgres:postgres would work, but it also won't as in this environment postgres is only a hostname with the ip of the container outside. There is not container here to be linked with the nested container
So all we can do is manually add a host with the correct ip to the nested container using --add-host as explained above.
I have this CircleCI configuration.
version: 2
jobs:
build:
docker:
- image: docker:18.09.2-git
- image: docker.elastic.co/elasticsearch/elasticsearch:6.6.0
name: elasticsearch
working_directory: ~/project
steps:
- checkout
- setup_remote_docker:
docker_layer_caching: true
- run:
name: test
command: |
docker run --rm \
--network host \
byrnedo/alpine-curl \
elasticsearch:9200
I'm looking for a way to allow my new container to access to the elasticsearch port 9200. With this configuration, the elasticsearch is not even a known host name.
Creating an extra network is not possible, so I have this error message container sharing network namespace with another container or host cannot be connected to any other network
Host network seems to be working only in the primary image
How could I do this?
That will not work. Containers started during a build via the docker run command are running via a remote Docker engine. The cannot talk to the containers running as part of the executor via TCP since they are isolated. Just docker exec.
The solution will ultimately depend on your end goal, but one option might be to remove the Elasticsearch image/container from the executor, and use Docker Compose to get both images to talk to each other within the build.
Let say I have running orchestration with docker-compose with docker-compose.yml looking like this:
version: '2.2'
services:
service1:
# ...
networks:
- compose_network
service2:
# ...
networks:
- compose_network
networks:
compose_network:
I aim to run and connect temporarily one container to compose_network_1. I tried using
$ docker run --net=compose_network <image for the job>
but I could not connect. I am also aware that docker-compose names the networks as [projectname]_default, so I also tried that variant, but with same result.
Is there a way I can accomplish that?
I'm not sure if the --net option ever existed but it's now --network.
From docker run --help:
--network string Connect a container to a network (default "default")
As #maxm notes you can find the network name, with the DIR prefix of the compose project directory, then simply run it as you were trying:
$ docker run --network=DIR_compose_network <image for the job>
I wanted to connect on run as my container is transient (running tests) so I can't use a second docker network command in time before it quits.
e.g. for my docker composition in a "dev" folder with no network name specified so uses the docker-compose "default" name, therefore I get the name dev_default.
docker network ls
NETWORK ID NAME DRIVER SCOPE
2c660d9ed0ba bridge bridge local
b81db348e773 dev_default bridge local
ecb0eb6e93a5 host host local
docker run -it --network dev_default myimage
This connects the new docker container to the existing docker-compose network.
The network name is going to be something like name-of-directory_compose_network. Find the name with docker network ls
I had success with:
docker-compose up # within directory ./demo
docker run -itd -p "8000:8000" --hostname=hello "crccheck/hello-world"
# outputs: 1e502f65070c9e2da7615c5175d5fc00c49ebdcb18962ea83a0b24ee0440da2b
docker network connect --alias hello demo_compose_network 1e502f65070c
I could then curl hello:8000 from inside my docker compose containers. Should be the exact same functionality as your commands, just with an added alias.
When creating a Jenkins Docker container, it is very useful to able to connect to the Docker daemon. In that way, I can start docker commands inside the Jenkins container.
For example, after starting the Jenkins Docker container, I would like to 'docker exec -it container-id bash' and start 'docker ps'.
On Linux you can use bind-mounts on /var/run/docker.sock. On Windows this seems not possible. The solution is by using 'named pipes'. So, in my docker-compose.yml file I tried to create a named pipe.
version: '2'
services:
jenkins:
image: jenkins-docker
build:
context: ./
dockerfile: Dockerfile_docker
ports:
- "8080:8080"
- "50000:50000"
networks:
- jenkins
volumes:
- jenkins_home:/var/jenkins_home
- \\.\pipe\docker_engine:\\.\pipe\docker_engine
# - /var/run/docker.sock:/var/run/docker.sock
# - /path/to/postgresql/data:/var/run/postgresql/data
# - etc.
Starting docker-compose with this file, I get the following error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is
the docker daemon running?
How can I setup the docker-compose file so that I can use the docker.sock (or Docker) inside the started container?
On Linux you can use something like volumes: /var/run/docker.sock:/var/run/docker.sock. This does not work in a Windows environment. When you add this folder (/var) to Oracle VM Virtualbox, it won't get any IP forever. And on many posts
You can expose the daemon on tcp://localhost:2375 without TLS in the settings. This way you can configure Jenkins to use the Docker API instead of the socket. I encourage you to read this article by Nick Janetakis about "Understanding how the Docker Daemon and the Docker CLI work together".
And then there are several Docker plugins for Jenkins that allows this connection:
Also, you can find additional information in the Docker plugin documentation on wiki.jenkins.io:
def dockerCloudParameters = [
connectTimeout: 3,
containerCapStr: '4',
credentialsId: '',
dockerHostname: '',
name: 'docker.local',
readTimeout: 60,
serverUrl: 'unix:///var/run/docker.sock', // <-- Replace here by the tcp address
version: ''
]
EDIT 1:
I don't know if it is useful, but the Docker Daemon on Windows is located to C:\ProgramData\docker according to the Docker Daemon configuration doc.
EDIT 2:
You need to say explicitly the container to use the host network because you want to expose both Jenkins and Docker API.
Following this documentation, you only have to add --network=host (or network_mode: 'host' in docker-compose) to your container/service. For further information, you can read this article to understand what is the purpose of this network mode.
First try was to start a Docker environment using "Docker Quickstart terminal". This is a good solution when running Docker commands within that environment.
When installing a complete CI/CD Jenkins environment via Docker means that WITHIN the Jenkins Docker container you need to access the Docker daemon. After trying many solutions, reading many posts, this did not work. #Paul Rey, thank you very much for trying all kinds of routes.
A good solution is to get an Ubuntu Virtual Machine and install it via the Oracle VM Virtualbox. It is then VERY IMPORTANT to install Docker via this official description.
Before installing Docker, of course you need to install Curl, Git, etc.
I'm having an issue here with the ansible docker_container command.
- name: Start Docker
docker_container:
name: postgres
image: "centos/postgresql-94-centos7"
state: started
recreate: yes
pull: yes
env:
POSTGRESQL_USER: "test"
POSTGRESQL_PASSWORD: "test"
POSTGRESQL_DATABASE: "test"
POSTGRESQL_MAX_CONNECTIONS: 100
POSTGRESQL_SHARED_BUFFERS: 32M
exposed_ports:
- 5432
ports:
- 5432:5432
become: yes
become_method: sudo
When this command runs, it boots the container, but the container immediately exits with some output that I can't see because ansible doesn't print sys out for that for some reason.
Here's an example of some output that is printed when the container refuses to boot (due to incorrect config):
$ sudo docker run --name postgres -e POSTGRESQL_USER="test" -p 5432:5432 centos/postgresql-94-centos7
You must either specify the following environment variables:
POSTGRESQL_USER (regex: '^[a-zA-Z_][a-zA-Z0-9_]*$')
POSTGRESQL_PASSWORD (regex: '^[a-zA-Z0-9_~!##$%^&*()-=<>,.?;:|]+$')
POSTGRESQL_DATABASE (regex: '^[a-zA-Z_][a-zA-Z0-9_]*$')
Or the following environment variable:
POSTGRESQL_ADMIN_PASSWORD (regex: '^[a-zA-Z0-9_~!##$%^&*()-=<>,.?;:|]+$')
Or both.
Optional settings:
POSTGRESQL_MAX_CONNECTIONS (default: 100)
POSTGRESQL_MAX_PREPARED_TRANSACTIONS (default: 0)
POSTGRESQL_SHARED_BUFFERS (default: 32MB)
For more information see /usr/share/container-scripts/postgresql/README.md
within the container or visit https://github.com/openshift/postgresql.
This is the output that I do not see when I run it from ansible, it just says its up (success) and then I check docker ps and its not up because presumably it immediately exited. What's more weird is that I can get this to work just fine with the same configuration passed to the ansible command module, but with the docker_container maybe its not passing the env correctly? I'm really not sure because I can't even get the error output.
Check
docker ps -a
to see if ansible managed to at least create the container. If it did, check
docker logs postgres
Hopefully that should give you a clue where it's failing. If you're confused with that maybe you can post some of the (redacted) logs to show where it's failing?
You can do this with registered.ansible_facts.docker_container.Output, where registered is the fact you used to register your docker_container play with.
More information here: https://wordpress.com/post/carlosonunez.wordpress.com/1440