I installed docker and issues a 'docker swarm init' command.
I'm trying to launch a stack using the following command: docker stack deploy -c docker-compose.yml mystack
The docker-compose file can be found here, the first docker file here and the second here
The output of 'docker ps' is:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f582b3e8d33e tons/ip2country:latest "/bin/sh -c 'java -D…" 8 seconds ago Up 6 seconds 8080/tcp ip2flag_country-service.1.t5rvuqaw8tj7v20u0xo0dgy6x
bbf2c8304f1a tons/ip2flag:latest "/bin/sh -c 'java -D…" 10 seconds ago Up 8 seconds 8080/tcp ip2flag_app.1.z00gz8adj2yshpgimaw2o55d3
cbc7eaace4bf portainer/portainer "/portainer" 39 minutes ago Up 39 minutes 0.0.0.0:9000->9000/tcp portainer
The output of 'docker service ls' is:
ID NAME MODE REPLICAS IMAGE PORTS
ex51pyh1oyyo ip2flag_app replicated 1/1 tons/ip2flag:latest *:8080->8080/tcp
yhbt97lmjqan ip2flag_country-service replicated 1/1 tons/ip2country:latest
Since I'm running this on localhost I'd expect http://localhost:8080/ to return some sort of data. But it just times out. If I attach to the container and execute something like wget localhost:8080/some/path it works as expected. So the service is running and within the container listening to port 8080. However the port isn't exposed outside of dockers net. Further more I can add that launching with 'docker-compose up' works just fine too. But not with 'docker stack deploy'. Any clue about what I'm doing wrong?
Related
I am using the official docker-compose file of airflow to spin it up.
Some of my containers seem unhealthy:
34d8698d67e7 apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 28 minutes (unhealthy) 0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp airflow_flower_1
a291cf238b9f apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 29 minutes 8080/tcp airflow_airflow-init_1
fdb20e9152f3 apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 29 minutes (unhealthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp airflow_airflow-webserver_1
abf5a16aa846 apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 29 minutes 8080/tcp airflow_airflow-worker_1
f6dc352f407b apache/airflow:2.0.2 "/usr/bin/dumb-init …" 31 minutes ago Up 28 minutes 8080/tcp airflow_airflow-scheduler_1
12dfc71e518f redis:latest "docker-entrypoint.s…" 31 minutes ago Up 29 minutes (healthy) 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp airflow_redis_1
However the logs of one of them for example do not seem very informative.
# docker logs -f fdb20e9152f3
WARNING! You should run the image with GID (Group ID) set to 0
even if you use 'airflow' user (UID=50000)
You started the image with UID=50000 and GID=50000
This is to make sure you can run the image with an arbitrary UID in the future.
See more about it in the Airflow's docker image documentation
http://airflow.apache.org/docs/docker-stack/entrypoint
BACKEND=postgresql+psycopg2
DB_HOST=my-db-endpoint
DB_PORT=5432
WARNING! You should run the image with GID (Group ID) set to 0
even if you use 'airflow' user (UID=50000)
You started the image with UID=50000 and GID=50000
This is to make sure you can run the image with an arbitrary UID in the future.
See more about it in the Airflow's docker image documentation
http://airflow.apache.org/docs/docker-stack/entrypoint
BACKEND=postgresql+psycopg2
DB_HOST=my-db-endpoint
DB_PORT=5432
Regardless of any airflow - specific issues, how can I check docker - wise what's going on?
Docker seems to be aware of a couple of containers not being healty.
edit: both failing containers have the healtcheck condition
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
and
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/"]
that seems to be failing by looking into their inspect output
Failed to connect to localhost port 8080: Connection refused
but I cannot pinpoint what is causing the failure.
edit: I have tried following the instructions to start the init service first as well
# docker-compose up airflow-init
Starting airflow_redis_1 ... done
Starting airflow_airflow-init_1 ... done
Attaching to airflow_airflow-init_1
airflow-init_1 | BACKEND=postgresql+psycopg2
airflow-init_1 | DB_HOST=my-db-endpoint
airflow-init_1 | DB_PORT=5432
but it never exits, it prints the above message and that's it...
I ran into similar issue and it was docker volume causing the issue. As I was running lots of containers on my mac, there wasn;t enough disk space. I managed to fixed this issue my pruning the docker volume.
docker volume prune
This will remove any unused volume on your mac book. Before running this command please check if you got any useful data.
For docker-compose, from the entrypoint, the default value of group id is 0.
"${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}"
Edit your docker-compose.yaml file or ad ass env.sh file in your Airflow project repository.
It does seems to be an error due to less memory allocated to docker for running this image, try to increase the resources available to docker and see the magic
I ran into similar issue and the healthchecks were "causing" this. I was running them using default container's user.
Just to give a try, I changed the healthckeck command to start using airflow user instead, as follow:
$ runuser -u airflow -- <healthckeck command>
And it solved. I'm gonna change the user whom runs docker compose up to airflow from now on.
Solved at bottom
But why do I have to append :4000?
I'm following the docker get-started Guide here, https://docs.docker.com/get-started/part4/
I'm fairly certain I've done everything correctly, but am wondering why I can't connect to view the app after deploying it.
I've set my env to my VM, myvm1, for reference to following commands.
docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
099e16249604 beresj/getting-started:part2 "python app.py" 12 seconds ago Up 12 seconds 80/tcp getstartedlab_web.5.y0e2k1r1ev47u24e5iufkyn3i
6f9a24b343a7 beresj/getting-started:part2 "python app.py" 12 seconds ago Up 12 seconds 80/tcp getstartedlab_web.3.1pls3osj3uhsb5dyqtt4ts8j6
docker image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
beresj/getting-started <none> e290b6208c21 22 hours ago 131MB
docker stack ls
NAME SERVICES ORCHESTRATOR
getstartedlab 1 Swarm
docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
myvm1 * virtualbox Running tcp://192.168.99.100:2376 v18.09.6
myvm2 - virtualbox Running tcp://192.168.99.101:2376 v18.09.6
docker stack ps getstartedlab
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
vkxx79fh3h85 getstartedlab_web.1 beresj/getting-started:part2 myvm2 Running Running 3 minutes ago
qexbaa3wz0pd getstartedlab_web.2 beresj/getting-started:part2 myvm2 Running Running 3 minutes ago
1pls3osj3uhs getstartedlab_web.3 beresj/getting-started:part2 myvm1 Running Running 3 minutes ago
ucuwen1jrncf getstartedlab_web.4 beresj/getting-started:part2 myvm2 Running Running 3 minutes ago
y0e2k1r1ev47 getstartedlab_web.5 beresj/getting-started:part2 myvm1 Running Running 3 minutes ago
curl 192.168.99.100
curl: (7) Failed to connect to 192.168.99.100 port 80: Connection refused
docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 1
Server Version: 18.09.6
...
Swarm: active
NodeID: 0p9qrax9h3by0fupat8ufkfbq
Is Manager: true
ClusterID: 7vnqdk85n8jx6fqck9k7dv2ka
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
...
Node Address: 192.168.99.100
Manager Addresses:
192.168.99.100:2377
...
Kernel Version: 4.14.116-boot2docker
Operating System: Boot2Docker 18.09.6 (TCL 8.2.1)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 989.4MiB
Name: myvm1
I would expect to see what I was able to see when I just ran it on my local machine instead of on a VM in a swarm (I think I have the lingo correct?)
Not sure how to check open ports.
Again: this works if I simply remove the stack, unset the docker-machine environment, and just run:
docker stack deploy -c docker-compose.yml getstartedlab
not on the vm.
Thank you in advance. (Also, I'm new hence the get-started guide so I appreciate any help)
Edit
It works if I append :4000 to the VM IP in my url, ex: 192.168.99.100:4000 or 192.168.99.101:4000. It shows the two container Id's listed in 'docker container ls' for myvm1, and the other three are from myvm2. Could anyone tell me why I have to append 4000? Is it because I have ports: "4000:80" in my docker-compose.yml?
Not sure if this will help but if you use docker inspect <instance_id_here>, you can see what ports are exposed.
Exposed ports aren't open ports. You would need to bind a host port to a container port in the docker-compose.yml in order for it to be to be open.
In Docker Swarm mode, how can I restart a single global service? Is it even possible? I know you can scale replicated services to zero then back to 1+, but there doesn't appear to be any documentation on how to have the same effect with global services.
I am updating my SSL certificate so would like to just restart our reverse proxy instead of restarting our entire app (via restarting the docker service).
The docs just mention you cannot scale global services:
The scale command enables you to scale one or more replicated services either up or down to the desired number of replicas. This command cannot be applied on services which are global mode.
You can force a rolling update of a service, either globally scheduled or replicated using docker service update --force ${service_name}. Here's an example compose file:
version: '3'
services:
busybox-global:
image: busybox
command: tail -f /dev/null
deploy:
mode: global
busybox-replicated:
image: busybox
command: tail -f /dev/null
deploy:
replicas: 2
Verify it has started:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
917aefdc910b busybox:latest "tail -f /dev/null" 50 seconds ago Up 31 seconds sched_busybox-global.q44zx0s2lvu1fdduk800e5ini.hzn6jnzh7x539timamphzzw8a
7187fbbde0da busybox:latest "tail -f /dev/null" About a minute ago Up 31 seconds sched_busybox-replicated.1.i4nm7lpr1spmf0aorh1dtcqrc
f04a0062b088 busybox:latest "tail -f /dev/null" About a minute ago Up 31 seconds sched_busybox-replicated.2.oc6zn0ziqg9wyzofokek8eb24
$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
gto0d5a6betb sched_busybox-global global 1/1 busybox:latest
yfq5mne0qhtj sched_busybox-replicated replicated 2/2 busybox:latest
$ docker service ps sched_busybox-global
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
hzn6jnzh7x53 sched_busybox-global.q44zx0s2lvu1fdduk800e5ini busybox:latest bmitch-asusr556l Running Running 49 seconds ago
Force the rolling update:
$ docker service update --force sched_busybox-global
sched_busybox-global
overall progress: 1 out of 1 tasks
q44zx0s2lvu1: running [==================================================>]
verify: Service converged
$ docker service ps sched_busybox-global
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
zcfocrfjvvux sched_busybox-global.q44zx0s2lvu1fdduk800e5ini busybox:latest bmitch-asusr556l Running Running 7 seconds ago
hzn6jnzh7x53 \_ sched_busybox-global.q44zx0s2lvu1fdduk800e5ini busybox:latest bmitch-asusr556l Shutdown Shutdown 10 seconds ago
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3c5fe0f79e3d busybox:latest "tail -f /dev/null" About a minute ago Up About a minute sched_busybox-global.q44zx0s2lvu1fdduk800e5ini.zcfocrfjvvuxz6tkge0pn0bq2
917aefdc910b busybox:latest "tail -f /dev/null" 3 minutes ago Exited (137) About a minute ago sched_busybox-global.q44zx0s2lvu1fdduk800e5ini.hzn6jnzh7x539timamphzzw8a
7187fbbde0da busybox:latest "tail -f /dev/null" 3 minutes ago Up 2 minutes sched_busybox-replicated.1.i4nm7lpr1spmf0aorh1dtcqrc
f04a0062b088 busybox:latest "tail -f /dev/null" 3 minutes ago Up 2 minutes sched_busybox-replicated.2.oc6zn0ziqg9wyzofokek8eb24
The same would have worked if I forced an update to the replicated service.
This is what ps -a gives us
NAMES
4514ea1b7b22 debian "--name gallant_spen…" 9 minutes ago Created peaceful_engelbart
df9bd2731a2b debian "--name gallant_spen…" 9 minutes ago Created happy_hodgkin
dd5b1f1b39ec redis "docker-entrypoint.s…" 32 minutes ago Up 31 minutes 6379/tcp myred
ffd6ef9d8bd5 redis "docker-entrypoint.s…" 32 minutes ago Exited (127) 32 minutes ago festive_jennings
9d01d321adad redis "docker-entrypoint.s…" 33 minutes ago Exited (0) 32 minutes ago agitated_shannon
eb7c13e7cdee debian "ls /data" 2 days ago Exited (0) 9 seconds ago gallant_spence
8991a31b1e38 debian "ls /data" 2 days ago Exited (0) 2 days ago determined_minsky
I have tried in this manner
docker start `docker ps -q -l` gallant_spence
But error ocurrs
Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"--name\": executable file not found in $PATH": unknown
gallant_spence
Error: failed to start containers: 4514ea1b7b22
I am interested in data volume that has been mounted in my previous work on this container
"Mounts": [
{
"Type": "bind",
"Source": "/home/mm/code/lesson_04",
"Destination": "/data",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
],
If I try
mm#6830s:~$ docker start -ai gallant_spence
one_sample.py
parallel_series.py
Python files are added to data folder so I thought that it worked.Then I try again
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dd5b1f1b39ec redis "docker-entrypoint.s…" About an hour ago Up About an hour 6379/tcp myred
Hot to fix this?
You start a stopped container with
docker start [OPTIONS] CONTAINER [CONTAINER...]
For your case, you can use:
docker start gallant_spence or
docker start eb7c13e7cdee
As it is shown by the docker ps -a result, your container is configured with this CMD:
"ls /data"
This means that every time you start your container, this command will run and the container will then exit. This is how containers work. When their primary process finishes, they exit.
About the error you get and docker ps -q -l:
-q (from "quiet") tells the command to only print the container IDs
-l (from "last") tells the command to print the latest created container
This means that the above command brings you back: 4514ea1b7b22. If we put together all things...
your command:
docker start `docker ps -q -l` gallant_spence
turns to:
docker start 4514ea1b7b22 gallant_spence
The fail message you get is for 4514ea1b7b22 because it doesn't have a CMD set properly to start. I see something like: --name gallant_spen…
I am trying to access a running app on a port that I defined using "EXPOSE".
Here is what I get:
docker#boot2docker:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
653d8ee23260 nginx:latest "nginx -g 'daemon of 2 minutes ago Up 2 minutes 80/tcp, 443/tcp
insane_thompson
007cfcd0f539 highlighter:latest "java -jar -Xmx1500m 8 minutes ago Up 8 minutes 7777/tcp
elated_kirch
docker#boot2docker:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
653d8ee23260 nginx:latest "nginx -g 'daemon of 2 minutes ago Up 2 minutes 80/tcp, 443/tcp insane_thompson
007cfcd0f539 highlighter:latest "java -jar -Xmx1500m 8 minutes ago Up 8 minutes 7777/tcp elated_kirch
docker#boot2docker:~$ docker port 007cfcd0f539
docker#boot2docker:~$ docker port 653d8ee23260
docker#boot2docker:~$ docker port 653d8ee23260 80
FATA[0000] Error: No public port '80/tcp' published for 653d8ee23260
docker#boot2docker:~$ docker port 007cfcd0f539 7777
FATA[0000] Error: No public port '7777/tcp' published for 007cfcd0f539
Am I misunderstanding how the "port" command works?
EXPOSE in Dockerfile is not enough.
You need to specifically tell docker to expose port upon docker run with the -P/-p flags.
A much more detailed answer can be found here.