Is there any way to obtain detailed logging info when executing 'docker stack deploy'? - docker

In Docker 17.03, when executing
docker stack deploy -c docker-compose.yml [stack-name]
the only info that is output is:
Creating network <stack-name>_myprivatenet
Creating service <stack-name>_mysql
Creating service <stack-name>_app
Is there a way to have Docker output more detailed info about what is happening during deployment?
For example, the following information would be extremely helpful:
image (i.e. 'mysql' image) is being downloaded from the registry (and provide the registry's info)
if say the 'app' image is unable to be downloaded from its private registry, that an error message (i.e. due to incorrect or omitted credentials - registry login required) be output
Perhaps it could be provided via either of the following ways:
docker stack deploy --logs
docker stack log
Thanks!

docker stack logs is actually a requested feature in issue 31458
request for a docker stack logs which can show the logs for a docker stack much like docker service logs work in 1.13.
docker-compose works similarly today, showing the interleaved logs for all containers deployed from a compose file.
This will be useful for troubleshooting any kind of errors that span across heterogeneous services.
This is still pending though, because, as Drew Erny (dperny) details:
there are some changes that have to be made to the API before we can pursue this, because right now we can only get the logs for 1 service at a time unless you make multiple calls (which is silly, because we can get the logs for multiple services in the same stream on swarmkit's side).
After I finish those API changes, this can be done entirely on the client side, and should be really straightforward. I don't know when the API changes will be in because I have started yet, but I can let you know as soon as I have them!

Related

Unable to pull logs from Airflow Worker

I've got a simple docker development setup for Airflow that includes separate containers for the Airflow UI and Worker. I'm encountering a 403 Forbidden error whenever I attempt to view the log for a task in the Airflow UI.
So far I've ensured they all have the same secret key (in fact, using Docker Volumes they're all reading the exact same configuration file) but this doesn't seem to help. I haven't done anything about time sync, but I'd expect that docker containers would effectively be sharing the system clock anyway so I don't see how they'd get out of sync in the first place.
I can find the log file on the airflow worker, and it has run successfully - but something is obviously missing that should be allowing the airflow UI to display that (and it would be much more convenient for my workflow to be able to see the logs in the UI rather than having to rummage around on the worker).

Rsyslog can't start inside of a docker container

I've got a docker container running a service, and I need that service to send logs to rsyslog. It's an ubuntu image running a set of services in the container. However, the rsyslog service cannot start inside this container. I cannot determine why.
Running service rsyslog start (this image uses upstart, not systemd) returns only the output start: Job failed to start. There is no further information provided, even when I use --verbose.
Furthermore, there are no error logs from this failed startup process. Because rsyslog is the service that can't start, it's obviously not running, so nothing is getting logged. I'm not finding anything relevant in Upstart's logs either: /var/log/upstart/ only contains the logs of a few things that successfully started, as well as dmesg.log which simply contains dmesg: klogctl failed: Operation not permitted. which from what I can tell is because of a docker limitation that cannot really be fixed. And it's unknown if this is even related to the issue.
Here's the interesting bit: I have the exact same container running on a different host, and it's not suffering from this issue. Rsyslog is able to start and run in the container just fine on that host. So obviously the cause is some difference between the hosts. But I don't know where to begin with that: There are LOTS of differences between the hosts (the working one is my local windows system, the failing one is a virtual machine running in a cloud environment), so I wouldn't know where to even begin about which differences could cause this issue and which ones couldn't.
I've exhausted everything that I know to check. My only option left is to come to stackoverflow and ask for any ideas.
Two questions here, really:
Is there any way to get more information out of the failure to start? start itself is a binary file, not a script, so I can't open it up and edit it. I'm reliant solely on the output of that command, and it's not logging anything anywhere useful.
What could possibly be different between these two hosts that could cause this issue? Are there any smoking guns or obvious candidates to check?
Regarding the container itself, unfortunately it's a container provided by a third party that I'm simply modifying. I can't really change anything fundamental about the container, such as the fact that it's entrypoint is /sbin/init (which is a very bad practice for docker containers, and is the root cause of all of my troubles). This is also causing some issues with the docker logging driver, which is why I'm stuck using syslog as the logging solution instead.

How to extract docker-compose file from running docker start

I have docker stack started with docker stack deploy --compose-file ...
and later manually edited via Docker Portainer UI.
I'd like to write a script that updates the docker image tag of one of the services.
To do that I need to "download" the latest "docker-compose" stack definition however I cannot find the appropriate docker command.
I do know that the best would be to stop changing stack manually and rely on its definition stored in git but unfortunately, it is not up to me.
Please point me to the appropriate docker command or confirm that it is not available.
As far as i know there is no command you could get the compose file from the running container directly. At least not implemented out of the box in docker. You could try to parse all the relevant information from docker inspect and few other commands to list/inspect all relevant objects?.
I have once came across the similar situation where we had a running container but no run/compose command which we needed to update. At the time (roughly a year ago) i found and used docker-autocompose which did very good job. We only had to manually verify and adjust few things,but it got all the difficult parts with run parameters done for us.
It could help in your case to automate it if your compose configs are simple enough.
But if you wanted to fully automate it to mimic CD, then i would not recommend the approach above. In that case i would check if you could use portainer api as #LinFelix recommended. Or store compose files somewhere - prepared with parameters ($IMAGE_TAG) (git/on server) so you can then generate temporary compose files with all configuration and then remove the current one.

Why docker not support the http api for docker stack?

I checked docker API doc:api/v1.40 and I couldn't find API support for docker stack stuff.
After I googled it, I found these GitHub issues:
Feature Req: docker stack deploy into API
Roadmap for docker deploy and Bundles
I still don't understand why docker stack stuff isn't supported.
If I want to manage my stack through HTTP API, what am I supposed to do? It looks like I can only ssh to the machine in a certain way to execute commands docker stack ls if I want to list all stacks on my web management page.
Also, I'd like to know how a product like portainer achieves this function?
Any answer will be appreciated.

what's the BestPractice for Docker logging?

Im using docker with my Web service.
when I deploy using Docker, loosing some logging files (nginx accesslog, service log, system log.. etc)
Cause, docker deployment system using down and up container architecures.
So I thought about this problem.
LoggingServer and serviceServer(for api) must seperate!
using these, methods..
First, Using logstash(in elk)(attaching all my logFile) .
Second, Using batch system, this batch system will moves logfiles to otherServer on every midnight.
isn't it okay?
I expect a better answer.
thanks.
There are many ways for logging which most the admin uses for containers
1 ) mount log directory to host , so even if docker goes up/down logs will be persisted on host.
2) ELK server, using logstash/filebeat for pushing logs to elastic search server with tailing option of file, so if new log contents it pushes to server.
3) if there is application logs like maven based projects, then there are many plugins which pushes logs to server
4) batch system , which is not recommended because if containers dies before mid-night then logs will be lost.

Resources