1) I am running Grafana v6.7.2 from Docker.
2) I wanted to enable grafana log. Since I am running from Docker, /etc/grafana/grafana.ini is read only
3) Now, cloned that grafana.ini to my host where docker is running from. I un-commented this line to enabling logging: logs = /var/log/grafana
#################################### Paths ####################################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
;data = /var/lib/grafana
# Temporary files in `data` directory older than given duration will be removed
;temp_data_lifetime = 24h
# Directory where grafana can store logs
logs = /var/log/grafana
4) I made sure to stop Grafana container. Then, issued following command to to re-start Grafana. This time it has volume mapping for config:
docker run -d -p 3000:3000 -v "$PWD/grafana.ini:/etc/grafana/grafana.ini" -v grafana-storage:/var/lib/grafana grafana_internal:latest
5) I made sure Grafana container running, and I can access the UI
6) Then, I went here to see if log is generated: /var/log/grafana/ using docker exec <yourimage> ls /var/log/grafana
The issue is that there was no Grafana log. Now, this led me to believe config volume mapping may not be working as expected.
Any pointers would be helpful.
thanks.
If you look at the running grafana instance using e.g. ps, you'll see this:
$ ps -fe | grep grafana
1 grafana 0:00 grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --packaging=docker cfg:default.log.mode=console cfg:default.paths.data=/var/lib/grafana cf
g:default.paths.logs=/var/log/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning
If you take a close look at those config options, you'll see:
cfg:default.log.mode=console
That means that Grafana will log only to the console. You can inspect these logs using docker logs. There's not really any reason to have Grafana log to a file also (or instead of).
If you really want Grafana to log to a file, you need to include the following in your grafana.ini:
[log]
mode = console file
With this in my grafana.ini, I see output on the docker console and I see logs in /var/log/grafana/grafana.log.
But like I said, I don't see any point in create the logfile when you can capture the same information from docker logs.
Related
I use Dokku to run my app, and for some reason, the container is dying every few hours and recreates itself.
In order to investigate the issue, I am willing to read the error logs to this container and understand why it's crashing. Since Docker clears logs of dead containers, this is impossible.
I turned on docker events and it shows many events (like container update, container kill, container die, etc.) But no sign of what triggered this kill.
How can I investigate the issue?
Versions:
Docker version 19.03.13, build 4484c46d9d
dokku version 0.25.1
Logs are deleted when the container is deleted. If you want the logs to persist, then you need to avoid deleting the container. Make sure you aren't running the container with an option like --rm that automatically deletes it on exit. And check for the obvious issues like running out of disk space.
There are several things you can do to investigate the issue:
You can run the container in the foreground and allow it to log to your console.
If you were previously starting the container in the background with docker run -d (or docker-compose up -d), just remove the -d from the command line and allow the container to log to your terminal. When it crashes, you'll be able to see the most recent logs and scroll back to the limits of your terminal's history buffer.
You can even capture this output to a file using e.g. the script tool:
script -c 'docker run ...`
This will dump all the output to a file named typescript, although you can of course provide a different output name on the command line.
You can change the log driver.
You can configure your container to use a different logging driver. If you select something like syslog or journald, your container logs will be sent to the corrresponding service, and will continue to be available even after the container has been deleted.
I like use the journald logging driver because this allows searching for output by container id. For example, if I start a container like this:
docker run --log-driver journald --name web -p 8080:8080 -d docker.io/alpinelinux/darkhttpd
I can see logs from that container by running:
$ journalctl CONTAINER_NAME=web
Feb 25 20:50:04 docker 0bff1aec9b65[660]: darkhttpd/1.13, copyright (c) 2003-2021 Emil Mikulic.
These logs will persist even after the container exits.
(You can also search by container id instead of name by using CONTAINER_ID_FULL (the full id) or CONTAINER_ID (the short id), or even by image name with IMAGE_NAME.)
I encountered an issue when running the docker container.
An error log was generated as below:
[Error] mysqld : unknown variable “wait_timeout = 288000”.
I wanted to test some docker container features.
So, I opened the docker bash and entered the directory /etc/mysql/my.cnf.
And I added the variable “wait_timeout = 288000” below [mysqld] option.
However, after rebooting, when I ran the container, it exited immediately with status code (1).
I knew that the error was caused by the variable I just added.
So, I wanted to delete the variable, but now the docker container bash won’t open.
Is there any way that I can delete the variable “wait_timeout” in this case?
If there isn’t, could you recommend other methods for troubleshooting?
Thanks for checking the issue.
Delete and recreate the container, and it will start fresh from a clean container filesystem.
That is probably also a better way to modify the database configuration (if you do, in fact, need a custom my.cnf). You can bind-mount a directory of configuration files into the container at startup time:
docker run -d -p 3306:3306 --name mysql \
-v $PWD/mysql-conf:/etc/mysql/conf.d \
mysql:8
Then when the configuration changes, you can delete and recreate this container:
docker stop mysql
docker rm mysql
docker run -d -p 3306:3306 ... mysql:8 # as above
(See "Using a custom MySQL configuration file" in the Docker Hub mysql image page for more information.)
Deleting and recreating Docker containers is very routine, and one of the benefits is that when a new container starts, it always has a "clean" filesystem. This particular setup also makes sure the modified configuration file is stored outside the container, so if you are forced to recreate the container (to upgrade MySQL to get a critical security fix, for example) it's something you're used to doing and you won't lose data or settings.
I am running a deployment which contains three containers the app, nginx and cloud sql instance. I have a lot of print statements in my python based app.
Every time a user interacts with the app, outputs are printed. I want to know if these logs are saved by default at any location.
I am worried that these logs might consume the space on the nodes in the cluster running it. Does this happen ? or Kubernetes deployments by default don't save any logs by default?
The applications run in containers usually under Docker and the stdout/stderr logs are saved for the lifetime of the container in the graph directory (usually /var/lib/docker)
You can look at the logs with either:
$ kubectl logs <pod-name> -c <container-in-pod>
Or:
$ ssh <node>
$ docker logs <container>
If you'd like to know more where they are stored you can go into the /var/lib/docker directory and see the logs stored in JSON format:
$ cd /var/lib/docker/containers
$ find . | grep json.log
./3454a0681100986248fd81856fadfe7cd95a1a6467eba32adb33da74c2c5443d/3454a0681100986248fd81856fadfe7cd95a1a6467eba32adb33da74c2c5443d-json.log
./80a87a9529a55f8d3fb9b814f0158dc91686704222e252b256455bcde48f56a5/80a87a9529a55f8d3fb9b814f0158dc91686704222e252b256455bcde48f56a5-json.log
...
If you'd like to do garbage collection on 'Exited' containers you can read more about it here.
Another way is to set up a cron job that runs periodically on your nodes that does this:
$ docker system prune -a --force
I'm new to log4j2 and the elastic stack.
I have a filebeat docker container that doesn't work exactly how I want and now I want to take a look at the logs. But when I do docker-compose logs I get a lot debug messages and json objects. It's unreadable how much there is.
How can I create a log4j2 properties setup to create some rolling log files. Maybe put the old logs into a monthly based folder or something? and where do I put this log4j2.properties file?
It's generating a lot of logs because you're running docker-compose logs, which will get the logs for all containers in your docker compose file.
What you want is probably:
docker logs <name-of-filebeat-container>. The name of the filebeat container can be found doing a docker ps.
docker compose logs <name-of-filebeat-service>. The name of the service can be found on your docker-composer.yml file.
Regarding the JSON outputs, you can query your Docker engine default logging driver with:
# docker info | grep 'Logging Driver'
Logging Driver: json-file
If your container have a different Logging Driver you can check with:
docker inspect -f '{{.HostConfig.LogConfig.Type}}' <name-or-id-of-the-container>
You can find all log drivers in this link
To run containers with a different log-driver you can do:
With docker run: docker run -it --log-driver <log-driver> alpine ash
With docker-compose:
`logging:
driver: syslog
options:
syslog-address: "tcp://192.168.0.42:123"`
Regarding your log rotation questio, I'd say the easyest way is to configure the logging driver with the syslog driver, configure it to your local machine (or your syslog server) and then logrotate the files.
You can find several logrotate articles for Linux (which I assume you're using), for example this one
I spent the weekend pouring over the Docker docs and playing around with the toy applications and example projects. I'm now trying to write a super-simple web service of my own and run it from inside a container. In the container, I want my app (a Spring Boot app under the hood) -- called bootup -- to have the following directory structure:
/opt/
bootup/
bin/
bootup.jar ==> the app
logs/
bootup.log ==> log file; GETS CREATED BY THE APP # STARTUP
config/
application.yml ==> app config file
logback.groovy ==> log config file
It's very important to note that when I run my app locally on my host machine - outside of Docker - everything works perfectly fine, including the creation of log files to my host's /opt/bootup/logs directory. The app endpoints serve up the correct content, etc. All is well and dandy.
So I created the following Dockerfile:
FROM openjdk:8
RUN mkdir /opt/bootup
RUN mkdir /opt/bootup/logs
RUN mkdir /opt/bootup/config
RUN mkdir /opt/bootup/bin
ADD build/libs/bootup.jar /opt/bootup/bin
ADD application.yml /opt/bootup/config
ADD logback.groovy /opt/bootup/config
WORKDIR /opt/bootup/bin
EXPOSE 9200
ENTRYPOINT java -Dspring.config=/opt/bootup/config -jar bootup.jar
I then build my image via:
docker build -t bootup .
I then run my container:
docker run -it -p 9200:9200 -d --name bootup bootup
I run docker ps:
CONTAINER ID IMAGE COMMAND ...
3f1492790397 bootup "/bin/sh -c 'java ..."
So far, so good!
My app should then be serving a simple web page at localhost:9200, so I open my browser to http://localhost:9200 and I get nothing.
When I use docker exec -it 3f1492790397 bash to "ssh" into my container, I see everything looks fine, except the /opt/bootup/logs directory, which should have a bootup.log file in it -- created at startup -- is instead empty.
I tried using docker attach 3f1492790397 and then hitting http://localhost:9200 in my browser, to see if that would generated some standard output (my app logs both to /opt/bootup/logs/bootup.log as well as the console) but that doesn't yield any output.
So I think what's happening is that my app (for some reason) doesn't have permission to create its own log file when the container starts up, and puts the app in a weird state, or even prevents it from starting up altogether.
So I ask:
Is there a way to see what user my app is starting up as?; or
Is there a way to tail standard output while the container is starting? Attaching after startup doesn't help me because I think by the time I run the docker attach command the app has already choked
Thanks in advance!
I don't know why your app isn't working, but can answer your questions-
Is there a way to see what user my app is starting up as?; or
A: Docker containers run as root unless otherwise specified.
Is there a way to tail standard output while the container is starting? Attaching after startup doesn't help me because I think by the time I run the docker attach command the app has already choked
A: Docker containers dump stdout/stderr to the Docker logs by default. There are two ways to see these- 1 is to run the container with the flag -it instead of -d to get an interactive session that will list the stdout from your container. The other is to use the docker logs *container_name* command on a running or stopped container.
docker attach 3f1492790397
This doesn't do what you are hoping for. What you want is docker exec (probably docker exec -it bootup bash), which will give you a shell in the scope of the container which will let you check for your log files or try and hit the app using curl from inside the container.
Why do I get no output?
Hard to say without the info from the earlier commands. Is your app listening on 0.0.0.0 or on localhost (your laptop browser will look like an external machine to the container)? Does your app require a supervisor process that isn't running? Does it require some other JAR files that are on the CLASSPATH on your laptop but not in the container? Are you running docker using Docker-Machine (in which case localhost is probably not the name of the container)?