I donn't want/need 'auditd' in my LINUX.
Problem is that this not a standard LINUX distribution rather a special, minimized version called 'CoreElec' (www.coreelec.org)
So there is no 'apt', 'auditctl'. crontab is empty.
Docker is issues the auditing. If I stop Docker, auditing messages also stop.
I can find no way to stop it or at least to stop its logging, resulting in a syslog full of audit log messages.
If I could at least get rid of logs, it would help.
Thanks for any help.
Gerry
Related
I have a multi-services environment that is hosted with docker swarm. There are multiple stacks that are created. All the docker containers which are running have an inbuild Spring Boot application. The issue is coming that all my stacks get restarted on their own. Now I know that in compose file I have mentioned that restart_policy as on failure. Hence it auto restarted. The issue comes that when services are restarted, I get errors from a particular service and this breaks everything.
I am not able to figure out what actually happens.
I did quite a lot of research and found out about these things.
Docker daemon is not restarted. I double-checked this with the uptime of the docker daemon.
I checked the docker service ps <Service_ID> and there I can see service showing shutdown and starting. No other information.
I checked the docker service logs <Service_ID> but no error in there too.
I checked for resource crunch. I can assure you that there was quite a good resource available at the host as well as each container level.
Can someone help where exactly to find logs for this even? Any other thoughts on this?
My host is actually a VM hosted on VMWare Vcenter.
After a lot of research and going through all docker logs, I could not find the solution. Later on, I discovered that there was a memory snapshot taken for backup every 24 hours.
Here is what I observe:
Whenever we take a snapshot, all docker services running on the host restart automatically. There will be no errors in that but they will just restart gracefully.
I found some questions already having this problem with VMware snapshots.
As far as I know, when we take a snapshot, it points to a different memory location and saves the previous one. I am not able to find why it's happening but yes Root cause of the problem was this. If anyone is a VMWare snapshots expert, please let us know.
I've got a docker container running a service, and I need that service to send logs to rsyslog. It's an ubuntu image running a set of services in the container. However, the rsyslog service cannot start inside this container. I cannot determine why.
Running service rsyslog start (this image uses upstart, not systemd) returns only the output start: Job failed to start. There is no further information provided, even when I use --verbose.
Furthermore, there are no error logs from this failed startup process. Because rsyslog is the service that can't start, it's obviously not running, so nothing is getting logged. I'm not finding anything relevant in Upstart's logs either: /var/log/upstart/ only contains the logs of a few things that successfully started, as well as dmesg.log which simply contains dmesg: klogctl failed: Operation not permitted. which from what I can tell is because of a docker limitation that cannot really be fixed. And it's unknown if this is even related to the issue.
Here's the interesting bit: I have the exact same container running on a different host, and it's not suffering from this issue. Rsyslog is able to start and run in the container just fine on that host. So obviously the cause is some difference between the hosts. But I don't know where to begin with that: There are LOTS of differences between the hosts (the working one is my local windows system, the failing one is a virtual machine running in a cloud environment), so I wouldn't know where to even begin about which differences could cause this issue and which ones couldn't.
I've exhausted everything that I know to check. My only option left is to come to stackoverflow and ask for any ideas.
Two questions here, really:
Is there any way to get more information out of the failure to start? start itself is a binary file, not a script, so I can't open it up and edit it. I'm reliant solely on the output of that command, and it's not logging anything anywhere useful.
What could possibly be different between these two hosts that could cause this issue? Are there any smoking guns or obvious candidates to check?
Regarding the container itself, unfortunately it's a container provided by a third party that I'm simply modifying. I can't really change anything fundamental about the container, such as the fact that it's entrypoint is /sbin/init (which is a very bad practice for docker containers, and is the root cause of all of my troubles). This is also causing some issues with the docker logging driver, which is why I'm stuck using syslog as the logging solution instead.
Technical details:
Docker versions: This has happened on docker 18, 19 and 20, at several minor versions.
19.03.12 and 20.10.3 are the most recent installs I've got running.
OS: Ubuntu 16.04/18.04/20.04, all display this behavior.
All installs use overlay2.
Description:
Running lsof | grep -i 'deleted reveals log files being held on to, from processes running inside docker containers. Grepping the PID in the output of ps -aef --forest reveals the processes are running inside a docker container.
Running the same process outside docker has thus far not reproduced this behavior.
This started to become apparent when tens of gigabytes of diskspace could not be found.
This happens both for java and nodejs processes, using logback/winston respectively.
These logger libraries take care of their own logrotating and are not being tracked by a distinct logrotate service.
I have no clue what causes this behavior, and why it doesn't happen when not dockerized.
Googling "docker logrotate files being held" just gives results on logrotating the log files of docker itself.
I'm not finding what I'm looking for with general searching, so I came here, to ask if maybe someone recognizes this and it's actually an easy fix? Some simple thing I forgot but has to be pointed out?
Here's hoping someone knows what's going on.
I want to make a faster deployment process than before. Always too much time spent in this step.
But I can't find any way to see detailed docker logs such as Downloading, Pulling Images, Starting Containers, ... etc. I want to see it in the machine; I want to debug it. How to check this?
These will be in various places.
docker events will show you each action the scheduler takes, and any actions on the node you've run that command on. You'll need to run this on all potential nodes while creating/updating a service to get a full accounting of manager and worker events.
On the node that's been assigned a task to create a container, the docker debug flag may give you more insight.
Using heroku logs --tail which works great for a few minutes. Then it stops displaying the logs. It seems that the ssh connection is timing out and dying. There is no error or message. Working in Ubuntu 11.04 on wired conneciton.
I added the following to ~/.ssh/config:
ServerAliveInterval 5
But it didn't work. Do I need anything else in the config file? How do I know if it is doing anything? How can I monitor the traffic and see the keepalive request? I am looking at System Monitor but don't see anything every 5 seconds.
Thanks.
Have you done all of this:
$ heroku config:add LOG_LEVEL=DEBUG
$ heroku addons:upgrade logging:expanded
$ heroku logs --tail
It turns out that I was looking for an answer to the wrong question. Why use the tail to save logs? It is problematic, labor intensive, and error prone.
The solution I found was Papertrail. Small sites are free. papertrailapp.com.
Here's the complete story from my blog: http://www.onlineinvestingai.com/blog/2011/08/07/better-logging-with-papertrail-on-heroku/
I have seen the same issue. I'm not certain that it is ssh that times out but something does. For now, we've placed our monitor in a loop so it resumes automatically in case of time-out.
We also use PaperTrail, however it has limits in the amount that you can transfer. We use PaperTrail for general purpose use, and tail the logs for the excrutiating-detail logs that quickly use up all avaiable PaperTrail capacity.