I am trying to use tee to log into two locations:
file in persistent storage
Docker stdout
Error log line from VirtualHost config:
ErrorLog "|/usr/bin/tee -a /var/log/apache/error.log /proc/1/fd/1"
Now the problem is that errors are logged twice in /proc/1/fd/1 (as docker logs states), yet error is only logged once into /var/log/apache/error.log
I´ve also tried run from cli:
echo 123 | /usr/bin/tee -a /tmp/test /proc/1/fd/1
This succesfully writes only once to both file and stdout.
Is there some reason why Apache logs writes twice to /proc/1/fd/1 while it logs only once to file and /usr/bin/tee also works as expected?
Related
I set my /etc/docker/daemon.json to send container logs to GCP:
{
"log-driver": "gcplogs",
"log-opts": {
"gcp-meta-name": "some-cool-name",
"gcp-project": "some-project-name"
}
}
This works fine, but it seems there is no distinction between STDERR and STDOUT, both entries have a Severity of 'Default'
In container:
root#0bbcf70a30ed:/var/www/app# echo 'xx' > /proc/1/fd/2
root#0bbcf70a30ed:/var/www/app# echo 'xx' > /proc/1/fd/1
In GCP:
Is there anything I can do to make the logs from STDERR have a Severity of 'Error' ?
And if not, is there anything I can do to make all STDERR entries have a string like 'ERROR' prepended, so I can at least filter on them?
For example, in my Dockerfile I do:
RUN touch /var/log/apache2/error.log
RUN ln -sf /proc/1/fd/2 /var/log/apache2/error.log
This makes sure the apache2 error logs go to the containers' STDERR. If I could somehow make all those logging entries have a string like 'ERROR' prepended, I would at least have a semi-workable solution.
But really, having STDERR entries automatically get Severity 'Error' is ideal.
I am using Docker 1.13 community edition on a CentOS 7 x64 machine. When I was following a Docker Compose sample from Docker official tutorial, all things were OK until I added these lines to the docker-compose.yml file:
volumes:
- .:/code
After adding it, I faced the following error:
can't open file 'app.py': [Errno 13] Permission denied. It seems that the problem is due to a SELinux limit. Using this post I ran the following command:
su -c "setenforce 0"
to solve the problem temporarily, but running this command:
chcon -Rt svirt_sandbox_file_t /path/to/volume
couldn't help me.
Finally I found the correct rule to add to SELinux:
# ausearch -c 'python' --raw | audit2allow -M my-python
# semodule -i my-python.pp
I found it when I opened the SELinux Alert Browser and clicked on 'Details' button on the row related to this error. The more detailed information from SELinux:
SELinux is preventing /usr/local/bin/python3.4 from read access on the
file app.py.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that python3.4 should be allowed read access on the
app.py file by default. Then you should report this as a bug. You can
generate a local policy module to allow this access. Do allow this
access for now by executing:
ausearch -c 'python' --raw | audit2allow -M my-python
semodule -i my-python.pp
when I run the application into the container, I can see this output:
docker logs 3aee2bd90da4
> Dot-Service initializing with env_type: CA [2018-01-11
> 09:47:30,487][16:MainProcess][webapps.py:102][INFO] DAS-conductor
> initialized with settings CAConfig
Now, from the docker instance, I see only the "stdout":
root#3acd2bd90da4:/usr/src/app# ./bin/run.sh
Dot-Service initializing with env_type: CA
[2018-01-11 09:47:30,487][16:MainProcess][webapps.py:102][INFO] DAS-conductor initialized with settings CAConfig
Traceback (most recent call last):
File "/usr/src/app/bin/..//run.py", line 25, in <module>
cmd_main()
File "/usr/src/app/bin/..//run.py", line 22, in cmd_main
it misses the error section beginning by "Traceback" (stderr) from the docker logs command
what could I do to print all the log as it appears when I run the command inside the container?
thanks
If I understood the question correctly, you can always put your application logs to Docker container logs by using below code snippet in your Dockerfile -
RUN ln -sf /dev/stdout /app/out/access.log \
&& ln -sf /dev/stderr /app/out/error.log
You need to make sure your application writes the output to respective files & you will be able to see them in your container logs. Official Nginx docker image does the same.
Ref - https://github.com/nginxinc/docker-nginx/blob/8921999083def7ba43a06fabd5f80e4406651353/mainline/jessie/Dockerfile#L21-L23
If you need to follow the log output, and not seeing only last lines of output, use: $docker logs -f
https://docs.docker.com/engine/reference/commandline/logs/
In my CI chain I execute end-to-end tests after a "docker-compose up". Unfortunately my tests often fail because even if the containers are properly started, the programs contained in my containers are not.
Is there an elegant way to verify that my setup is completely started before running my tests ?
You could poll the required services to confirm they are responding before running the tests.
curl has inbuilt retry logic or it's fairly trivial to build retry logic around some other type of service test.
#!/bin/bash
await(){
local url=${1}
local seconds=${2:-30}
curl --max-time 5 --retry 60 --retry-delay 1 \
--retry-max-time ${seconds} "${url}" \
|| exit 1
}
docker-compose up -d
await http://container_ms1:3000
await http://container_ms2:3000
run-ze-tests
The alternate to polling is an event based system.
If all your services push notifications to an external service, scaeda gave the example of a log file or you could use something like Amazon SNS. Your services emit a "started" event. Then you can subscribe to those events and run whatever you need once everything has started.
Docker 1.12 did add the HEALTHCHECK build command. Maybe this is available via Docker Events?
If you have control over the docker engine in your CI setup you could execute docker logs [Container_Name] and read out the last line which could be emitted by your application.
RESULT=$(docker logs [Container_Name] 2>&1 | grep [Search_String])
logs output example:
Agent pid 13
Enter passphrase (empty for no passphrase): Enter same passphrase again: Identity added: id_rsa (id_rsa)
#host SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6
#host SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6
parse specific line:
RESULT=$(docker logs ssh_jenkins_test 2>&1 | grep Enter)
result:
Enter passphrase (empty for no passphrase): Enter same passphrase again: Identity added: id_rsa (id_rsa)
I am trying to run ruby on rails under passenger with apache2 under fedora 19 and I got this error in log:
[Tue Feb 25 09:37:52.367683 2014] [passenger:error] [pid 2779] ***
Passenger could not be initialized because of this error: Unable to
start the Phusion Passenger watchdog because it encountered the
following error during startup: Cannot change the directory
'/tmp/passenger.1.0.2779/generation-1/buffered_uploads' its UID to 48
and GID to 48: Operation not permitted (errno=1)
That directory (/tmp/passenger.1.0.2779) doesn't even exist. I think that problem is with selinux. I tried to solve it about 4 hours. Httpd is running under user apache and group apache, I tried:
cat /var/log/audit/audit.log | grep passenger | audit2allow -M
passenger semodule -i passenger.pp
but still nothing.
In your case, you should switch SELinux into Permissive mode at first, then try to capture the audit log from starting Apache to run your application.1
Once you got the home page of your application, you can build your custom policy with the logs.
Switch SELinux into Permissive mode and clean audit.log
]# setenforce 0
]# rm /var/log/audit/audit.log
]# service auditd restart
Restart Apache
]# service httpd restart
Try to open your application with a web browser
It might give more information about what is happenning when you application is running.
Make a custom policy module to allow these actions
]# mkdir work
]# cd work
]# grep httpd /var/log/audit/audit.log | audit2allow -M passenger
]# ls
passenger.pp passenger.te
Load postgrey policy module using the 'semodule' command into the current SELinux policy:
]# semodule -i passenger.pp
]# setenforce 1
Restart Apache
]# service httpd restart
References:
http://wiki.centos.org/HowTos/SELinux#head-faa96b3fdd922004cdb988c1989e56191c257c01
I ran into a similar error, with a startup error about being unable to create a directory that did not exist. (logs, not tmp, but same sort of thing) I, too, battled with it for an hour and couldn't make sense of it. I created/deleted/chmod the directory many ways without success.
The fix for me was to change the parameters to passenger-start. Initially, my Docker container started passenger with:
exec bundle exec passenger start --auto --disable-security-update-check --min-instances 20 --max-pool-size 20 --max-request-queue-size 500
I removed all parameters, leaving just this:
exec bundle exec passenger start
At this point, passenger could create the log folder and file, and all was well. I could have restored the params at this point, but we decided they were not needed for the development environment so left them out moving ahead.
In hindsight, I have a hunch that I deleted the log directory while a file in it was still open, and the file system persisted that condition in some way. But that's just a hunch. Perhaps simply rebooting my Mac would have fixed it...