I have a python script called script.py. When I run this script, it creates a logs folder on the Desktop and downloads all the necessary logs from a website and writes them as .log files in this logs folder. I want Fluentd to run this script every 5 minutes and do nothing more. The next source I have on the config file does the real job of sending this log data to another place. If I already have the logs folder on the Desktop, this log files are uploaded correctly to the next destination. But the script never runs. If I delete my logs folder locally, this is the output fluentd gives:
2020-07-27 10:20:42 +0200 [trace]: #0 plugin/buffer.rb:350:enqueue_all: enqueueing all chunks in buffer instance=47448563172440
2020-07-27 10:21:09 +0200 [trace]: #0 plugin/buffer.rb:350:enqueue_all: enqueueing all chunks in buffer instance=47448563172440
2020-07-27 10:21:36 +0200 [debug]: #0 plugin_helper/child_process.rb:255:child_process_execute_once: Executing command title=:exec_input spawn=[{}, "python /home/zohair/Desktop/script.py"] mode=[:read] stderr=:discard
This never gives a logs folder on my Desktop which the script normally does output if run locally like python script.py
If I already have the logs folder, I can see the logs on the stdout normally. Here is my config file:
<source>
#type exec
command python /home/someuser/Desktop/script.py
run_interval 5m
<parse>
#type none
keys none
</parse>
<extract>
tag_key none
</extract>
</source>
<source>
#type tail
read_from_head true
path /home/someuser/Desktop/logs/*
tag sensor_1.log-raw-data
refresh_interval 5m
<parse>
#type none
</parse>
</source>
<match sensor_1.log-raw-data>
#type stdout
</match>
I just need fluentd to run the script and do nothing else, and let the other source take this data and send it to somewhere else. Any solutions?
Problem was solved by creating another #type exec for pip install -r requirements.txt which fulfilled the missing module error which was not being shown on the fluentd error log (Was running fluentd as superuser).
Related
I am trying to use tee to log into two locations:
file in persistent storage
Docker stdout
Error log line from VirtualHost config:
ErrorLog "|/usr/bin/tee -a /var/log/apache/error.log /proc/1/fd/1"
Now the problem is that errors are logged twice in /proc/1/fd/1 (as docker logs states), yet error is only logged once into /var/log/apache/error.log
I´ve also tried run from cli:
echo 123 | /usr/bin/tee -a /tmp/test /proc/1/fd/1
This succesfully writes only once to both file and stdout.
Is there some reason why Apache logs writes twice to /proc/1/fd/1 while it logs only once to file and /usr/bin/tee also works as expected?
On Ubuntu 18.04, I am running td-agent v4 which uses Fluentd v1.0 core. First I configured it with TCP input and stdout output. It receives and outputs the messages fine. I then configure it to output to file with a 10s flush interval, yet I do not see any output files generated in the destination path.
This is my file output configuration:
<match>
#type file
path /var/log/td-agent/test/access.%Y-%m-%d.%H:%M:%S.log
<buffer time>
timekey 10s
timekey_use_utc true
timekey_wait 2s
flush_interval 10s
</buffer>
</match>
I perform this check every 10s to see if log files are generated, but all I see is a directory with a name that still has the placeholders that I set for the path param:
ls -la /var/log/td-agent/test
total 12
drwxr-xr-x 3 td-agent td-agent 4096 Feb 5 23:14 .
drwxr-xr-x 6 td-agent td-agent 4096 Feb 6 00:17 ..
drwxr-xr-x 2 td-agent td-agent 4096 Feb 5 23:14 access.%Y-%m-%d.%H:%M:%S.log
From following the Fluentd docs, I was expecting this should be fairly straight forward since the file output and buffering plugins are bundled with Fluentd's core.
Am I missing something trivial here?
I figured it out, and it works now. I had two outputs, one to file and another to stdout. Apparently that won't work if they're both defined separately in the config file with their own <match> ... </match>. I believe output to stdout was read first in the config, so Fluentd outputted to that and not to file. They should both instead be nested under the copy output like this:
<match>
#type copy
<store>
#type file
...
</store>
<store>
#type stdout
</store>
</match>
I have a script called script.py. After running the script, I get different/various .log files in a folder. How do I write a config file which runs the script and then sends a log file successfully?
Here is my configuration which appears to be getting no output from match #type stdout:
<source>
#type exec
tag sensor_1.log-raw-data
command python /home/cool/Desktop/script.py
run_interval 5m
<parse>
keys something
</parse>
</source>
<source>
#type tail
read_from_head true
path /home/cool/Desktop/logs/0slaprunner.log
tag foo.*
<parse>
#type none
</parse>
</source>
<match pattern>
#type stdout
</match>
The argument for match section is pattern. It means that this would cater for the events with tag pattern. But, none of the tags in source sections is pattern; hence, no routing to stdout.
From your description, it looks like you want to route events from the tail input plugin to the stdout output plugin, so the relevant configuration would be something like this:
<source>
#type tail
# ...
tag foo # tag for the events
# ...
</source>
<match foo> # cater events with tag `foo`
#type stdout
</match>
For debugging purposes, run fluentd with -v or -vv command-line option.
I'm trying to run a docker container on Compute Engine, everything works fine, my PHP app is correctly returning all data but i want to Increase log verbosity.
For now I've added two config files for fluentd inside a container config dir:
This one for nginx:
<source>
#type tail
format nginx
path /var/log/feedbacks/nginx-access.log
pos_file /var/lib/google-fluentd/pos/nginx-access.pos
read_from_head true
tag nginx-access
</source>
<source>
#type tail
format none
path /var/log/feedbacks/nginx-error.log
pos_file /var/lib/google-fluentd/pos/nginx-error.pos
read_from_head true
tag nginx-error
</source>
And this one for PHP log output :
<source>
#type tail
format /^\[(?<time>[\d\-]+ [\d\:]+)\] (?<channel>.+)\.(?<level>(DEBUG|INFO|NOTICE|WARNING|ERROR|CRITICAL|ALERT|EMERGENCY))\: (?<message>[^\{\}]*) (?<context>(\{.+\})|(\[.*\])) (?<extra>(\{.+\})|(\[.*\]))\s*$/
path /var/log/feedbacks/structured.log
pos_file /var/lib/google-fluentd/pos/feedbacks.pos
read_from_head true
tag feedbacks
</source>
I've mounted this 2 config files as follow with the corresponding logs files:
container path: /usr/src/app/var/logs/, host path: /var/log/feedbacks/, mode: r/w
container path: /usr/src/app/docker/runnable/fluentd/, host path: /etc/google-fluentd/config.d/, mode: r/w
But when I /bin/bash to these directories inside the stackdriver-logging-agent there is nothing inside, maybe i'm missing something ...
Thanks for helping !
stackdriver-logging-agent reads a container's logs through the equivalent of docker logs [container]. This provides a consistent API for processes on the host OS to gather container logs.
By default, the container's stdout|stderr are sent to docker logs and it's this stream that the stackdriver-logging-agent is collecting and onsending to the Stackdriver service.
IIUC correctly, you'd need to ensure that your PHP app is generating the richer logs and that these are being sent to stdout|stderr.
If you were to use Nginx's stock Docker image, it does this:
lrwxrwxrwx 1 root root 11 May 8 03:01 access.log -> /dev/stdout
lrwxrwxrwx 1 root root 11 May 8 03:01 error.log -> /dev/stderr
See Docker's documentation here:
https://docs.docker.com/config/containers/logging/
I was unable to find a good explanation for this for Container OS on Google's site.
Here's my td-agent.conf file
<source>
#type http
port 8888
</source>
<match whatever.access>
#type file
path /var/log/what.txt
</match>
But when I try to start the server using
sudo /etc/init.d/td-agent start
it gives the following error:
'2016-02-01 10:45:49 +0530 [error]: fluent/supervisor.rb:359:rescue in >main_process: config error file="/etc/td-agent/td-agent.conf" error="out_file: >/var/log/what.txt.20160201_0.log is not writable"
Can someone explain what's wrong?
If you installed td-agent v2, it creates its own user and group called td-agent. I believe that when you run the td-agent service, it switches to this user and hence it expects the directory to have write permissions for this user. I faced the same issue and did something like: (Use sudo if needed for below commands.)
mkdir /logs
chown td-agent:td-agent /logs
and update your section to:
<match whatever.access>
#type file
path /logs/what.txt
</match>
I think when you try to start td-agent, you do not have permission to access /var/log/, using ls -l to check its permission mode and change it with chmod.
I got the same problem, after change directory's access permission, td-agent can be started.