I want to write error logs to GCP. But can't find out how to filter messages with string, ex: level='error'
I have read this documentation.
Currently, I can't think anyway but write it to Fluentd => filter message => write GCP. But it has added an unnecessary step in my case.
Do we have a straightforward way to filter and send logs directly to GCP?
Most simple way is to just go to logs explorer and change "severity" to "error" - like so:
This way you will only see error messages for all your VM's.
It's another matter if you want fluentd to send just errors to GCP. In this case you need to reconfigure it. Have a look at the documentation on how to send structured logs to GCP and make proper changes.
Depending on your needs fist method will work out of the box. Second one needs some tinkering but will also work.
Related
I'm trying to find a way to get response times from Traefik per route.
For instance:
/api/asdf/.* 123.5ms
/blub/narf/.* 70.5ms
/blub/.* 1337.8ms
and so on.
This doesn't seem to be a very unusual requirement but after googeling a lot, I didn't find anything that could do the job.
I even had a look at the middlewares but there is no way to get response time of a request because it only hooks itself into the request but there is no hook that would be called after the request completed.
The traefik log files actually contain the information (in debug log level) and if I could tail somehow with a e.g. python script, I could run a list of regexs on them and collect this way the response times. But tailing on docker logs is quite messy imho and I think there should be some more obvious way I havn't found yet 🤔
Can't imaging that I'm the first person trying to track response times per route - but why can't I find anything 🤷♂️
Does someone perhaps an idea in which direction I should search?
Thank you in advance!
If you dont find any, you can take help from this project and https://github.com/traefik/plugin-rewritebody. It enables the user to replace the body but you get the idea, there is an example to get the response and you can add your own logic to write the response time in the file or whatsoever
You can try using Prometheus with Traefik. Here is a sample docker-compose which will do the job
You might wanna checkout this open-source repo
We calculated the average response time of API's by enabling prometheus in traefik.
The expression which we are using for the same is like
expr: sum(traefik_service_request_duration_seconds_sum{instance="company.com:12345",service=~"backend-module-test.*"}) / sum(traefik_service_request_duration_seconds_count{instance="company.com:12345",service=~"backend-module-test.*"}) * 1000
This expression is evaluated for a period of 1m, 5m, 10m etc and the resulting graph is displayed on the dashboard.
Other solutions include traefik-docs
Currently, I have a docker container sending logs to a Logstash using gelf. Pretty standard configuration set in the docker-compose file used to create the container.
I'm investigating the feasibility of sending the logs of a docker container to more than one instance of ELK. This is not needed for production, but will greatly improve the quality of life of our dev teams.
Reading the docs, it seems that what i need is not possible (at least, they don't mention whether the gelf-address property accepts a list of URIs or not, and I must assume it doesn't while I look for more info).
Does anyone know if this can be achieved? Thanks!
Looking at docker's code here and here I'd say that currently only a single target address is supported.
// New creates a gelf logger using the configuration passed in on the
// context. The supported context configuration variable is gelf-address.
func New(info logger.Info) (logger.Logger, error) {
// parse gelf address
address, err := parseAddress(info.Config["gelf-address"])
Potentially, this project andviro/grayproxy looks like it could help you take a single GELF input, and forward it to multiple GELF collectors:
By default grayproxy configures the input at udp://0.0.0.0:12201 and no outputs. Outputs are added using -out flag and may be specified multiple times
I am developing a UI in which I need to show the live logs (stdout and stderr) of jobs running in a mesos slave. I am finding out a way in which I will be able to generate a URL which will point to the mesos logs for the job. Is there a way to do the same? Basically, I need to know the slave id, executor id, master id etc. for generating the URL. Is there a way to find these information?
The sandbox URL is of the form http://
$slave_url:5050/read.json?$work_dir/work/slaves/$slave_id/frameworks/$framework_id/executors/$executor_id/runs/$container_id/stdout, and you can even use the browse.json endpoint to browse around within the sandbox.
Alternatively, you can use the mesos tail $task_id CLI command to access these logs.
For more details, see the following mailing list thread: http://search-hadoop.com/m/RFt15skyLE/Accessing+stdout%252Fstderr+of+a+task+programmattically
How about using reverse approach. You need to present live logs from stderr and stdout. How about storing them outside of mesos slave e.g., elastic-search? You will get nearly live updates, old logs available after, nice search options.
From version 0.27.0 Mesos supports ContainerLogger. You can write your own implementation of ContainerLogger that will push logs to central logs repository (Graylog, Logstash, e.t.c) and then expose it in your UI.
Mesos offers a REST interface where you get the information you want. Visit with your browser http://<MESOS_MASTER_IP>:5050/help (using default port) to check the options you have to query (for example, you can get the information you need from http://<MESOS_MASTER_IP>:5050/master/state.json). Check this link to see an example using it.
We have custom Docker web app running in Elastic Beanstalk Docker container environment.
Would like to have application logs be available for viewing outside. Without downloading through instances or AWS console.
So far neither of solutions been acceptable. Maybe someone achieved centralised logging for Elastic Benastalk Dockerized apps?
Solution 1: AWS Console log download
not acceptable - requires to download logs, extract every time. Non real-time.
Solution 2: S3 + Elasticsearch + Fluentd
fluentd does not have plugin to retrieve logs from S3
There's excellent S3 plugin, but it's only for log output to S3. not for input logs from S3.
Solution 3: S3 + Elasticsearch + Logstash
cons: Can only pull all logs from entire bucket or nothing.
The problem lies with Elastic Beanstalk S3 Log storage structure. You cannot specify file name pattern. It's either all logs or nothing.
ElasticBeanstalk saves logs on S3 in path containing random instance and environment ids:
s3.bucket/resources/environments/logs/publish/e-<random environment id>/i-<random instance id>/my.log#
Logstash s3 plugin can be pointed only to resources/environments/logs/publish/. When you try to point it to environments/logs/publish/*/my.log it does not work.
which means you can not pull particular log and tag/type it to be able to find in Elasticsearch. Since AWS saves logs from all your environments and instances in same folder structure, you cannot chose even the instance.
Solution 4: AWS CloudWatch Console log viewer
It is possible to forward your custom logs to CloudWatch console. Do achieve that, put configuration files in .ebextensions path of your app bundle:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html
There's a file called cwl-webrequest-metrics.config which allows you to specify log files along with alerts, etc.
Great!? except that configuration file format is neither yaml,xml or Json, and it's not documented. There is absolutely zero mentions of that file, it's format either on AWS documentation website or anywhere on the net.
And to get one log file appear in CloudWatch is not simply adding a configuration line.
The only possible way to get this working seem to be trial and error. Great!? except for every attempt you need to re-deploy your environment.
There's only one reference to how to make this work with custom log: http://qiita.com/kozayupapa/items/2bb7a6b1f17f4e799a22 I have no idea how that person reverse engineered the file format.
cons:
Cloudwatch does not seem to be able to split logs into columns when displaying, so you can't easily filter by priority, etc.
AWS Console Log viewer does not have auto-refresh to follow logs.
Nightmare undocumented configuration file format, no way of testing. Trial and error requires re-deploying whole instance.
Perhaps an AWS Lambda function is applicable?
Write some javascript that dumps all notifications, then see what you can do with those.
After an object is written, you could rename it within the same bucket?
Or notify your own log-management service about the creation of a new object?
Lots of possibilities there...
I've started using Sumologic for the moment. There's a free trial and then a free tier (500mb /day, 7 day retention). I'm not out of the trial period yet and my EB app does literally nothing (it's just a few HTML pages serve by Nginx in a docker container. Looks like it could get expensive once you hit any serious amount of logs though.
It works ok so far. You need to create an IAM user that has access to the S3 bucket you want to read from and then it sucks the logs over to Sumologic servers and does all the processing and searching over there. Bit fiddly to set up, but I don't really see how it could be simpler and it's reasonably well-documented.
It lets you provide different path expressions with wildcards, then assign a "sourceCategory" to those different paths. You then use those sourceCategories to filter your log searching to a specific type of logging.
My plan long-term is to use something like your solution 3, but this got me going in very short order so I can move on to other things.
You can use a Multicontainer environment, sharing the log folder to another docker container with the tool of your preference to centralize the logs, in our case we connected an Apache Flume to move the files to an HDFS. Hope this helps you with this.
The easiest method I found to do this was using papertrail via rsyslog and .ebextensions, however it is very expensive for logging everything.
The good part is with rsyslog you can essentially send your logs anywhere and you are not tied to papertrail.
example ebextension
I've found loggly to be the most convenient.
It is a hosted service which might not be what you want. However if you check out their setup page you can see a number of ways your situation is supported (docker specific solutions, as well as like 10 amazon specific options). Even if loggly isn't to your taste, you can look at those solutions and easily see how some of them could be applied to most any centralized logging solution you might use or write.
I am running a couple of Rails 3.0 sites on a Debian server. Currently all logs go into RAILS_ROOT/log/production.log (RAILS_ROOT for each site)
I would like to change this, so that all logging goes through rsyslog and is placed in:
/var/log/rails/site1.log
/var/log/rails/site2.log
etc.
And also to have these logs auto-rotated every day.
How is this done?
/ Carsten
In the past I have used the syslog-logger gem for this
You can set this up in an initializer:
config.logger = Logger::Syslog.new("site1", Syslog::LOG_LOCAL5)
In recent ruby versions, there's also syslog/logger in the standard library - usage is pretty much the same.
To start logging to syslog rather than the default text file. What happens next is a syslog configuration thing - you need to create rsyslog rules that define where your data goes
The simplest possible thing would be something like
!site1 /var/log/site1.log
Which directs everything with the program name "site1" ( the first argument to Logger::Syslog).
There is lots more you can do, for example you could forward log messages across to a central logging server that aggregates them all into one file so tht you don't have Iook at one log file for each of your app instances.
For log rotation I use logrotate - I believe rsyslog can actually handle that sort of stuff internally but I don't know the details.