We have application deployed in K8S pod and all logs are being monitored in ELK stack. Now we have one application which is using external *.jar which is writing logs in one file local to container path. How I can send this logs to kubernetes console so that it will come to elastic search monitoring.
Any help is much appreciated!.
Now we have one application which is using external *.jar which is writing logs in one file local to container path. How I can send this logs to kubernetes console so that it will come to elastic search monitoring.
There are three ways, in increasing order of complexity:
Cheat and symlink the path it tries to log to as /dev/stdout (or /proc/1/fd/0); sometimes it works and it's super cheap, but if the logging system tries to seek to the end of the file, or rotate it, or catches on that it's not actually a "file", then you'll have to try other tricks
If the app uses a "normal" logging framework, such as log4j, slf4j, logback, etc, you have a better-than-average chance of being able to influence the app's logging behavior via some well placed configuration files or in some cases environment variables
Actually, you know, ask your developers to configure their application according to the 12 Factor App principles and log to stdout (and stderr!) like a sane app
Without more specifics we can't offer more specific advice, but that's the gist of it
Related
The issue
I am using the same container (similar resources) on 2 projects -- production and staging. Both have custom domains setup with cloud flare DNS and are on the same region. Container build is done in a completely different project and IAM is used to handle the access to these containers. Both project services have 80 concurrency and 300 seconds time out for all 5 services.
All was working good 3 days back but from yesterday almost all cloud run services on staging (thankfully) started throwing 503 randomly and for most requests. Some services were not even deployed for a week. The same containers are running fine on production project, no issues.
Ruled out causes
anything to do with Cloudflare (I tried the URL cloud run gives it has the issue of 503)
anything with build or containers (I tried the demo hello world container with go - it has the issue too)
Resources: I tried giving it 1 GB ram and 2 cpus but the problem persisted
issues on deployment (deploy multiple branches - didn't work)
issue in code (just routed traffic to old 2-3 days old revision but still issue was there)
Issue on service level ( I used the same container to create a completely new service, it also had the issue)
Possible causes
something on cloud run or cloud run load balancer
may some env vars but that also doesn't seem to be the issue
Response Codes
I just ran a quick check with vegeta (30 secs with 10 rps) same container on staging and production for a static file path and below are the responses:
Staging
Production
If anyone has any insights on this it would help greatly.
Based on your explanation, I cannot understand what's going on. You explained what doesn't work but didn't point out what works (does your app run locally? are you able to run a hello world sample application?)
So I'll recommend some debugging tips.
If you're getting a HTTP 5xx status code, first, check your application's logs. Is it printing ANY logs? Is there logs of a request? Does your application have and deployed with "verbose" logging setting?
Try hitting your *.run.app domain directly. If it's not working, then it's not a domain or dns or cloudflare issue. Try debugging and/or redeploying your app. Deploy something that works first. If *.run.app domain works, then the issue is not in Cloud Run.
Make sure you aren't using Cloudflare in proxy mode (e.g. your DNS points to Cloud Run; not Cloudflare) as there's a known issue about certificate issuance/renewals when domains are behind Cloudflare, right now.
Beyond these, if a redeploy seems to solve your problem, maybe try redeploying. It could be very likely some configuration recently became different two different projects.
See Cloud Run Troubleshooting
https://cloud.google.com/run/docs/troubleshooting
Do you see 503 errors under high load?
The Cloud Run (fully managed) load balancer strives to distribute incoming requests over the necessary amount of container instances. However, if your container instances are using a lot of CPU to process requests, the container instances will not be able to process all of the requests, and some requests will be returned with a 503 error code.
To mitigate this, try lowering the concurrency. Start from concurrency = 1 and gradually increase it to find an acceptable value. Refer to Setting concurrency for more details.
We have .net app that writes logs with help of Nlog logger to file source, logs are recorded all the time, every second. If you run it on Windows, no dockers, everything works fine: log records appear in the file immediately, but being deployed under our cluster of Linux dockers it takes from several minutes to hours to flash data into the file, which is shared with our host system. I can see data in the database, indicating that app ran successfully, but log file is not changed for a while. Having very little experience with dockers, not sure what it could be caused by, and even where to look at. I found yaml file that looks like this:
mount -v -t cifs //10.153.1.61/apps/configs/stage/testApp/logs
/logs/ -o credentials=/smb/smbcredentials;
As it works fine without docker, I believe something is wrong in the way we create images and deploy dockers. Any ideas on where to direct the investigation is very appreciated.
I think your see the slowness because you are trying to write your logs over the CIFS network share, that's a lot of overhead. You should consider using one of the commonly available distributed log handling solutions out there, such as gray log, ELK or Splunk(https://www.splunk.com/)
Looks like NLog has a lot of integrations to choose from. There are very detailed step by step tutorials available that explain the process in fine detail.
Using the centralized log collection not only will speed things up for you, it will let you query / combine logs from multiple containers and build graphs / dashboards giving more insight to you about the current status of your system.
We deployed a rails app in Google Cloud Run using their managed platform. The app is working fine and it is able to serve requests.
Now we want to get access to the rails console of the deployed app. Can anyone suggest a way to achieve this?
I'm aware that currently, Cloud Run supports only HTTP requests. If no other way is possible I'll have to consider something like rails web console
I think you cannot.
I'm familiar with Cloud Run but I'm not familiar with rails.
I assume you'd need to be able to shell into a container in order to be able to run IRB. Generally, you'd do this by asking the runtime (Docker Engine, Kubernetes, Cloud Run) to connect you to the container so that you could do this.
Cloud Run does not (appear) to permit this. I think it's a potentially useful feature request for the service. For those containers that contain shells, this would be the equivalent of GCE's gcloud compute ssh.
Importantly, your app may be serviced by multiple, load-balanced containers and so you'd want to be able to console into any of these.
However, you may wish to consider alternatives mechanisms for managing your app: monitoring, logging, trace etc. These mechanisms should provide you with sufficient insight into your app's state. Errant container instances should be terminated.
This follows the concept of "pets vs. cattle" whereby, instead of nurturing individual containers (is one failing?), you nurture the containers holistically (is the service comprising many containers failing?)
For completeness, if you think that there's an issue with a container image that you're unable to resolve through other means, you could run the image elsewhere (e.g. locally) where you can use IRB. Since the same container image will behave consistently wherever it's run, you should be able to observe the issue using IRB locally too.
Im using docker with my Web service.
when I deploy using Docker, loosing some logging files (nginx accesslog, service log, system log.. etc)
Cause, docker deployment system using down and up container architecures.
So I thought about this problem.
LoggingServer and serviceServer(for api) must seperate!
using these, methods..
First, Using logstash(in elk)(attaching all my logFile) .
Second, Using batch system, this batch system will moves logfiles to otherServer on every midnight.
isn't it okay?
I expect a better answer.
thanks.
There are many ways for logging which most the admin uses for containers
1 ) mount log directory to host , so even if docker goes up/down logs will be persisted on host.
2) ELK server, using logstash/filebeat for pushing logs to elastic search server with tailing option of file, so if new log contents it pushes to server.
3) if there is application logs like maven based projects, then there are many plugins which pushes logs to server
4) batch system , which is not recommended because if containers dies before mid-night then logs will be lost.
I have a DockerFile that starts 2 processes in a single docker container using a jar file and a config file as an argument
java -jar process1.jar process1.cfg &
java -jar process2.jar process2.cfg
process1.cfg and process2.cfg are residing in mounted directories. Now whenever there is a change in any of the cfg files, I would need to restart the corresponding process for the new change to take effect. All these to be done programmatically using Java in a REST microservice that updates the config file and restarts the process. Any idea on how to go about it ?
The problem can be generically solved by your Java app starting a config change monitoring service/thread, which manages the actual business service/thread(s) by starting it in the beginning and restarting on any change (if the change actually needs a restart). File change monitoring is standard Java functionality. The solution does not need any REST, it is not bound to microservice architecture (although it is more sensible within it) and it is not limited by or to docker containers.
If you do not want any file-based configs, do the same, but the monitoring bit can be e.g. a vert.x-based web server listening for external REST requests supplying configs, on start or for any update. The rest remains the same.
In my current workplace we actually have a module that functions in exactly this way, it is deployed to a docker and uses both file system monitoring and vert.x web server for config changes.
You can even go further and make the monitoring bit start multiple instances internally if multiple configs need to be supported.