Using heroku logs --tail which works great for a few minutes. Then it stops displaying the logs. It seems that the ssh connection is timing out and dying. There is no error or message. Working in Ubuntu 11.04 on wired conneciton.
I added the following to ~/.ssh/config:
ServerAliveInterval 5
But it didn't work. Do I need anything else in the config file? How do I know if it is doing anything? How can I monitor the traffic and see the keepalive request? I am looking at System Monitor but don't see anything every 5 seconds.
Thanks.
Have you done all of this:
$ heroku config:add LOG_LEVEL=DEBUG
$ heroku addons:upgrade logging:expanded
$ heroku logs --tail
It turns out that I was looking for an answer to the wrong question. Why use the tail to save logs? It is problematic, labor intensive, and error prone.
The solution I found was Papertrail. Small sites are free. papertrailapp.com.
Here's the complete story from my blog: http://www.onlineinvestingai.com/blog/2011/08/07/better-logging-with-papertrail-on-heroku/
I have seen the same issue. I'm not certain that it is ssh that times out but something does. For now, we've placed our monitor in a loop so it resumes automatically in case of time-out.
We also use PaperTrail, however it has limits in the amount that you can transfer. We use PaperTrail for general purpose use, and tail the logs for the excrutiating-detail logs that quickly use up all avaiable PaperTrail capacity.
Related
I donn't want/need 'auditd' in my LINUX.
Problem is that this not a standard LINUX distribution rather a special, minimized version called 'CoreElec' (www.coreelec.org)
So there is no 'apt', 'auditctl'. crontab is empty.
Docker is issues the auditing. If I stop Docker, auditing messages also stop.
I can find no way to stop it or at least to stop its logging, resulting in a syslog full of audit log messages.
If I could at least get rid of logs, it would help.
Thanks for any help.
Gerry
I tried uploading a csv with 2700 rows and my service (running in a docker container on A.W.S) stopped running after a some seconds but the upload was complete (as all the data is present in my database). The logs (cloudwatch) do not show any error, instead the service is stopped and restarted (sometimes successfully and sometimes not).
I found a similar issue that Heroku has here, where the answer says Heroku has a 30 second timeout on all request, does A.W.S have something similar? If not how can overcome this as CSV uploads are frequent at my workplace.
Thank you.
I would suggest to use sidekiq for uploading data in bulk amount. It does this job at background. Moreover I have run the issue where even sidekiq stopped if you face similar issue then I would recommend God gem for monitoring purpose for sidekiq.
Regarding AWS timeout information kindly have a look on this. Elastic Load Balancing
As described here, Heroku logs metrics for your dynos every ~20 seconds.
source=web.1 dyno=heroku.2808254.d97d0ea7-cf3d-411b-b453-d2943a50b456 sample#load_avg_1m=2.46 sample#load_avg_5m=1.06 sample#load_avg_15m=0.99
source=web.1 dyno=heroku.2808254.d97d0ea7-cf3d-411b-b453-d2943a50b456 sample#memory_total=21.00MB sample#memory_rss=21.22MB sample#memory_cache=0.00MB sample#memory_swap=0.00MB sample#memory_pgpgin=348836pages sample#memory_pgpgout=343403pages
These pile up quickly and distract from the log view, especially when they are every 20 seconds and there are 10+ dynos for a small web app.
Is it possible to change the frequency at which these are logged? Something like once per minute would cut the quantity down by a factor of 3 and would still be sufficient for my app's needs.
Thanks!
Edit: Heroku does provide a way of filter out certain types of logs while tailing, so in theory I could get rid of them entirely by excluding the system logs. But if there are other valuable things in the system logs, the question of how to specifically reduce (not filter out) these metrics still stands.
We had a similar problem and decided to just turn off the runtime-metrics logging altogether.
You can check to see if it's enabled with:
$ heroku labs -a you-app-name
=== App Features (your-app-name)
[+] log-runtime-metrics Emit dyno resource usage information into app logs
and remove it with
heroku labs:disable log-runtime-metrics -a your-app-name
heroku restart -a your-app-name
Situation: I am using Rails + Unicorn, deploying with Capistrano. Sometimes Rails app fails to start in production mode (though it is not the real production, but a staging env). This usually happens due to errors in deploy scripts or configuration (thus usually not detectable by tests). When this happens, unicorn master process kills the worker that failed and spawns a new one, which also fails and so on and so forth. During all that time unicorn consumes lots of CPU and pollutes logs with the same message.
Manual way (not good): Go to your home page to see if it works. Look at the htop. Tail the logs. Kill unicorn manually. Cons: easy to forget. Logs are polluted, CPU is loaded while you are reacting.
Another solution: Use unicorn's preload_app true. This will cause master process to fail fast. Cons: higher memory consumption in happy scenario.
Best practice: - ???
Is there any way to cleverly detect that unicorn master uselessly tries to spawn failing children and stop it?
You have something like "unicorn start" in your Capistrano script right? Make your Capistrano script ping Unicorn right after invoking that command. If Unicorn does not return an expected response within a timeout, then you know that something went wrong, and you can choose to rollback the deploy or performing some other action.
As for how to ping Unicorn, that depends. If you have Unicorn listening on a TCP socket then you can use curl. If you have Unicorn listening on a Unix domain socket then you have to write a little script that connects to it, like this:
require 'socket'
sock = UNIXSocket.new('/path-to-unicorn.sock')
sock.write("HEAD / HTTP/1.0\r\n")
sock.write("Host: www.foo.com\r\n")
sock.write("Connection: close\r\n")
sock.write("\r\n")
if sock.read !~ /something/
exit 1
end
But it sounds like Phusion Passenger Enterprise solves your problem beautifully. It has this feature called "deployment error resistance". When you deploy a new version and Phusion Passenger detects that it cannot spawn any processes for your new codebase, it will stop trying to spawn your new version and keep the processes for the old versions around indefinitely, until you manually give the signal that it's okay to spawn processes for the new version. In the mean time it will log all errors into the log file so that you can analyze the problem.
I would suggest brushing off your bash skills. The functionality you need is already in Unicorn as it leverages the Unix-y master/worker process.
You need a init.d script. Or at the very least godrb or monit. I recommend the init.d script route AND monitoring. Its more complex, but it can more easily be leveraged by your monitoring software and also gives you an automatic start on reboot.
The gist of it is:
Send the USR2 signal to the unicorn master process, this will fork the master process.
Then send the WINCH to the old master process that gets created, this will kill each worker.
Then you can send the old master process the QUIT signal.
Unicorn Signals
This will spin up a new master process running the new code and label the old one as (old). If it fails the old one should be returned to its prior state and you shouldn't suffer an outage, just a restart error. This is the beauty of unicorn. You can almost get instantaneous deploys of your code.
I'm using a lot of hedge words because I did this work on my apps over a year ago so there are a lot of cobwebs upstairs. Hope this helps!
This is by no mean a correct script. Its a good starting point though ... feel free to update the gist if you can improve upon it! :-)
Example Unicorn Control Script
So right now I have an implementation of delayed_job that works perfectly on my local development environment. In order to start the worker on my machine, I just run rake jobs:work and it works perfectly.
To get delayed_job to work on heroku, I've been using pretty much the same command: heroku run rake jobs:work. This solution works, without me having to pay anything for worker costs to Heroku, but I have to keep my command prompt window open or else the delayed_job worker stops when I close it. Is there a command to permanently keep this delayed_job worker working even when I close the command window? Or is there another better way to go about this?
I recommend the workless gem to run delayed jobs on heroku. I use this now - it works perfectly for me, zero hassle and at zero cost.
I have also used hirefireapp which gives a much finer degree of control on scaling workers. This costs, but costs less than a single heroku worker (over a month). I don't use this now, but have used it, and it worked very well.
Add
worker: rake jobs:work
to your Procfile.
EDIT:
Even if you run it from your console you 'buy' worker dyno but Heroku has per second biling. So you don't pay because you have 750h free, and month in worst case has 744h, so you have free 6h for your extra dynos, scheduler tasks and so on.
I haven't tried it personally yet, but you might find nohup useful. It allows your process to run even though you have closed your terminal window. Link: http://linux.101hacks.com/unix/nohup-command/
Using heroku console to get workers onto the jobs will only create create a temporary dyno for the job. To keep the jobs running without cli, you need to put the command into the Procfile as #Lucaksz suggested.
After deployment, you also need to scale the dyno formation, as heroku need to know how many dyno should be put onto the process type like this:
heroku ps:scale worker=1
More details can be read here https://devcenter.heroku.com/articles/scaling