As part of the security procedures for our Rails application, we want to have an audit log of the commands that were run from rails console in our production environment.
This audit log should be persisted somewhere, such as a database, a file in an S3 bucket, or similar. If it could be written to the standard rails log, that might be OK too, as we already have a way to persist that.
We're currently hosted on Heroku. Heroku will log the event of firing up a console, but (a) Heroku provides no functionality for logging commands run from an active console, and (b) we'd like a more general-purpose solution.
Are there any pre-existing solutions for this out there?
If there is none, I'm trying to figure out how to monkey-patch IRB or Rails Console.
I've discovered that the data I need is already in
Readline::HISTORY
but I'm struggling to figure out where/how to hook into it.
Ideally, I'd like to capture each entry as (before?) it is sent to the interpreter, and quickly persist it somewhere (we already have Resque, so that may be a good solution). FWIW, sending the contents of Readline::HISTORY on Kernel.exit seems like it may be unreliable. For example, if a SIGKILL stops the process, then the history contents wouldn't be saved.
Related
I would like to incorporate company policy into the Rails console so that any customer data being accessed via the console is logged and saved.
Ideally, whenever a developer opens up the console, it should ask him/her for the reason they are doing so along with the customer context if any, and then save any commands run in that session. This log should be stored somewhere and ideally land up in the Inbox of someone who monitors these things.
My question is if such a paradigm exists already on Heroku, and addon, or through a gem, or if not, some direction on how this can be implemented?
Heroku doesn't give you any way to do that automatically.
Consoles are code being executed though. So you could log the start of a bin/console wherever you want, and ask the person for their reason as the beginning of the session.
As mentioned by spickermann in the comments, people with push access would be able to override that. But Heroku logs all deployments. So if someone tried doing that, you'd know about it too.
The archer gem can log Rails console activity to a database.
I have an application hosted on Amazon EC2 on a Ubuntu machine, written in Ruby (on Rails), deployed with Capistrano and running on Nginx.
Last friday one module of my application has crashed and nobody in the company noticed until this morning. We spent some money with Facebook and Google ads and received a few hundreds of visits, but nobody created an account due to this bug.
I wonder if this configuration is saving the HTTP requests and its bodies somewhere in a log file. But we didnt explicitly set it, so it would only happen if any of these technologies do it by default.
Do you know whether there is such log or not?
Nope, that wouldn't be anywhere in a usable form (I'm inferring you want to try to create the accounts from request bodies in log files). You'll have the requests themselves in your nginx logs, and the rails logs will contain more info about the request, but as a matter of security, by default, any sensitive information (e.g. passwords) would be scrubbed from them. You may still be able to get some info from them.
To answer your question a little more specifically, the usual place for these logs on your system would be:
/var/log/nginx/
/path/to/your/rails/app/log/production.log
On a separate note, I would recommend looking into an error reporting service like Honeybadger, Airbrake, Raygun, Appsignal, or others so that you don't have silent failures like this moving forward.
We have custom Docker web app running in Elastic Beanstalk Docker container environment.
Would like to have application logs be available for viewing outside. Without downloading through instances or AWS console.
So far neither of solutions been acceptable. Maybe someone achieved centralised logging for Elastic Benastalk Dockerized apps?
Solution 1: AWS Console log download
not acceptable - requires to download logs, extract every time. Non real-time.
Solution 2: S3 + Elasticsearch + Fluentd
fluentd does not have plugin to retrieve logs from S3
There's excellent S3 plugin, but it's only for log output to S3. not for input logs from S3.
Solution 3: S3 + Elasticsearch + Logstash
cons: Can only pull all logs from entire bucket or nothing.
The problem lies with Elastic Beanstalk S3 Log storage structure. You cannot specify file name pattern. It's either all logs or nothing.
ElasticBeanstalk saves logs on S3 in path containing random instance and environment ids:
s3.bucket/resources/environments/logs/publish/e-<random environment id>/i-<random instance id>/my.log#
Logstash s3 plugin can be pointed only to resources/environments/logs/publish/. When you try to point it to environments/logs/publish/*/my.log it does not work.
which means you can not pull particular log and tag/type it to be able to find in Elasticsearch. Since AWS saves logs from all your environments and instances in same folder structure, you cannot chose even the instance.
Solution 4: AWS CloudWatch Console log viewer
It is possible to forward your custom logs to CloudWatch console. Do achieve that, put configuration files in .ebextensions path of your app bundle:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html
There's a file called cwl-webrequest-metrics.config which allows you to specify log files along with alerts, etc.
Great!? except that configuration file format is neither yaml,xml or Json, and it's not documented. There is absolutely zero mentions of that file, it's format either on AWS documentation website or anywhere on the net.
And to get one log file appear in CloudWatch is not simply adding a configuration line.
The only possible way to get this working seem to be trial and error. Great!? except for every attempt you need to re-deploy your environment.
There's only one reference to how to make this work with custom log: http://qiita.com/kozayupapa/items/2bb7a6b1f17f4e799a22 I have no idea how that person reverse engineered the file format.
cons:
Cloudwatch does not seem to be able to split logs into columns when displaying, so you can't easily filter by priority, etc.
AWS Console Log viewer does not have auto-refresh to follow logs.
Nightmare undocumented configuration file format, no way of testing. Trial and error requires re-deploying whole instance.
Perhaps an AWS Lambda function is applicable?
Write some javascript that dumps all notifications, then see what you can do with those.
After an object is written, you could rename it within the same bucket?
Or notify your own log-management service about the creation of a new object?
Lots of possibilities there...
I've started using Sumologic for the moment. There's a free trial and then a free tier (500mb /day, 7 day retention). I'm not out of the trial period yet and my EB app does literally nothing (it's just a few HTML pages serve by Nginx in a docker container. Looks like it could get expensive once you hit any serious amount of logs though.
It works ok so far. You need to create an IAM user that has access to the S3 bucket you want to read from and then it sucks the logs over to Sumologic servers and does all the processing and searching over there. Bit fiddly to set up, but I don't really see how it could be simpler and it's reasonably well-documented.
It lets you provide different path expressions with wildcards, then assign a "sourceCategory" to those different paths. You then use those sourceCategories to filter your log searching to a specific type of logging.
My plan long-term is to use something like your solution 3, but this got me going in very short order so I can move on to other things.
You can use a Multicontainer environment, sharing the log folder to another docker container with the tool of your preference to centralize the logs, in our case we connected an Apache Flume to move the files to an HDFS. Hope this helps you with this.
The easiest method I found to do this was using papertrail via rsyslog and .ebextensions, however it is very expensive for logging everything.
The good part is with rsyslog you can essentially send your logs anywhere and you are not tied to papertrail.
example ebextension
I've found loggly to be the most convenient.
It is a hosted service which might not be what you want. However if you check out their setup page you can see a number of ways your situation is supported (docker specific solutions, as well as like 10 amazon specific options). Even if loggly isn't to your taste, you can look at those solutions and easily see how some of them could be applied to most any centralized logging solution you might use or write.
I recently had a short term project served off of Heroku that approached 10,000 Postgres records and needed to be upgraded from hobby-dev to hobby-basic. Being new to Heroku, I did my best in finding out how to do so and followed heroku's upgrade using PG Backup guide. With a bit of trial and error, I thought I got it working. I erred on the side of caution and didn't do the last deletion of the old DB step. However, I soon got a message from Heroku telling me I've passed the 10,000 record limit. I went to my Heroku dashboard and checked out the DB size. It doesn't look like the Hobby-Basic DB had anything stored in it. I waited a couple weeks and checked again, thinking it may have been a realtime updating issue. Still no tables in the new Hobby-Basic DB.
So my question is, is there a way for me to find out which DB my Heroku app is set up to use currently? How do I force it to switch to a different DB in my app?
This is the guide I followed to upgrade from Hobby-Dev to Basic: https://devcenter.heroku.com/articles/upgrade-heroku-postgres-with-pgbackups
Presumably your app is reading from the environment variable DATABASE_URL. If you have multiple database then you should also have other environment variables like HEROKU_POSTGRESQL_(color). You can access those environment variables from within your app to inspect the configured database information, and you can change them in the Heroku panel. If you change DATABASE_URL to one of the values of HEROKU_POSTGRESQL_(color), then your app should pick up on that and switch to the different database. You can also edit your configuration and access another environment variable while bypassing DATABASE_URL entirely.
By default thin logs in a separate file (something like thin.0.log). This makes debugging hard since exceptions will be logged in thin.0.log while all the other information is in the Rails log. How can you force thin to log in the same file as Rails?
As annoying as it may be, that data is meant to be separated from the development.log or production.log (which is what I assume you're referring to). Thin logs handle everything related directly to the thin server as far as reasons why it can't start up, saying that it did, what port it's bound to, etc. The other logs are for your application itself, such as exceptions, page renders, requests, etc...
You'll be glad for this when you run into an issue with thin not starting due to a syntax error in a model or something. It's nicer to have one clean log for those instances, rather than filtering through thousands of lines of served static asset...
Rails is all about everything being organized and having its own place.