I am trying to disable logging of IP addresses while handling the request. But I am not able find a way to do this. I want to disable logging of IP for limited portion of my app where user is not yet authenticated.
So my questions is
How to disable logging of IP in rails log for specific pages(so the IP will not be saved in any log)
I am using Rails 3.2.17
EDIT:
Here is sample log (from environment.log)
Started GET "/my_path" for 192.168.0.109 at 2014-03-28 11:53:20 +0530
I do not want to save 192.168.0.109 in log file
In config/initializers, add file log_fomat.rb with:
class ActiveSupport::BufferedLogger
def formatter=(formatter)
#log.formatter = formatter
end
end
class Formatter
SEVERITY_TO_TAG_MAP = {'DEBUG'=>'meh', 'INFO'=>'fyi', 'WARN'=>'hmm', 'ERROR'=>'wtf', 'FATAL'=>'omg', 'UNKNOWN'=>'???'}
SEVERITY_TO_COLOR_MAP = {'DEBUG'=>'0;37', 'INFO'=>'32', 'WARN'=>'33', 'ERROR'=>'31', 'FATAL'=>'31', 'UNKNOWN'=>'37'}
USE_HUMOROUS_SEVERITIES = true
def call(severity, time, progname, msg)
if USE_HUMOROUS_SEVERITIES
formatted_severity = sprintf("%-3s","#{SEVERITY_TO_TAG_MAP[severity]}")
else
formatted_severity = sprintf("%-5s","#{severity}")
end
formatted_time = time.strftime("%Y-%m-%d %H:%M:%S.") << time.usec.to_s[0..2].rjust(3)
color = SEVERITY_TO_COLOR_MAP[severity]
"\033[0;37m#{formatted_time}\033[0m [\033[#{color}m#{formatted_severity}\033[0m] #{msg.strip} (pid:#{$$})\n"
end
end
Rails.logger.formatter = Formatter.new
References:
http://rubyjunky.com/cleaning-up-rails-4-production-logging.html
http://cbpowell.wordpress.com/2012/04/05/beautiful-logging-for-ruby-on-rails-3-2/
Rails logger format string configuration
Finally did this by using emaillenin's answer thanx emaillenin :D.
Here is solution
# Overriding Rails logger to not save IP addresses for specific paths
# Put this file in <app_root>/config/initializers
# defining setter for Rails default log formatter, so later we can set our custom logger using '='
class ActiveSupport::BufferedLogger
def formatter=(formatter)
#log.formatter = formatter
end
end
# Modified Formatter Class with custom 'call' method
class Formatter
Format = "%s\n"
# Remove IP while getting request on below specified Path
FilteredActionRegexp = /app_path|another_path/i
# reference for regexp of IP address
# http://answers.oreilly.com/topic/318-how-to-match-ipv4-addresses-with-regular-expressions/
IPRegexp = /\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b/
FilteredString = '**FILTERED**'
def call(severity, time, progname, msg)
Format % [msg2str(filter_ip(msg))]
end
private
def msg2str(msg)
case msg
when ::String
msg
when ::Exception
"#{ msg.message } (#{ msg.class })\n" <<
(msg.backtrace || []).join("\n")
else
msg.inspect
end
end
# Replace IP Address with custom string if action is filtered
def filter_ip(msg)
# Replace only if message contains filtered action
if msg =~ FilteredActionRegexp
# If log string contains IP address then remove it with custom string
msg.gsub(IPRegexp, FilteredString )
else
msg
end
end
end
# Override Rails default logger formatter
Rails.logger.formatter = Formatter.new
I use Lograge
Taming Rails' Default Request Logging
Instead of having an unparsable amount of logging output like this:
Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
Processing by HomeController#index as HTML
Rendered text template within layouts/application (0.0ms)
Rendered layouts/_assets.html.erb (2.0ms)
Rendered layouts/_top.html.erb (2.6ms)
Rendered layouts/_about.html.erb (0.3ms)
Rendered layouts/_google_analytics.html.erb (0.4ms)
Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
you get a single line with all the important information, like this:
method=GET path=/jobs/833552.json format=json controller=jobs action=show status=200 duration=58.33 view=40.43 db=15.26
Related
Rails mailer instances do not have any context about the request. However this is needed to service multiple hosts. I was hoping to invoke url_for for the mailer, but I am confused to both the placing of it AND how it should be constructed (the api documentation provides examples for controllers, not mailers).
The form submission has a hidden_field :host, value: #site.host which percolates to the request
Parameters: {"authenticity_token"=>"[FILTERED]", "user"=>{"email"=>"some#quack.com", "host"=>"localhost"}
environments/development.rb has config.action_mailer.default_url_options commented out.
initializer for devise has config.parent_mailer = 'DeviseMailer'
The users/passwords_controller.rb has been edited
class Users::PasswordsController < Devise::PasswordsController
def create
puts params[:host]
super
end
and mailers/devise_mailer.rb sets
class DeviseMailer < ActionMailer::Base
layout 'mailer'
before_action :set_mailers_url_host
def set_mailers_url_host
puts 'host'
puts params
ActionMailer::Base.default_url_options[:host] = params[:user][:host]
end
I did not expect the mailer to know the parameter as it is designed to inherit from < ActionMailer::Base
However, the log is indicating that the password controller generated is not being invoked. the 'host' string is being put, then an empty line indicates the mailer know nothing of the params
Processing by Devise::PasswordsController#create as HTML
[...]
↳ app/controllers/application_controller.rb:284:in `get_departments'
User Load (2.5ms) SELECT "users"...
User Load (1.2ms) SELECT "users"...
TRANSACTION (1.0ms) BEGIN
User Update (1.7ms) UPDATE "users" SET "reset_password_token" = $1, "reset_password_sent_at" = $2 WHERE "users"."id" = $3 [["reset_password_token", "..."], ["reset_password_sent_at", "..."], ["id", 45]]
TRANSACTION (6.1ms) COMMIT
host
Devise::Mailer#reset_password_instructions: processed outbound mail in 0.6ms
Completed 500 Internal Server Error in 745ms (ActiveRecord: 81.6ms | Allocations: 203970)
NoMethodError (undefined method `[]' for nil:NilClass):
app/mailers/devise_mailer.rb:8:in `set_mailers_url_host'
The error is expected given params[:user][:host] is an unknown entity to the mailer. The bypassing of the passwords controller, not.
Also attempted: commenting out the devise_mailer before_action and adding to the application_controller.rb, where #site is set in before_action :set_site :
def default_url_options
{ host: #site.host, locale: I18n.locale }
end
While this is the most succinct way of dealing with the case, that fails with error ActionView::Template::Error (Missing host to link to! Please provide the :host parameter, set default_url_options[:host], or set :only_path to true)
Why are the checks (via puts) in the passwords controller not being invoked?
How should url_for thus establish the required string based on params[:user][:site]
This is a way to solve your issue but I'm not quite satisfied with the beauty of this. 🤔
around_action :change_host
def change_host
default_options = Rails.application.routes.default_url_options
new_options = { host: param[:user][:host], port: :thing }
Rails.application.routes.default_url_options = new_options
Rails.application.configuration.action_mailer.default_url_options = new_options
yield
Rails.application.routes.default_url_options = default_options
Rails.application.configuration.action_mailer.default_url_options = default_options
end
Given all the moving parts, a quick review:
comment out config.action_mailer.default_url_options
submission of param in form for host
no need for DeviseMailer class
setting default_url_options in application_controller not necessary
url_for is uncalled for
First element of answer: get the routing established correctly:
devise_for :users, controllers: { passwords: 'users/passwords' }
Second element of solution (preliminary - works in development on remote server), modify the devise passwords controller.
def create
ActionMailer::Base.default_url_options[:host] = params[:user][:host]
super
end
I use the gem “Lograge” 0.3.6, and Rails 4.2. I have this configured in my config/environments/development.rb file
config.lograge.enabled = true
config.lograge.formatter = CustomLogstash.new
However, I notice the output in my log/development.log file doesn’t contain date/times in front of each line. How do I configure lograge (or maybe just my Rails logger?) to prefix each line in that file with a date and time?
As per the document, the lograge gem provides below log formatters.
Lograge::Formatters::Lines.new # need to install "lines" gem
Lograge::Formatters::Cee.new
Lograge::Formatters::Graylog2.new
Lograge::Formatters::KeyValue.new # default lograge format
Lograge::Formatters::Json.new
Lograge::Formatters::Logstash.new # need to install "logstash-event" gem
Lograge::Formatters::LTSV.new
Lograge::Formatters::Raw.new # Returns a ruby hash object
By default the lograge gem uses Lograge::Formatters::KeyValue.new format for log.
You can customize this and make it universal by using your CustomLogStash class with some changes.
class CustomLogStash < Lograge::Formatters::KeyValue
def call(data)
# I'm using timestamp key here, you can choose whatever you want.
data_hash = { timestamp: Time.now.utc.strftime("%Y-%m-%dT%H:%M:%S.%3N")}.merge!(data)
super(data_hash)
end
end
Same way you can use any Lograge::Formatters class and apply the custom format to the log.
Now add below code to your config/initializers/lograge.rb file.
Rails.application.configure do
config.lograge.enabled = true
config.lograge.formatter = CustomLogStash.new
end
Now restart your server and load a page in your browser. You will see the logs something like below:
timestamp=2021-11-21T17:14:10.726 method=GET path=/categories format=html controller=categories action=index status=200 duration=259.11 view=244.91 db=2.60
EDITED
If you are looking for logs something like
then you don't need any gem for this. You can achieve this by adding below lines to your preferred environment development/production/test
config.log_level = :debug
config.log_formatter = ::Logger::Formatter.new
If you want to apply this across all environments then add above lines to the config/application.rb file.
Let me know if it helps.
Can you add the following in config if it helps.
config.lograge.formatter = ->(data) { data.reverse_merge({time: Time.now}) }# data is a ruby hash.
It will give you output like following
{:time=>2021-11-16 12:26:24.65362 +0000, :method=>"GET", :path=>"/", :format=>:html, :controller=>"Controller", :action=>"index", :status=>200, :duration=>393.41, :view=>85.55, :db=>1.38}
Can you paste contents of CustomLogstash class? From docs, this class should respond to call method and return Hash.
This works for me:
class CustomLogstash
def call(data)
{ time: Time.now, controller: data[:controller] } # this can be anything as long it is Hash, eg. data.merge(time: Time.now)
end
end
Sample output from above:
{:time=>"2021-11-18T20:31:41.486+01:00", :controller=>"calendar_events"}
As per official documentation for lograge, you can make use of custom_options
EDIT 1 : custom_options using time: Time.now or time:event.time
Rails.application.configure do
config.lograge.enabled = true
config.lograge.formatter = Lograge::Formatters::Logstash.new
# add time to lograge
config.lograge.custom_options = lambda do |event|
{ time: Time.now } #or use time:event.time
end
end
Note: When using the logstash output, you need to add the additional gem logstash-event. You can simply add it to your Gemfile like this
gem "logstash-event"
EDIT 2: Update based on comments custom_options using :time => event.time
#config/environments/production.rb
MyApp::Application.configure do
config.lograge.enabled = true
# add time to lograge
config.lograge.custom_options = lambda do |event|
{:time => event.time}
end
end
OR the below custom options which was a fix in lograge issue to ensure both date and time logged using time: event.time.to_s(:db)
config.lograge.custom_options = lambda do |event|
unwanted_keys = %w[format action controller utf8]
params = event.payload[:params].reject { |key,_| unwanted_keys.include? key }
{time: event.time.to_s(:db), user: event.payload[:user], params: params}
end
ALTERNATIVELY you can use this Custom logger
# Define a setter to pass in a custom log formatter
class ActiveSupport::BufferedLogger
def formatter=(formatter)
#log.formatter = formatter
end
end
# Defines a custom log format (time, severity, message, PID, backtrace)... all with color!
class Formatter
SEVERITY_TO_TAG = {'DEBUG'=>'meh', 'INFO'=>'fyi', 'WARN'=>'hmm', 'ERROR'=>'wtf', 'FATAL'=>'omg', 'UNKNOWN'=>'???'}
SEVERITY_TO_COLOR = {'DEBUG'=>'37', 'INFO'=>'32', 'WARN'=>'33', 'ERROR'=>'31', 'FATAL'=>'31', 'UNKNOWN'=>'37'}
HUMOR_FOR_ENV = {development: true, test: true, production: false}
DEPTH_FOR_ENV = {development: 3, test: 3, production: 1}
EXCLUSION_REGEX = /log|active_support|active_record/
def humorous?
return #is_humorous if defined? #is_humorous
#is_humorous = HUMOR_FOR_ENV[ Rails.env.to_sym ]
end
def depth
#depth ||= DEPTH_FOR_ENV[ Rails.env.to_sym ]
end
def call(severity, time, progname, msg)
t = time.strftime("%Y-%m-%d %H:%M:%S.") << time.usec.to_s[0..2].rjust(3)
color = SEVERITY_TO_COLOR[severity]
sev = humorous? ? "%-3s" % SEVERITY_TO_TAG[severity] # pad to at least 3 characters
: "%-5s" % severity # pad to at least 5 characters
# 2013-05-01 19:16:00.785 [omg] oh noes! (pid:30976) (admin/user.rb:45:in `block (4 levels) in <top (required)>') <- `call' <- `content_for' <- `block (2 levels) in row' <- `block in build_tag'
"\033[0;37m#{t}\033[0m [\033[#{color}m#{sev}\033[0m] #{msg.strip} (pid:#{$$}) #{whodunit}\033[0m\n"
end
def whodunit
latest, *others = caller.select{ |a| a !~ EXCLUSION_REGEX }[0, depth]
latest = latest[/(lib|app)\/(.*)/,-1] || latest
string = ""
string << "\033[36m(#{latest})"
string << "\033[35m <- " + others.map{ |s| s[/`.*/] }.join(' <- ') if others.any?
string
end
end
Rails.logger.formatter = Formatter.new
For Rails 4.2 don’t forget to add ActiveSupport::TaggedLogging to be able to call custom logger like a default rails logger
ActiveSupport::TaggedLogging is used to wrap any standard logger instance to add "tags" to a log statement. A "tag" in this case usually describes a subdomain, and is used by the default Rails.logger to allow you to tag log statements with subdomains, request ids, etc. in your multi-user, multi-instance production applications.
include
ActiveSupport::TaggedLogging::Formatter
to the Formatter class.
I'm trying to request the json pages for multiple subreddits and take the title and link from each page for a college project. here's the code in question:
require 'rufus-scheduler'
require 'json'
require 'httparty'
ENV['TZ'] = 'Europe/Dublin'
scheduler = Rufus::Scheduler::singleton
scheduler.every '12h00m', :first_at => Time.now + 10 do
array_of_subreddits = ["pics", "memes", "funny", "aww", "memes",
"birdswitharms"]
array_of_subreddits.each do |category|
sleep 10 #wait 10 seconds between each request
#response = JSON.parse(HTTParty.get("http://reddit.com/r/#{category}/.json?limit=25").body)
#response['data']['children'].each do |data|
#link = data['data']['url']
#title = data['data']['title']
#category = category
Pic.create([{:title => "#{#title}", :link => "#{#link}", :category => "#{#category}"}])
end
end
end
this works perfectly sometimes, it will run through each one and end as it should. more often than not however it will give me this message after after one or two passes:
NoMethodError (undefined method `[]' for nil:NilClass):
app/controllers/home_controller.rb:17:in `block in index'
app/controllers/home_controller.rb:9:in `each'
app/controllers/home_controller.rb:9:in `index'
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_source.erb (4.8ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_trace.html.erb (2.2ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_request_and_response.html.erb (1.2ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/diagnostics.html.erb within rescues/layout (66.2ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_markup.html.erb (0.4ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_inner_console_markup.html.erb within layouts/inlined_string (0.3ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_prompt_box_markup.html.erb within layouts/inlined_string (0.3ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/style.css.erb within layouts/inlined_string (0.5ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/console.js.erb within layouts/javascript (51.6ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/main.js.erb within layouts/javascript (0.3ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/error_page.js.erb within layouts/javascript (0.5ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/index.html.erb (124.8ms)
Creating client classes is a much better way to work with httparty:
class RedditClient
include HTTParty
format :json
base_uri "http://reddit.com/r/"
def self.get_category(category, *opts)
opts.reverse_merge(limit: 25)
get("/#{category}.json", opts)
end
end
This way HTTParty handles JSON parsing for us and does not attempt to an convert an empty response. Its also much easier to test separately.
However you should still check if the response was successful before trying to use it:
#response = RedditClient.get_category(category)
if #response.success?
attrs = #response['data']['children'].map do |child|
{
category: category,
link: child['data']['url'],
title: child['data']['title']
}
end
Pic.create!(attrs)
else
# log it or raise some sort of error
end
Note that you where passing an array containing a single hash to .create. You can instead pass an array of hashes and it will insert the records in a single SQL insert statement.
When you get errors like this, you should always dump the actual response so you can inspect it. The fact you got a error for a nil with code doing stuff like ['data']['children'] means Id guess you got a JSON response, but one that is missing one of the first items (e.g. ['data'] returned nil).
Dont just assume every request is successful, many things can make a HTTP fail. Its possible that you get a valid JSON response back, just not the one you are expecting, for example an error message which would have told you the problem.
Also even with a 10 second delay, you may be hitting a rate limit (never tested Reddit personally), but read the rules
Many default User-Agents (like "Python/urllib" or "Java") are drastically limited to encourage unique and descriptive user-agent strings.
This kind of errors are most common in ruby or rails. Can be handled in multiple ways. As #Stefan suggested you can use any of the bellow.
Most simply like this
response = HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
if response.success?
response_body = response.body
# continue
end
or
response = HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
case response.code
when 200
puts "Good!"
# Continue your parsing
when 404
puts "NOT FOUND!"
when 500...600
puts "ERROR #{response.code}"
end
or
begin
HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
rescue HTTParty::Error
# HTTParty errors like Not found
rescue StandardError
# StandardError like Timeout
else
# continue
end
I am working on rails json webservices. I am using warden for authentication in that, its working fine with html format , but in json its not working. because passed parameter is not working in config/intializers/wrden.rb file .below is the code
config/intializers/warden. rb
def authenticate!
Rails.logger.info '!!!!!!!!!!!!!!!!!!!!!!!!!!!!'
Rails.logger.info params['emailID']
user = User.find_by_emailID(params['emailID'])
if user && user.authenticate(params['password'])
success! user
else
fail "Invalid email or password"
end
end
here is log of this
Parameters: {"emailID"=>"xyz#gmail.com", "password"=>"123456", "session"=>{"emailID"=>"xyz#gmail.com", "password"=>"123456"}}
(0.2ms) BEGIN
!!!!!!!!!!!!!!!!!!!!!!!!!!!!
nil
User Load (12.2ms) SELECT "users".* FROM "users" WHERE "users"."emailID" IS **NULL** LIMIT 1
(0.3ms) COMMIT
Completed in 276ms
Well, I just ran into this, too. This GitHub thread fixed it for me:
https://github.com/hassox/warden/issues/84
Basically, tell Warden to use Rails' request object instead of whatever it's using by default. You can use the rails-warden gem or mix this into Warden:
module Warden::Mixins::Common
def request
#request ||= ActionDispatch::Request.new(#env)
end
end
This is my index.html.haml:
= stylesheet_link_tag 'user'
.title
%h1 Port Testing
= form_tag('port_testing/test', method: 'get') do
= text_field_tag :hostname, 'localhost', size: 50
= check_box_tag('Port 80', '80')
= label_tag('80')
= check_box_tag('Port 443', '443')
= label_tag('443')
= check_box_tag('Port 28009', '28009')
= label_tag('28009')
= check_box_tag('Port 2195', '2195')
= label_tag('2195')
%button(type="submit") Test
In my routes.rb, I have this:
match 'port_testing/test', :controller => :port_testing, :action=> :test
This is my port_testing_controller.rb:
class PortTestingController < ApplicationController
def index
end
def test
puts "\n"
puts #params["hostname"]
end
end
Right now when I click the "Test" button, I get this:
Started GET "/port_testing/test?utf8=%E2%9C%93&hostname=localhost" for 127.0.0.1
at 2012-03-07 13:51:33 -0500
Processing by PortTestingController#test as HTML
Parameters: {"utf8"=>"G£ô", "hostname"=>"localhost"}
Completed 500 Internal Server Error in 4ms
NoMethodError (You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.[]):
app/controllers/port_testing_controller.rb:7:in `test'
Rendered vendor/bundle/jruby/1.9/gems/actionpack-3.1.2/lib/action_dispatch/middl
eware/templates/rescues/_trace.erb (6.0ms)
Rendered vendor/bundle/jruby/1.9/gems/actionpack-3.1.2/lib/action_dispatch/middl
eware/templates/rescues/_request_and_response.erb (3.0ms)
Rendered vendor/bundle/jruby/1.9/gems/actionpack-3.1.2/lib/action_dispatch/middl
eware/templates/rescues/diagnostics.erb within rescues/layout (155.0ms)
How do I pass to the controller which checkboxes are checked along with what is entered in the text field?
The params variable in Rails isn't an instance variable, so your controller method should say:
def test
puts "\n"
puts params["hostname"]
end
I know I did it.. you can puts all the params, by looping through them.. do this.. I'd bet your value is in there already.