Consider the following. From my heroku console:
>> Rails.cache.stats
=> {"server_id"=>{"evictions"=>"0", "curr_items"=>"2064", "total_items"=>"18793", "bytes"=>"7674501", ...
>> Rails.cache.clear
=> [true]
>> Rails.cache.stats
=> {"server_id"=>{"evictions"=>"0", "curr_items"=>"2064", "total_items"=>"18793", "bytes"=>"7674501",
Super weird -- how can I clear my cache!!
Similar Issue ? : https://stackoverflow.com/q/7122513/192791
If you connect directly to the Dalli/memcahced client through the console and flush_all the cache clears.
i.e.
dc = Dalli::Client.new('localhost:11211')
dc.flush_all
NOTE: the stats take a while to update, but the cache will definitely clear.
The Expiring Cache section at http://devcenter.heroku.com/articles/building-a-rails-3-application-with-the-memcache-addon suggests using filters
after_save :expire_contact_all_cache
after_destroy :expire_contact_all_cache
def expire_contact_all_cache
Rails.cache.delete('Contact.all')
end
Related
I have 2 databases defined in my YAML: primary and datawarehouse.
With the following code:
pp ActiveRecord::Base.connection_config
ActiveRecord::Base.connected_to(database: :datawarehouse) do
ActiveRecord::Base.connection.execute('select 1')
end
pp ActiveRecord::Base.connection_config
I would expect the output to be the same before and after the block and let it get back to the default.
However the output before gives me:
=> {:adapter=>"postgresql", :host=>"postgres", :encoding=>"unicode", :migrations_paths=>"db/migrate", :database=>"rails_devise_production"}
And the output after the block keeps the connection to the datawarehouse.
=> {:adapter=>"postgresql", :host=>"postgres-archive", :encoding=>"unicode", :migrations_paths=>"db/datawarehouse_migrate", :database=>"archive" }
How is this possible? I would expect only the code in the block to be run against the other database connection.
There's a similar question in Rails repo issues. In short, connected_to with database key is specifically for one-off connections, and you need to use the role key in your application.
I'd like to start over with errbit - there are millions of records in our mongodb database and we hadn't been cleaning them up. I'd like to start over, but I don't want to lose my user accounts.
I've tried to run these routines (https://mensfeld.pl/2015/01/making-errbit-work-faster-by-keeping-it-clean-and-tidy/):
bundle exec rake errbit:clear_resolved
desc 'Resolves problems that didnt occur for 2 weeks'
task :cleanup => :environment do
offset = 2.weeks.ago
Problem.where(:updated_at.lt => offset).map(&:resolve!)
Notice.where(:updated_at.lt => offset).destroy_all
end
but the second one (deleting problems and notices over 2 weeks old), just seems to run forever.
Querying problems and notices collections via mongo shell doesn't seem to show any being deleted... we're using errbit V 0.7.0-dev and mongodb version 3.2.22.
Fastest way would be to get a mongo console and drop most of the collections. I'd say stop your errbit server, get a mongo console, connect to the db you use and run:
> db.errs.drop()
true
> db.problems.drop()
true
> db.backtraces.drop()
true
> db.notices.drop()
true
> db.comments.drop()
Problem.where(:updated_at.lt => 2.months.ago).destroy_all
runs too long because of N+1 problem with recursive deletion of Err, Notice and Comment, also mongoid does not support nested eager loading, so only way to delete faster - is to manually take these ids and delete directly, without callbacks:
problem_ids = Problem.where(:updated_at.lt => 2.months.ago).pluck(:id)
err_ids = Err.where(problem_id: {:$in => problem_ids}).pluck(:id)
Notice.where(err_id:{:$in => err_ids}).delete_all
Err.where(id:{:$in => err_ids}).delete_all
Comment.where(problem_id: {:$in => problem_ids}).delete_all
Problem.where(id: {:$in => problem_ids}).delete_all
Major edit: Since originally finding this issue I have whittled it down to the below. I think this is now a marginally more precise description of the problem. Comments on the OP may therefore not correlate entirely.
Edit lightly modified version posted in rails/puma projects: https://github.com/rails/rails/issues/21209, https://github.com/puma/puma/issues/758
Edit Now reproduced with OS X and Rainbows
Summary: When using Puma and running long-running connections I am consistently receiving errors related to ActiveRecord connections crossing threads. This manifests itself in message like message type 0x## arrived from server while idle and a locked (crashed) server.
The set up:
Ubuntu 15 / OSX Yosemite
PostgreSQL (9.4) / MySQL (mysqld 5.6.25-0ubuntu0.15.04.1)
Ruby - MRI 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux] / Rubinius rbx-2.5.8
Rails (4.2.3, 4.2.1)
Puma (2.12.2, 2.11)
pg (pg-0.18.2) / mysql2
Note, not all combinations of the above versions have been tried. First listed version is what I'm currently testing against.
rails new issue-test
Add a route get 'events' => 'streaming#events'
Add a controller streaming_controller.rb
Set up database stuff (pool: 2, but seen with different pool sizes)
Code:
class StreamingController < ApplicationController
include ActionController::Live
def events
begin
response.headers["Content-Type"] = "text/event-stream"
sse = SSE.new(response.stream)
sse.write( {:data => 'starting'} , {:event => :version_heartbeat})
ActiveRecord::Base.connection_pool.release_connection
while true do
ActiveRecord::Base.connection_pool.with_connection do |conn|
ActiveRecord::Base.connection.query_cache.clear
logger.info 'START'
conn.execute 'SELECT pg_sleep(3)'
logger.info 'FINISH'
sse.write( {:data => 'continuing'}, {:event => :version_heartbeat})
sleep 0.5
end
end
rescue IOError
rescue ClientDisconnected
ensure
logger.info 'Ensuring event stream is closed'
sse.close
end
render nothing: true
end
end
Puma configuration:
workers 1
threads 2, 2
#...
bind "tcp://0.0.0.0:9292"
#...
activate_control_app
on_worker_boot do
require "active_record"
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[rails_env])
end
Run the server puma -e production -C path/to/puma/config/production.rb
Test script:
#!/bin/bash
timeout 30 curl -vS http://0.0.0.0/events &
timeout 5 curl -vS http://0.0.0.0/events &
timeout 30 curl -vS http://0.0.0.0/events
This reasonably consistently results in a complete lock of the application server (in PostgreSQL, see notes). The scary message comes from libpq:
message type 0x44 arrived from server while idle
message type 0x43 arrived from server while idle
message type 0x5a arrived from server while idle
message type 0x54 arrived from server while idle
In the 'real-world' I have quite a few extra elements and the issue presents itself at random. My research indicates that this message comes from libpq and is subtext for 'communication problem, possibly using connection in different threads'. Finally, while writing this up, I had the server lock up without a single message in any log.
So, the question(s):
Is the pattern I'm following not legal in some way? What have I mis[sed|understood]?
What is the 'standard' for working with database connections here that should avoid these problems?
Can you see a way to reliably reproduce this?
or
What is the underlying issue here and how can I solve it?
MySQL
If running MySQL, the message is a bit different, and the application recovers (though I'm not sure if it is then in some undefined state):
F, [2015-07-30T14:12:07.078215 #15606] FATAL -- :
ActiveRecord::StatementInvalid (Mysql2::Error: This connection is in use by: #<Thread:0x007f563b2faa88#/home/dev/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.3/lib/action_controller/metal/live.rb:269 sleep>: SELECT `tasks`.* FROM `tasks` ORDER BY `tasks`.`id` ASC LIMIT 1):
Warning: read 'answer' as 'seems to make a difference'
I don't see the issue happen if I change the controller block to look like:
begin
#...
while true do
t = Thread.new do #<<<<<<<<<<<<<<<<<
ActiveRecord::Base.connection_pool.with_connection do |conn|
#...
end
end
t.join #<<<<<<<<<<<<<<<<<
end
#...
rescue IOError
#...
But I don't know whether this has actually solved the problem or just made it extremely unlikely. Nor can I really fathom why this would make a difference.
Posting this as a solution in case it helps, but still digging on the issue.
I have a fairly simple Rails app. It listens for requests in the form
example.com/items?access_key=my_secret_key
My application controller looks at the secret key to determine which user is making the call, looks up their database credentials, and connects to the appropriate database to get that person's items.
However we need to have this support multiple requests at a time, and Puma seems like everyone's favorite / the fastest server for us to use. We started running into problems when benchmarking it with ApacheBench. FYI, puma is configured to have 3 workers and min=1, max=16 threads.
If I were to run
ab -n 100 -c 10 127.0.0.1:3000/items?access_key=my_key
then this error is thrown with a whole lot of stack trace after it:
/home/user/.gem/ruby/2.0.0/gems/mysql2-0.3.16/lib/mysql2/client.rb:70: [BUG] Segmentation fault
ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]
Edit: This line also appears in the enormous amount of info that the error contains:
*** glibc detected *** puma: cluster worker 1: 17088: corrupted double-linked list: 0x00007fb671ddbd60
And it looks to me like that's tripping multiple times. I have been unable to determine exactly when (on which requests) it trips.
The benchmarking seems to still finish, but it seems quite slow (from ab):
Concurrency Level: 10
Time taken for tests: 21.085 seconds
Complete requests: 100
Total transferred: 3620724 bytes
21 seconds for 3 megabytes? Even if mysql was being slow, that's... bad. But I think it's worse than that - the amount of data isn't high enough. There are no segfaults when I run concurrency 1, and the amount of data for -n 10 -c 1 is 17 megabytes. So puma is responding with some error page that I can't see - running 'curl address' gives me the expected data, and I can't manually do concurrency.
It gets worse when I run more requests or higher concurrency.
ab -n 1000 -c 10 127.0.0.1:3000/items?access_key=my_key
yields
apr_socket_recv: Connection reset by peer (104)
Total of 199 requests completed
and
ab -n 100 -c 50 127.0.0.1:3000/items?access_key=my_key
yields
apr_socket_recv: Connection reset by peer (104)
Total of 6 requests completed
Running top in another putty window shows me that very often (most times I try to benchmark) only one of the three workers puma created is performing any work. Rarely, all three do.
Because it seems like the error might be somewhere in here, I'll show you my application_controller. It's short, but the bulk of the application (which, like I said, is fairly simple).
class ApplicationController < ActionController::Base
# Prevent CSRF attacks by raising an exception.
# For APIs, you may want to use :null_session instead.
protect_from_forgery with: :exception
def get_yaml_params
YAML.load(File.read("#{APP_ROOT}/config/ecommerce_config.yml"))
end
def access_key_login
access_key = params[:access_key]
unless access_key
show_error("missing access_key parameter")
return false
end
access_info = get_yaml_params
unless client_login = access_info[access_key]
show_error("invalid access_key")
return false
end
status = ActiveRecord::Base.establish_connection(
:adapter => "mysql2",
:host => client_login["host"],
:username => client_login["username"],
:password => client_login["password"],
:database => client_login["database"]
)
end
def generate_json (columns, footer)
// config/application.rb includes the line
// require 'json'
query = "select"
columns.each do |column, name|
query += " #{column}"
query += " AS #{name}" unless column == name
query += ","
end
query = query[0..-2] # trim ','
query += " #{footer}"
dbh = ActiveRecord::Base.connection
results = dbh.select_all(query).to_hash
data = results.map do |result|
columns.map {|column, name| result[name]}
end
({"fields" => columns.values, "values" => data}).to_json
end
def show_error(msg)
render(:text => "Error: #{msg}\n")
nil
end
end
And an example of a controller that uses it
class CategoriesController < ApplicationController
def index
access_key_login or return
columns = {
"prd_type" => "prd_type",
"prd_type_description" => "description"
}
footer = "from cin_desc;"
json = generate_json(columns, footer)
render(:json => json)
end
end
That's pretty much it as far as custom code goes. I can't find anything making this not threadsafe, so I don't know what the cause of the segfaults is. I don't know why not all of the workers spin up when requests are made. I don't know what error is getting returned to ApacheBench. Thanks for helping, I can post more information as you need it.
It appears that the stable version of mysql2 library, 0.3.17, is NOT threadsafe. Until it is updated to be threadsafe, using it with multithreaded puma will be impossible. An alternative would be to use Unicorn.
Is there a way to check the size of the Rails cache?
Something in the vein of: Rails.cache.size => 390 MB
I assume there's some slight variation between data stores, but right now I'm not sure how to even start to check the disk space a cache is taking up.
that totally depends on your cache store and the backend you use.
this is an example from my heroku instance running memcachier:
Rails.cache.stats
# => {"xxx.memcachier.com:11211"=>{"curr_items"=>"278", "bytes"=>"3423104", "evictions"=>"0", "total_items"=>"7373", "curr_connections"=>"7", "total_connections"=>"97", "cmd_get"=>"141674", "cmd_set"=>"7373", "cmd_delete"=>"350", "cmd_flush"=>"6", "get_hits"=>"63716", "get_misses"=>"77958", "delete_hits"=>"162", "delete_misses"=>"188", "incr_hits"=>"0", "incr_misses"=>"0", "decr_hits"=>"0", "decr_misses"=>"0"}}
FileStore does not have such a method:
Rails.cache.stats
# => NoMethodError: undefined method `stats' for #<ActiveSupport::Cache::FileStore:0x007ff1cbe905b0>
And when running a memcached locally, i get a different result set:
Rails.cache.stats
# => {"127.0.0.1:11211"=>{"pid"=>"327", "uptime"=>"517931", "time"=>"1392163858", "version"=>"1.4.16", "libevent"=>"2.0.21-stable", "pointer_size"=>"64", "rusage_user"=>"2.257386", "rusage_system"=>"4.345445", "curr_connections"=>"15", "total_connections"=>"16", "connection_structures"=>"16", "reserved_fds"=>"20", "cmd_get"=>"0", "cmd_set"=>"0", "cmd_flush"=>"0", "cmd_touch"=>"0", "get_hits"=>"0", "get_misses"=>"0", "delete_misses"=>"0", "delete_hits"=>"0", "incr_misses"=>"0", "incr_hits"=>"0", "decr_misses"=>"0", "decr_hits"=>"0", "cas_misses"=>"0", "cas_hits"=>"0", "cas_badval"=>"0", "touch_hits"=>"0", "touch_misses"=>"0", "auth_cmds"=>"0", "auth_errors"=>"0", "bytes_read"=>"48", "bytes_written"=>"30", "limit_maxbytes"=>"67108864", "accepting_conns"=>"1", "listen_disabled_num"=>"0", "threads"=>"4", "conn_yields"=>"0", "hash_power_level"=>"16", "hash_bytes"=>"524288", "hash_is_expanding"=>"0", "malloc_fails"=>"0", "bytes"=>"0", "curr_items"=>"0", "total_items"=>"0", "expired_unfetched"=>"0", "evicted_unfetched"=>"0", "evictions"=>"0", "reclaimed"=>"0"}}
In addition to #phoet's answer, for a Redis cache you can use the following to get a human readable format:
Rails.cache.stats["used_memory_human"] #=> 178.32M
Where used_memory_human can actually be any key that's returned when running an INFO command on the redis server.