I've got an application that's been running in production for many months now, with tens of thousands of attachments. This morning, I tried to do an operation on one of these attachments, and got the following error:
Azure::Core::Http::HTTPError: BlobNotFound (404): The specified blob does not exist.
I can easily recreate this blob, but this situation makes me want to write a script to check the integrity of all of my attachments, and verify that no others have gone missing. (I expect that this was a transitory network error, and I expect to find very few, but I need the peace of mind.)
I can see that there is a method to call that seems to do exactly what I need: exist?(key), which is documented here:
https://github.com/rails/rails/blob/master/activestorage/lib/active_storage/service/disk_service.rb
However, I can't figure out how I'm supposed to call it. According to this, it's implemented as an instance method. So how do I reference my Rails application's active ActiveStorage instance (depending on environment) to use this method?
I wondered if it were possible to backtrack to get the instance of ActiveStorage::Service::AzureStorageService (in production), and found this answer on finding each instance of a class.
From there, I found that I could:
asass = ObjectSpace.each_object(ActiveStorage::Service::AzureStorageService).first
Which then allowed me to:
2.5.5 :015 > asass.exist?(c.json.blob.key)
AzureStorage Storage (313.3ms) Checked if file exists at key: ....PTLWNbEFLgeB8Y5x.... (yes)
=> true
Further digging around in the bowels of Rails' GitHub issues led me to see that the reference to the ActiveStorage service instance can be reached through an instance of a stored object, and that I could always be using the send() method to call one of its methods:
2.5.5 :045 > c.json.blob.service.send(:exist?, c.json.blob.key)
AzureStorage Storage (372.4ms) Checked if file exists at key: ....PTLWNbEFLgeB8Y5x.... (yes)
=> true
I'm sure both approaches are lighting up the same code paths, but I'm not sure which one I prefer at the moment. I still think there must exist some way to traverse the Rails instance through app. or Rails.application. to get to the ActiveStorage instance, but I haven't been able to suss that out.
I have this setup
config.cache_store = :redis_store, ENV['REDIS_CACHE_URL']
$ redis-cli
127.0.0.1:6379> set random_key 1
OK
Now I go to the console and do Rails.cache.clear which returns nil
And I am still able to access the key random_key in the redis-cli. It did not clear the cache.
I could not read what Rails.cache returns here too ruby/2.3.4/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails.rb
Is Rails.cache.clear is supposed to return true?
Can someone please help me out if my understanding is wrong?
redis-cache stores data under a particular namespace.
For example, if you've configured redis-store according to Documentation, then cache keys will be stored under cache namespace. That means, that when you Rails.cache.write("random_key", "key") a key cache:random_key will appear in the Redis. Therefore, when you Rails.cache.clear, only keys under cache namespace will be deleted.
Hence, if you manually create random_key in Redis, Rails.cache.clear won't remove it. But if you manually create cache:random_key, it will.
Be careful when using Rails.cache.clear it will invalidate all the keys for the application (source)
[~Not sure if this is the best place for this answer~]
This helpful article was a great way for me to understand caching when changing versions of Rails from 5.1+ to Rails 6.1+. The article talks about options for generating a cache key with or without versioning.
In the instance of my application, versioning was needed but not turned on when upgraded to Rails 6.1:
#in application.rb
config.active_record.collection_cache_versioning = true
Then within the application code where object.cache_key is called, I had to change it to object.cache_key_with_version (source)
I'm trying to debug stale entries in a cached view in a Rails (5.0.0.beta2) running on Heroku. I'd like to look at the entries in the cache to confirm that they are named the way that I expect and are getting expired when they should.
Is there any way to do this? I found this question, HOW do i see content of rails cache, which suggests Rails.cache.read("your_key"). So, using bin/rails c (on Heroku) I tried:
Rails.cache.read(User.find(19).cache_key) => nil
Where 19 is the :id of one of the users for whom I'm seeing stale data. This has me kind of stumped...
If I try:
User.find(19).cache_key => "users/19-20160316151228266421"
But when a cache entry is supposedly expired the log line looks like:
Expire fragment views/users/19-20160316151228266421 (0.2ms)
So I tried doing a Rails.cache.read on that path, this also returned nil – I also tried doing the same with a user that had not be expired, and got nil again.
I'm wondering if that difference in path signals a problem, or if there is a way to see the path of the key that is created (I've been assuming that it matches at least the part after the slash).
Cache has the following instance variables:
[:#options, :#data, :#key_access, :#max_size, :#max_prune_time, :#cache_size, :#monitor, :#pruning]
You can examine the data with:
Rails.cache.instance_variable_get(:#data)
I have a RoR application (ruby v1.8.7; rails v2.3.5) that is caching a page in the development environment. This wouldn't be so much of an issue, but the cached page's a elements are incorrect.
I haven't made any changes to the development.rb file and I haven't knowingly added any caching commands to the controllers.
I've tried clearing the browser's (Firefox 3.5 on OSX) cookie and page caches for this site (localhost). I've also restarted Mongrel. Nothing seems to help.
What am I missing?
This line in development.rb ensures that caching is not happening.
config.action_controller.perform_caching = false
You can clear the Rails cache with
Rails.cache.clear
That said - I am not convinced this is a caching issue. Are you making changes to the page and not seeing them reflected? You aren't perhaps looking at the live version of that page? I have done that once (blush).
Update:
You can call that command from in the console.
Are you sure you are running the application in development?
The only alternative is that the page that you are trying to render isn't the page that is being rendered.
If you watch the server output you should be able to see the render command when the page is rendered similar to this:
Rendered shared_partials/_latest_featured_video (31.9ms)
Rendered shared_partials/_s_invite_friends (2.9ms)
Rendered layouts/_sidebar (2002.1ms)
Rendered layouts/_footer (2.8ms)
Rendered layouts/_busy_indicator (0.6ms)
rake tmp:cache:clear might be what you're looking for.
I was able to resolve this problem by cleaning my assets cache:
$ rake assets:clean
Check for a static version of your page in /public and delete it if it's there. When Rails 3.x caches pages, it leaves a static version in your public folder and loads that when users hit your site. This will remain even after you clear your cache.
More esoteric ways:
Rails.cache.delete_matched("*")
For Redis:
Redis.new.keys.each{ |key| Rails.cache.delete(key) }
If you're doing fragment caching, you can manually break the cache by updating your cache key, like so:
Version #1
<% cache ['cool_name_for_cache_key', 'v1'] do %>
Version #2
<% cache ['cool_name_for_cache_key', 'v2'] do %>
Or you can have the cache automatically reset based on the state of a non-static object, such as an ActiveRecord object, like so:
<% cache #user_object do %>
With this ^ method, any time the user object is updated, the cache will automatically be reset.
Running a rails site right now using SQLite3.
About once every 500 requests or so, I get a
ActiveRecord::StatementInvalid (SQLite3::BusyException: database is locked:...
What's the way to fix this that would be minimally invasive to my code?
I'm using SQLLite at the moment because you can store the DB in source control which makes backing up natural and you can push changes out very quickly. However, it's obviously not really set up for concurrent access. I'll migrate over to MySQL tomorrow morning.
You mentioned that this is a Rails site. Rails allows you to set the SQLite retry timeout in your database.yml config file:
production:
adapter: sqlite3
database: db/mysite_prod.sqlite3
timeout: 10000
The timeout value is specified in miliseconds. Increasing it to 10 or 15 seconds should decrease the number of BusyExceptions you see in your log.
This is just a temporary solution, though. If your site needs true concurrency then you will have to migrate to another db engine.
By default, sqlite returns immediatly with a blocked, busy error if the database is busy and locked. You can ask for it to wait and keep trying for a while before giving up. This usually fixes the problem, unless you do have 1000s of threads accessing your db, when I agree sqlite would be inappropriate.
// set SQLite to wait and retry for up to 100ms if database locked
sqlite3_busy_timeout( db, 100 );
All of these things are true, but it doesn't answer the question, which is likely: why does my Rails app occasionally raise a SQLite3::BusyException in production?
#Shalmanese: what is the production hosting environment like? Is it on a shared host? Is the directory that contains the sqlite database on an NFS share? (Likely, on a shared host).
This problem likely has to do with the phenomena of file locking w/ NFS shares and SQLite's lack of concurrency.
If you have this issue but increasing the timeout does not change anything, you might have another concurrency issue with transactions, here is it in summary:
Begin a transaction (aquires a SHARED lock)
Read some data from DB (we are still using the SHARED lock)
Meanwhile, another process starts a transaction and write data (acquiring the RESERVED lock).
Then you try to write, you are now trying to request the RESERVED lock
SQLite raises the SQLITE_BUSY exception immediately (indenpendently of your timeout) because your previous reads may no longer be accurate by the time it can get the RESERVED lock.
One way to fix this is to patch the active_record sqlite adapter to aquire a RESERVED lock directly at the begining of the transaction by padding the :immediate option to the driver. This will decrease performance a bit, but at least all your transactions will honor your timeout and occurs one after the other. Here is how to do this using prepend (Ruby 2.0+) put this in a initializer:
module SqliteTransactionFix
def begin_db_transaction
log('begin immediate transaction', nil) { #connection.transaction(:immediate) }
end
end
module ActiveRecord
module ConnectionAdapters
class SQLiteAdapter < AbstractAdapter
prepend SqliteTransactionFix
end
end
end
Read more here: https://rails.lighthouseapp.com/projects/8994/tickets/5941-sqlite3busyexceptions-are-raised-immediately-in-some-cases-despite-setting-sqlite3_busy_timeout
Just for the record. In one application with Rails 2.3.8 we found out that Rails was ignoring the "timeout" option Rifkin Habsburg suggested.
After some more investigation we found a possibly related bug in Rails dev: http://dev.rubyonrails.org/ticket/8811. And after some more investigation we found the solution (tested with Rails 2.3.8):
Edit this ActiveRecord file: activerecord-2.3.8/lib/active_record/connection_adapters/sqlite_adapter.rb
Replace this:
def begin_db_transaction #:nodoc:
catch_schema_changes { #connection.transaction }
end
with
def begin_db_transaction #:nodoc:
catch_schema_changes { #connection.transaction(:immediate) }
end
And that's all! We haven't noticed a performance drop and now the app supports many more petitions without breaking (it waits for the timeout). Sqlite is nice!
bundle exec rake db:reset
It worked for me it will reset and show the pending migration.
Sqlite can allow other processes to wait until the current one finished.
I use this line to connect when I know I may have multiple processes trying to access the Sqlite DB:
conn = sqlite3.connect('filename', isolation_level = 'exclusive')
According to the Python Sqlite Documentation:
You can control which kind of BEGIN
statements pysqlite implicitly
executes (or none at all) via the
isolation_level parameter to the
connect() call, or via the
isolation_level property of
connections.
I had a similar problem with rake db:migrate. Issue was that the working directory was on a SMB share.
I fixed it by copying the folder over to my local machine.
Most answers are for Rails rather than raw ruby, and OPs question IS for rails, which is fine. :)
So I just want to leave this solution over here should any raw ruby user have this problem, and is not using a yml configuration.
After instancing the connection, you can set it like this:
db = SQLite3::Database.new "#{path_to_your_db}/your_file.db"
db.busy_timeout=(15000) # in ms, meaning it will retry for 15 seconds before it raises an exception.
#This can be any number you want. Default value is 0.
Source: this link
- Open the database
db = sqlite3.open("filename")
-- Ten attempts are made to proceed, if the database is locked
function my_busy_handler(attempts_made)
if attempts_made < 10 then
return true
else
return false
end
end
-- Set the new busy handler
db:set_busy_handler(my_busy_handler)
-- Use the database
db:exec(...)
What table is being accessed when the lock is encountered?
Do you have long-running transactions?
Can you figure out which requests were still being processed when the lock was encountered?
Argh - the bane of my existence over the last week. Sqlite3 locks the db file when any process writes to the database. IE any UPDATE/INSERT type query (also select count(*) for some reason). However, it handles multiple reads just fine.
So, I finally got frustrated enough to write my own thread locking code around the database calls. By ensuring that the application can only have one thread writing to the database at any point, I was able to scale to 1000's of threads.
And yea, its slow as hell. But its also fast enough and correct, which is a nice property to have.
I found a deadlock on sqlite3 ruby extension and fix it here: have a go with it and see if this fixes ur problem.
https://github.com/dxj19831029/sqlite3-ruby
I opened a pull request, no response from them anymore.
Anyway, some busy exception is expected as described in sqlite3 itself.
Be aware with this condition: sqlite busy
The presence of a busy handler does not guarantee that it will be invoked when there is
lock contention. If SQLite determines that invoking the busy handler could result in a
deadlock, it will go ahead and return SQLITE_BUSY or SQLITE_IOERR_BLOCKED instead of
invoking the busy handler. Consider a scenario where one process is holding a read lock
that it is trying to promote to a reserved lock and a second process is holding a reserved
lock that it is trying to promote to an exclusive lock. The first process cannot proceed
because it is blocked by the second and the second process cannot proceed because it is
blocked by the first. If both processes invoke the busy handlers, neither will make any
progress. Therefore, SQLite returns SQLITE_BUSY for the first process, hoping that this
will induce the first process to release its read lock and allow the second process to
proceed.
If you meet this condition, timeout isn't valid anymore. To avoid it, don't put select inside begin/commit. or use exclusive lock for begin/commit.
Hope this helps. :)
this is often a consecutive fault of multiple processes accessing the same database, i.e. if the "allow only one instance" flag was not set in RubyMine
Try running the following, it may help:
ActiveRecord::Base.connection.execute("BEGIN TRANSACTION; END;")
From: Ruby: SQLite3::BusyException: database is locked:
This may clear up the any transaction holding up the system
I believe this happens when a transaction times out. You really should be using a "real" database. Something like Drizzle, or MySQL. Any reason why you prefer SQLite over the two prior options?