Disable WAL in influxdb v0.13 or force flush - influxdb

I am trying to setup a simple cluster for influxdb on AWS using EFS (elastic file storage which is similar to NFS) where 2 different influx processes will share database.
I have 2 EC2 instances (named: EC2_1, EC2_2) running their own influxdb processes.
Both EC2 instances are mounted with a common EFS directory called "efs" at /opt/efs.
I modified influxdb.conf on both ec2 instances to have it point to common /opt/efs/influxdb/data directory for data, /opt/efs/influxdb/wal for wal,/opt/efs/influxdb/meta for meta directories.
What I am trying to achieve by this is when EC2_1 influx process writes something to database it can be immediately read by influx process on EC2_2.
To achieve this I am trying to find out a setting that can disable WAL or make writing to disk very frequent (once/sec) so that another process can read directly from disk. Read/Write performance or data loss is not a major concern for me right now.
I tried changing below setting hoping that it will make WAL flush frequent but it did not work the way I expected:
# CacheSnapshotMemorySize is the size at which the engine will
# snapshot the cache and write it to a TSM file, freeing up memory
# cache-snapshot-memory-size = 26214400
cache-snapshot-memory-size = 2
# CacheSnapshotWriteColdDuration is the length of time at
# which the engine will snapshot the cache and write it to
# a new TSM file if the shard hasn't received writes or deletes
# cache-snapshot-write-cold-duration = "1h"
cache-snapshot-write-cold-duration = "1s"
If EC2_1 writes something to influxdb, querying EC2_2 does not show that data unless influxdb process on EC2_2 is restarted and vice versa.

So there's not a way to disable the WAL entirely or to make it flush every second.

Related

Remove all data from Active Storage?

I would like to know how can I delete all data from Active Storage or even resetting Active Storage? There is any way to do that? Thank you in advance!
NOTE: I'm using Rails 5.2
This question challenged me, so I did some test on my dummy app with local storage.
I have the usual model User which has_one_attached :avatar
On local storage files are saved on /storage folder, under subfolders named randomly with a string of two characters.
Informations related to files are stored in two tables:
ActiveStorage::Attachment
ActiveStorage::Blob
To completely clean the two tables, I did in rails console:
ActiveStorage::Attachment.all.each { |attachment| attachment.purge }
This command deletes
All record in that table: ActiveStorage::Attachment.any? #=> false
All the blobs: ActiveStorage::Blob.any? #=> false
All the files located under /storage subfolders; of course, subfolders are still there empty.
The ActiveStorage still works poperly.
I expect the same behaviour for remote storage, having the right privileges.
No doubt ActiveStorage::Attachment.all.each { |attachment| attachment.purge } will purge all records, but it will take long time if you have lots of files.
For development env, you can simply remove all the attachment records from the database and remove files from the storage folder.
run rails dbconsole and execute the following queries to delete the attachment records.
delete from active_storage_attachments;
if you are storing variant in your database:
delete from active_storage_variant_records;
then finally,
delete from active_storage_blobs;
On my local development environment, whenever I perform rails db:reset, I also run rm -rf storage to clear all previously-saved files. Rails will automatically recreate the storage directory the next time a file is uploaded.

Creating only one log every day using Ruby standard Logger

I'm using ruby standard logger, I want rotational daily one, so in my code I have :
Logger.new("#{$ROOT_PATH}/log/errors.log", 'daily')
It is working perfectly, but it created two files errors.log.20130217 and errors.log.20130217.1.
How can I force it to create just one file a day ?
Your code is correct for a long-running application.
What's happening is you're running code more than once on a given day.
The first time you run it, Ruby creates a log file "errors.log".
When the day changes, Ruby renames the file to "errors.log.20130217".
But somehow you ran the code again, perhaps you're running two apps (or processes, or workers, or threads) that use similar code, and your logger saw that the file name "errors.log.20130217" already existed.
Your logger didn't want to clobber that file, but still needed to rename "errors.log" to a date, so the logger instead created a different file name "errors.log.20130217.1"
To solve this, run your code just once.
If you're running multiple apps called "foo" and "bar" then use log file names like "foo-errors.log" and "bar-errors.log". Or if you're using multiple workers, give each worker its own log file name (e.g. by using the worker's process id, or worker pool array index, or however you're keeping track of your workers).
If you really want to solve this using the Ruby logger, you'll need to override the logger #shift_log_period so it doesn't choose a ".1" suffix. You could subclass Logger and write your worn #shift_log_period to detect that there is an existing log file for the date, and if so, use it instead of doing the file rename.
This is the code causing it from the logger:
def shift_log_period(period_end)
postfix = period_end.strftime("%Y%m%d") # YYYYMMDD
age_file = "#{#filename}.#{postfix}"
if FileTest.exist?(age_file)
# try to avoid filename crash caused by Timestamp change.
idx = 0
# .99 can be overridden; avoid too much file search with 'loop do'
while idx < 100
idx += 1
age_file = "#{#filename}.#{postfix}.#{idx}"
break unless FileTest.exist?(age_file)
end
end
#dev.close rescue nil
File.rename("#{#filename}", age_file)
#dev = create_logfile(#filename)
return true
There is no solution (AFAIK) using the Ruby logger, with its built-in rotator, to manage logs written by multiple apps (a.k.a. workers, processes, threads) simultaneously. This is because each of the apps gets it own log file handle.
Alternatively, use any of the good log rotator tools, such as logrotate as suggested by the Tin Man user in the question comments: http://linuxcommand.org/man_pages/logrotate8.html
In general, logrotate will be your best bet IMHO.

Are there Mongo Admin fsync + lock commands available in Mongoid?

If I wanted to call the fsync + lock methods on my database, is there a way to do this with Mongoid in a Rails app? Is there also a way to only specify the replica node that I want to perform this operation on?
I'm trying to create a rake task to perform backups nightly using cron.
Mongoid 2 uses the 10gen supported driver.
Mongoid::Config.master.connection corresponds to the connection object of class Mongo::MongoClient (was Mongo::Connection).
This class has an instance method lock! which does the fsyncLock command, and unlock! is its mate.
http://api.mongodb.org/ruby/current/Mongo/MongoClient.html#lock!-instance_method
http://api.mongodb.org/ruby/current/Mongo/MongoClient.html#unlock!-instance_method
There are no options to these methods to specify member/s of a replica set,
only by socket which is essentially for internal use.
So if you need to fsyncLock a specific replica set member,
I recommend that you connect to it explicitly via an explicit connection,
e.g., Mongo::MongoClient.new(host, port).
client = Mongo::MongoClient.new(host, port)
client.lock!
# ...
client.unlock!
client.close
Mongoid 3 uses Moped and not the 10gen driver.
But you can still use the 10gen driver independently for your rake tasks even if you move to Mongoid 3.
I'm interested in your results and any followup questions.

One Redis server per Rails app?

I've a bunch of rails app on my server that should be able to use Redis as cache engine.
Do I've to start one instance of Redis for each of my application, or does the Redis support scoping?
I'm worried that if I remove one value in one app, the value with the same key will be removed for all of my apps.
I do NOT for example want this to happen.
App 1
Rails.cache.write("key", "value")
App 2
Rails.cache.read("key") => "value"
App 3
Rails.cache.delete("key")
App 1
Rails.cache.read("key") => nil
I suggest running a server for every application. Every additional Redis instance only uses 1 megabyte more memory when empty, so the overhead is small and it is ok to run tens of servers in a single instance. Also an idle Redis server is going to use a minimal amount of memory.
So basically by running multiple servers you are not wasting resources, but instead gaining speed as you'll use all your CPUs or CPU cores, given that Redis is single threaded.
A common method for this is one of two things:
Prefix your keys with some sort of "type" identifier, like app1:key, app2:key.
Use separate DBs for each using SELECT. By default, connections start out against DB 0. If you do SELECT 1 for app1, SELECT 2 for app2, etc., you are guaranteed that no other application will clobber that data.
The solution was to use redis-store with the namespace param.
Here is my config/production.rb file.
# App 1
config.cache_store = :redis_store, {path: "/tmp/redis.sock", db:1, namespace: "app1"}
# App 2
config.cache_store = :redis_store, {path: "/tmp/redis.sock", db:2, namespace: "app2"}

SQLite3::BusyException

Running a rails site right now using SQLite3.
About once every 500 requests or so, I get a
ActiveRecord::StatementInvalid (SQLite3::BusyException: database is locked:...
What's the way to fix this that would be minimally invasive to my code?
I'm using SQLLite at the moment because you can store the DB in source control which makes backing up natural and you can push changes out very quickly. However, it's obviously not really set up for concurrent access. I'll migrate over to MySQL tomorrow morning.
You mentioned that this is a Rails site. Rails allows you to set the SQLite retry timeout in your database.yml config file:
production:
adapter: sqlite3
database: db/mysite_prod.sqlite3
timeout: 10000
The timeout value is specified in miliseconds. Increasing it to 10 or 15 seconds should decrease the number of BusyExceptions you see in your log.
This is just a temporary solution, though. If your site needs true concurrency then you will have to migrate to another db engine.
By default, sqlite returns immediatly with a blocked, busy error if the database is busy and locked. You can ask for it to wait and keep trying for a while before giving up. This usually fixes the problem, unless you do have 1000s of threads accessing your db, when I agree sqlite would be inappropriate.
// set SQLite to wait and retry for up to 100ms if database locked
sqlite3_busy_timeout( db, 100 );
All of these things are true, but it doesn't answer the question, which is likely: why does my Rails app occasionally raise a SQLite3::BusyException in production?
#Shalmanese: what is the production hosting environment like? Is it on a shared host? Is the directory that contains the sqlite database on an NFS share? (Likely, on a shared host).
This problem likely has to do with the phenomena of file locking w/ NFS shares and SQLite's lack of concurrency.
If you have this issue but increasing the timeout does not change anything, you might have another concurrency issue with transactions, here is it in summary:
Begin a transaction (aquires a SHARED lock)
Read some data from DB (we are still using the SHARED lock)
Meanwhile, another process starts a transaction and write data (acquiring the RESERVED lock).
Then you try to write, you are now trying to request the RESERVED lock
SQLite raises the SQLITE_BUSY exception immediately (indenpendently of your timeout) because your previous reads may no longer be accurate by the time it can get the RESERVED lock.
One way to fix this is to patch the active_record sqlite adapter to aquire a RESERVED lock directly at the begining of the transaction by padding the :immediate option to the driver. This will decrease performance a bit, but at least all your transactions will honor your timeout and occurs one after the other. Here is how to do this using prepend (Ruby 2.0+) put this in a initializer:
module SqliteTransactionFix
def begin_db_transaction
log('begin immediate transaction', nil) { #connection.transaction(:immediate) }
end
end
module ActiveRecord
module ConnectionAdapters
class SQLiteAdapter < AbstractAdapter
prepend SqliteTransactionFix
end
end
end
Read more here: https://rails.lighthouseapp.com/projects/8994/tickets/5941-sqlite3busyexceptions-are-raised-immediately-in-some-cases-despite-setting-sqlite3_busy_timeout
Just for the record. In one application with Rails 2.3.8 we found out that Rails was ignoring the "timeout" option Rifkin Habsburg suggested.
After some more investigation we found a possibly related bug in Rails dev: http://dev.rubyonrails.org/ticket/8811. And after some more investigation we found the solution (tested with Rails 2.3.8):
Edit this ActiveRecord file: activerecord-2.3.8/lib/active_record/connection_adapters/sqlite_adapter.rb
Replace this:
def begin_db_transaction #:nodoc:
catch_schema_changes { #connection.transaction }
end
with
def begin_db_transaction #:nodoc:
catch_schema_changes { #connection.transaction(:immediate) }
end
And that's all! We haven't noticed a performance drop and now the app supports many more petitions without breaking (it waits for the timeout). Sqlite is nice!
bundle exec rake db:reset
It worked for me it will reset and show the pending migration.
Sqlite can allow other processes to wait until the current one finished.
I use this line to connect when I know I may have multiple processes trying to access the Sqlite DB:
conn = sqlite3.connect('filename', isolation_level = 'exclusive')
According to the Python Sqlite Documentation:
You can control which kind of BEGIN
statements pysqlite implicitly
executes (or none at all) via the
isolation_level parameter to the
connect() call, or via the
isolation_level property of
connections.
I had a similar problem with rake db:migrate. Issue was that the working directory was on a SMB share.
I fixed it by copying the folder over to my local machine.
Most answers are for Rails rather than raw ruby, and OPs question IS for rails, which is fine. :)
So I just want to leave this solution over here should any raw ruby user have this problem, and is not using a yml configuration.
After instancing the connection, you can set it like this:
db = SQLite3::Database.new "#{path_to_your_db}/your_file.db"
db.busy_timeout=(15000) # in ms, meaning it will retry for 15 seconds before it raises an exception.
#This can be any number you want. Default value is 0.
Source: this link
- Open the database
db = sqlite3.open("filename")
-- Ten attempts are made to proceed, if the database is locked
function my_busy_handler(attempts_made)
if attempts_made < 10 then
return true
else
return false
end
end
-- Set the new busy handler
db:set_busy_handler(my_busy_handler)
-- Use the database
db:exec(...)
What table is being accessed when the lock is encountered?
Do you have long-running transactions?
Can you figure out which requests were still being processed when the lock was encountered?
Argh - the bane of my existence over the last week. Sqlite3 locks the db file when any process writes to the database. IE any UPDATE/INSERT type query (also select count(*) for some reason). However, it handles multiple reads just fine.
So, I finally got frustrated enough to write my own thread locking code around the database calls. By ensuring that the application can only have one thread writing to the database at any point, I was able to scale to 1000's of threads.
And yea, its slow as hell. But its also fast enough and correct, which is a nice property to have.
I found a deadlock on sqlite3 ruby extension and fix it here: have a go with it and see if this fixes ur problem.
https://github.com/dxj19831029/sqlite3-ruby
I opened a pull request, no response from them anymore.
Anyway, some busy exception is expected as described in sqlite3 itself.
Be aware with this condition: sqlite busy
The presence of a busy handler does not guarantee that it will be invoked when there is
lock contention. If SQLite determines that invoking the busy handler could result in a
deadlock, it will go ahead and return SQLITE_BUSY or SQLITE_IOERR_BLOCKED instead of
invoking the busy handler. Consider a scenario where one process is holding a read lock
that it is trying to promote to a reserved lock and a second process is holding a reserved
lock that it is trying to promote to an exclusive lock. The first process cannot proceed
because it is blocked by the second and the second process cannot proceed because it is
blocked by the first. If both processes invoke the busy handlers, neither will make any
progress. Therefore, SQLite returns SQLITE_BUSY for the first process, hoping that this
will induce the first process to release its read lock and allow the second process to
proceed.
If you meet this condition, timeout isn't valid anymore. To avoid it, don't put select inside begin/commit. or use exclusive lock for begin/commit.
Hope this helps. :)
this is often a consecutive fault of multiple processes accessing the same database, i.e. if the "allow only one instance" flag was not set in RubyMine
Try running the following, it may help:
ActiveRecord::Base.connection.execute("BEGIN TRANSACTION; END;")
From: Ruby: SQLite3::BusyException: database is locked:
This may clear up the any transaction holding up the system
I believe this happens when a transaction times out. You really should be using a "real" database. Something like Drizzle, or MySQL. Any reason why you prefer SQLite over the two prior options?

Resources