Creating a multi-tenant application using PostgreSQL's schemas and Rails - ruby-on-rails

Stuff I've already figured out
I'm learning how to create a multi-tenant application in Rails that serves data from different schemas based on what domain or subdomain is used to view the application.
I already have a few concerns answered:
How can you get subdomain-fu to work with domains as well? Here's someone that asked the same question which leads you to this blog.
What database, and how will it be structured? Here's an excellent talk by Guy Naor, and good question about PostgreSQL and schemas.
I already know my schemas will all have the same structure. They will differ in the data they hold. So, how can you run migrations for all schemas? Here's an answer.
Those three points cover a lot of the general stuff I need to know. However, in the next steps I seem to have many ways of implementing things. I'm hoping that there's a better, easier way.
Finally, to my question
When a new user signs up, I can easily create the schema. However, what would be the best and easiest way to load the structure that the rest of the schemas already have? Here are some questions/scenarios that might give you a better idea.
Should I pass it on to a shell script that dumps the public schema into a temporary one, and imports it back to my main database (pretty much like what Guy Naor says in his video)? Here's a quick summary/script I got from the helpful #postgres on freenode. While this will probably work, I'm gonna have to do a lot of stuff outside of Rails, which makes me a bit uncomfortable.. which also brings me to the next question.
Is there a way to do this straight from Ruby on Rails? Like create a PostgreSQL schema, then just load the Rails database schema (schema.rb - I know, it's confusing) into that PostgreSQL schema.
Is there a gem/plugin that has these things already? Methods like "create_pg_schema_and_load_rails_schema(the_new_schema_name)". If there's none, I'll probably work at making one, but I'm doubtful about how well tested it'll be with all the moving parts (especially if I end up using a shell script to create and manage new PostgreSQL schemas).
Thanks, and I hope that wasn't too long!

Update Dec 5, 2011
Thanks to Brad Robertson and his team, there's the Apartment gem. It's very useful and does a lot of the heavy lifting.
However, if you'll be tinkering with schemas, I strongly suggest knowing how it actually works. Familiarize yourself with Jerod Santo's walkthrough , so you'll know what the Apartment gem is more or less doing.
Update Aug 20, 2011 11:23 GMT+8
Someone created a blog post and walks though this whole process pretty well.
Update May 11, 2010 11:26 GMT+8
Since last night I've been able to get a method to work that creates a new schema and loads schema.rb into it. Not sure if what I'm doing is correct (seems to work fine, so far) but it's a step closer at least. If there's a better way please let me know.
module SchemaUtils
def self.add_schema_to_path(schema)
conn = ActiveRecord::Base.connection
conn.execute "SET search_path TO #{schema}, #{conn.schema_search_path}"
end
def self.reset_search_path
conn = ActiveRecord::Base.connection
conn.execute "SET search_path TO #{conn.schema_search_path}"
end
def self.create_and_migrate_schema(schema_name)
conn = ActiveRecord::Base.connection
schemas = conn.select_values("select * from pg_namespace where nspname != 'information_schema' AND nspname NOT LIKE 'pg%'")
if schemas.include?(schema_name)
tables = conn.tables
Rails.logger.info "#{schema_name} exists already with these tables #{tables.inspect}"
else
Rails.logger.info "About to create #{schema_name}"
conn.execute "create schema #{schema_name}"
end
# Save the old search path so we can set it back at the end of this method
old_search_path = conn.schema_search_path
# Tried to set the search path like in the methods above (from Guy Naor)
# [METHOD 1]: conn.execute "SET search_path TO #{schema_name}"
# But the connection itself seems to remember the old search path.
# When Rails executes a schema it first asks if the table it will load in already exists and if :force => true.
# If both true, it will drop the table and then load it.
# The problem is that in the METHOD 1 way of setting things, ActiveRecord::Base.connection.schema_search_path still returns $user,public.
# That means that when Rails tries to load the schema, and asks if the tables exist, it searches for these tables in the public schema.
# See line 655 in Rails 2.3.5 activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
# That's why I kept running into this error of the table existing when it didn't (in the newly created schema).
# If used this way [METHOD 2], it works. ActiveRecord::Base.connection.schema_search_path returns the string we pass it.
conn.schema_search_path = schema_name
# Directly from databases.rake.
# In Rails 2.3.5 databases.rake can be found in railties/lib/tasks/databases.rake
file = "#{Rails.root}/db/schema.rb"
if File.exists?(file)
Rails.logger.info "About to load the schema #{file}"
load(file)
else
abort %{#{file} doesn't exist yet. It's possible that you just ran a migration!}
end
Rails.logger.info "About to set search path back to #{old_search_path}."
conn.schema_search_path = old_search_path
end
end

Change line 38 to:
conn.schema_search_path = "#{schema_name}, #{old_search_path}"
I presume that postgres is trying to lookup existing table names when loading schema.rb and since you've set the search_path to only contain the new schema, it fails. This of course, is presuming you still have the public schema in your database.
Hope that helps.

Is there a gem/plugin that has these things already?
pg_power provides this functionality to create/drop PostgreSQL schemas in migration, like this:
def change
# Create schema
create_schema 'demography'
# Create new table in specific schema
create_table "countries", :schema => "demography" do |t|
# columns goes here
end
# Drop schema
drop_schema 'politics'
end
Also it takes care about correctly dumping schemas into schema.rb file.

Related

How do I ensure correctness when using find_in_batches?

current my application have stat needs and I
make up a background job using rufus-scheduler and runs at 3:00
to batch process these records into CacheStat table. It's just like
any normal application's Weekly/Monthly Stat needs.
And I found out using find_each(say using User.find_each to iterate
all users), which invokes find_in_batches, I checkout the source code
of rails,
while records.any?
records_size = records.size
primary_key_offset = records.last.id
yield records
break if records_size < batch_size
if primary_key_offset
records = relation.where(table[primary_key].gt(primary_key_offset)).to_a
else
raise "Primary key not included in the custom select clause"
end
end
which the implentation is by comparing the primary-key,
my concern is the cocurrency,while I processing the batch,
whatif some records be inserted in-between?
does anybody have this kind of problem?
While I think, this code implementation may be be problemic,
because new records will always have larger PK and later in the
end will be find.
So this is what this kind of needs be implemented? If I want to
implement a batch stat processing by myself(without rails), then I
need to ensure have an integer primary key and using these fields to
compare(better not to use other kind of fields)?
(I was thinking of this because I'm kind of in the middle of switching
from mysql to mongo, so maybe later I need to implement this kind of
functionality by myself).
If I understand correctly, you can ensure correctness here by enforcing transactional isolation, e.g.
User.transaction do
User.find_each do |user|
user
end
end

Having lowercase method names with mixed case column names

I need to connect to a legacy SQL Server 2000 database using their own conventions and specially CamelCase columns and Tables.
For tables it seems fine, Rails is asking it with lowercase and the database find it nicely. The issue is with the columns because Rails fetch their name with SQL and thus get whatever case their name is.
I'm dealing with 500+ tables with some dozen columns in each of them and several legacy applications running in production above them so renaming the columns is no solution.
Using alias_attribute is also a way-too-much-work solution.
I don't want to have some weird case in my code too like client.AccountId (just looks like Java code).
So my final question is: is there any way to have Rails dealing with lowercase methods and symbols which are then used in whatever-case the database uses when dealing with SQL ?
I'm looking for any existing solution or even a direction to the sensible area of ActiveRecord where all this mechanics is done (I've been searching but the source code is huge ...)
OKay some time after posting the question I had a flash idea that alias_attribute was actually the solution but just needed a bit of magic over it.
Here is the solution to my own problem:
module LegacyDatabase
module ClassMethods
def aliased_attributes
#aliased_attributes ||= {}
end
def alias_attribute(new_name, old_name)
self.aliased_attributes[new_name.to_sym] = old_name.to_sym
super(new_name, old_name)
end
end
module InstanceMethods
private
def read_attribute(attr_name)
attr_name = self.class.aliased_attributes[attr_name.to_sym] if self.class.aliased_attributes.has_key?(attr_name.to_sym)
super(attr_name)
end
def write_attribute(attr_name, value)
attr_name = self.class.aliased_attributes[attr_name.to_sym] if self.class.aliased_attributes.has_key?(attr_name.to_sym)
super(attr_name, value)
end
end
def self.included(base)
base.instance_eval do
extend(ClassMethods)
include(InstanceMethods)
end
base.columns.each do |column|
legacy_name = column.name
rails_name = column.name.underscore
if legacy_name != rails_name
base.alias_attribute rails_name, legacy_name
end
end
end
end
I think this is the minimum code modification possible to avoid messing all ActiveRecord code. I'd like your opinion on this and your comments if you see a wall I'm going to hit and I don't !
To describe the solution, I'm using the columns method of ActiveRecord to generate snake_case looking aliases for each column. I'm also giving alias_column a memory of the aliases, that way read and write attribute methods know when they are dealing with alias names.
Since in my legacy database the convention for the ID or the table Table is TableID, my solution will create a table_id alias found by ActiveRecord using the "table_name_with_underscore" convention, so the id method is working as expected.
I presume it's not going to work with all the SQL fetches, even with Squeel of something but I don't think there is any simple solution for this.

Using and Editing Class Variables in Ruby?

So I've done a couple of days worth of research on the matter, and the general consensus is that there isn't one. So I was hoping for an answer more specific to my situation...
I'm using Rails to import a file into a database. Everything is working regarding the import, but I'm wanting to give the database itself an attribute, not just every entry. I'm creating a hash of the file, and I figured it'd be easiest to just assign it to the database (or the class).
I've created a class called Issue (and thus an 'issues' database) with each entry having a couple of attributes. I was wanting to figure out a way to add a class variable (at least, that's what I think is the best option) to Issue to simply store the hash. I've written a rake to import the file, iff the new file is different than the previous file imported (read, if the hash's are different).
desc "Parses a CSV file into the 'issues' database"
task :issues, [:file] => :environment do |t, args|
md5 = Digest::MD5.hexdigest(args[:file])
puts "1: Issue.md5 = #{Issue.md5}"
if md5 != Issue.md5
Issue.destroy_all()
#import new csv file
CSV.foreach(args[:file]) do |row|
issue = {
#various attributes to be columns...
}
Issue.create(issue)
end #end foreach loop
Issue.md5 = md5
puts "2: Issue.md5 = #{Issue.md5}"
end #end if statement
end #end task
And my model is as follows:
class Issue < ActiveRecord::Base
attr_accessible :md5
##md5 = 5
def self.md5
##md5
end
def self.md5=(newmd5)
##md5 = newmd5
end
attr_accessible #various database-entry attributes
end
I've tried various different ways to write my model, but it all comes down to this. Whatever I set the ##md5 in my model, becomes a permanent change, almost like a constant. If I change this value here, and refresh my database, the change is noted immediately. If I go into rails console and do:
Issue.md5 # => 5
Issue.md5 = 123 # => 123
Issue.md5 # => 123
But this change isn't committed to anything. As soon as I exit the console, things return to "5" again. It's almost like I need a .save method for my class.
Also, in the rake file, you see I have two print statements, printing out Issue.md5 before and after the parse. The first prints out "5" and the second prints out the new, correct hash. So Ruby is recognizing the fact that I'm changing this variable, it's just never saved anywhere.
Ruby 1.9.3, Rails 3.2.6, SQLite3 3.6.20.
tl;dr I need a way to create a class variable, and be able to access it, modify it, and re-store it.
Fixes please? Thanks!
There are a couple solutions here. Essentially, you need to persist that one variable: Postgres provides a key/value store in the database, which would be most ideal, but you're using SQLite so that isn't an option for you. Instead, you'll probably need to use either redis or memcached to persist this information into your database.
Either one allows you to persist values into a schema-less datastore and query them again later. Redis has the advantage of being saved to disk, so if the server craps out on you you can get the value of md5 again when it restarts. Data saved into memcached is never persisted, so if the memcached instance goes away, when it comes back md5 will be 5 once again.
Both redis and memcached enjoy a lot of support in the Ruby community. It will complicate your stack slightly installing one, but I think it's the best solution available to you. That said, if you just can't use either one, you could also write the value of md5 to a temporary file on your server and access it again later. The issue there is that the value then won't be shared among all your server processes.

Creating tables & models on the fly (dynamically) in Ruby on Rails 3

Bear with me on this one.
In an app I'm working on users are able to upload CSV files into the system, with any headers they like and any columns in the data. The CSV is then used to generate a table in the database and the data written to it, it can then be accessed through the system for various uses, searches, sorts updates etc.
The old (and now defunct system) was in PHP and handled this fine, although quite messily with lots of raw sql to create the tables and the framework supported magic-models (if the table existed so did the object without a class being defined in a model file)
The new version is being written in RoR3, and I am yet to figure out a way to do this. I've managed to sort out the table creation by calling migration tools inside a model (not very Rails-y I know, but needs must...), but I cannot find a way to link to the new table once it's created to write in the data, build relationships or anything else.
What I'm hoping for is either,
a) someone on here has a better way of doing this than creating tables and models on the fly (a warning here, these files can contain 100'000's of records and different fields so a single table option doesn't work so well) i.e a better database design for this issue.
or
b) can tell me how to sort out the model issue.
I've looked at Dr Nic's Magic Model gem for RoR but it doesn't seem to work in RoR3 unless I'm doing something wrong
Sorry for the wall of text, look forward to any suggestions
Thanks in advance
OK, i got a solution i think, but if is nice thats different thing.
Basically you create a table on the fly by executing SQL thru Rail ActiveRecord.
Next use a Model and change its name (Model.table_name).
Something like this:
// SQL Create table
sql = "CREATE TABLE my_table (id int(11) NOT NULL AUTO_INCREMENT, code varchar(3) NOT NULL)"
ActiveRecord::Base.connection.execute(sql)
Then with a model you can change the table name on the fly, like:
MyModel.table_name = "my_table"
records = MyModel.all
Ok, one of your problems is the model logic & associations. You are kind of limited, but maybe you can workaround that.
Not really best practices i guess, but if you need this. I think it might work!
I have implemented app which is used to upload a csv file and convert the file into a active record model. you can checkout this repository.
Visit https://github.com/Athul92/Salary_Sheet_Comparison/blob/master/app/models/makemodel.rb
here is a small idea about how it was achieved:
class Makemodel < ActiveRecord::Migration
def self.import(file,project_name,file_name,start_row,end_row,unique,last_column)
spreadsheet = open_spreadsheet(file)
header = spreadsheet.row(start_row.to_i)
i=0
header.count.times do
unless header[i].nil?
header[i]= header[i].gsub(/[^0-9A-Za-z]/, '').downcase
end
i+=1
end
name = "#{project_name.downcase}"+"#{file_name.downcase}"
create_table name.pluralize do |t|
header.each do |head|
t.string head
end
end
model_file = File.join("app", "models", name.singularize+".rb")
model_name = name.singularize.capitalize
File.open(model_file, "w+") do |f|
f << "class #{model_name} < ActiveRecord::Base\nend"
end
((start_row.to_i+1)..end_row.to_i).each do |i|
row = Hash[[header, spreadsheet.row(i)].transpose]
#should find a logic to find the model class that is being created
product = Object.const_get(name.capitalize).new
product.attributes = row.to_hash
product.save!
end
end
def self.open_spreadsheet(file)
case File.extname(file.original_filename)
when ".csv" then Csv.new(file.path, nil, :ignore)
when ".xls" then Roo::Excel.new(file.path)
when ".xlsx" then Roo::Excelx.new(file.path)
else raise "Unknown file type: #{file.original_filename}"
end
end
end
There are also some glitches with this it is difficult to add association and also validations

Rails - Data Migration vs. value in DB

I have a question about where values in dropdowns are coming from:
I have a migration that set up the original table with some initial values:
add_column :contracts, :signature_status_id, :integer
# lookup data
sig = SignatureStatus.new(:name => "Delivered")
sig.save!
sig = SignatureStatus.new(:name => "Signed")
sig.save!
I have a table called signature_statuses that contains the updated values:
id, name
1, 'Delivered; awaiting signature'
2, 'Delivered; awaiting full execution'
3, 'Terms being negotiated'
4, 'Fully executed and filed'
I have a form that contains the code to pull out the signature status:
<%= collection_select(:contract, :signature_status_id, #signature_statuses, :id, :name) %>
The collection select is pulling in "Signed" and "Delivered" when I want it to be from the DB. How do I make it do that.
Note: I think that the data was edited manually rather than a migration, but I'm not sure. I also searched the code for "signed" and "delivered", but the only place it shows up is in the migration.
I'm just wondering how are you getting that list of values in the signature_statuses table? Are you querying your development database? Is your application running in development mode? Is the database.yml file setup correctly to point to your development database?
Also can you post the controller code that populates the #signature_statuses variable.
A little more info and I'm sure people will be able to help.
Hmmm, this is a bit odd, but i suspect the following: there might be a method called name inside your signature_status model which is overriding the default one and which returns yes and no.
The key to debugging this is to look where
#signature_statuses
is being set in the controller. If it's pulling from the database, then that is what is in the database. I wonder if there is more than one database involved, where your migration updated the development database, but you are running the query against production (or something like that).
It turns out I needed to run "rake db" and that fixed it.

Resources