Our app currently spawns a new database for each client. We're starting to wonder whether we should consider refactoring this to a multi-tenant system.
What benefits / trade-offs should we be considering? What are the best practices for implementing a multi-tenant app in Rails?
I've been researching the same thing and just found this presentation to offer an interesting solution: Using Postgre's schemas (a bit like namespaces) to separate data at the DB level while keeping all tenants in the same DB and staying (mostly) transparent to rails.
Writing Multi-Tenant Applications in Rails - Guy Naor
Multi-tenant systems will introduce a whole range of issues for you. My quick thoughts are below
All SQL must be examined and
refactored to include a ClientId
value.
All Indexes must be examined to
determine if the ClientId needs to be
included
An error in a SQL statement by a
developer/sysadmin in production will
affect all of your customers.
A database corruption/problem will
affect all of your customers
You have some data privacy issues
whereby poor code/implementation could
allow customerA to see data belonging
to CustomerB
A customer using your system in a
heavy/agressive manner may affect
other customers perception of performance
Tailoring static data to an individual customers preference becomes more complex.
I'm sure there are a number of other issues but these were my initial thoughts.
It really depends upon what you're doing.
We are making a MIS program for the print industry that tracks inventory, employees, customers, equipment, and does some serious calculations to estimate costs of performing jobs based on a lot of input variables.
We are anticipating very large databases for each customer, and we currently have 170 tables. Adding another column to almost every table just to store the client_id hurts my brain.
We are currently in the beta stage of our program, and here are some things that we have encountered:
Migrations: A Rails assumption is that you will only have 1 database. You can adapt it for multiple databases, and migrations is one of them. You need a custom rake task to apply migrations to all existing databases. Be prepared to do a lot of trouble shooting because a migration may succeed on one DB, but fail on another.
Spawning Databases: How do you create a new db? From a SQL file, copying an existing db, or running all migrations? How do you keep you schema consistent between your table creation system, and your live databases?
Connecting to the appropriate database: We use a cookie to store a unique value that maps to the correct DB. We use a before filter in an Authorized controller that inheirits from ActionController that gets the db from that unique value and uses the establish_connection method on a Subclass of ActiveRecord::Base. This allows us to have some models pull from a common db and others from the client's specific db.
If you have specific questions about any of these, I can help.
I don't have any experience with this personally, but during the lightning talks at the 2009 Ruby Hoedown, Andrew Coleman presented a plugin he designed and uses for multi-tenant databases in rails w/ subdomains. You can check out the lightning talk slides and here's the acts_as_restricted_subdomain repository.
Why would you? Do you have heavy aggregation between users or are you spawning too many DBs? Have you considered using SQLite files per tenant instead of shared DB servers (since multitenant apps often are low-profile and don't need that much concurrency)?
Related
I am planning on using Devise and Apartment in my upcoming application to create subdomains for each organization that creates an account. I would like to host my application on Heroku, but ran across the following quote:
The most common use case for using multiple schemas in a database is
building a software-as-a-service application wherein each customer has
their own schema. While this technique seems compelling, we strongly
recommend against it as it has caused numerous cases of operational
problems. For instance, even a moderate number of schemas (> 50) can
severely impact the performance of Heroku’s database snapshots tool,
PG Backups.
What technique would work well with Heroku to host basecamp-style subdomains in rails 4 where many users can log in to the subdomain which they are part of?
If Heroku does not work, what other PaaS options are there that would do this well?
Domain
Firstly, you need to be sure that you're using your own custom domain for the subdomains.
Heroku's standard xxx.herokuapp.com won't be able to handle another subdomain on top of that - so you'll basically need to use your custom domain from the get-go
It will be good to reference this documentation for more information!
Multi Tenancy
Although I don't have experience with PGSQL's schemas, I do have some with multi tenancy as a whole.
There are a number of great resources here:
Basecamp-style Subdomains (by DHH)
Multitenancy Railscasts (Pro)
Apartment Gem Documenatation
Essentially, multi-tenancy is just a way to scope the data so that it's only the tenant's that you see / interact with. In the sense of the DB, the two ways to achieve this are either to use different DB's (as you would with MYSQL), or use a schema (like with PGSQL)
Whilst I can't give you a direct fix for your issue, I can help you with some ideas:
Models
One way to achieve multi-tenancy, especially with the likes of MYSQL, is to do it through the model:
How do i work with two different databases in rails with active records?
#lib/admin.rb
class Admin < ActiveRecord::Base
self.abstract_class = true
establish_connection "#{Rails.env}_admin"
end
#app/models/option.rb
Class Option < Admin
# do stuff
end
This works very well for us, although we have not got it working for scoped accounts yet. We've been thinking of setting a ##class_variable for the Account or something, but haven't been working on that right now.
This works very well for MYSQL - powered databases, but also means you'll have to create db's for every account, which will not work with PGSQL (as far as I'm aware)
PGSQL Schemas
I feel this is kind of a cheat way to do this, as all the data is still stored in 1 database - it's basically just scoped around different types of data.
The problem here is that real multi tenancy should be where you completely separate the user's data, so you could cut it out of the app completely if they wanted. From a security & access perspective, it's the most flexible & modular way.
The problem for Heroku is they can only use one database (they give everyone access to their AWS database instances), meaning they can't allow you to create 50+ free databases (it just won't work very well).
You can, of course, use your own stack to create the databases you require, but in terms of PGSQL, it's just about creating the schemas for your data & then using something like -Apartment to make it happen:
PostgreSQL works slightly differently than other databases when
creating a new tenant. If you are using PostgreSQL, Apartment by
default will set up a new schema and migrate into there. This provides
better performance, and allows Apartment to work on systems like
Heroku, which would not allow a full new database to be created.
I originally wrote my Ruby on Rails application for one client. Now, I am changing it so that it can be used for different clients. My end-goal is that some user (not me) can click a button and create a new project. Then all the necessary changes (new schema, new tables, handling of code) are generated without anyone needing me to edit a database.yml file or add new schema definitions. I am currently using the SCOPED access. So I have a project model and other associated models have a project_id column.
I have looked at other posts regarding multi-tenant applications in Rails. A lot of people seem to suggest creating a different schema for each new client in Postgres. For me, however, it is not much useful for a new client to have a different schema in terms of data model. Each client will have the same tables, rows, columns, etc.
My vision for each client is that my production database first has a table of different projects/clients. And each one of those tables links to a set of tables that are pretty much the same with different data. In other terms a table of tables. Or in other terms, the first table will map to a different set of data for each client that has the same structure.
Is the way I explained my vision at all similar to the way that Postgres implements different "schemas"? Does it look like nested tables? Or does Postgres have to query all the information in the database anyway? I do not currently use Postgres, but I would be willing to learn if it fits the design. If you know of database software that works with Rails that fits my needs, please do let me know.
Right now, I am using scopes to accomplish multi-tenant applications, but it does not feel scalable or clean. It does however make it very easy for a non-technical user to create a new project provided I give them fillable information. Do you know if it is possible with the multi-schema Postgres defintion to have it work automatically after a user clicks a button? And I would prefer that this be handled by Rails and not by an external script if possible? (please do advise either way)
Most importantly, do you recommend any plugins or that I should adopt a different framework for this task? I have found Rails to be limited in some cases of abstraction as above and this is the first time I have ran into a Rails-scaling issue.
Any advice related to multi-tenant applications or my situation is welcome. Any questions for clarification or additional advice are welcome as well.
Thanks,
--Dave
MSDN has a good introduction to multi-tenant data architecture.
At one end of the spectrum, you have one database per tenant ("shared nothing"). "Shared nothing" makes disaster recovery pretty simple, and has the highest degree of isolation between tenants. But it also has the highest average cost per tenant, and it supports the fewest tenants per server.
At the other end of the spectrum, you store a tenant id number in every row of every shared table ("shared everything"). "Shared everything" makes disaster recovery hard--for a single tenant, you'd have to restore just some rows in every shared table--and it has the lowest degree of isolation. (Badly formed queries can expose private data.) But it has the lowest cost per tenant, and it supports the highest number of tenants per server.
My vision for each client is that my production database first has a
table of different projects/clients. And each one of those tables
links to a set of tables that are pretty much the same with different
data. In other terms a table of tables. Or in other terms, the first
table will map to a different set of data for each client that has the
same structure.
This sounds like you're talking about one schema per tenant. Pay close attention to permissions (SQL GRANT and REVOKE statements. And ALTER DEFAULT PRIVILEGES.)
There are two railscasts on multitenancy that using scopes and subdomains and another to help with handling multiple schemas.
There is also the multitenant gem which could help with your scopes and apartment gem for handling multiple schemas.
Here is also a good presentation on multitenancy-with-rails.
Dont forget about using default scopes, while creating named scops the way you are now works it does feel like it could be done better. I came across this guide by Samuel Kadolph regarding this issue a few months ago and it looks like it could work well for your situation and have the benefit of keeping your application free of some PgSQL only features.
Basically the way he describes setting the application up involves adding the concepts of tennants to your application and then using this to scope the data at query time using the database.
I'm about to finish building a simple subscription based support ticket Web app. I'm setting up authorization. But since this it's going to be my very own Web app that I'm going to deploy I'm wondering about this.
Do you create a separate database per account opened?
Let's say you have this support ticket Web app. You have ONE and ONLY ONE account owner. Account owner can setup agents that can respond to support tickets. Also, there are customer roles that open support tickets.
So as you can see the database will contain users, support tickets and more.
What is the best way to go?
1) Create one database for the whole application? That way every time somebody signs up, everything is added to the same database with the other tickets and users data and everything else or...
2) Everytime someone signs up, create a separate database per account subscription.
I'm thinking that maybe option number 2 would be a best choice for security and data integrity purposes. If so, how have you gone about tackling this issue?
It sounds like what you want is Multitenancy:
Multitenancy refers to a principle in software architecture where a
single instance of the software runs on a server, serving multiple
client organizations (tenants). Multitenancy is contrasted with a
multi-instance architecture where separate software instances (or
hardware systems) are set up for different client organizations. With
a multitenant architecture, a software application is designed to
virtually partition its data and configuration, and each client
organization works with a customized virtual application instance.
- Wikipedia article on Multitenancy
This article while a little dated is the general idea of how I would go about doing it. Simple Rails Multi-Tenancy. It's clean and efficient and saves you from writing code that you don't need to.
You should go for option #1. Number 2 is (almost (there are probably cases where it is good, but I can't find one at the moment)) never an option.
You are right in security purposes (well, in a sense), but it also creates a lot of other problems that you will have to think about.
Having a different database for each user means that for each request (remember, HTTP is state-less) you will have to open up a new connection to the database, do whatever needs to be done and then close the connection again, instead of using the connection pooling that is in Rails. This affects the performance a great deal.
Administration will be a hassle the more databases you have. Also, having multiple databases on a server do require more resources than just a bigger database.
You would have to circumvent the entire connection handling in Rails since there it is usually one database per application. It is easy to change the database for specific models, but it adds additional places where things can go wrong.
Rails do have good functionality for scoping and handling of separating data within the same database, just for that kind of use-case that you are mentioning.
I am about to create an application with Ruby on Rails and I would like to use multiple databases, basically is an accounting app that will have multiple companies for each user. I would like to create a database for each company
I found this post http://programmerassist.com/article/302
But I would like to read more thoughts about this issue.
I have to decide between MySQL and PosgreSQL, which database might fit better my problem.
There are several options for handling a multi-tenant app.
Firstly, you can add a scope to your tables (as suggested by Chad Birch - using a company_id). For most use-cases this is fine. If you are handling data that is secure/private (such as accounting information) you need to be very careful about your testing to ensure data remains private.
You can run your system using multiple databases. You can have a single app that uses a database for each client, or you can have actually have a seperate app for each client. Running a database for each client cuts a little against the grain in rails, but it is doable. Depending on the number of clients you have, and the load expectations, I would actually suggest having a look at running individual apps. With some work on your deployment setup (capistrano, chef, puppet, etc) you can make this a very streamlined process. Each client runs in a completely unique environment, and if a particular client has high loads you can spin them out to their own server.
If using PostgreSQL, you can do something similar using schemas.
PostgresQL schemas provide a very handy way of islolating your data from different clients. A database contains one or more named schemas, which in turn contain tables. You need to add some smarts to your migrations and deployments, but it works really well.
Inside your Rails application, you attach filters to the request that switch the current user's schema on or off.
Something like:
before_filter :set_app
def set_app
current_app = App.find_by_subdomain(...)
schema = current_app.schema
set_schema_path(schema)
end
def set_schema_path(schema)
connection = ActiveRecord::Base.connection
connection.execute("SET search_path TO #{schema}, #{connection.schema_search_path}")
end
def reset_schema_path
connection = ActiveRecord::Base.connection
connection.execute("SET search_path TO #{connection.schema_search_path}")
end
The problem with answers about multiple databases is when they come from people who don't have a need or experience with multiple databases. The second problem is that some databases just don't allow for switching between multiple databases, including allowing users to do their own backup and recovery and including scaling to point some users to a different data server. Here is a link to a useful video
http://aac2009.confreaks.com/06-feb-2009-14-30-writing-multi-tenant-applications-in-rails-guy-naor.html
This link will help with Ruby on Rails with Postgresql.
I currently have a multi-tenant, multi-database, multi-user (many logons to the same tenant with different levels of access), and being an online SaaS application. There are actually two applications one is in the accounting category and the other is banking. Both Apps are built on the same structure and methods. A client-user (tenant) can switch databases under that user's logon. An agent-user such as a tax accountant can switch between databases for his clients only. A super-user can switch to any database. There is one data dictionary i.e. only one place where tables and columns are defined. There is global data and local data. Global data such as a master chart-of-accounts which is available to everyone (read only). Local data is the user's database. A new user can get a clone of a master database. There are multiple clones to choose from. A super-user can maintain the clone databases.
The problem is that it is in COBOL and uses ISAM files and uses the CGI method. The problem with this is a) there is a perception that COBOL is outdated, b) getting trained people, c) price and d) online help. Otherwise it works and I'm happy with it.
So I'm researching what to replace it with and what a minefield that is.
It has past time and the decission for this has been to use PostgreSQL schemas, making multitenant applications, I have a schema called common where related data is stored.
# app/models/organisation.rb
class Organisation < ActiveRecord::Base
self.table_name = 'common.organisations'
# set relationships as usual
end
# app/models/user.rb
class User < ActiveRecord::Base
self.table_name = 'common.users'
# set relationships as usual
end
Then for migrations I have done that with this excellent tutorial. http://timnew.github.com/blog/2012/07/17/use-postgres-multiple-schema-database-in-rails/ use this, this is way better than what I saw in other places even the way Ryan Bates did on railscasts.
When a new organisation is created then a new schema is created with the name of the subdomain the organisation. I have read in the past that it's not a good idea to use different schemas but it depends on the job you are doing, this app has almost no soccial component so it's a good fit.
No, you shouldn't use multiple databases.
I'm not really sure what advice to give you though, it seems like you have some very basic misunderstandings about database design, you may want to educate yourself on the basics of databases first, before going further.
You most likely just want to add a "company id" type column to your tables to identify which company a particular record belongs to.
I have developed a fully functional ruby-on-rails application which utilizes numerous mysql tables. I would like to turn this into a subscription based service but I have some general, probably basic, conceptual questions:
In a setup such as Basecamp does each user have access to her/his own (as in unique) db tables or are the tables shared with millions of users and Identified by some variable?
If this is the case how well does it scale? What would be the best db to use (mysql, oracle etc)?
If each user is given his/her own unique db tables; how is this accomplished? Is it through a rake task?
Are there any resources you would suggest (books, media, etc) that explain how to accomplish either of these methods?
Thanks!
I believe it is achieved using an overall account. Whereby the resources in your current system will be scoped by that account. i.e in your index actions something like #projects = #account.projects. Looking at basecamp I would say it scales very well! If you hit this problem then you have a good problem to solve, don't worry about it too much until then. I should image the database is a cluster but very much doubt each user has their own set of tables, that would become a nightmare to manage!
A quick google and I've found a this: http://www.robbyonrails.com/articles/2009/01/11/subdomain-accounts-with-ruby-on-rails-explained which also links to a post by DHH which looks like it explains how they did it.
There are probably newer write ups but I'm guessing they would be a great place start.
Good luck!
The tables are shared and identified to a "parent" using a foreign key value. Having separate tables per-user would be a nightmare. It's more likely that good database normalisation fixes most of these issues. To-dos are related to projects, projects are related to an account and then each account has many users.
The best db to use would be entirely up to you. If you're using rails and db migrations, you're only limited to what that interface can utilise. To begin with, go with either MySQL or PostgreSQL (my preference). They're free and there's a wealth of knowledge available for hobby projects.
I personally would not create separate tables per user
Reading the wikipedia entries on database normalisation and database design would be a good start. Following that you should read up as much as you can on good database design, perhaps even starting with the common mistakes developers make when it comes to database design.