I need to find a way for scaling my rails monolith application with the help of rails engines.
Goal: I have database connection timeout issue and monolith has more than 200+ models. what we want to do is divide our models into the tree-like structure of engines. and we will be able to use a separate database for each engine.
UseCase: let's say we have engine A as the base engine and which is included in engine B and C respectively. Both B and C engines live on the same level of the tree.
So i have models seggrated in different engines.
Engine A: has all data related to user.
Class User
end
Engine B: has all data related to products
class Product
end
Engine C: has all data related to reports.
class Report
end
now the main issue comes while defining the associations. earlier we were having associations and several other methods which access associations. For eg.
class User
has_many products
def get_title_product
products.pluck(:title)
end
end
now I cannot define it in engine A as products table doesn't live there.
Option:
what I know is I have to open that User model inside Engine B and define all the association and get_title_product logic related to this Domain in engine B itself.
I can't even include Engine B in Engine A because it will result in circular dependency.
I don't want to follow above approach because it will get messy and my application is significantly large, additionally I don't think it is good as per rails best practices.
Thanks in advance.
Your post contains many questions. Your asking about database sharding, architecture with rails engines and performances / scalability with the timeout issue.
Performances / Timeout
First your timeout issue is not related to the number of models as you suggest. The number of models as no impact at all on performances. To know where is the performance problem or the bottleneck you should use a monitoring tool. This one is my favorite https://www.rorvswild.com (DISCLAIMER: I'm the author ^_^). There is other competitors, use the one your like more.
For the timeouts maybe you should check around your database config. We have not enough informations here to go deeper.
Database sharding
That is not trivial at all since you cannot JOIN and reference foreign keys when they are not in the same db. That is why you have to carefully chose where to shard your db. Ideally it's where you have the least joins. It's a long work which has an important impact on your code base. A stack overflow post is not enough to talk about sharding. Fortunately there is a lot of articles and gems to help you for that.
Just be sure you understand that you can split the load across many databases, but it comes at an extra price on your code base.
For the relationships across databases you cannot use the Rails builtin has_many and so on. You have to define the relationships yourself or use a
gem which will help for that. To give you an idea:
class User
def products
Product.where(user_id: id)
end
end
Rails engines
They are great to built reusable piece of features across applications (http://guides.rubyonrails.org/engines.html). Looks like reusing is not your goal. So I'm afraid your going in the wrong direction.
If you don't want to reopen the class you can use a module:
module HasProducts
def self.included(model)
model.has_many(:products)
end
def get_title_product
products.pluck(:title)
end
end
Related
I'm having a hard time wrapping my head around Rails' MVC system and its relationship to Ruby classes. In Rails, it feels like objects go out the window and are replaced by databases.
I'm working on an interview question right now and it requires me to create a website that accepts HTTP requests and returns responses. I'm not super familiar with JS and I like Ruby a lot so I've been using Rails. But I feel so limited by the database... it feels like things that I could do so easily in Ruby (e.g., look through a hash of objects and filter by multiple constraints until I got what I wanted) are so much more difficult with a database.
The interview question says I don't need to use a durable data store, but that seems like a big no-no in Rails. Does anyone have advice for how to change my mindset around this? (Or, in the short term, recommendations for other easy-to-use tools?)
BTW - I know this may beg the question, "why are you interviewing if you don't know this stuff?" It's an internship and I'm just trying to start interviewing as much as possible so I get used to the process.
But I feel so limited by the database... it feels like things that I could do so easily in Ruby (e.g., look through a hash of objects and filter by multiple constraints until I got what I wanted) are so much more difficult with a database.
Working with databases is challenging since SQL is a completely different paradigm. Instead of being object oriented SQL is based on tables and relational algebra.
Everyone finds it easier to hack out loops and create if/else constructs to munge data as a beginner. But its not really a viable solution at scale. Any time you spend learning SQL will pay off immensely in the future.
I'm having a hard time wrapping my head around Rails' MVC system and its relationship to Ruby classes. In Rails, it feels like objects go out the window and are replaced by databases.
This doesn't really have that much do with Rails or MVC. Its rather ActiveRecord which is the object relational mapper (ORM) which is built into Rails by default.
In comparison with other ORM's ActiveRecord has a lot of magic. It reads your table definitions from the database and automatically sets up attributes - in other ORMs you do this explicitly in your model classes, a repository class, XML files or annotations. This magic is great for productivity but can be very difficult to wrap your head around as a beginner.
ActiveRecord is still very much an object oriented design - just with more metaprogramming then you may be used to.
The interview question says I don't need to use a durable data store,
but that seems like a big no-no in Rails.
This is a misconception. Rails is a very modular framework and can be used without ActiveRecord, with other object relational mappers (or object-document mappers) or even without any storage at all.
Models are also are not necissarily even persisted at all. They are just objects that encompass the buisness logic of your application.
Most beginner guides start with ActiveRecord as an overwhelming majority web applications use a SQL database*.
I agree that this is probably one of the hardest things newbies face in Rails. In the Rails guide, models are almost always Active Record models (aka map to a database table). However, it's totally normal that not all of your models are Active Record models and / or need to be persisted. I think the term you're missing are Plain Old Ruby Objects (PORO) and Active Model.
A PORO could be anything from something super simple, a mapping to an API or composing several other (ActiveRecords) models together.
For example, you could have a versions controller which displays the latest version of your web app. The version would be determined by reading the SHA from the git repository, file or env variable. So it would never be persisted in the database.
class Version
def self.from_git
new(`git rev-parse HEAD`)
end
def initialize(name)
#name = name
end
def to_s
#name
end
end
class VersionsController
def show
#version = Version.from_git
end
end
Here are some links you might find useful
https://www.codewithjason.com/code-without-service-objects/
https://www.oreilly.com/library/view/build-chatbot-interactions/9781680507089/f_0047.xhtml
https://thecodest.co/blog/hi-im-poro-2/
https://www.toptal.com/ruby-on-rails/decoupling-rails-components
https://guides.rubyonrails.org/active_model_basics.html
https://www.codewithjason.com/extracting-tidy-poro-messy-active-record-model/
So I am a fairly seasoned Django developer and I've been using Ruby on Rails for about a year. I'm working on a project with, let's say, 100 models. In my actual scenario, I'm writing a "component" to the app, which requires 2 models, one of which I wanted to call Event, however there is another piece of the software which uses an Event model in a totally different context. In Django, this is resolved by splitting the "components" into "apps", and models can have the same name because they are in a different app.
Is there is a similar pattern in Rails to separate some models in their own package to avoid name conflicts, but also so that each separate component can have it's own README which describes how that component behaves? I don't require that it have it's own controllers/views, since right now we have a single frontend interface, but I would be interested to hear if there is a pattern for that too. I've heard the term "engines" in Rails, but that seems to imply a total decoupling and unidirectional dependency scheme like for a generic 3rd party app, where as in my case, there is still some coupling between this "component" and the rest of the app, but it still seems reasonable to have some way of grouping models logically.
Also, I'm wondering how this case is handled in general. When a web application grows organically to have tons and tons of models, is there a standard pattern for managing this complexity?
You can use namespacing using modules for your models.
For example Foo::Bar or Baz::Bar like this:
module Foo
class Bar < ApplicationRecord
end
end
To associate this model with table foo_bars, you can implement the method:
module Foo
class << self
def table_name_prefix
'foo_'
end
end
end
I have a problem that I have been trying to solve for a while now. I have 4 different Rails applications using the same database, meaning they need to use the same models and have the same migrations. I initially solved the problem by creating a Rails engine packaged into a gem, which then carries all the models and migrations with it. Now I realize that there are pieces of functionality that only one application needs, but the others do not - like for example the admin application needs to have methods for providing sortable tables for all models - the other applications do not need this functionality at all.
So my idea was to find a way where I can provide the "base" models from the gem, while augmenting these base models in my specific applications to add additional functionality when needed. What I tried first was inheritance:
class User < Base::User
end
This does not work though, because now you have 2 User models in your load path (User and Base::User) and when querying associations, it always picks the "closest" class for the associated record class - meaning when you have an Model::Account which belongs_to :user, it will pick Model::User as the association class, not User. I tried reversing the AR type compute method but this only resulted in more problems.
I can technically provide all of my models from the base engine (gem), but the issue here is that how do i extend these models in my application? .class_eval feels really really dirty, inheritance does not work, providing base functionality as mixins means the "base" models do not feel and look like models at all. My goal would be to cause as little friction as possible for the other developers, I want them to be able to define their models in the gem like they do normally and then also have an easy way to extend that functionality in other applications.
Has anyone solved this problem before or have any suggestions? Or how do you guys solve this problem in your larger applications? Any help would be appreciated.
This is mentioned in the Rails guides. It describes class modification with the Decorator pattern.
I'm very interested to see what Rails veterans on Stack Overflow have to say about this.
Basically, as a Rails newb, and as someone with no "formal" background in programming, one of my biggest challenges has to do with proper architecture. How do I know what the right or better to build an app are?
I would love to see what some resources are.
I know the basic ones - but a lot of these don't give me the high level architecture overview I want:
http://www.amazon.com/Agile-Development-Rails-Third-Edition/dp/1934356166/ref=cm_lmf_tit_7
http://www.amazon.com/Ruby-Programming-Language-David-Flanagan/dp/0596516177/ref=cm_lmf_tit_3
Railscasts
Here's a quick example:
Say I'm building a Rails app that has merchants and shoppers - each set of users has its own authentication, different permissions, etc. Would it be proper to build a single app for this or multiple that communicate through APIs? Is there any added benefit to this multi-app abstraction?
Thanks!
this is not an easy question. The complete answer would largely depend on your project.
If you are a starter I'd recommend you keep it all in one app, you have enough "separation" using models wisely. It's hard to immagine a scenario where the complexity introduced by inter-app communication is beneficial.
In your example you should ask yourself wether it's better to use one single parent model for the Merchants and Shoppers or two separate models.
In the former case you can consider STI:
user (main class, defined as class User < ActiveRecord::Base)
merchant (defined as class Merchant < User)
shopper (similarly class Shopper < User)
Google for STI for further details.
Then in your controllers/view you can check permissions quickly, for example:
if user.class == Merchant
do_something
else
do_something_else
end
Similarly the two classes might authenticate with different algorithms. You might also include a "standard" authentication in the base User class and specialize it in the subclasses, if required.
Cheers,
I am about to create an application with Ruby on Rails and I would like to use multiple databases, basically is an accounting app that will have multiple companies for each user. I would like to create a database for each company
I found this post http://programmerassist.com/article/302
But I would like to read more thoughts about this issue.
I have to decide between MySQL and PosgreSQL, which database might fit better my problem.
There are several options for handling a multi-tenant app.
Firstly, you can add a scope to your tables (as suggested by Chad Birch - using a company_id). For most use-cases this is fine. If you are handling data that is secure/private (such as accounting information) you need to be very careful about your testing to ensure data remains private.
You can run your system using multiple databases. You can have a single app that uses a database for each client, or you can have actually have a seperate app for each client. Running a database for each client cuts a little against the grain in rails, but it is doable. Depending on the number of clients you have, and the load expectations, I would actually suggest having a look at running individual apps. With some work on your deployment setup (capistrano, chef, puppet, etc) you can make this a very streamlined process. Each client runs in a completely unique environment, and if a particular client has high loads you can spin them out to their own server.
If using PostgreSQL, you can do something similar using schemas.
PostgresQL schemas provide a very handy way of islolating your data from different clients. A database contains one or more named schemas, which in turn contain tables. You need to add some smarts to your migrations and deployments, but it works really well.
Inside your Rails application, you attach filters to the request that switch the current user's schema on or off.
Something like:
before_filter :set_app
def set_app
current_app = App.find_by_subdomain(...)
schema = current_app.schema
set_schema_path(schema)
end
def set_schema_path(schema)
connection = ActiveRecord::Base.connection
connection.execute("SET search_path TO #{schema}, #{connection.schema_search_path}")
end
def reset_schema_path
connection = ActiveRecord::Base.connection
connection.execute("SET search_path TO #{connection.schema_search_path}")
end
The problem with answers about multiple databases is when they come from people who don't have a need or experience with multiple databases. The second problem is that some databases just don't allow for switching between multiple databases, including allowing users to do their own backup and recovery and including scaling to point some users to a different data server. Here is a link to a useful video
http://aac2009.confreaks.com/06-feb-2009-14-30-writing-multi-tenant-applications-in-rails-guy-naor.html
This link will help with Ruby on Rails with Postgresql.
I currently have a multi-tenant, multi-database, multi-user (many logons to the same tenant with different levels of access), and being an online SaaS application. There are actually two applications one is in the accounting category and the other is banking. Both Apps are built on the same structure and methods. A client-user (tenant) can switch databases under that user's logon. An agent-user such as a tax accountant can switch between databases for his clients only. A super-user can switch to any database. There is one data dictionary i.e. only one place where tables and columns are defined. There is global data and local data. Global data such as a master chart-of-accounts which is available to everyone (read only). Local data is the user's database. A new user can get a clone of a master database. There are multiple clones to choose from. A super-user can maintain the clone databases.
The problem is that it is in COBOL and uses ISAM files and uses the CGI method. The problem with this is a) there is a perception that COBOL is outdated, b) getting trained people, c) price and d) online help. Otherwise it works and I'm happy with it.
So I'm researching what to replace it with and what a minefield that is.
It has past time and the decission for this has been to use PostgreSQL schemas, making multitenant applications, I have a schema called common where related data is stored.
# app/models/organisation.rb
class Organisation < ActiveRecord::Base
self.table_name = 'common.organisations'
# set relationships as usual
end
# app/models/user.rb
class User < ActiveRecord::Base
self.table_name = 'common.users'
# set relationships as usual
end
Then for migrations I have done that with this excellent tutorial. http://timnew.github.com/blog/2012/07/17/use-postgres-multiple-schema-database-in-rails/ use this, this is way better than what I saw in other places even the way Ryan Bates did on railscasts.
When a new organisation is created then a new schema is created with the name of the subdomain the organisation. I have read in the past that it's not a good idea to use different schemas but it depends on the job you are doing, this app has almost no soccial component so it's a good fit.
No, you shouldn't use multiple databases.
I'm not really sure what advice to give you though, it seems like you have some very basic misunderstandings about database design, you may want to educate yourself on the basics of databases first, before going further.
You most likely just want to add a "company id" type column to your tables to identify which company a particular record belongs to.