We have a Ruby on Rails application that is getting quite big. And we are trying to split it up into separate logical domains, and we are wondering how to enforce that the logical domain is not crossed.
Let's say we have models A, B, C, D. Model A, B are used in one part of the app, Model C, D in another. We would like to be able to draw some logical system boundaries, for the sake of faster continuous integration and independent teams.
We can re-organize the code, and break some dependencies.
For example,
let's say model B is a join or mapping table, that joins A and C in a many-to-many relationships
class A
has_many :C, through: B
We could change this to:
class A
has_many :B
From A, you get a list of c_id through the mapping table B, and then you can call a GET API on C, to get the list of C objects. This is similar to if we were to A and C physically in separate databases, and we want to display a list of C objects, when we show A.
Eventually, we would probably want A, B in one service, and C be in a separate service. In the meanwhile, we do want to keep A, B, C are in the same Postgres database, but just make a logical extraction.
Question: What are ways we can enforce logical system boundary in Rails + Postgres, so that we don't have team members that take class A, and accidentally joins with C, and introduce unintended coupling before we do a full service split.
This blog suggests using test suites and selectively loading the necessary dependencies https://medium.com/#dan_manges/the-modular-monolith-rails-architecture-fb1023826fc4. But I feel that might not prevent developer from using SQL directly to join tables across logical boundary. I am wondering what others might have tried.
Related
I am trying to come up with a convenient data structure that would allow me to capture all associations among my Rails models (I already got all associations using some metaprogramming). Any ideas? The closest I got is a tree with the root node being a special node called root, and then the first children are models with no parent models (that is, not belong_to), and the second level down is the rest.
The reason I want this is that I want to process the models in the following fashion:
Start with models with no parent models, that is ones that have no belong_to association to another model.
Move to those that have belong_to dependence on models from 1.
Deal with nesting, so if C belongs to B belongs to A, I want to process A first, then B, then C.
A breadth first traversal comes to mind, but I can't quite get the data
structure to reflect this. Perhaps I am looking for a better data structure.
The data structure I am looking for is a directed acyclic graph.
This is an architectural and implementation question, but I'm more focused on getting ideas for big picture strategy.
The scenario
The app lists Foo records in a table. There are several ways to sort them as follows:
By attribute b_domains, each Foo has_many :b_domains, :through => :b_domain_foos
Easy enough, simply get all foos by b_domains, this is working.
By a set of drop-down pills. Foos belong_to each of the objects represented by a pill. For example, a foo may belong to an instance of M called m_one.
Each foo should be able to be sorted with any number of pills of type M, N, O, or P.
Freeform text search.
All of this should be done with js and never reload the page.
What are some strategies to keep the active list of foos in memory? Is this a common Rails pattern? Where can I find an example?
What I find myself beginning to do is completely replace the list of foos rather than refining the list in memory on new requests.
Thanks.
I have a Ruby on Rails app with a PostgreSQL database which has this structure:
class A < ActiveRecord::Base
has_many :B
end
class B < ActiveRecord::Base
has_many :C
end
class C < ActiveRecord::Base
attr_accessible :x, :y :z
end
The are only a few A's, and they grow slowly (say 5 a month). Each A has thousands of B's, and each B has tens of thousands of C's (so each A has millions of C's).
A's are independent and B's and C's from different A's will never be needed together (i.e. in the same query).
My problem is that now that I have only a couple of A's, ActiveRecord queries take pretty long. When the table for C has tens of millions of rows, queries will take forever.
I am thinking about scaling the database horizontally (i.e a table for A's, one table of B's and one table of C's for each A). But I don't know how to do it. It is a kind of sharding i guess, but I can't figure out how to create DB tables dynamically and use ActiveRecord to access the data if the table depends on which A im working with.
Thank you very much.
If you have performance concerns with only a few rows, or even with several million rows, you need to take a step back before trying to atmosphere engineer a solution. The problem you are describing is very easily solved by indexing; there is no advantage to creating additional physical tables and you'd be introducing incredible complexity.
As #mu-is-too-short already stated: pay attention to your query plans. Use your tools to analyze performance.
That being said you can use table partitioning to physically and transparently house the storage of data into different sharded tables which is especially useful for data that grows very fast but is only useful in a given time box (like a month). You can also do this with an archive bit flag column to shuttle old or deleted records onto some slower storage (say, standard RAID comprised of spinning rust) while keeping active records on faster storage (like a RAID of SSDs).
So it seems you have a tree-like structure. If there is really no need to pull them out of the database in a some kind of cross-referenced manner, then your A's have exactly the properties of a "document", have a look at MongoDB. A's would be saved with all of their B's and there C's in a single record.
http://www.mongodb.org/
If you are looking for an ORM, check
http://mongoid.org/en/mongoid/index.html
I have two models which are one-to-one to each other. A currently has one B.
Lately I encounter many cases where it is desirable if A keeps the id of B in order to simplify logic and boost performance. However I wonder if:
this is possible
would breach and convention
any thoughts really
UPDATE
I was wrong, the left outer join would not be benefited by the extra foreign key.
The only place I can think of is to find all A's which does not have B, an inner join is required on each of my 100000+ records. But if there is an id then I can know straight away which A has a B.
I don't believe this is possible - you have to decide where to keep the foreign key. Would it make sense for you to use a joining through relationship and for both your existing models to be has_one through the join?
Another alternative is to put A id on B, i.e. denormalize. This would allow you to figure out which A's without a B, and A's with a B. This might be appropriate for reporting scenarios when B's don't move between A's often.
I am working on a project that currently has tons of HABTM associations. Essentially, everything is related to everything else. I am considering setting up a single intermediate table/model that has two polymorphic fields. This way, if I add another model I can easily connect it to the remaining models. Is this a good idea? If not, why not? If it is, why don't all rails projects have this kind of intermediate table?
I see two other options. I could keep adding intermediate tables or I could add a table that contains one of each type. The former option is kind of a hassle and the latter option does not allow for self joins.
While a polymorphic join table sounds like it would make things easier, I think you will end up creating more headache for yourself than it's worth. Here are a few potential challenges/problems off the top of my head:
You will not be able to use ActiveRecord's has_and_belongs_to_many association or related helpers without a ton of hacking/monkeypatching which will immediately eclipse the time it would take to setup individual pairwise link tables.
Your join table will have two id columns, let's call them a_id and b_id. For any given pair of models you will have to ensure that the ids always end up in the same column.
Example: If you have two models called User and Role, you would have to ensure for that pair that the user_id is always stored in col a_id and the role_id is always stored in col b_id, otherwise you will not be able to index the table in any kind of meaningful way (and will run the risk of defining the same relationship twice).
If you ever want to use database enforcement of FOREIGN KEY constraints it is unlikely that this polymorphic link table scheme will be supported.
The universal link table will get n times larger than n separate link tables. It shouldn't matter much with good indexing but as your application and data grow this could become a headache and limit some of your options in regards to scaling. Give your DB a break.
Most or least importantly (I can't decide) you will be bucking the norm which means a lot fewer (if any) resources out there to help you when you run into trouble. Basically the Adam Sandler "they're all gonna laugh at you" rationale.
Last thought: Can you eliminate any of the link tables by using has_many :xxx, :through => :xxx relationships?
Thinking it all through, you could actually do this, but I wouldn't. Join tables grow fast enough as it is and i like to keep model relationships simple and easy to alter.
I'm used to working on very large systems / data sets though, so if you're going going to have much in each join then ok. I'd still do it separately for joins however and i really like my polymorphics.
I think it would be cleaner and more flexible if you were to use multiple join tables as opposed to one giant multipurpose join table.