I'm wondering if anyone could shed some light on the best way to test code that works with database models, but does not care what that model is.
Our app has a kind of synchronisation engine that keeps data on a remote server in sync with data on our system, and operates on a number of database models. It is also designed to work with multiple remote systems, and as such contains a number of internal data models which later get mapped to our actual database records.
As an example, we have something like this:
class RemoteResource < Syncable
maps_to ::InternalDbModel
....
end
Here, the sync engine would pull data from any of the remote servers and convert it into a RemoteResource instance, and later this would get translated into an InternalDbModel for persistence.
Due to the generic design, the sync engine doesn't care what database models it is converting to. So long as they meet certain criteria, they are valid.
My question revolves around testing this sort of code. Due to time constraints, I've had to write tests that I simply don't like - the tests involve creating records of actual models used by the app, but I don't like this for a number of reasons:
Each test has to know about the model in order to create the records
If the model has other requirements, they must be fulfilled first, which is completely irrelevant for the test itself
It is not the model that I want to test
It makes the tests brittle to any changes in those models
The syncable class is fine, as I am able to simply stub a class in the test and that works fine. But how can I do something similar for the active record model? Is there a way that I can create a kind of anonymous model class for these kind of tests? I will need to be able to persist the data as some of the sync engine process tests will require that data already exists, but i want this data to be completely independent of the main app itself.
I hope this makes sense
Example
Sync engine resource
class Integration::Resources::Account < Syncable
maps_to ::Account
string :name
string :description
belongs_to :company, Integration::Resources::Company, required: true
belongs_to :currency, Integration::Resources::Currency, required: true
end
Part being tested
I want to test a bit of code that checks that the required belongs_to relations of the Account resource as present. If they're not, it simply ignores the resources, but otherwise it continues.
Part of this involves loading the database records that are associated with the resources. So if the currency association has been set to an instance of the Integration::Resources::Account class, then the engine will retrieve that record from the database. If that record is not found, then the association is considered missing and thus the resource is ignored as it is required.
In this scenario, I currently have to:
Create a currency record
Create a company record
Instantiate a currency resource
Instantiate a company resource
THEN instantiate the account resource with the other resources, and run the test
To me, there is far too much knowledge here about the Account resource, as this is not what I am testing. I simply want to test the code that is ignoring resources with a missing association.
What I want to do is
- Create a dummy model class
- Only a single association
- Create a dummy resource class (this is ok)
- Only a single association
- Test
This way the test only knows what it needs to know. Any changes to any other code would then not break this test which stops it being so brittle.
Related
TL;DR: I don't know how organise my logic domain classes.
I have the model "Application", this model is in the "core" of the App and is the way I "enter" and operate over other models like:
#application = Application.find(params[:application_id])
#application.payment.update_attribute 'active', true
or
unless #application.report.status
or
#application.set_income(params[:income][:new_income])
so the models Payment, Income and Report are basically empty because I initialise the Application model and from there I do things "on cascade" to change the "subordinated" models. But now the Application model has more than forty methods and 600 lines.
I'm doing it right? For instance when I want to add a new Payment I like to do :
payment = Payment.create params
inside the Application model because ActiveRecord "knows" how to handle the foreign keys automatically. I could create the payment inside the Payment model using:
application = Application.find(application_id)
params[:application_id] = application.id
self.create params
but this way, I need to set the Application.id manually and that looks more verbose and not elegant.
So --if I want to reduce my Application model--, should I create modules in APP/lib directory or should I move methods to the other models?
should I create modules in APP/lib directory
Basically, yes, that's what you should do. Although I'd probably make them classes rather than modules. The pattern it sounds like you're after is called "service Objects" (or sometimes "use cases"). What this does is takes the logic from a specific operation you want to perform, and puts it in it's own self-contained class. That class then collaborates with whatever models it needs to. So, your models stay quite small, and your "Service Classes" follow the Single Responsibility Principle. Your controllers then usually call a single "service class" to do what they need to do - so your controllers stay pretty minimal too.
If you google "rails service objects" or similar, you'll find lots of great stuff, but here's some resources to get you started.
Service objects rails casts: https://www.youtube.com/watch?v=uIp6N89PH-c
https://webuild.envato.com/blog/a-case-for-use-cases/
https://blog.engineyard.com/2014/keeping-your-rails-controllers-dry-with-services
http://blog.codeclimate.com/blog/2012/10/17/7-ways-to-decompose-fat-activerecord-models/ (there's one section on service objects there)
Keep in mind, once you do start using service objects, you don't necessarily have to ALWAYS go through your Application model to get to the related ones. A service object might take an application_id and then do eg. #payment = Payment.find_by(application_id: application_id) and so you don't have to fetch the application instance at all and can manipulate the #payment variable directly.
The fact that Rails makes it "easy" and "pretty" to get to related models doesn't necessarily mean you should do it.
I would not worry about long controller and spec files in Rails.
These files tend to get very long and the usual advice of keeping classes and methods short does not necessarily apply for controllers and their specs.
For example, in our production system user_controller.rb is 8500 lines long and the corresponding user_controller_spec.rb is 7000 lines long.
This is the length of our top 10 controllers
1285 app/controllers/*********_controller.rb
1430 app/controllers/***********_controller.rb
1444 app/controllers/****_controller.rb
1950 app/controllers/****_controller.rb
1994 app/controllers/********_controller.rb
2530 app/controllers/***********_controller.rb
2697 app/controllers/*********_controller.rb
2998 app/controllers/*****_controller.rb
3134 app/controllers/application_controller.rb
8737 app/controllers/users_controller.rb
TL;DR: If your app has four models that are all tied to tables in your database (ie. leveraging ActiveRecord and inheriting from ActiveModel::Base), the framework is pretty opinionated toward using model classes.
Abstractions of the service class pattern can be useful in some cases, but give yourself a break. One of the advantages of Rails is that its supposed to remove a lot of the barriers to development, among many things, by making organization decisions for you. Leverage your model classes.
Let's see if this starts an epic developer bickering war.
Also, its ok to create interfaces in your models for related model creation:
class Application < ActiveModel::Base
has_one :payment
def create_payment(attrs)
payment.create(attrs)
end
end
And by ok, i mean that the framework will allow this. But remember, you're already inheriting from ActiveModel::Base which defines many instance methods, including create.
I would recommend, esp. if this is a small project and you're just getting your feet wet, to use well-named rails controllers to read and write objects to the database:
class ApplicationPaymentsController < ActionController::Base
def create
application = Application.find(params[:id])
application.create_payment(payment_params)
end
private
def payment_params
params.require(:payment).permit(:x, :y) - whatever your attr names are.
end
end
The sleekness you're looking for in abstracting foreign keys in creating a relational record is taken care of for you with Rails associations:
http://guides.rubyonrails.org/association_basics.html (good starting point)
http://apidock.com/rails/ActiveRecord/Associations/ClassMethods/has_one (more explicit docs)
That will help you slim down models if that is your goal. Just for clarification, this is one of those things that devs are extremely opinionated on, one way or another, but the truth is that there are code smells (which should be addressed) and then there are folks who arbitrary preach file length maxes. The most important thing in all of this is readable code.
A good litmus test for refactoring working code is put it down for a few weeks, come back to it, and if its confusing then put in some time to make it better (hopefully guided by already written test coverage). Otherwise, enjoy what you do, especially if you're working solo.
I have a scenario in which I need to dump/transfer data of one user from my Rails App to another database of same configuration and tables. For instance, The application is built as
class Company < ActiveRecord::Base
has_many :depots
has_many :users
end
class Depot < ActiveRecord::Base
belongs_to :company
has_many :products
end
class User < ActiveRecord::Base
belongs_to :company
end
class Product < ActiveRecord::Base
belongs_to :depot
end
My requirement is, if companyA stops paying, I want to dump their data into another DB (databse2 for instance) to keep my actual DB clean and once they come back and start paying, I want this data back.
Second requirement is, database2 can already have some data in it. So I need to retain all the records and I want to change the IDs of companyA (as there can already be a company with the same ID) while saving in to database2 keeping associations intact. It might seem silly to do this, but that is my requirement.
I am using Postgres as my application DB.
Any helps???
You have a few options here worth investigating:
Output everything in a singular JSON file that encodes everything the client had in the database, complete with ID fields.
Dump out a series of CSV files that can be imported on the destination server.
Dump out a single .sql file that will properly restore the data simply by running it.
The first option is the most elegant, but probably requires the most work. It gives you the ability to archive your data in a neat, tidy file that's easily parsed.
The second option might be fine or could be severely ugly depending on the sorts of data you have. If there's any binary data involved that's probably not going to work, but for clean, text-only columns and tabular data it's usually fairly efficient. The advantage here is you can selectively load in parts of your data without having to commit to parsing all of it.
The third option isn't easily parsed, you need to restore it to be able to use it, but it does make insertion really, really simple. You will only have to write an archiver, no specific restoration tool is required.
Whatever approach you take you'll need to be absolutely certain that ID numbers are never, ever reissued. Do not reset your sequence generators to fill in holes, and when moving databases port these over as well and test that they're set correctly. The last thing you need is ID conflicts.
If you're really worried about ID conflicts you might want to switch to non-numeric IDs, like use a UUID for everything where conflicts are basically irrelevant, though this does not come without a cost.
I am creating a SAAS app using Rails 3. When a user creates a new account, the database needs to populate with a lot of data. Some of the data will be specific to the newly created account. I don't necessarily want to do this all with Models within the controller "sign up" action. What would be the best way to do this?
From the sounds of things you should be using a callback within your User model. Most likely: a before_create or after_create (depending on your exact needs). You can then have the user model handle the creation of the account specific data, rather than your controller and thus adhere to the ideals of 'fat-model, skinny-controller'.
class User < ActiveRecord::Base
after_create :setup_account_data
private
def setup_account_data
# create other data as required
end
end
Pretty simple really, after the user model is created - the setup_account_data method will be called. A list of the other available callbacks is here.
A couple of approaches come to mind.
simple ruby. This is similar to what is done when you run rake db:seed -- execution of a ruby script.
fixtures. If you dump a database to fixtures [http://snippets.dzone.com/posts/show/4468], you can modify the fixtures so that the data that needs to be customized is done in erb blocks. This is the same technique that is used often in test fixtures.
The data source I am working with is terrible. Some places where you would expect integers, you get "Three". In the phone number field, you may get "the phone # is xxx". Some fields are simply blank.
This is OK, as I'm parsing each field so "Three" will end up in my model as integer 3, phone numbers (and such) will be extracted via regex. Users of the service KNOW that the data is sketchy and incomplete, as it's an unfortunate fact of the way our data source is maintained and there's nothing we can do about it but step up our parsing game! As an aside, we are producing our own version of the data slowly as we parse more and more of the original data, but this poor source has to do for now.
So users select the data they wish to parse, and we do what we can, returning a partial/incorrect model. Now the final model that we want to store should be validated - there are certain fields that can't be null, certain strings must adhere to a format and so on.
The flow of the app is:
User tells the service which data to
parse.
Service goes off and grabs
the data, parses what it can and
returns a partial model with
whatever data it could retrieve.
We display the data to the user,
allowing them to make corrections
and to fill in any mandatory fields
for which no data was collected.
This user-corrected data is to be
saved, and therefore validated.
If validation fails, show data again
for user to make fixes, rinse &
repeat.
What is the best way to go about having a model which starts off being potentially completely invalid or containing no data, but which needs to be validated eventually? The two ways I've thought of (and partially implemented) are:
2 models - a Data model, which has validations etc, and an UnconfirmedData model, which has no validations. The original data is put into an UnconfirmedData model until the user has made their corrections, at which point it it put into a Data model and validation is attempted.
One model, with a "confirmed data" flag, with validation being performed manually rather than Rails' validation.
In practice I lean towards using 2 models, but I'm pretty new to Rails so I thought there me be a nicer way to do this, Rails has a habit of surprising me like that :)
Must you save your data in between requests? If so, I would use your two model format, but use Single Table Inheritance (STI) to keep things dry.
The first model, the one responsible for the parsing and the rendering and the doing-the-best-it-can, shouldn't have any validations or restrictions on saving it. It should however have the type column in the migration so you can use the inheritance goodness. If you don't know what I'm talking about, read up on the wealth of information on STI, a good place to start would be a definitive guide.
The second model would be the one you would use in the rest of the application, the strict model, the one which has all the validations. Every time a user submitted reworked and potentially valid data, your app would try and move your instance of the open model created from the params, to an instance of the second model, and see if it was valid. If it was, save it to the database, and the type attribute will change, and everything will be wonderful. If it isn't valid, save the first instance, and return the second instance to the user so the validation error messages can be used.
class ArticleData < ActiveRecord::Base
def parse_from_url(url)
# parses some stuff from the data source
end
end
class Article < ArticleData
validates_presence_of :title, :body
validates_length_of :title, :greater_than => 20
# ...
end
You'll need a pretty intense controller action to facilitate the above process, but it shouldn't be too difficult. In the rest of your application, make sure you run your queries on the Article model to only get back valid ones.
Hope this helps!
Using one model should be straightforward enough. You'll need an attribute/method to determine whether the validations should be performed. You can pass :if => to bypass/enable them:
validates_presence_of :title, :if => :should_validate
should_validate can be a simple boolean attribute that returns false when the model instance is "provisional", or a more complicated method if necessary.
What is the best way to test a model that is using a different database connection in Rails. For example I have a model FooBar that is read only:
class FooBar < ActiveRecord::Base
establish_connection configurations['foo_bars']
# ...
end
Are there any good conventions, hacks, or plugins out there?
In my experience, once the connection is established you can treat the model just like any other model. Since you are just consuming the data that will simplify some of the testing as you wont need to test for data validations.
Obviously because Rails is talking to two different databases from two different models, you won't be able to do joins between the databases and so there will be nothing to test there either.
So, to answer the question: what is the best way to test a model that is using a second database? I would say, exactly the same way that you would test it if it was your only database.
I find that in my models that wrap my legacy databases I have to add some special tie in code that makes things a little more "Rails-y" and makes the view and controller code look like there are has_one and belongs_to type code in there. I do have tests that exercise those custom methods.
Hope that helps.