I have a rails app the works with multiple 3rd party APIs. Some of these APIs are rate limited and many have methods that can take hours to complete and you have to continually check to see if your request is ready. As such, I have built a queueing system to manage all the baggage.
Each request that goes into this queue has a JSON string that details which 3rd party API the request pertains to, which method, what arguments and a callback to handle the response. I'm going to need a good many callback methods and they handle wildly different tasks. Some update the service status of the 3rd party, some update customer information, some create another queued request to download a large CSV or parse a CSV, etc.
I'm not sure where to put all these unrelated callback methods. They are so varied and meddle with so many different models that I'm not sure it's right to put them under the QueuedRequest model (although that seems the easiest especially regarding testing). I'd like to consolidate them in a single place (again to make testing easier) so trying to shoehorn them into their related models (though often they may not be related to a model at all) isn't going to work.
A utility class of sorts seems to be the best place but where would I put that file?
Details:
Rails 3.2.6
Update: To elaborate on the answer below, what I ended up doing was putting a module of utility methods in /lib/api. I added /lib/api to my autoload_paths in /config/application.rb and, unmentioned, I created an initializer to load the module into my app for me.
/lib/api/callbacks.rb
module Callbacks
extend self
def some_method
# magic
end
end
/config/application.rb
config.autoload_paths += %W(#{config.root}/lib/api)
/config/initializers/application.rb
require 'callbacks'
Under the lib directory, and don't forget to add it to your autoload paths (or eagerload paths if your app is threadsafe!). Make a folder called api and then stick that stuff in a namespace
For example, lib/api/stack_overflow.rb
module API
class StackOverflow
# schtuff
end
end
You could always just throw it into the lib folder.
I am working on an application which queries an external API and presents the data to the user without actually saving it to the DB. There are various elements involved in this process (passing the API key and user ID to the API, validating the query parameters in the request, parsing the response, etc.) which don't have a lot to do with the actual rails application (which will be mainly focused on presenting this data to the user), so I split off all the API stuff into its own directory under lib.
By doing it this way I can test it separately from the main application, which makes for a clean separation of concerns. Also, since my classes for the library are not based on activemodel, the library can be spun off into its own gem and potentially used in other non-rails projects. I have considered, for example, switching rails for sinatra, which is easy to do given the current setup.
Related
I need to generate xml from a model and send it to a web service on model save.
I'm sure this is a common case and should be straight forward. Just create a job on after_save callback that generates the xml and sends it to the endpoint.
Since I'm new to Ruby on Rails I'm not to sure how to handle this though. My questions are more about code organization. It's not unlikely that this api connection will be discontinued in the future so I need a clean modular way to get rid of it. Would it be best practice/convention to put this in a separate gem? Can gems actually add jobs to an existing rails queue? Can gems create migrations on install? I'll probably need to add a model to keep track of the api sync. How about dropping a table on gem uninstall? Or should I not use a gem for this at all?
I realize these are broad and basic Ruby on Rails questions but I'm kind of drowning in documentation. I'm just hoping for some examples and/or advice and maybe some pointers to relevant documentation. Thanks.
Gem installs/uninstalls are unrelated to apps, they live on different level and do not khow anything about your app code, db and so on unless they are loaded.
Gems for rails can provide rake tasks and/or generators, for example you can look into devise gem structure on how it does this.
But i'd advise against moving code to a gem before you know you have to, like for example when you need to reuse it in different project.
To reuse code inside single project - use mixins/concerns
In general:
don't make it a gem
it's an unnecessary world of pain, pretty much always,
never make anything a gem unless you intend to use it in the same way in 3+ applications
don't extract it into a concern either,
it doesn't seem very likely that you'll do the same operation on multiple models, code reuse seems to not be an issue here and you can actually reuse code more efficiently using service classes too
a lot of experienced Rails programmers regard this practice as concerning, forgive the pun. It seems this view is not shared by the Rails development team, but at least from my experience writing service classes seems like unnecessary complexity until your project grows enough and then you need to refactor a BUNCH of stuff and you realize you would have been better off ditching concerns from the start
use a service class instead and delegate the necessary methods to it from the model
this will leave you with a clean interface to extract later and will also allow you to use dependency injection if you need to mock your XML service for tests
don't tie API requests to model callbacks, there's usually just 2-3 places where you need to do something with the API and a bunch of other cases where that may not be the case, imagine:
tests,
or if you get a requirement to implement cache column,
or a "number of visits" column
or a gem like Paperclip that thought that it wanted to add something to the model but changed his mind and instead of that just touched updated_at
or any such trickery which will make you a grandiose API spammer and a sufferer of VERRRRY slow database updates
if you DO tie API requests to model callbacks,
then you better make sure that error handling is done properly and that timeouts etc don't rollback or delay your DB operation,
best way from my experience is to run these things through ActiveJob + one of the backends (though obviously not the :inline backend and ideally one of the backends which don't use your main database and allow asynchronous job submission - sidekiq comes to mind as a candidate)
I am creating a Rails app that will use quite a few external APIs such as SalesForce, FolderGrid(like Dropbox) etc., which would all sync to my database. I've never worked with external api calls so I created a few basic Ruby scripts with scattered methods to test the call to those resources. Now I would like to implement them with my full Rails app.
So far, I started off by creating a directory in my /lib folder for holding the api call modules/classes..
/lib/apis/foldergrid.rb
Now I'm not sure what the best approach would be towards organizing the code. In my standalone Ruby scripts, I have methods for authenticating, creating a folder, auditing files, downloading files etc.
What should be in a Module? Do I even need a Module? What should be in a class? How do I make sure I can use these methods in my models and controllers where needed? Is there a best practices concerning external APIs?
Any resources, links, and/or deeper understanding is highly appreciated.
I have an app that does this too (pulls data from multiple APIs) so I can tell you what I would did (or in some cases what I would do now if I was to start over).
Data Storage
First off if you are already persisting the data pulled from those into a Rails app, then you will have models representing the data themselves. So that's what you would refer to in your controllers unless your controller is in charge of calling the APIs.
Use Gems where possible
Second make sure to leverage existing gems that existing for the APIs so you're not recreating the wheel of authentication, parsing the JSON to and from the app, etc. For example: there is a rest-force gem for salesforce.
Use Ruby Objects in Lib for mapping between your models and API responses
Third, I would make the classes that map data in between API responses and your models into "Plain old ruby objects" and store them in a lib/apis folder (no need in these cases to store in a module, yet IMHO). They might end up very functional and in the case of mapping data, that is ok.
APIs without existing Gems
In those cases, I would definitely keep them in one or more classes under a module, in their own folder in lib. That way you can parse them out if needed as a Gem (or if you to take the time make them a Gem to start with, I'm sure others could benefit). I just started down this road myself this week using Infoconnect's API which didn't have a Gem (or any code samples in any language). But I'm not very far along.
If needed, create service objects for code in between V and C (view and controllers)
If you do end up needing to do anything to interact with users for the API information or anything that calls those request, I would put those in service objects so your code doesn't get bloated in the view or controller.
What should be in a Module?
A module should contain code that is shared by different classes, or it can be added to a single class.
Do I even need a Module?
No, you can achieve the same thing with inheritance. If there is no code to share, then you probably don't need a module.
How do I make sure I can use these methods in my models and controllers where needed?
You can access them using the class from within a controller:
SalesForceAPI.get_data
"Module or class?" is an age-old question in Ruby, and any other language that supports both.
I would encapsulate the functionality for each API in a class. For example, you can have a SalesForceAPI class. Then, if you need to share functionality between the classes for authenticating, creating a folder, auditing files, or downloading files, you can create an API class.
Each of the classes that needs access to the API class would inherit from it:
SalesForceAPI < API
You can achieve the same thing by creating an API module, and mixing it into the other classes. In this case, it's largely a matter of preference, because either solution is good.
FWIW, I don't know if there would be much shared functionality for authenticating, creating a folder, auditing files, or downloading files, because each API can work quite differently. However, if they are REST API's then you may create some REST helper methods and put them in the API class.
A great, easy to understand, and complete resource on the topic is Practical Object Oriented Design Ruby.
I'm looking for the "rails" design pattern for code that fetches data from other websites.
I have a rails controller in my app that fetches data not from the database, but from external API's or scraped from the web.
Where's the "rails" place to put this code.
For quick implementation, I just stuck it in a model, but the model doesn't interact with the database - or support standard model functionality - so that feels wrong, but my understanding of rails and ruby isn't yet solid enough to know where it should go.
The way the code works roughly is
controller calls model.fetchData args
the model uses HTTParty or similar to make the call
processes data
passes it back to the controller
Any advice?
Broadly-speaking I think there are two possible ways to do this:
Create a plain ruby class to contain the methods for making requests to the API(s) and processing responses from it(them). You can include the HTTParty module in this class with include HTTParty. The usual place to put this code is in lib/ (make sure that wherever you put it, the path is in autoload_paths).
If you're doing anything really complex, or the API itself is complex, you might want to consider creating a separate gem to handle interaction with the API(s). The term for this type of gem is an "API wrapper" -- if you look around, you'll see there are lots of them out there for popular services (Twitter, LinkedIn, Flickr, etc.)
Notice I haven't mentioned activerecord. If you're not going to be saving anything to the DB, I don't see any need to even create any activerecord models. You can get by with just controllers and views, and then (if needed) pick and choose components from activemodel (validations, internationalization, etc.) to make your ruby API wrapper class feel more like a Rails model. For example, one thing that I've done in an app I'm working on is to apply validations to query strings before actually making requests to an external API, which is a bit like running validations on database queries before querying a DB. See this article by Yehuda Katz for more on how to make plain ruby objects feel like activerecord models.
Hope that helps. I answered another question very similar to this one just yesterday, you might want to have a look at that answer as well: Using rails to consume web services/apis
I'm just starting out in Rails and there's a lot I still need to learn so I'm likely to be on Stackoverflow more often than normal asking beginner Rails / Ruby questions.
I'm just trying to figure out how Helpers work in Rails. From what I've seen so far, Helpers are intended to be used with Views and not so much with your Controllers.
However I would like to make a simple function that will validate the user input given in params (check if certain params are defined and optionally check if their value is valid).
Can anyone explain to me what would be the best way of implementing this? (Keeping in mind that I will want to use this in many different controllers so it should be globally available.)
I also noticed that by default Rails does not generate a lib folder in the main application folder. Are developers to place their libs outside the app folder in the main folder, or does Rails use libraries differently?
With regards to your validation issue, it depends on what you are validating.
If the data makes up objects from your problem domain, also known as models, then you should use the built in validators from ActiveModel. This is probably what you should do, but its hard to say without knowing the exact problem. See the Rails Guides on Validations. You can tell if this is the case by asking yourself if the data that needs validation will be stored after you get it. If so, its most definitely a model. An example of this kind of data would be the title and text fields of a blog post being sent to Rails from a browser form.
If the data is something tertiary to your models, or specific to presentation, then you should be fine using helpers. You noticed that helpers are used mostly in the views, and although this is true, theres nothing stopping you from using them in the controllers, you just have to declare that you will use them using the ActiveController#helper method. Inside the ApplicationController class, a lot of devs will put helper :all to just include all the helpers in all the controllers. Once the code has been required once, it doesn't really incur that big a performance hit.
Do note that almost all incoming data can be modeled using a model. A big school of thought in the Rails world subscribes to the Fat Model idea. People say that putting as much code as possible in the model and as little in the controller as possible separates concerns properly and leads to more maintainable code. This suggests that even if you don't think the incoming data is modelable (in the sense that you can create a model to represent it), you should try to make it a model and encapsulate the logic around validating it. However, you may find that making a helper function is faster, and either will work.
Your notion of validating user input is a good one. I get the feeling that as you are new to Rails you are used to doing these things yourself, but that doesn't quite apply here. In the Rails world, a lot of the common stuff like validations is handled by the framework. You don't have to check for presence in the params array, instead you call validates_presence_of on the model and let Rails spit the error out to the user. It makes things easier in the long run if you let the framework do what it is designed to.
With regards to your question about the lib folder, it doesn't really matter. You can put miscellaneous support files and libraries in the lib folder in the root directory and they will be available for use in your application (the files in the app folder). You can also choose to abstract your code into a plugin or a gem and include it that way, which a lot of people opt to do. My suggestion for this would be to read up on the notion of gems and plugins before diving in.
Want you want is probably a custom validator (in Rails3):
http://railscasts.com/episodes/211-validations-in-rails-3
You can either add libs in a lib folder you create, or add them to config/initializers in a file you add. Files in the initializers directory are automatically loaded by Rails.
Let's say I have a Rails app that gets most of it's functionality from a gem (for instance, a CMS).
If I now need to add some customisation (for instance, add a property to a user) what is the best practice way of doing this? If I customise the gem, then I will have issues with updating the gem in the future.
What is the best approach to take here?
This question is quite old, but I feel it could use a bit more fleshing out. It is true that you can monkeypatch rails (and ruby) at run-time. That means it's easy to reopen a class or module and inject new code. However, this is somewhat trickier in rails due to all the dynamic class loading and unloading that goes on development mode.
I won't go into details, but you really want to put your extensions into an initializer, or a gem, since they get reloaded between requests in dev mode. If you put the code into a plugin it won't get reloaded and you'll get very mysterious errors such as "A copy of XXX has been removed from the module tree but is still active!"
The easiest thing to do is throw the code into an initializer (e.g. config/initializers/user_extensions.rb). You can just use class_eval to inject the code.
User.class_eval do
... new code ...
end
One major drawback of ruby's extensibility is tracking down where code is coming from. You might want to add some kind of log message about the extensions being loaded, so people can track it down.
Rails.logger.info "\n~~~ Loading extensions to the User model from #{ __FILE__ }\n"
User.class_eval do
... new code ...
end
Further reading:
http://airbladesoftware.com/notes/monkey-patching-a-gem-in-rails-2-3
Ruby allows you to extend classes in runtime, so you can often hack in to a library without touching the source code. Otherwise I would suggest that you download the gem, create some hooks in the library and submit that back as a patch.
Update:
Note that these customisations are application specific
Yes. What I meant was to modify the generic api in a way, such that it is possible to customise per application. For example, by allowing the user to pass a block to certain methods etc.