I'm in the process of setting up a centralised data management system + CAS for a series of apps that currently have duplicated data across many databases that is a nightmare to manage.
I've got the CAS system implemented nicely with rubycas-server and rubycas-client, but I'm running into problems when it comes to actually sharing the user data and other models between client apps. For example:
CAS/Central system (share a DB) expose a base User as JSON
App1 extends user with information about favourite colours
App2 just cares about having a user
I want to be able to let App1 add the data it cares about the object returned from the API, without burdening App2 with that info.
I'm currently playing around with writing a gem for the client apps which can consume the JSON (with ActiveResource or ROAR, not sure which yet), however the client apps need to decorate/extend their models with additional data which is currently managed via ActiveRecord. So I'm not sure what the best way to go about extending those gem provided instances would be.
So far I've experimented a bit with having the gem contain a bunch of modules which the ActiveRecord models can then include, but this has issues when needing to deal with class methods (not insurmountable but any means). I've tried looking around for resources/guides for best practices in this sort of situation, but haven't been able to find anything.
tl;dr summary: Multiple client apps need to use and extend data provided by a JSON api. What's the recommended practice for doing this?
Related
Background
I have a fairly typical Rails application, which uses Devise for authentication management. While building this app, I realized that realtime chat would be a great feature to have. Ideally, of course, this would make use of Websockets, in order to reduce the polling load on the server (as well as making it marginally easier to implement, as you don't have to manage polling).
I realized quickly that Ruby isn't really a great fit for having a large number of concurrent connections open at one time. Phoenix, however, is written in Elixir, so I can make use of the Erlang VM, which is quite good at long connections. It also seems like it could be greatly beneficial if all the chat data was stored separate from the main application database, which should also reduce load in the future.
The Problem
I want to be able to make this separation completely invisible to the user. They visit www.example.com/chat, and it loads all the relevant data in from chat.example.com and starts up the websockets, without requiring them to login to a separate service. I think using an <iframe> is probably the way to go about doing this.
My problem is sharing authentication and data between the two applications. The Rails app needs to be able to create conversations on the Phoenix app in response to certain events. The Phoenix app needs to know what user is currently authenticated into Rails, as well as general data about the user.
An OAuth flow with the Rails app as the ID provider seemed like a good fit at first, but I can't figure out a way for the Phoenix app to automatically be granted access. I also have some concerns about user records existing inside the Phoenix app—it should be aware of all users on the main application, so you can start a chat with a user even if they haven't ever opened chat.
What would be the best way to go about doing this? My intuition says that this is going to involve window.postMessage and some kind of token system, but I wanted to ask what the generally accepted way of doing this was before I accidentally created an insecure mess.
Sharing the session isn't too hard, assuming you are running at least Rails 4.1 and using JSON serialization (default for apps created with >=4.1). A quick google search finds PlugRailsCookieSessionStore, which accomplishes this.
For more information on what it takes to share a session between Rails and another language, Matt Aimonetti has an excellent blog post with detailed information.
Lastly, if you would prefer to stay entirely in Ruby, it's definitely doable. Ryan Stout discusses scalability around persistent connections in the FAQ for Volt, which uses a persistent connection for every user. The article he links is also a great read. Just mentioning it to help you weigh the trade off of building a separate app in another language.
I've built a database in Neo4j and use Rails with Neo4jrb as an easy way to manipulate the database. For reasons I explain below, below I call this a knowledgebase (kb) instead of database.
I starting working with a friend who wants to provide access to the kb for users of his app. So I built an API in rails so he can access it.
Now my friend and I are talking about building a membership site with subscription payments. So I figured I needed to figure out how to build Stripe subscription payments into my app. I found this great tutorial by RailsApps which in terms of functionality does exactly what I need, it relies on a gem called Payola which makes Stripe integration easy. Payola is great but it works with ActiveRecord, not Neo4j. I was thinking I'd have to figure out how to do what Payola does with Neo4j.
But now I have the idea of just building a separate app using the RailsApp+Payola approach, and then just hooking that app up to my kb's API.
The reason I say knowledgebase is because its purpose is to structure knowledge within a particular domain. I use the graph database Neo4j because the graph-based data model suits this goal, for example (object of type A) -[has a certain influence on]-> (object of type B).
So I like my idea of using a separate app for managing users and subscriptions because then I avoid mixing app specific data with domain knowledge in the database.
So I guess my concern is, what might I be missing? Will speed be a concern if this membership site has to access an API every time a page loads? Would there be unusual security concerns?
It's perfectly fine to use another database next to neo4j. You should always use the database that fits your needs I think :)
Suppose we have this scenario:
www.main.com - Main interface where admin (foo, bar, etc..) can store products, based on their own e-commerce
www.foo.com - Sample store that sells items from the "foo" store
www.bar.com - Sample store that sells items from the "bar" store
The problem is finding a way to centralize the database structure and models.
I prefer to keep every single store in separated apps (so I exclude rails engines).
For instance, if a user buys something in the "foo" store, I need to interact with the main db and update it.
How can I do this?
Rails works much better with a "one database, one app" model. There are lots of ways how you can share models across apps (gems, engines, git submodules, etc), but none of those ways is great. You end up introducing lots of overhead in your development, deployment, and testing process. You also invite lots of hidden dependencies between code, as Rails doesn't give you easy way to keep clean abstraction (for example, you wrote a helper for store Foo, and then your coworker used that helper in store Bar, and then every time you change Foo, Bar breaks).
I recommend a centralized API approach instead:
api.foobarmain.com - a app/service that provides RESTful API for all the functionality of all stores.
This app has all the db models, and it exposes them as resources in the API for other apps to interact with.
This app can have an admin UI for all the stores, if you need it. Alternatively, admin UI may be another client of an API.
www.foo.com - a full stack app that interacts with API at api.foobarmail.com
There is no shared database connection to API, everything that you need to interact has to be exposed via API.
There will be no shared code between www.foo.com and www.bar.com. Code reuse happens only by virtue of using the same API app/service.
From the perspective of www.foo.com, the model layer (in MVC) is powered by API, not by database.
You can still have its own database on www.foo.com if you need to store data specific to www.foo.com only.
www.bar.com - another full stack app that interacts with API
so on ...
Another way is using multiple schema if you are using Postgresql. I have a similar issue with a new project that I about to start.
You can use gem 'apartment' to deal with different schemas. The queries will be a bit complicated but with different schemas you will end up with one database and you can create different namespaces to respond accordingly.
You can set up so app select the correct schema based on the Domain or Subdomain.
Here is the link: https://github.com/influitive/apartment
I am creating a Rails app that will use quite a few external APIs such as SalesForce, FolderGrid(like Dropbox) etc., which would all sync to my database. I've never worked with external api calls so I created a few basic Ruby scripts with scattered methods to test the call to those resources. Now I would like to implement them with my full Rails app.
So far, I started off by creating a directory in my /lib folder for holding the api call modules/classes..
/lib/apis/foldergrid.rb
Now I'm not sure what the best approach would be towards organizing the code. In my standalone Ruby scripts, I have methods for authenticating, creating a folder, auditing files, downloading files etc.
What should be in a Module? Do I even need a Module? What should be in a class? How do I make sure I can use these methods in my models and controllers where needed? Is there a best practices concerning external APIs?
Any resources, links, and/or deeper understanding is highly appreciated.
I have an app that does this too (pulls data from multiple APIs) so I can tell you what I would did (or in some cases what I would do now if I was to start over).
Data Storage
First off if you are already persisting the data pulled from those into a Rails app, then you will have models representing the data themselves. So that's what you would refer to in your controllers unless your controller is in charge of calling the APIs.
Use Gems where possible
Second make sure to leverage existing gems that existing for the APIs so you're not recreating the wheel of authentication, parsing the JSON to and from the app, etc. For example: there is a rest-force gem for salesforce.
Use Ruby Objects in Lib for mapping between your models and API responses
Third, I would make the classes that map data in between API responses and your models into "Plain old ruby objects" and store them in a lib/apis folder (no need in these cases to store in a module, yet IMHO). They might end up very functional and in the case of mapping data, that is ok.
APIs without existing Gems
In those cases, I would definitely keep them in one or more classes under a module, in their own folder in lib. That way you can parse them out if needed as a Gem (or if you to take the time make them a Gem to start with, I'm sure others could benefit). I just started down this road myself this week using Infoconnect's API which didn't have a Gem (or any code samples in any language). But I'm not very far along.
If needed, create service objects for code in between V and C (view and controllers)
If you do end up needing to do anything to interact with users for the API information or anything that calls those request, I would put those in service objects so your code doesn't get bloated in the view or controller.
What should be in a Module?
A module should contain code that is shared by different classes, or it can be added to a single class.
Do I even need a Module?
No, you can achieve the same thing with inheritance. If there is no code to share, then you probably don't need a module.
How do I make sure I can use these methods in my models and controllers where needed?
You can access them using the class from within a controller:
SalesForceAPI.get_data
"Module or class?" is an age-old question in Ruby, and any other language that supports both.
I would encapsulate the functionality for each API in a class. For example, you can have a SalesForceAPI class. Then, if you need to share functionality between the classes for authenticating, creating a folder, auditing files, or downloading files, you can create an API class.
Each of the classes that needs access to the API class would inherit from it:
SalesForceAPI < API
You can achieve the same thing by creating an API module, and mixing it into the other classes. In this case, it's largely a matter of preference, because either solution is good.
FWIW, I don't know if there would be much shared functionality for authenticating, creating a folder, auditing files, or downloading files, because each API can work quite differently. However, if they are REST API's then you may create some REST helper methods and put them in the API class.
A great, easy to understand, and complete resource on the topic is Practical Object Oriented Design Ruby.
The iTunes Enterprise Partner Feed is "a data feed of the complete set of metadata from iTunes and the App Store" and "is available in two different formats - either as the files necessary to build a relational database or as stand-alone flat files that are country and media dependent."
I need to consume the data from this feed (which is essentially exported into flat files) and allow linking of my own Model objects (User, Activity, etc.) to data provided by the feed (App, Developer, etc.) The data is provided as a weekly full export and a daily incremental export.
I have two ideas for ways to implement this:
Create all of the models in my rails app and write my own importer that will insert/update records directly into my app's database daily via cron using models I've created (App, Developer, etc.)
Keep this database entirely separate and open up REST API that my own app will consume
My naive approach with #1 to keep everything in the Rails app is based on the need to be able to observe changes in the data I'm getting from the EPF. For example, if an App's description is updated, I want to be able to create an Activity object via an observer to track that update.
On one hand #2 feels like a better approach because it creates a standalone API into the data that can be consumed from several different apps I create. On the other hand, I'm just not sure how I'd accomplish the data change notifications without using observers directly on my own models. Or, how I would even consume the data in an object oriented way that could still be used with my own models. It feels like a lot of duplicate work to have to query the API for, say, an App's data, create a proper Active Record object for it and then save it just so it can be linked to one of my own models.
Is there a standard way to do this that I'm totally missing? Any pointers?
EDIT: Rails engines sound interesting but it would mean that each app would still need to consume and insert the data separately. That doesn't sound so DRY. It sounds more and more like a REST API is the way to go. I just don't know how to bridge the gap from API to Active Record model.
Rails Engines might be a good fit for this. You can create a Rails Engine gem and add all of your models and rake tasks to consume the data. Then you can include this gem in any app that uses it and also create an API app which includes the gem. You should be able to create observers in your other apps that interact with the gem.
I have quite a few apps that interact with each other and this approach works well for me. I have one central app that includes all the engines that consume data and I run all of my cronjobs from this app. I use the use_db plugin which allows my app to communicate with different databases. Each engine has use_db as a dependency and I keep the database configuration inside the gem. One example:
Engine Gem = transaction_core
This gem consumes transaction data from a source and inserts it into my transaction database.
The gem is included in my central app and I pull the transaction data using a rake task on the cron
I include this gem in several other apps that need to use the transaction data. Since the engine automatically adds the models and database config to the app, there is no additional work required to use the models in the app.
I have not used observers inside an app that includes my engines, but I see no reason why it would not work. With the engine the models work as if they are in your app/models directory. Hope this helps!
Modest Rubyist has a good 4 part tutorial on Rails 3 plugins that includes Engines:
http://www.themodestrubyist.com/2010/03/05/rails-3-plugins---part-2---writing-an-engine/