I'm looking for some resources on taking an existing monolithic Rails 3.0 application (35K LOC) and breaking it apart into an SOA design. Any books, blogs, screencasts, or example applications would be awesome.
The main questions I'm looking to answer are:
Is SOA even the right design?
Where do I start?
What are some common pitfalls I can avoid?
What should I be thinking about now vs what can I do later? (ie performance)
Some resources I've seen but not totally sure if they are the right places to start:
Service Oriented Design with Rails
RESTful Web Services
Enterprise Rails (not sure if this book is still relevant)
SOA Design Patterns (maybe too "enterprisey"?)
Is SOA even the right design?
It depends. Don't you hate these kinds of answers?
Breaking up your app to loosely coupled services using messaging or API calls would, by definition, be implementing SOA.
The beauty of it is that you can interchange service implementation without changing their interfaces and allow for independent deployments without having to bring down the whole app. Also, I'd implement the SOA via specialised API controllers that are versioned and expose custom state rather than their whole state which you'd reserve for authenticated users or role-based sessions.
The dilemma, from my experience, is whether to implement synchronous or asynchronous calls. Synchronous calls are obviously easier to program, but may leave your user hanging while they are being executed, and you'd have to handle timeouts for long-running queries. Watch out for database and web server timeouts.
If you implement asynchronous calls, let's say via ActiveMessaging or alike, youd have to handle callbacks or some kind of notifications to bubble up to your user. It also entails setting up primary and secondary message brokers and maybe some JavaScript or pollers to check for status. It's all fun though!
Where do I start?
I'd first see if it's "worth it": after all, SOA is cool, but does introduce multiple points of failure that you do not have currently.
If you think your broken up app will result in discrete services that are HA and will serve other projects, I'd start with "the druby book" and "service oriented" as you mentioned.
What are some common pitfalls I can avoid?
I think the biggest concern would be transactions across multiple services and the ability to roll-back the whole operation if a distant service fails. The problems begins if you are in some operation where you call A and it calls B and B calls C and C fails.
Who knows that C failed? How will you tell B and A to roll back? Can they? Do they save state? Tough questions for upfront design.
One other issue is that life gets complicated when you throw a workflow on top of your SOA: who's the keeper of the business process? Centralised or distributed? It's all absolutely cool stuff again, but heaviness creeps in, no? But that's life if you must move to SOA.
What should I be thinking about now vs what can I do later? (ie performance)
I'd factor out the obvious generic services that can be used in other apps right now. I would not over-SOA your environment to avoid adding points of failure and keep the ones SOA introduces to a minimum.
This is an excellent resource from my friend who is the CTO at Crowdtap. They did the same thing and it has really helped them massively improve the velocity of product development and give them better test coverage. Hope it helps https://www.youtube.com/watch?v=KsiQXAXsQDQ
Related
I’m working on an iOS app that aggregates contents (i.e. events) from different services (Google Calendar, Basecamp…) using their REST APIs.
Instead of querying these services in the client, I wonder if it wouldn’t make more sense to do it server side. This would allow me to unify the different models beforehand, reduce the client side logic significantly, and IMO make it both more scalable and easier to maintain.
But I have no experience of this kind of backend and no idea where to start, or even if it’s a good idea at all? Do you have any idea? I was thinking of using Node.js and Express/LoopBack…
Thank you in advance for your help!
David
I am afraid this question will be soon marked as opinionated, and myself most likely be down voted... but maybe I will be on time to pass a little of my experience working on a back-end for my employer.
If you do not need to impose any control over who and when and what data is being pulled from those multiple external services it makes more sense to free yourself from the hurdle of developing and maintaining your own back end. Keep in mind the costs associated with having to support your own servers...
Additionally, having this logic on the client scales proportionate to the external services being used while with your own back end you will have to make sure that it is programmed to scale well, which could be a considerable undertaking if you never did it (not saying that it is impossible to learn in a reasonable timeframe).
Basically, what I am saying is it is always better to delegate security/control agnostic logic to the client side. This way by the way, you will be following along the lines of currently buzzing micro-services trend :)
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I was intrigued by Robert Martin's talk about "Architecture: The Lost Years". In it he discusses the Entity, Boundary, Control design pattern on which MVC is based. I love the idea of deferring architectural decisions. He described deferring the decision about how to implement the DB layer in his own wiki app FitNesse. I have organically deferred decisions like this in my own coding, though there wasn't a preconceived modular design that brought this about.
I want to better understand this EBC architecture (which seems closely related to DCI) from a practical standpoint so that I can begin using in a small project. I want to capitalize on "deferring decisions" and the ability to swap out aspects of the design like the UI.
Rails, for example, uses a form of EBC (MVC) but it's so heavily baked in that one could not easily substitute an alternate UI thus converting a Rails app to a console app or a desktop app. The intriguing thing about design for me is this ability to transform applications by swapping one thing out and plugging another in. That is, I wonder at the idea of designing an architecture so that one can, in a manner of speaking, swap out the UI or the persistence layer. I feel that if the architecture is well designed, the coupling will be low, and such a feat will be within grasp.
I've ordered the book by Ivar Jacobson that Bob mentioned in his talk. I've search online quite a bit but all of the examples I've found show simple diagrams. I speak code. I would benefit more from looking over a few simple classes that demonstrate the concept and show how one might swap out one layer (UI, DB) for some other implementation through the use of boundary classes.
If someone can't point me to a good resource illustrating this, would this be hard to whip up? Maybe we could use the standby example used in lots of software books: a video rental store (almost a relic these days). Please demonstrate how the UI or DB layer could be swapped. One thing that's confusing me is views. I can't tell from the diagrams I've seen if the views are the boundary classes themselves or if they just communicate with them. Also, Bob mentioned that the original intent of EBC was that we'd have lots of micro-views not a single macro-view (as we do in typical MVC); I'm curious what this might look like. (I prefer Ruby or JavaScript but, as beggars can't be choosers, any example would be fine.)
Thank you.
As far as I understand the video by Uncle Bob using "EBI" (Entity, Boundary, and Interactor) you should completely decouple your business behavior/state from frameworks/OS and services.
So in case of an Rails app your business behavior/state is completly free of dependencies to the Rails framework and hence can be tested like with rspec without firing Rails!
So on the business side you have Boundary classes wich interact with the Rails side using request and response models (very simple dataholders, not to be exchanged with the usual models from Rails). Only the Boundary classes interact with the Interactor classes which implement the (business) use cases / scenarios. And only the Interactor classes interact with the Entity classes which encapsulate the business state.
On the Rails side you find Controller classes interacting with Boundary classes (using Request models) and backwards a Boundary class interacts with a Presenter (using a Response model). Only Presenters/Controllers interact with Views (with the help of models (again simple data-holders). Note that in the realm of Rails Presenters are most likely Controllers.
Where does this leave AR? Well AR just provides the persistant service. On the same level as the Presenter/Controller level you will find Service classes which provide their services to the Boundary classes. So they provide all the necessary services which are frameworks/OS/technology dependent like persistance, security, timing, notifaction, etc..
With this architecture you are really able to reuse your business logic and completely replace the UI or database technology. For example, porting to mobile (iOS, Android, Windows) should be pretty straight forward.
With Rails, your app folder could look like:
app/
controllers/ Only these interact with Boundary classes
models/ simple data-holders, no AR here! (see services)
views/
services/ AR-stuff
boundaries/ To be tested without Rails
models/ Request & Response
interactors/ use cases / scenarios, to be tested without Rails
entities/ "the real business model without technical dependencies"
With this architecture, you need to code a bit more but don't forget the benefits of a good architecture:
A good architecture allows major changes to be deferred
A good architecture maximizes (major) changes not made
Last note: compared to the MVC pattern, its more like the M is replaced by EBI, the C can be splitted in CP/resenter), and an S(ervice) is added. So this could be called: VCPS/EBI but that sounds ugly to me ;-) BEPVICS maybe?
#Seralize, thanks for your feedback.
Let me try to answer your questions, so far I understand them: the stuff in services are coupled to Rails. They provide the implementation for the logic in EBI side. In the usecase of security, you have to be clear what (quantified) requirements you have, so you know what logic you can implement on EBI side, for instance (business) rules about when a user(role) has access to what content(and needs to be authenticated).
This means to implement authentication will be implemented using Rails, this service will be used by EBI. This security related logic in EBI is then pretty easy to reuse in your Java GUI example. There you have only to reimplement the authentication service.
To be clear in the security example:
The EBI side has the logic: what stuff needs what kind of security and when and how. The Rails knows nothing about this, it request what to do from the EBI side and or the EBI side request the Rails side to act.
The Rails side only implements the way how to do security like asking the user to authenticate (when needed) and passing the result of this to EBI so the logic can decide what should be done next.
EBI demands both sides to be decoupled & independent. It were as you
are developing the EBI as a library with a defined API.
Ask and you shall receive. I kept my eyes open and discovered this resource by Avdi Grimm:
http://avdi.org/devblog/2011/11/15/early-access-beta-of-objects-on-rails-now-available-2/
In it, he covers some of the reason that Rails projects get so tightly coupled to both the framework and ActiveRecord. He uses TDD to assure loose coupling with techniques like
Dependency Injection
Presenters
Strategy Pattern
DCI
It provides a good start to answering this question in a practical way. (It costs $5 for the early beta but will eventually be free.) In fact, it's the first resource I've found that does. Please add any others you find.
Here's the real gem that elucidates the heart of the problem:
One day, after years of witnessing and addressing the technical debt incurred in various maturing Rails codebases as a result of ActiveRecord-inspired tight coupling, I had an epiphany. What if we stopped treating ActiveRecord as the backbone of our model classes, and instead, programmed as if ActiveRecord were merely a private implementation detail?
Corey Haines puts it another way:
I pull the behavior out of my models into other objects that wrap the models.
I prefer to make the AR objects simple wrappers around the db-access stuff in AR.
I have a fairly strict rule that controller actions cannot use AR finders or,
in fact, interact with AR at all. AR should be accessed within api methods
inside your model, not from the outside.
This should be of interest too. It's the other book, not referred to by name in "Architecture: The Lost Years"
"Agile Software Development: Principles, Patterns, and Practices", by "Uncle Bob" Martin.
Taken from this SE question & answer. Read the other answers too.
So I've been tasked at work to write windows services to replace some old legacy VB6 WinForms apps currently running as services, consistently repeating tasks day-to-day. To give some general background, they have there own state machines built in to handle decision basing and not utilizing threading.
A lot of the senior developers here thought it would be worth a try to look into WorkFlow to replace the state machines rather than write my own business logic and try threading it programmaticly. So it's WF vs. the "Old College Try" I suppose.
My concern is that there aren't many books on the topic, and since it was implemented in .Net I've heard very little about it being used. I brought this up at work and another developer mentioned that it's because Biz Talk never really caught on and it was designed for that.
So is it broken? Do you think it will be supported long enough to not worry so much? I don't want an ill-functioning process injected into my services, my new babies at work, and then have WF's keel over. Leaving me with having to replace them with my own code in the event of an emergency; which does not seem like much of a grand scenario to me.
Any suggestions, recommendations would be super.
Workflow Foundation is used in Microsoft SharePoint, so I think they will continue supporting it.
There is an open source project called Stateless by Nicholas Blumhardt. It is quite flexible and very light weight. See my SO answer for details.
I chose this over Windows Workflow simply because I could define a state as State and thereby persist the state of my workflows back to the database using SubSonic. Configuration consists of one XML file. If I need to add tasks, I simply add nodes to the XML.
The each state can have a series of triggers that once satisfied will advance to appropriate state. This framework is a single assemble and fits nicely in your domain logic.
I have a Rails webapp [deployed on Heroku] which makes a series of HTTP calls to other sites on a repeated basis, using Heroku's rake:cron feature. The current situation isn't ideal; the rake:cron process is executed in a single thread, which means HTTP calls are made sequentially; which means in turn that there's a long time between calls to the same site [typically 2 mins].
I'd like to execute this process in parallel, and reduce the time between calls to 10 secs. Having seen Kevin Smith's 'Erlang in Practice' I'm sold on the idea of using Erlang as a replacement backend. What I'm trying to figure out [given Damien Katz's comments], is whether I should a) re-write the entire webapp in Erlang, front end and all or b) maintain a split structure, with a Rails frontend / Erlang backend.
I like the idea of using a 100% Erlang stack for the project; I'll need to use some kind of Erlang web framework [Nitrogen ? Erlyweb ?]; I'm concerned they're not mature enough and I'll spend my time bogged down on the web part of the project with them.
Anyone any views ? Thanks.
What's the actual impact on your visitors (of the two-minute interval between HTTP backend calls)?
If there isn't much of a difference, I'd say this sounds like premature optimization and that you'd be much better off skipping Erlang for now.
The two previous posters have pretty much covered they philosophical aspects of your question. So I'll answer the framework maturity/getting bogged down part of your question.
In the event that you decide you do want to rewrite the webapp in erlang for whatever reason then I wouldn't be too concerned about the framework slowing you down. Both erlyweb and nitrogen are already feature complete enough that you can work pretty quickly with them. I've developed a fairly complex agile project management app in nitrogen and found it to be quite intuitive and not really lacking in features that I needed. A few hours in the evenings and a few weeks later and I had a working app up and running.
As to which one to use that depends on the type of app you want to build.
Nitrogen's target is extremely dyamic web applications. Most of the page is rendered using javascript and it is highly event driven.
ErlyWeb is more suited to a site where the content is the primary focus less so than a rich client type of application. It uses the MVC style of architecure.
Good luck on whatever you decide.
It depends. How much Erlang do you know? How much code have you already written?
How much project experience do you have? Is this for work or for fun?
Rewriting projects from scratch is often a recipe for disaster, especially if you are trying
to learn a new language along the way. It seems to me like you would not be asking this question if you were already fluent in both languages, in which case I would recommend that you just stick to Ruby if it's a work project.
I disagree with the above poster that changing the language is a premature optimization, if it is necessary.
Changing the language is a big deal. It can't be done at the last minute.
However, I would probably not change the language at all for the reason you outlined.
If you don't have any other reasons than performance for switching, you should probably just
look at multi-threading in Ruby or some other optimization.
I'm all about using the right tool for the job. Unless you have an absolutely dead on reason to port the front, there's absolutely nothing wrong with hooking the two together.
I have a memory of talking to people who have got so far in using Ruby on Rails and then had to abandon it when they have hit limits, or found it was ultimately too rigid. I forget the details but it may have had to do with using more than one database.
So what I'd like is to know is what features/requirements fall outside of Ruby on Rails, or at least requires such contortions that it is better to use another more flexible framework, even though you may have to lose some elegance or write extra boilerplate code.
Rails (not ruby itself) is proud to be "Opinionated Software".
What this means in practice is that the authors of rails have a certain target audience in mind (themselves basically) and aim rails specifically at that. If X feature isn't needed for that target audience, it doesn't get added.
Off the top of my head, things that rails explicitly doesn't support that people may care about:
Foreign keys in databases
Connections to multiple DB's at once
SOAP web services (since rails 2.0)
Connections to multiple database servers at once
That said, it is very easy to extend rails with plugins, and there are plugins which add all of the above functionality to rails, and a lot more, so I wouldn't really count these as limits.
The only other caveat is that rails is built around the idea of creating CRUD web applications using MVC. If you're trying to do something which is NOT a CRUD web app (like twitter, which is actually a messaging system, or if you are insane and want to use a model like ASP.NET webforms) then you will also encounter problems. In this case you're better off not using rails, as you're essentially trying to build a boat out of bicycle parts.
In all likelihood, the problems you will run into that can't just be fixed with a quick plugin or a day or 2 of coding are all inherent problems with the underlying C Ruby runtime (memory leaks, green threads, crap performance, etc).
Ruby on Rails does not support two-phase commits out of the box, which maybe required if your database-backed application needs to guarantee immediate consistency AND you need to use two or more database schemas.
For many web applications, I would venture that this is not a common use-case. One can perfectly well support eventual consistency with two or more databases. Or one could support immediate consistency with one database schema. The former case is a great problem to have if your app has to support a mondo amount of transactions (note the technical term :). The latter case is more typical, and Rails does just fine.
Frankly, I wouldn't worry about limits to using Ruby on Rails (or any framework) until you hit real scalability problems. Build a killer app first, and then worry about scalability.
CLARIFICATION: I'm thinking of things that Rails would have a hard-time supporting because it might require a fundamental shift in its architecture. I'll be generous and include some things that are part of the gem/plugin ecosystem such as foreign key enforcement or SOAP services.
By two-phase commits, I mean attempting to make two commits to physically distinct servers within one transactional context.
Use case #1 for a two-phase commit: you've clustered your database, so that you have 2 or more database servers and your schema is spread across both servers. You may want to commit to both servers, because you want to allow ActiveRecord think do a "foreign key map" that traverses across the different servers.
Use case #2 for a two-phase commit: you're attempting to implement a messaging solution (sorry, I'm J2EE developer by day). The message producer commits to the messaging broker (one server) and to the database (a different server).
Also found some good discussion about the limits of ActiveRecord.
I think there is a greater “meta-question” here, that could be answered and that is “when is it OK to lean on external libraries to speed up development time?”
Third party libraries are often great and can drastically reduce development time, however there is a major problem, Joel Spolsky calls this “the law of leaky abstractions.” If you look that up on Google his post will come up first. Essentially this means that the trade off in development time means that you have no idea what is going on under the covers. So when something breaks you are completely stuck and have very limited methods of debugging. This also means that if you hit one of the features that are simply unsupported in RAILS, that you really need, you’ll have no next step except to write the feature yourself, if you’re lucky. Many libraries can make this difficult to do.
We’ve been burned badly in my dev shop by this issue. Our solutions worked fine under normal load, but we found that the third party subscription libraries that we were using simply could not stand up to the kind of load that we experienced once our site started to get a large number of concurrent users. This puts us in a very difficult place; essentially we have to rewrite the entire subscription service ourselves, with performance in mind. Doing this means that we’ve wasted all the time that we spent using the library.
Third party libraries can be great for small to medium sized applications; they can drastically reduce development time and hide complexities that aren’t necessary to deal with in the early stages of development. However eventually they will catch up with you and you’ll likely have to rewrite or re-engineer your solution to get past the “law of leaky absctractions”
Ruby don't have a functionality like IsPostBack in ASP.Net
Orion's answer is right on. There are few hard limits to AR/Rails: deploying to Windows, AR connectors that aren't frequently used, e.g. Firebird, ), but even the things he mentioned, multiple databases and DB servers, there are gems and plugins that address those for legacy, sharding, and other reasons.
The real limitation is how time-consuming it is to keep on top of all the things that rails devs are working on, and researching specific issues, given how many blogs, and how much mailing list volume there are.