I have recently been come to for advise on an idea of rewriting an existing site due to massive maintenance problems in their old design.
Basically, the company is considering a complete rewrite of aprox. 90% of their site which is currently written in PHP using an in-house framework.
The company would like to rebuild the backend and some way down the road the front-end as well in order to minimize their maintenance problems and make it easier to bring in new tallent which doesn't need to spend months learning the architecture before they can become affective developers.
We've come up with several possible architectures, some involving rewriting the whole site using an existing scripting web framework such as Cake, Django or RoR and some compiled language frameworks in Java or even .Net.
In addition we have come up with some cross technology solutions - such as a web application built in Django with a Scala backend.
I was wondering what merit would there be to using a single technology stack (such as RoR) as apposed to using a cross between two (such as RoR with Scala, like Twitter now do) and vise versus.
Take into consideration the fact that this company's site is a high traffic site with over 1 million unique visitors a day, which will be transitioned onto the new architecture slowly over a long period (several month to a year)...
Thanks
Generally speaking, I don't think any particular technology stack is better than any other in terms of performance; Facebook runs on PHP and I know first hand that Java and .Net scale well too. Based on what you've said I'd be worrying more about the maintainability related issues than performance and scalability just now.
Generally speaking, I would keep within one well known technology stack if possible:
It'll be easier to find (good) staff for a well known platform / technology stack; there will be more in the market, and rates will not be as expensive as the skills are too rare.
Splitting your technology means you need a wider range of knowledge; by sticking with a single technology stack you can focus on it, with better / faster results.
People tend to focus on one platform / technology stack, so it'll be easier to find developers for technology X, rather than technologies X, Y and Z.
It's easier for team members to work on different parts of the system as it's all written in the same technology - presumably in a similar way.
In terms of integation, items within the same technology stack play nicer together, crossing into different stacks can quickly become more difficult and harder to support.
Where you do want to use different technology, ensure the boundary is clean - something standards based or technology agnostic like web service / JSON calls.
Rewriting your whole codebase will require significant effort and lots of pressure, and for a start you would be best to start by doubling or maybe tripling the initial time estimate.
You can think about your problem from two perspectives :
Number of platforms. In order to minimize and manage complexity of this task, it is most definitely your imperative to reduce mental strain by using as less new technologies/platforms as possible. For example, an advantage of RoR over PHP+Smarty that has been cited often is that with RoR you don't have to learn a new presentation language.
Team effort required to learn new techs. If your existing team is already versatile with PHP, Django etc, but not RoR, then you might be better off reusing existing skills, since the mental strain for developers will be lesser.
Single technology means less moving targets; simpler is always better as long as it meets the requirements. So, use as many technologies as you need, but not more than that. The technology is not important; the right technology is the one that makes your job easier. So, ask yourself what are your current pain points, and how would each of those technologies help.
Getting the architecture right and the code clean is the easiest with Smalltalk and Seaside, especially when you do the persistence with Gemstone. At this scale, you'll have to talk to them about license costs. You might know them from the Ruby work they do with Maglev.
Related
I'm dealing with a rails application which is growing too big. It takes long time to start and lots of memory. We are having performance issues. Tests are very slow. Managing the codebase, debugging and introducing new futures becoming harder everyday. We are thinking to split the application into smaller components (Rails engines or different Rails apps) where all components will share a single database.
Considering such scenario we will need to share the models, maybe some libs, tests, and some gems, etc... It doesn't sound right for me!
Is there any pattern can be applied?
There are some great suggestions here about to to split up a Rails code base however I think that before you apply any of then you need to stop and seriously consider how you got into this position. All of these solutions introduce new complexities and challenges. They might be worthwhile trade offs but only if you make sure they also solve a problem you have today.
Take each of the pain points you listed (startup time is slow, memory use if high, performance is poor, rest performance is poor, development speed is slow) and run a "5 whys" exercise on them. Why are these things happening. Why did the app get into this state.
Most importantly before you commit to any plan for splitting up a large app consider if the app should be large in the first place. If your app is more complex than your product demands then switching to an equally complex cluster of services is not an improvement.
More concretely I would recommend against shared database access between apps/services/whatever. A shared database has a shared schema which becomes fragile. It also leads to tightly coupled services which lack the separation of concerns you need to see any improvement in your pace of development. Just as splitting one massive class into several tightly coupled files does not improve it nor does splitting one app into coupled services.
If you must maintain a large app you need to isolate separate concerns. In order to do so you need to break the dependencies between them. Based on the efforts I have seen to repair Rails monoliths you'll have better success creating clean interface within your existing app an then splitting out components than if you split the app apart and then hope the resulting pieces can be improved independently.
Yes, there are several ways to do it:
Micro services Erin Swenson-Healey has a nice post for rails.
Hexagonal Architecture GoRuCo 2012 Hexagonal Rails by Matt Wynne and Refactoring with Hexagonal Rails
Rails Engines Approach. Dealing with Rails Application Complexity - A Report from MWRC
Use other framework more suitable for high complexity and more PORO oriented. http://lotusrb.org/
Also see Ruby Midwest 2011 - Keynote: Architecture the Lost Years by Robert Martin
Read "Practical Object Oriented Design in Ruby" by Sandi Metz. Also, watch every presentation and talk by her on the internet. Youtube is a great place to start. She's a great speaker. Rails is written in Ruby and Ruby is an object oriented language which brings with it massive benefits. But grokking the object oriented part takes a little work. Sandi will get you there.
We've got an Excel spreadsheet floating around right now (globally) at my company to capture various pieces of information about each countries technology usage. The problem is that it goes out, gets changes, but they're never obvious, and often conflicting - and then we have to smash them together. To me, the workbook is no more than a garbage in/garbage out type application waiting to be written.
In a company that has enough staff and knowledge to dedicate to Enterprise projects, for some reason, agile and language/frameworks such as Rails, Grails, etc. are frowned upon. That said, I can't help but think that this is almost a perfect fit for the need, given the scaffolding features for extremely simple implementations of capturing raw fields with only a couple lookups (i.e. a pre-defined category). I'm thinking this would be considered a very appropriate use of these frameworks.
Has anyone worked on these types of quick and dirty apps before in normally large-scale, heavy-handed enterprise environments with success? Any tips for communicating this need/appropriateness to non-technical management?
The only way to get this implemented in a rigid organization is to get this working and demo it -- without approval. It's very hard for management to say no to a finished project.
I work for a really big company & have written many utility apps based on Rails (as well as contributed to some larger Rails projects). That said, the biggest concern is not the quality of the app, but who's going to support/maintain it when you leave or get hit by the bus.
IMHO, The major fear that an enterprise organization has - especially if the application becomes more critical to it's core business - is how to support it. If it doesn't fit into it's neat little box of supported technologies, it's less likely to happen.
Corporations have been bitten by this many times in the past & are cautious when bringing in new technology.
So, if you can drum up more folks to learn Ruby/Rails in your group (or elsewhere in your company), you may be able to make a good case for it. Otherwise, sad to say, your probably better off implementing something on Sharepoint :-(.
If you already have a Java infrastructure, then creating a Grails app will require little to no additional IT ramp up to support and maintain. The support and maintenance cost and effort should be the same as for a Java application (i.e. Grails apps run on Tomcat, use the same JVM, use the same diagnostic/profiling tools, etc.).
In my experience, larger IT organizations have a harder time supporting Ruby when its not already in the toolchain because its a new language, new deployment environment, and requires a considerable amount of support and maintenance ramp up.
I would develop a minimal viable product, then make friends with someone in IT who can help you deploy it into a staging or production environment. Then get a few of the users to hop on board and test it like its a Beta product. After that, open it up to a larger audience.
So as others have said, forgiveness over permission, but be smart about the impact on the IT organization.
----edit-----
QUICK QUESTION: Does Grails take too much resources for high traffic website, and is it expensier to host?
For example: if I can make a site that has millions of users/m easier in CakePHP does it worth to make it in Grails just to save some webserver resources- or will it need more servers?
---------------
Hello,
I know there are a lot of similar questions on the net, but because I am a newbie in web development I didn't find the solution for my specific problem.
I am planing on creating a flash games portal from scratch. It is a big chance that there will be big traffic from the beginning (millions of pageviews). I want to reduce the server costs as much as possible but in the same time to not be tide to an expensive contract as there is a chance that the project will not be as successfully as I want and in that case the money would be very little.
The question is : what technology to use? I don't know any web dev technology yet so it doesn't matter what I will learn. My web dev experience is a little php 8 years ago, and from then I programmed in C++ / Java- game and mobile development. I like Java and C syntax and language very much and I tend to dislike dynamic typing or non robust scripting (like php)- but I can get along if these are the best
choices.
The candidates are now: -
Grails (my best for now)
Ruby on Rails
Cake PHP
Other technologies (Google App
Engine, Python/Django etc...)
I was considering at first using pure C and compiling the web app in the server- just to squeeze more from the servers, but soon I understand that this is overkill.
Next my eyes came on Ruby - as there is a lot of buzz for it's easiness of use. Next I discovered Grails and looked at Java because it is said that it is "faster". But I don't know what this "Faster" really means on my needs, so here comes the first question:
1) What will be my biggest consumption on the server, other than bandwidth, for a lot of flash content requests? Is it memory? I heard that Java needs a lot of memory, but is faster. Is it CPU? I am planning to take some daily VPS.NET nodes at first, to see if there is a demand, and if the "spike" is permanent to move to a dedicated server (serverloft.com has some good offers), else to remain with less nodes.
I was also considering developing in Google App Engine- cheap or free hosting to use at first - so I can test my assumption- and also very easy to use (no need for sys administration) but the costs became high if used more (> 3 million games played / month .. x mb/ each). And the issue with Google is that it looks me in this technology.
My other concern is scalability (not only for traffic/users, but as adding functionality) My plans are to release a functional site in just 4 weeks (just the basics frontend and some quick basic backend - so I can be able to modify some things and add games manually) - but then to raise it and add more things to it. I am planning to take a little different approach than other portals so I need to write it from scratch (a script will not do).
2) Will Grails take much more resources than RoR or Php server wise? I heard that making it on Java stack will be hardware expensive and is overkill if you don't make a bank application. My application will not be very complex (I hope and i will try to) but will have a lot of traffic.
I also took in account using CDN for files, but the cheapest CDN found was 5c/GB (vps.net) and the cost per gb on serverloft (http://www.serverloft.com/dedizierte-server/server-details.php?products=4) is only 1.79 cents/GB and comes with the other resources either.
I am new to this domain (web). I am learning the ropes and searching on the web for ~half of year but don't have any really practical experience, so I know that I must have some naive thinking and other issues that i don't know from now, so please give me any advice you want regarding anything, not just the specific questions asked.
And thank you so much for such great community!
This is how I (on my blog) view web performance, especially for highly abstracted frameworks like Grails.
I don't understand the obsession with
runtime performance. Given most
project scenarios your primary focus
should be on your performance, as in
your ability to get things done with a
chosen technology.
For example, you will get more done in
a given period of time with Groovy
than with Java any day. Often one line
of Groovy code will equate to 10 lines
of Java code etc etc
Very rarely will byte code execution
time be your performance issue, most
often its...
Bad algorithm implementation or
design. Bad DB design and / or queries.
Taking to long to get things done and
then having all sorts of commercial
relationship issues because of it.
With web applications you are usually
not performing lots of long running
CPU bound operations. Most of your
request / response time is spent in
the wire (internet routing etc) and in
the DB (executing queries).
Choose a technology that takes a load
off your mind and one that frees you
from writing mountains of boiler plate
code, so that you can rather
concentrate on designing and
implementing good algorithms, DB's and
queries etc etc
I want to learn a framework that promotes good programming practices and is respected by the programming community.
However, I also want a framework that I can use for a day job.
Which one would you recommend?
This question comes from my experience of learning the basics of Django because it was highly acclaimed by developers on Stack Overflow and Hacker News. However.. there's hardly any jobs in my area (NYC) that are asking for Django developers.
As a long-time ASP.NET guy, I've recently gone through a similar decision process to figure out what other web frameworks I should try. Here's what I learned so far which may apply to your case too:
framework/platofrm choices (and hence job opportunities) are highly regional-- the Bay Area job market differs alot from what you'll find in NYC, Chicago, Montreal, or London. Look at local job listings (craigslist and indeed are good places to start) to get a good sense at what's in demand.
similarly, usage varies alot based on the size and type of company. if you want to get a job in a large company, Spring MVC and ASP.NET MVC may be your best bets. In small companies, DJango and (especially) Rails seem to be on the rise.
usage also sometimes varies by industry. for example, many HR apps seem be to .NET based, while financial/banking apps seem to favor Java. if you want to work in a particular industry, check out what up-and-coming companies in that industry are using.
when investing your scarce time in learning something new, favor technologies which are on the upswing of the adoption curve (e.g. Rails) rather than frameworks with wider adoption which may not be growing as fast. Also be wary of very early or niche frameworks which may not ever gain wide adoption.
the one common thread between most (or almost all) frameworks gaining in popularity is that they're MVC frameworks and rely heavily on a solid understanding of REST. Learning those concepts in depth is a good idea.
before deciding to invest a lot of time in one framework, gain a basic understanding of several of them, so you can get a reasonable sense of what you like and don't about each-- and so if you end up applying for a job using a framework you haven't learned, at least you'll be able to talk intelligently about it.
If you focus on what you enjoy, you'll be more motivated to learn it. For example, personally I found Rails (regarless of employment opportunities) more interesting than Spring or Django, so I decided to focus on Rails first. Others may have different impressions-- follow your programmer instincts. That said, there are often few jobs using technologies you find fascinating, so try to strike the right balance: technology you like that many companies are actually hiring people to use!
once you answer the basic "what framework" question, there are many more questions lurking, including picking a javascript framework, validation framework, an ORM, etc. Don't worry too much about those choices yet-- when starting, just pick the default implementation for your framework. But as you get more advanced, the same argument about frameworks also hold for those other things-- e.g. it's useful to know a few ORMs.
Personally, I decided on this approach:
continue building stuff in what I knew best (ASP.NET) but transition all work to ASP.NET MVC, where I can better understand MVC and REST concepts which apply cross-platform
learn JQuery (again, platform neutral)
blow off the ORM choice alltogether for now-- too many other things to worry about
build a few projects in Rails, which is the framework I see used most in the newer SF-Bay-Area startups I've been looking at
learn the basics (e.g. read a book or two, try a few samples) about Python/Django, Java/Spring, and Groovy/Grails.
I've encountered real projects at cool, small companies using Django, Ruby on Rails and (eiuw!) even Zope. .NET is for teletubbies - I've only ever heard of it being used by big corporations that don't know better.
I would say that knowing two or three is better than knowing one that is widely used because you will gain a better understanding of how it works as a concept. For instance if you've only used Java, there is something probably missing in your understanding of OOP, because you're pigeon-holed into thinking about it in one way. If you already know Django though you Spring would probably be a good compliment to that.
i'd probably say ASP.NET MVC. I always see lots of .NET jobs around and this seems to be a solid framework which i think in fact powers all the stackoverflow family. As a PHP developer i must also make a mention of Zend Framework which is used by a number of big sites including bbc.co.uk and is now frequently mentioned in advertisements for PHP jobs.
I want to learn a framework that promotes good programming practices and is respected by the programming community. However, I also want a framework that I can use for a day job.
Sorry to be the bearer of bad news here, but those two desires tend to conflict. IMHO most business managers tend to go for (ugly) rapid development on top of CRMs or other higher-level 3rd party codebases. Building elegant websites from the ground up mostly happens in startups, or true web companies where the website is the sole product. There are not that many of those companies; and many of those that seem to fit are actually a mess on the inside, i.e. due to time pressure, messy legacy code and many other reasons you often don't get to write according to "good programming practices" anyway.
I agree with Kaleb Brasee that Java and .NET are the two main platforms when job availability is a priority.
Every job market is unique, so look at job openings in your area, or call a handful of recruiters and ask what they see a need for / could easily place you in a junior position for. What I'm seeing is that Microsoft Sharepoint is in demand, and a few other regional CMS'es are in demand (in Denmark I see Sitecore regularly).
I think ASP.NET MVC 2.0 together with MVC Areas and ASP.NET Dynamic Data will have a good story, a good solution, for many of those bosses who want rapid development. And I think the resulting code could be quite okay, or at least not bad compared to many of the "CMS beaten into something else" sites that exist. But this is a brand new thing for the .NET platform, and it will need to be sold to the decision makers first...
Bottom line: If you want job security first and foremost, then look at large CMS's like Sharepoint, and work on other technologies in your spare time. Optionally you could take a job at a startup / a web company later; but look before you leap.
Have you tried Spring MVC? Many companies do use Java for web-apps (or .NET) and web service based applications.
Since you mentioned Ruby on Rails, you might want to learn Ruby on Rails. It has got some good programming practices in it and a very well thought architecture. The Ruby community itself have also (in my personal opinion) created very innovative frameworks and highly favor testing and quality. You can see this by the innovative testing framework like Cucumber, webrat, shoulda, coulda, rspec, test/spec. Many startups also uses Rails as their platform, so it should be easier for you to get a job. You can start looking at Working With Rails and 37signals job board. So there is a good ecosystem inside Rails and Ruby community.
But the downside of Rails compare to Django is mainly there are too much magic (less explicit) and the docs is not as good as Django. If you want to get a Django job, try looking at several news site because Django grew up from a newspaper site so it is adopted alot in news based sites.
I would recommend ASP.NET MVC, Ruby on Rails, or Python/Django, they all seem to be popular and successful, and based on the MVC paradigm which is definitely the right tool for the job when it comes to the web.
.NET and Java are by far the 2 largest platforms used by employers, and hence the most in-demand when searching for a job. Java has a few popular frameworks, with JSF, Spring MVC and Struts all seeming to be about equal in demand. I don't use .NET, but from what I've seen, ASP.NET and ASP.NET MVC are the major ones.
I would say that most of the frameworks mentioned here promotes good practices. But that doesn't neccesarily mean that the companies using those frameworks are actually following those good practices! In fact most probably aren't. So don't expect too much.
You see, places like Stack Overflow, Hacker News etc. are a great way to connect with people who really care about their craft. Sadly this is a minority. There are millions of programmers in the world. Most of them suck. The code they write sucks. They don't care. They are not interested in improving their skills. They just want to learn the bare minimum required to collect their paycheck, go home, feed the dog, spend some time with the family, watch some TV, go too bed and do it all over again the next day.
Okay that was a bit harsh :) What I'm getting at is that you are probably better off asking this question to some of the managers at the companies where you would like to work. My guess is that most of them will answer .NET or Java. If you are up for a laugh ask them why they chose that particular technology over something else, and see how many buzzwords they throw at you ;)
What language, framework, and hosting considerations should one make before starting development of a scalable web application?
The most important consideration is not to over-engineer to the point that it gets in the way of building and launching something. Analysis paralysis is the single biggest inhibitor to productivity, progress and results.
Yes, do some planning. Pick a framework. Perfection in a framework will be impossible to find because it doesn't exist, partially because you don't know what you need until you build it anyways. Chances are, if you pick something, it will be better than picking nothing.
Yes, try to pick flexible, inter-operable tools for where you see yourself going.
Yes, look for a good built-in feature set where you see yourself going in the next 6-18 Months. Trying to look beyond that is not really realistic anyways as most projects change so much anyways going towards the first release.
So, pick what you're comfortable with or what is familiar. Don't follow the crowd, do what gets you the best results, quickest, and often. Understand that you might have to change in the future. So, whatever you build now, try to use unit testing so you can re-factor if ever needed.
If what you're building is going to be super successful, it will be a great problem to have, and an easy one to work on once it's making money as you'll be able to get other talent to help you.
Share what you end up picking and why for your situation -- it helps the us learn from you too!
Don't necessarily marry yourself to one language or framework. It may be that some parts of your site work better with different languages and frameworks than others. For example, all of 37signals' sites are based on Ruby on Rails, but they recently wrote a blog post about how the underlying technology of one is actually written in Erlang now because it's much easier to do concurrency that way.
Obviously there's a level of complexity where things turn into a mishmash, but using the right tool for the job — even if that means different tools for different jobs — can simplify things.
Firstly on language, it largely doesn't matter. PHP, Java and .Net being probably the biggest three are all proven in the sense that they run some of the largest sites on the Web so don't listen to anyone who tells you one is more suited than any of the others.
Some might also put Ruby and Django/Python in this list. I have nothing against them but I'm not aware of any big (say top 50) sites using either.
Hosting considerations depend on how low you want to start but basically the order is:
Shared;
Virtual Private Server;
Dedicated.
Scalability will largely be about your application's design than any language, framework or provider. Efficient database schema, efficient delivery and use of Javascript/CSS and in-memory caching are all issues common to any language or framework.
Language - I'd recommend something with good frameworks and good testing libraries like Perl or Java.
Framework - it depends on what do you plan to do. If you start with a hosting that does not allow FastCGI, it is best to avoid such frameworks like Catalyst or Rails. That's why I love CGI::Application (primarily Perl, but ported to other languages too) - it can run as CGI, FastCGI or mod_perl. For development it can be run from it's own web server.
Hosting - nothing is better than you own server. It can be your own server, leased server or virtual server. But you can start with cheapest hosting and when you need more, you should be able to afford it.
It depends.
Start by looking at your requirements (Functional or user defined) (Non Functional - aspects that describe your desired system link text)
Next I would clarify what it means to have a scalable web application. Define it as test cases that can be clearly tested (must support X page views / second with response time < Y seconds).
Once I had those pieces in place I would look at what type of skills my development team can support (for the intial project and on going maintenance). Then find some case studies of applications out in the wild that use similar language or framework. If someone else has made a specific language / framework scale then chances are good that you can too.
Finally go out and look for some hosting providers that support your chosen language, framework and requirements.