I am working on an application which has multi-level caching. So its guavacache->redis->database to fetch anything without directly going to database. But I read somewhere that having multi level caching might always not be a good idea. I am totally confused now. As in my understanding this will decrease the throughput on the database. Can someone help me in understanding this?
PS: I am new to this platform, so please excuse for this vague way of asking question.
Related
I'm releasing my first large app soon, and I've been thinking of writing a small app to manage the databases for my own app with master privileges, to clean up data and the like. This will just be a dev project shared with a few people within the company.
Is this something that people do?
Has anyone had any experience with this?
It feels like a good idea, but I'd also really love feedback.
I'm assuming when you say database you are not referring to coredata specifically but rather some remote database. There is certainly nothing wrong with you creating tools to help you do your job.
I think really this answer is going to come down to time management. If your database has complicated relationships,etc that a simple database editor is not going to be able to maintain or will be highly prone to human error mistakes, I'd say go for it. But, if you are just making trivial changes it might be a more efficient use of time to use a existing solution.
I am looking to start developing a relatively simple web application that will pull data from various sources and normalizing it. A user can also enter the data directly into the site. I anticipate hitting scale, if successful. Is it worth putting in the time now to use scalable or distributed technologies or just start with a LAMP stack? Framework or not? Any thoughts, suggestions, or comments would help.
Disregard my vague description of the idea, I'd love to share once I get further along.
Later. I can't remember who said it (might have been SO's Jeff Atwood) but it rings true: your first problem is getting other people to care about your work. Worry about scale when they do.
Definitely go with a well structured framework for your own sanity though. Even if it doesn't end up with thousands of users, you'll want to add features as time goes on. Maintaining an expanding codebase without good structure quickly becomes fairly horrible (been there, done that, lost the client).
btw, if you're tempted to write your own framework, be aware that it is a lot of work. My company has an in-house one we're quite proud of, but it's taken 3-4 years to mature.
Is it worth putting in the time now to use scalable or distributed technologies or just start with a LAMP stack?
A LAMP stack is scalable. Apache provides many, many alternatives.
Framework or not?
Always use the highest-powered framework you can find. Write as little code as possible. Get something in front of people as soon as you can.
Focus on what's important: Get something to work.
If you don't have something that works, scalability doesn't matter, does it?
Then read up on optimization. http://c2.com/cgi/wiki?RulesOfOptimization is very helpful.
Rule 1. Don't.
Rule 2. Don't yet.
Rule 3. Profile before Optimizing.
Until you have a working application, you don't know what -- specific -- thing limits your scalability.
Don't assume. Measure.
That means build something that people actually use. Scale comes later.
Absolutely do it later. Scaling pains is a good problem to have, it means people like your project enough to stress the hardware it's running on.
The last company I worked at started fairly small with PHP and the very very first versions of CakePHP that came out (when it was still in beta). Some of the code was dirty, the admin tool was a mess (code-wise), and sure it could have been done better from the start. But do you know what? They got it out the door before their competitors did, and became extremely successful.
When I came on board they were starting to hit the limits of their current potential scalability, and that is when they decided to start looking at CDN's, lighttpd caching techniques, and other ways to clean up the code and make things run smoother when under heavy load. I don't work for them anymore but it was a good experience in growing an architecture beyond what it was originally scoped at.
I can tell you right now if they had tried to do the scalability and optimizations before selling content and getting a website live - they would never have grown to the size they are now. The company is www.beatport.com if you're interested in who I'm talking about (To re-iterate, I'm not trying to advertise them as I am no longer affiliated with them, but it stands as a good case study and it's easier for people to understand what I'm talking about when they see their website).
Personally, after working with Ruby and Rails (and understanding the separation!) for a couple of years, and having experience with PHP at Beatport - I can confidently say that I never want to work with PHP code again =p
Funny to ask "scale now or later?" and label it "ruby on rails".
Actually, Ruby on Rails was created by David Heinemeier Hansson, who has a whole chapter in his book labeled "Scale later" :))
http://gettingreal.37signals.com/ch04_Scale_Later.php
I agree with the earlier respondents -- make it useful, make it work and get people motivated to use it first. I also agree that you should pick off-the shelf components (of which there are many) rather than roll your own, as much as possible. At the same time, make sure that you choose components for your infrastructure that you know to be scalable so that you can go there when you need to, without having to re-write major chunks of your application.
As the Product Manager for Berkeley DB, I've seen countess cases of developers who decided "Oh, we'll just write that to a flat file" or "I can write my own simple B-tree function" or "Database XYZ is 'good enough', I don't have to worry about concurrency or scalability until later". The problem with that approach is that a) you're re-inventing the wheel (and forgoing what others have learned the hard way already) and b) you're ignoring the fact that you'll have to deal with scalability at some point and going with a 'good enough' solution.
Good luck in your implementation.
For a new web project like kijiji, if I want a faster development(the faster the better), which one should I go with? I know some ruby and javascript, but not an elite and never did a project on either of them. Coding is my hobby, not my job. Any suggestion? Thanks in advance.
Express is more along the lines of Sinatra, not Rails. It's much smaller, simpler and to the point than Rails. I don't think it's a fair comparison.
But what the heck, here are my thoughts while I'm here:
The Node community is fast growing, but it's still nowhere near the size or momentum of the Rails community. Overall you'll probably find more help and more kindred spirits with Rails. Node and Express will require more figuring out things on your own.
In my experience, the Rails/Ruby community have much better tools for testing than Node/Express. Hell, they have the best tools for this out of any community I've seen :)
Express is not well documented at all. The guide on their page is a decent start but it's just that. You will find yourself digging through code, obscure blogs and github repos looking for info.
Data access? It's built right into Rails and a core part of it. Express isn't concerned about your data layer at all (again it's more like Sinatra). You will need to decide on what kind of data access/layer you want to go with and see what kind of support Node has for it.
I really like both Rails and Express. But if the project was of any decent size, I'd probably pick Rails myself.
They are both vastly different so it depends on your goals. I suggest Rails as it's more mature and you'll get better community support if you have any issues. Rails's Active Support Core Extensions will also give you a ton of useful helpers which could save you a lot of time developing a web app.
Otherwise just go for the one that interests you the most.
I have learned the language itself as well as html/css/javascript. But with no database knowledge and (a bit)network knowledge. When I read the Pragmatic book on ROR, I found it confusing at the very beginning where they creates a project and setting up databases and models. Should I learn some database(and network knowledge)knowledge first in order to fully understanding the code and to fully grasp rails? If so, points out some book suited for these subject. Thanks in advance.
sorry for my bad english..
Unless you're doing network specific stuff, all you need to know concerning networking is a basic understanding of how the HTTP protocol works.
Models have nothing to do with databases per se, they're just regular classes in your language, that happen to map to a database table in Object Relational Mapping (ORM) type frameworks such as Rails.
You can view them as the gateway to your database.
The database you'll be using is most likely a relational database. You don't need to know much about the theory, but you can get a very basic overview on wikipedia.
Probably the thing you'd need the most is a basic understanding of SQL, although the point of an ORM is to abstract that SQL away. You'll see glimpse of it when you look at the log file where you see the actual database queries. And when you get to more complicated, specific stuff, you might need some SQL.
In any case, you'll need general knowledge about databases and SQL in pretty much any type of development.
The short answer is YES. Even though ActiveRecord abstracts away much of the dirty work with the database, it is still important to understand what is going on. It will be useful, sometimes vital, when debugging, deploying, and/or maintaining.
You almost certainly will need some general understanding of SQL to work with RoR (and as others have said, probably for programming itself).
I'm a fan of the book Simple SQL. It covers SQL generically, as much as it can. That is, it goes over the basics of SQL itself rather than the details of one or the other SQL implementation. Also, the majority of the examples are web related which should fit you well.
I'd agree that a basic (at least) understanding of SQL is essential when doing any sort of database work. I found the "Sams Teach Yourself SQL in 24 Hours" book helpful when i first started dipping into database work. It covers pretty much everything you need to get you started.
My advice would be to pick a DB (MySQL, Sqlite, MsSQL etc.), learn some of the basics for that particular DB (i found it useful to know how to create new users and set permissions), learn some generic SQL and really get to know what ActiveRecord can and can't do.
Hope this helps a little.
What is really important is to understand the concept of MVC (model view controller)
I recommend you reading this getting started guide
RailsGuides in general are really helpful and more condensed than the book. You might prefer it
(source: rubyonrails.org)
When writing code for a new system I don't want to introduce unnecessary complexity in the design that I might never have any need for. So I'm following YAGNI here, and rather refactoring as I see the need for more flexibility or as responsibilities becomes more clear. This allows me to move faster.
But there is a problem here with junior devs, in that they will not recognize when to refactor or where build out the design. They just stuff more code into the existing design.
So, what are the best ways to tackle this? Should I more often build a more future-proof design so when adding to it they have a good example to follow, even if we might never have to add anything? Or should I just go ahead with more code reviews, education, etc? Or both?
Have any of you had any experience with this type of problem? How did you solve it?
I would recommend code reviews or pair programming. It gives you the chance to educate your fellow developers and increase the overall quality.
Perhaps you begin by recognizing explicitly that part of your job is to help develop the junior devs. If you're not the boss, management should sign off on this. Management needs to recognize that your choices are to develop them now or clean up after them later, and you need management's backing for the time this will take.
Code reviews and pair programming are fine ideas. They are especially good because they are not "just for junior people"–I do both with one of my close colleagues; together we are nearly 100 years old and have more than 70 years of programming experience :-)
But there's a larger problem here: the programming methodology that enables you to be most effective (YAGNI + refactor) is not effective for your junior partners. My experience is that it takes people years to learn the benefits of YAGNI, so if you expect them just to learn your way of doing things, you are setting yourself up for disappointment.
I would encourage you to identify some methodology that you think is going to be useful with your junior partners. The particular methodology probably doesn't matter (heresy!); I've had success with composite/structured design, object-based design, algebraic specification (!), and extreme programming. But
Do pick something that has a name and some literature devoted to it, that your juniors can take pride in learning, and that is a skill they can carry to future projects.
In order to show that it is tasty, you may need to eat the dog food yourself. Pick something you can live with and be productive in.
Observe your juniors carefully and teach them a decision procedure they can use to identify when they should ask you for guidance.
Good luck!
There is a reason they are junior and you are senior.
The ability to realise when a change in design is needed is one of them.
I would carry on as you are but encourage them to come to you when things are getting difficult. You can then work with them to alter the design if needed, this will be easier for you than refactoring it and will help you pass on knowledge to your junior developers.
A very good way to show how far to build out a design is to be specify about what the design will do when built out, then write tests to cover the new functionality. When the tests pass, development is done.
You might realize along the way that you forgot to test for something. That's fine, and is useful feedback to help you specify better next time. Write the missing test(s), and only enough code to make them pass.
Then refactor. Knowing what to look for when refactoring takes a bit of practice. Start with
Is there duplication in the code we've just written that we can eliminate?
Is there duplication between what we've just written and pre-existing code?
Does the code we've just written concern itself with too many things? (I.e., should we break out collaborators?)
Repeat this a few dozen times, and it'll get easier.
Another way of looking at YAGNI is that any changes to code need to be justified.
Might it be helpful to require any commit needs an associated unit test (or BDD user story, choose your poison)? It helps to communicate your intent (you want people to think about why they are adding this feature) and you get regression tests for free.
Also gets the noobs to start thinking about modularity (usually needed to make your code testable) and will help a lot if you do need to refactor later on.
I'm all for code reviews and teaching, but I think futureproof design is also important. Maybe you could think of it in terms of you designing an API and the junior developers using the API. In this way you are the one who does the hard work that they would screw up (identifying duplicated code and eliminating it) while they do all the grunt work that isn't a productive use of your time.
Of course this has to be balanced with a need to develop your junior developers skills. Neither side of the equation can be neglected.
It may help to map out what work they will do and then verify it to help build their sense of judgement which is really what you are asking, to my mind. Pairing is one option but if you can't spare that much time then having a sort of "check point" to see how they are doing and preventing them from going down the wrong path.