I am writing a program which will put a lot of documents into a Couchbase store. I'm using F# and I'm interested in the actor pattern as I feel this is a job that could be sped up greatly with actors. What I'm unsure of is how I handle the Couchbase client with this. I want to ensure that as many connections to Couchbase are available as possible but without causing any problems due to too many actors. How do I go about instantiating clients or is there something that can take care of it for me? I couldn't find any guides on the internet about people inserting docs into Couchbase with the actor pattern (doesn't have to be .NET). Does anyone know of any good resources to start me off?
Related
I'm looking for some resources on taking an existing monolithic Rails 3.0 application (35K LOC) and breaking it apart into an SOA design. Any books, blogs, screencasts, or example applications would be awesome.
The main questions I'm looking to answer are:
Is SOA even the right design?
Where do I start?
What are some common pitfalls I can avoid?
What should I be thinking about now vs what can I do later? (ie performance)
Some resources I've seen but not totally sure if they are the right places to start:
Service Oriented Design with Rails
RESTful Web Services
Enterprise Rails (not sure if this book is still relevant)
SOA Design Patterns (maybe too "enterprisey"?)
Is SOA even the right design?
It depends. Don't you hate these kinds of answers?
Breaking up your app to loosely coupled services using messaging or API calls would, by definition, be implementing SOA.
The beauty of it is that you can interchange service implementation without changing their interfaces and allow for independent deployments without having to bring down the whole app. Also, I'd implement the SOA via specialised API controllers that are versioned and expose custom state rather than their whole state which you'd reserve for authenticated users or role-based sessions.
The dilemma, from my experience, is whether to implement synchronous or asynchronous calls. Synchronous calls are obviously easier to program, but may leave your user hanging while they are being executed, and you'd have to handle timeouts for long-running queries. Watch out for database and web server timeouts.
If you implement asynchronous calls, let's say via ActiveMessaging or alike, youd have to handle callbacks or some kind of notifications to bubble up to your user. It also entails setting up primary and secondary message brokers and maybe some JavaScript or pollers to check for status. It's all fun though!
Where do I start?
I'd first see if it's "worth it": after all, SOA is cool, but does introduce multiple points of failure that you do not have currently.
If you think your broken up app will result in discrete services that are HA and will serve other projects, I'd start with "the druby book" and "service oriented" as you mentioned.
What are some common pitfalls I can avoid?
I think the biggest concern would be transactions across multiple services and the ability to roll-back the whole operation if a distant service fails. The problems begins if you are in some operation where you call A and it calls B and B calls C and C fails.
Who knows that C failed? How will you tell B and A to roll back? Can they? Do they save state? Tough questions for upfront design.
One other issue is that life gets complicated when you throw a workflow on top of your SOA: who's the keeper of the business process? Centralised or distributed? It's all absolutely cool stuff again, but heaviness creeps in, no? But that's life if you must move to SOA.
What should I be thinking about now vs what can I do later? (ie performance)
I'd factor out the obvious generic services that can be used in other apps right now. I would not over-SOA your environment to avoid adding points of failure and keep the ones SOA introduces to a minimum.
This is an excellent resource from my friend who is the CTO at Crowdtap. They did the same thing and it has really helped them massively improve the velocity of product development and give them better test coverage. Hope it helps https://www.youtube.com/watch?v=KsiQXAXsQDQ
I am trying to ilustrate the concept of distributed applications using Erlang. My system currently has one server and one ATM. I try to keep it as simple as possible.
For the moment my application runs locally. I am using gen_server for a client-server relationship between the banking server and the ATM. I also have a gen_fsm module to suggest different states my ATM has. In order to store any data i use the dict module (I don't want to make things more complicated using databases). To keep processes alive (the gen_server and gen_fsm) I am using a supervisor process. I've wrapped all modules as an application but for the moment it's all local. Any ideas would be highly appreciated.
I was thinking to start the same application on two different nodes and to illustrate the distributed concept to use some kind of failover/takeover mechanism but I have no ideas on what modules to use.
Is it mandatory to use target systems?(at some point i must do a hot upgrade to the application)
What's the correct order to do these things: first to upgrade and then distribute?
I would be very grateful if someone could give me some ideas on how to accomplish all those things.
I never tried it myself, but docs seem to point to:
Erlang Release Handling (11.3 Distributed Systems).
It's a really short paragraph showing the sync_nodes command, I suggest you to read the whole chapter because I noticed that often the concepts of concurrency and distribution are so persavive in Erlang that problems like yours have already been solved and included in OTP.
BTW, Erlang user guide also has a whole chapter dedicated to Distributed Applications that seems related to distributed applications config options, I think that the two should do the trick.
Hope this helps, if you need more help just ask!
I was wondering if someone had already use this three technologies together. I know Erlang and Comet are widely used, buy I can't find anything related of Comet + Membase or Erlang + Membase. Are them together a bad idea for some reason?
I'm doing a research using Freemind to map the ideas about these three technologies (Erlang, Membase and Comet). As I'm new to the three I am not certain if they are a good combination.
Basically, the application I have in mind will have many clients ("clients A") sending small amounts of data to the server. The server needs to save this data as fast as possible, and send it on request to another set of clients ("clients B"), where clients B are quite fewer than clients A.
This application is just an idea I have had for a while (nothing new, it's been done already), but I would like to experiment with Erlang and Comet and they seem to fit.
If anyone can provide my with some hints I would appreciate it very much. This is my first question on this site, it my be to open. If it's so, please let me know and I will post it somewhere else.
Thank you!
For membase, you just need to use any of several memcached clients. It should work quite well for what you're describing.
Give it a try. Describe what doesn't work for you. :)
I have to confirm the detail of my gradutaion project recently.
My setup a goal for myself, that is it should have values( maybe as a opensource project or tools that can be use by others).
Can you suggest some ideas or projects pertaining to one of :
Web architect, Social Media, Ruby, ROR, Testing.
Thanks!:D
First choose something that both interests you and is in the scope of your abilities.
After you have made such a choice, formalize the decision, perform research and, build requirements; at this stage one can still set "how big a bite they can chew". Most professors I have dealt with are understanding of partial implementations as long as the expectations have been previously established.
Finally, decided on the tools/language and approach for implementation that best fits in the requirement and resources (this includes your time, desired level of effort vs payout, and ability).
I personally find web work absolutely dull, but if I were to write something new, by choice, that was "web-related" and "social" it would be a multi-user interactive whiteboard which is in turn an extension of a real-time collaborative document. (I actually used this as one of my own projects, albeit I focused on a specific protocol implementation.)
i just had this thing a while ago .... and i really needed some help with that same problem ....
i gut a couple of ideas, witch i already used one of them, so i'll suggest the other:
its a network monitoring system based on "SNMP" protocol, gets it's data from the snmp agent on the desired machine (witch can be a computer, a router, a printer, ... ,any thing connected to the network), and alert the administrator (when somthing wrong is there like too many ports are open, or denial of service problem, or too many tcp packets, so it might be a tcp ping problem, ...) with any way u would like to (email, sms, a live ajax warning, ...) ...
sorry .... it sounds messy, but basically it will be like the "CACTI" or the "openNMS" systems (just google them), and it's based on alot of technologies ,thing like: ruby, mysql(to save the actions and to have users DB), linux(i would use Debian), SNMP agents, cron (to schedule the basic system working), SSH/telnet (take a reaction of some harmful action), PHP/RubyonRails to build an web interface that can also connect to your database, ...
i know it sounded like a big fat thing to do, but it's not that hard .... i can provide more things if u want, caus i worked some kind of a specification for this thing.
When I was in college, I used to look into a lot of the programming contests (which involved 3-4 months of projects). Recently came across https://tgmc.in/project_scenario.php. Quite possible that you can get some ideas after reading these project descriptions!
I am considering Erlang as a potential for my upcoming project. I need a "Highly scalable, highly reliable" (duh, what project doesn't?) web server to accept HTTP requests, but not really serve up HTML. We have thousands of distributed clients (other systems, not users) that will be submitting binary data to central cluster of servers for offline processing. Responses would be very short, success, fail, error code, minimal data. We want to use HTTP since it is our best chance of traversing firewalls.
Given this limited information about the project, can you provide any weaknesses that might pop up using a technology like Erlang? For instance, I understand Erlang's text processing capabilities might leave something to be desired.
You comments are appreciated.
Thanks.
This sounds like a perfect candidate for a language like Erlang. The scaling properties of the language are very good, but if you're worried about the data processing abilities, you shouldn't be. It's a very powerful language, with many libraries available for developers. It's an old language, and it's been heavily used/tested in the past, so everything you want to do has probably already been done to some degree.
Make sure you use erlang version R11B5 or newer! Earlier versions of erlang did not provide the ability to timeout tcp sends. This results in stalled or malicious clients being able to execute a DoS attack on your application by refusing to recv data you send them, thus locking up the sending process.
See issue OTP-6684 from R11B5's release notes.
With Erlang the scalability and reliability is there but from your project definition you don't outline what type of text processing you will need.
I think Erlang's main limitation might be finding experienced developers in your area. Do some research on the availability of Erlang architects and coders.
If you are going to teach yourself or have your developers learn it on the job keep in mind that it is a very different way of coding and that while the core documentation is good a lot of people do wish there were more examples. Of course the very active community easily makes up for that.
I understand Erlang's text processing
capabilities might leave something to
be desired.
The starling project already provides basic unicode support and there is a EEP (Erlang Enhancement Proposal) currently in draft, but going in to bring it into the mainstream of Erlang/OTP support.
I encountered some problems with Redis read performance from Erlang. Here is my question. I tend to think the reason is Erlang-written module, which has troubles while processing tons of strings during communication with Redis.