I have a distributed monitoring system that collects and gathers monitoring data like CPU utilization, database performance metrics, network performance into a backend store. Other applications need to consume these data like real-time calculating(for a resource scheduler) , for system monitoring(to system administrator using monitoring dashboard), for historical analytic(to operation and analyzer program to modeling the resource using pattern for future capacity planning and business system activity analysis).
The dataset size is about 1.2 billion entries in the data store for 9 months. (all in OpenTSDB like format)
Previously I used an Elasticsearch cluster as the backend data store solution and decide to find a better one.
I am looking at Couchbase or VoltDB cluster but still in investigation stage so need some input from here who has the similar experience.
Major questions are as below:
Which backend store solution is good for my scenario? (Couchbase or VoltDB)?
I have to rewrite my data aggregator code (which is in golang). Couchbase provide a good golang SDK client but VoltDB's go driver is only in community level with limited function. So are there any better implementation to communicate with voltdb in golang?
Any suggestion or best practice on it?
There isn't too much in the way of usage patterns here, but it sounds like the kind of app people use VoltDB for.
As for the Golang client, we'd love some feedback as to how to make it more suitable if it's specifically missing something you need. You can also use the HTTP/JSON query interface from any language, including Golang. More info on that here:
http://docs.voltdb.com/UsingVoltDB/ProgLangJson.php
If you would like to leverage your existing model, take a look at Axibase Time-Series Database. It supports both tcollector network and http protocols. Rule engine and visualization are built-in.
The fact that ATSD is based on HBase may be an asset or a liability depending on your prior experience with it :)
URL to tcollector integation: http://axibase.com/products/axibase-time-series-database/writing-data/tcollector/
Disclosure: I work for the company developing ATSD.
I have searched so much on net but could not find a good document or site on mnesia database dimesioning.Any pointers?
The use of Mnesia in ejabberd is intended for easy to deploy corporate messaging service. If you ask question about volume, it means you should plug another of the available ejabberd backend.
We are planning to use Azure Storage in a virtual Environment. Azure already offers very high availability, load balancing and replication of data around the world. If we install NEO4j Graph (using Java embedded) in a IIS server with high availability. Do we still need to design and use the NEO4j high availability architecture?
We still need the backup option of NEO4J (enterprise license for sure) but it would be nice not to get in the business of load balancing and replication. Azure Cloud does a very good job already. Thank you!
Best,
George Moon
The short answer is probably yes.
Windows Azure provide you infrastructure that allow you to build high availability system, it won't make any system high available by magic.
As NEO4J is state-full, each node (with only one node Azure don't give you any SLA, you instance will be down) will need to share some state and the way to do it will be dependent on how NEO4J is working. So you will need to rely on NEO4J mechanism to do it.
I don't know how NEO4J is working but you won't be able to skip designing an high available architecture around NEO4J using Windows Azure infra.
Cloud may be a magic buzz word that can make things append on management level, but when we are on hard real world level Harry magic wand doesn't exist.
I'm looking at using CouchDB as a backend of an ASP.NET web API/web app which uses JSON heavily.
I'm interested in the pros and cons in terms of:
replication
platform dependency (currently all our servers are Windows 2003)
ease of use (are there .NET libraries around?)
The application is not very transaction-intensive, probably about 100-200 CRUDs an hour.
There is a little comparism of CouchDB with other NoSQL databases. Here you'll find a practical experience report comparing MongoDB and CouchDB. And you might also have a look into this article from a web developer perspective. (Maybe they are all a little outdated.)
You might be interested in couchbrowse (SharpCouch) and Ottoman, too.
Replication :Its really easy to configure the replication and mark for replication in couchDB.
Platform dependency : I tried setup-couchdb-1.2.0_otp_R15B.exe on Windows 2003 x86 SP2 and succeeded.
Library : You can try LoveSeat/DreamSeat .Net client
I just watched the Windows Azure intro video and it left me feeling like it was a front end shell for hosted IIS instances. Can anyone who know more (possibily that was part of the beta) shed on why you would use this vs. EC2.
it seemed easy enough but really didnt give specifics on how it works, why it works or why you would use this vs the traditional solutions out there?
According to the vision (and I can only talk about the vision here since the product isn't really out yet), here's a couple of reasons you might consider Azure over EC2.
Azure includes built-in load balancing abilities. If you want to do that in Amazon, you have to roll your own solution or buy a third-party solution like www.RightScale.com.
Azure-friendly-coded apps can be delivered internally or in Microsoft's cloud. If you write apps that have confidential information like financial data or health care data, not all of your clients will be willing to put their data in the public cloud. In that case, they can deploy your apps internally on Windows. That's sold as a skillset win, because you can go from public to private projects. Don't get me wrong - if you master Amazon EC2 development, then you can deploy your apps internally with Linux virtual servers in your datacenter, but it's not as turnkey. (Hard to describe a tech preview as turnkey when it's not licensed yet, hahaha.)
Having said that, it wasn't clear that the load balancing functionality is included in the box with internal deployments. If you have to do a combination of Azure plus ISA Server, that'll be a tougher deployment and management sell.
AppHarbor is a .NET cloud hosting environment that sits on Amazon EC2. The nice thing is they offer a free plan (much like Heroku does) so you can check it out yourself with very little friction.
My company is using Amazon EC2 now and I am down at the PDC watching the details on Azure unfold. I have not seen anything yet that would convince us to move away from Amazon. Azure definitely looks compelling, but the fact is I can now utilize Windows and SQL server on Amazon with SLAs in place. Ray Ozzie made it clear that Azure will be changing A LOT based on feedback from the developer community. However, Azure has a lot of potential and we'll be watching it closely.
Also, Amazon will be adding load balancing, autoscaling and dashboard features in upcoming updates to the service (see this link: http://aws.amazon.com/contact-us/new-features-for-amazon-ec2/). Never underestimate Amazon as they have a good headstart on Cloud Computing and a big user base helping refine their offerings already. Never underestimate Microsoft either as they have a massive developer community and global reach.
Overall I do not think the cloud services of one company are mutually exclusive from one another. The great thing is that we can leverage all of them if we want to.
Microsoft should offer up the ability to host Linux based servers in their cloud. That would really turn the world upside down!
Well it's more than just web services. It will also allow you to host other types of connected applications. Plus it provides integrated access to other MS software on the cloud; i.e. SharePoint, Exchange, CRM, SQL data sevices, and will allow you to fully customize and extend those offerings in the same way that you would be able to customize and extend them if they were hosted on-premises.
At the Archtect Insight Conference last year they mentioned that they have started to alter core server products to deal with the large scale failover environment which is very interesting to me at least.
Its bunch of stuff that is coming into the Cloud. I think of this as more of Platform in the Cloud.
Sql Server
CRM
MOSS
Exchange
BizTalk
Geneva (identity)
The terms that are mentioned here are "STORE" and "COMPUTE"
For me this get really intersting around the IDEA of a Internet Service Bus.
It is also about moving to the development workflow process too.
OSLO DSLs and Qudrant - Moving to a Model Driven View
Entity Framework - giving developers strong typed model in code at a click of button
ADO Data Services and Data Dynamic Webtemplates using MVC
Then with the Azure Templates and the new "WebRoles" moving to deployment of the applications to the Cloud.
Then for the Admins one click provisioning of servers is awsome.
On the Data Privacy Rules... which is the one big elephant in the room and has been mentioned... Typically there is the often a ruling in each Country about information security.
UK RIPA
US Patriot Act
Are these really conceptully different? And these 2 countries do share information anyway...IMHO (legally they are different, but to a customer both laws give access to customer data its just question of who)
At this point, information on Windows Azure is pretty scarce. I was in the keynote during the announcement, and my best guess at this point is that they're trying to provide a more extensive virtualization environment than simply hosted IIS instances.
At this point, though, I can't say more than that.
We use S3 for storage very successfully and I've always kept an eye on EC2 for Windows and SQL Server support. So now these are available I dug further.
I was pretty worried when I read this:
http://www.brentozar.com/archive/2008/11/bad-storage-performance-on-amazon-ec2-windows-servers/
Perhaps, as we're developing what will hopefully become a very popular website, we should be considering the new data store models - Azure's or Amazon's SimpleDB. Hmmmmm - complete rewrite!
The major difference going forward is that Amazon EC2 is free from today Nov 1, Check this out.
http://www.buzzingup.com/2010/10/amazon-announces-free-cloud-services-for-new-developers/