We are planning to use Azure Storage in a virtual Environment. Azure already offers very high availability, load balancing and replication of data around the world. If we install NEO4j Graph (using Java embedded) in a IIS server with high availability. Do we still need to design and use the NEO4j high availability architecture?
We still need the backup option of NEO4J (enterprise license for sure) but it would be nice not to get in the business of load balancing and replication. Azure Cloud does a very good job already. Thank you!
Best,
George Moon
The short answer is probably yes.
Windows Azure provide you infrastructure that allow you to build high availability system, it won't make any system high available by magic.
As NEO4J is state-full, each node (with only one node Azure don't give you any SLA, you instance will be down) will need to share some state and the way to do it will be dependent on how NEO4J is working. So you will need to rely on NEO4J mechanism to do it.
I don't know how NEO4J is working but you won't be able to skip designing an high available architecture around NEO4J using Windows Azure infra.
Cloud may be a magic buzz word that can make things append on management level, but when we are on hard real world level Harry magic wand doesn't exist.
Related
I have a distributed monitoring system that collects and gathers monitoring data like CPU utilization, database performance metrics, network performance into a backend store. Other applications need to consume these data like real-time calculating(for a resource scheduler) , for system monitoring(to system administrator using monitoring dashboard), for historical analytic(to operation and analyzer program to modeling the resource using pattern for future capacity planning and business system activity analysis).
The dataset size is about 1.2 billion entries in the data store for 9 months. (all in OpenTSDB like format)
Previously I used an Elasticsearch cluster as the backend data store solution and decide to find a better one.
I am looking at Couchbase or VoltDB cluster but still in investigation stage so need some input from here who has the similar experience.
Major questions are as below:
Which backend store solution is good for my scenario? (Couchbase or VoltDB)?
I have to rewrite my data aggregator code (which is in golang). Couchbase provide a good golang SDK client but VoltDB's go driver is only in community level with limited function. So are there any better implementation to communicate with voltdb in golang?
Any suggestion or best practice on it?
There isn't too much in the way of usage patterns here, but it sounds like the kind of app people use VoltDB for.
As for the Golang client, we'd love some feedback as to how to make it more suitable if it's specifically missing something you need. You can also use the HTTP/JSON query interface from any language, including Golang. More info on that here:
http://docs.voltdb.com/UsingVoltDB/ProgLangJson.php
If you would like to leverage your existing model, take a look at Axibase Time-Series Database. It supports both tcollector network and http protocols. Rule engine and visualization are built-in.
The fact that ATSD is based on HBase may be an asset or a liability depending on your prior experience with it :)
URL to tcollector integation: http://axibase.com/products/axibase-time-series-database/writing-data/tcollector/
Disclosure: I work for the company developing ATSD.
I develop a commercial Asp.Net MVC Web Application. The application is standard, runs on IIS Web Server and utilizes SQL Server database. Our business model is such that we deploy our application on-site in our customers’ Intranet or data center. That is, for each such customer (account) we supply the complete setup, usually installed in a dedicated standalone server. Each such account has its own private content, users, configurations and so on.
We wish to expand and offer our service on the WWW (public Internet). After some research, I’ve chosen Microsoft’s Azure cloud platform to host our application. With some minor efforts (mainly teaching the application to work with Azure’s File Storage using blobs) I’ve managed to fully deploy to the cloud, using three cloud services: Web Site, Data base and File Storage.
Please note that I employ the same code base for the two deployment types (Intranet and Cloud), by using different configurations for Debug, Release – Intranet, Staging – Azure, Production – Azure.
However, the application (as is it written now) can serve only a single customer account, while I need our cloud version to serve numerous accounts (hopefully a lot ;) … each with its own private data set.
Question: which of the following strategies should I employ here?
Change the application so it will support multiple accounts. This means changes both in the data model (adding an Account entity in the data layer, bind it to all content types, etc.) and in the business logic.
Create for each Account its own site on the cloud (web site + data base + file storage services). This means deploying several times the same application to different Azure services.
It’s obvious that the amount of development needed for the 1st approach here is very large as is the risk in stability of the system, while the 2nd approach requires much less effort.
However, im not clear about how to manage a set of many identical services (applications) each serving a different customer account. I’ve started looking for some tools to help me here (e.g. Red Gate), and would love to hear of more.
Another question is the cost – is such a solution, using many cloud services instead of only few, more costly that the more standard “one application for all accounts” approach.
thanx,
I would recommend (1). It is a higher short term cost in terms of development effort, but will be better for two reasons long term:
Cheaper. The costs will go up quite a bit by adding more cloud services. I suppose you could pass this cost on to your client though?
It does become harder to manage releases across many clients.
I would say you can either spend your time refactoring existing code you know - or - you can learn how to do dev ops against Azure to manage releases. Probably easier to refactor what you know instead of learn something new.
As a note there are many great SDKs to automate deployments, scaling cloud services, etc.
Found some good reading about this issue:
MSDN: Developing Multi-tenant Applications for the Cloud, 3rd Edition
Provides good comparison between the two approaches:
Single Instance, Multi-tenant (my 1st option)
Multi Instance, Single-tenant (my 2nd option)
As it looks i will go with the Multi-tenant approach. All things considered it looks like it will require less development effort and will need less effort in maintenance. Also, my expertise lies in application development and less in system administration (which is required for truly implementing multi-instance solution).
Other reasons: there's a need to share some of the content between accounts (tenants), and this will be more easily achieved when using a single database. Also, there are future plans for the product which can utilize this solution.
Segregation of data will be done (high level):
File storage (blobs): use a separate container for each tenant (account).
Database: use tenant unique key to associate content to tenant
Cahce: use tenant unique key to generate cache keys for cached data items.
Scalability: it is easier using one instance to simply scale up its capabilities, or even move the Web Site to a dedicated virtual machine. In the future can also enhance the system to a Multi-instance, Multi-tenant structure, creating new separate instances for large tenants.
I want to know what is the difference between these two implementations of neo4j. Of-course names of both techniques is self-explanatory,but still what are the main differences?
What factors should be considered in deciding which technique to use in the project?
Pros and cons.
P.S. Sorry if it is a repeat question but I searched and was not able to find any ques which answers my question.
Because the standalone server is built on the embedded server, the general rule of thumb is that the embedded server is more capable and has (obviously) lower latency. Either can operate in High-Availability mode, allow monitoring, and even accept connections from the neo4j-shell. With the server though, you get more functionality out-of-the-box, like remoting, basic visualization, monitoring interface, etc.
The differences are otherwise the practical ones you'd imagine. Choosing a deployment approach is influenced by two things:
Language - embedded mode requires that you're implementing your application with a JVM compatible language. The server supports any language/framework that can send HTTP requests.
Hardware - sharing physical resources between your application and Neo4j can be demanding. Scaling may argue for a dedicated machine to split out the persistence layer. The server obviously has a remote API to support segmenting your application.
It's otherwise difficult to give guidance without a specific usage scenario. Deploying into an existing Service Oriented Architecture? Probably server. Running on an copier machine? Go embedded. From scratch web application? What's the rest of your stack?
I would like to create a simple ASP MVC3 web applications for learning and have this hosted. I am familiar with ASP MVC but don't yet know much about the Azure cloud. If possible I would like to have these hosted on the cloud but I'm not sure where to start.
Can anyone give me any advice. For example for something very very low usage and making use of a SQL CE database what kind of azure account do I need and approximately how much would it cost.
Thanks in advance for your help.
For ASP.NET MVC 3 the thing to do is "bin deploy" it - Scott Hanselman has a post on doing exactly that on his blog:
http://www.hanselman.com/blog/BINDeployingASPNETMVC3WithRazorToAWindowsServerWithoutMVCInstalled.aspx
I recommend you download the Windows Azure platform training kit and work through the first few labs just to get the feel of Azure.
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=413E88F8-5966-4A83-B309-53B7B77EDF78&displaylang=en
I agree with Nathan and XIII - you have to ditch SQL CE it just doesn't work in the elastic cloud model, look at Azure table storage instead, it's cheap and scalable. (And it's in the first lab)
If you're just playing around then grab the introductory offer on Nathan's link, it's a free start. Unfortunately you can't use it to run something permanently as there aren't enough hours. (Apparently that may change in the new year). If you want to run a very small instance for a month it's about $40, a small is $90.
If you're more serious than that, join Bizspark and get free access to more resources.
You can easily create an ASP.NET MVC project and host it on a single role. While #XIII is correct that you need to have two instances to get the Azure SLA, if you are just playing around a single instance will be fine. You should check out the special offers here: http://www.microsoft.com/windowsazure/offers/ With the Development Accelerator Core you get 750 hours of a small compute instance, which I believe you could swapped for 2x extra small insances for the same cost. (Double check this before you do it!) Additionally, you get 1 million storage transaction and 10gb of storage space. If you are starting with a new Azure app, I would recommend looking at blob storage and table storage rather than a traditional database. Azure Storage is much more scalable and will be much less expensive than SQL Azure for small scale deployments. I would suggest using the Windows Azure Toolkit to get started with table and blob storage.
I would stay away from trying to use SQL CE or any other database running on your Azure roles. That is going to be a lot more headache than it is worth for a simple deployment. If you must have a SQL database, you can get a 1gb SQL Azure instance for $10 per month.
You need at least two instances if you want to have an SLA.
Depending on the load you expect I would assume that a Small instance variant would be enough for you.
making use of a SQL CE database
That would rather become either SQL Azure or table storage instead. SQL Azure costs quite some more money than table/blob storage so try to get more information about what you want to use if costs matter to you.
To calculate the costs I suggest you check out this great tool: http://azureroi.cloudapp.net/.
I just watched the Windows Azure intro video and it left me feeling like it was a front end shell for hosted IIS instances. Can anyone who know more (possibily that was part of the beta) shed on why you would use this vs. EC2.
it seemed easy enough but really didnt give specifics on how it works, why it works or why you would use this vs the traditional solutions out there?
According to the vision (and I can only talk about the vision here since the product isn't really out yet), here's a couple of reasons you might consider Azure over EC2.
Azure includes built-in load balancing abilities. If you want to do that in Amazon, you have to roll your own solution or buy a third-party solution like www.RightScale.com.
Azure-friendly-coded apps can be delivered internally or in Microsoft's cloud. If you write apps that have confidential information like financial data or health care data, not all of your clients will be willing to put their data in the public cloud. In that case, they can deploy your apps internally on Windows. That's sold as a skillset win, because you can go from public to private projects. Don't get me wrong - if you master Amazon EC2 development, then you can deploy your apps internally with Linux virtual servers in your datacenter, but it's not as turnkey. (Hard to describe a tech preview as turnkey when it's not licensed yet, hahaha.)
Having said that, it wasn't clear that the load balancing functionality is included in the box with internal deployments. If you have to do a combination of Azure plus ISA Server, that'll be a tougher deployment and management sell.
AppHarbor is a .NET cloud hosting environment that sits on Amazon EC2. The nice thing is they offer a free plan (much like Heroku does) so you can check it out yourself with very little friction.
My company is using Amazon EC2 now and I am down at the PDC watching the details on Azure unfold. I have not seen anything yet that would convince us to move away from Amazon. Azure definitely looks compelling, but the fact is I can now utilize Windows and SQL server on Amazon with SLAs in place. Ray Ozzie made it clear that Azure will be changing A LOT based on feedback from the developer community. However, Azure has a lot of potential and we'll be watching it closely.
Also, Amazon will be adding load balancing, autoscaling and dashboard features in upcoming updates to the service (see this link: http://aws.amazon.com/contact-us/new-features-for-amazon-ec2/). Never underestimate Amazon as they have a good headstart on Cloud Computing and a big user base helping refine their offerings already. Never underestimate Microsoft either as they have a massive developer community and global reach.
Overall I do not think the cloud services of one company are mutually exclusive from one another. The great thing is that we can leverage all of them if we want to.
Microsoft should offer up the ability to host Linux based servers in their cloud. That would really turn the world upside down!
Well it's more than just web services. It will also allow you to host other types of connected applications. Plus it provides integrated access to other MS software on the cloud; i.e. SharePoint, Exchange, CRM, SQL data sevices, and will allow you to fully customize and extend those offerings in the same way that you would be able to customize and extend them if they were hosted on-premises.
At the Archtect Insight Conference last year they mentioned that they have started to alter core server products to deal with the large scale failover environment which is very interesting to me at least.
Its bunch of stuff that is coming into the Cloud. I think of this as more of Platform in the Cloud.
Sql Server
CRM
MOSS
Exchange
BizTalk
Geneva (identity)
The terms that are mentioned here are "STORE" and "COMPUTE"
For me this get really intersting around the IDEA of a Internet Service Bus.
It is also about moving to the development workflow process too.
OSLO DSLs and Qudrant - Moving to a Model Driven View
Entity Framework - giving developers strong typed model in code at a click of button
ADO Data Services and Data Dynamic Webtemplates using MVC
Then with the Azure Templates and the new "WebRoles" moving to deployment of the applications to the Cloud.
Then for the Admins one click provisioning of servers is awsome.
On the Data Privacy Rules... which is the one big elephant in the room and has been mentioned... Typically there is the often a ruling in each Country about information security.
UK RIPA
US Patriot Act
Are these really conceptully different? And these 2 countries do share information anyway...IMHO (legally they are different, but to a customer both laws give access to customer data its just question of who)
At this point, information on Windows Azure is pretty scarce. I was in the keynote during the announcement, and my best guess at this point is that they're trying to provide a more extensive virtualization environment than simply hosted IIS instances.
At this point, though, I can't say more than that.
We use S3 for storage very successfully and I've always kept an eye on EC2 for Windows and SQL Server support. So now these are available I dug further.
I was pretty worried when I read this:
http://www.brentozar.com/archive/2008/11/bad-storage-performance-on-amazon-ec2-windows-servers/
Perhaps, as we're developing what will hopefully become a very popular website, we should be considering the new data store models - Azure's or Amazon's SimpleDB. Hmmmmm - complete rewrite!
The major difference going forward is that Amazon EC2 is free from today Nov 1, Check this out.
http://www.buzzingup.com/2010/10/amazon-announces-free-cloud-services-for-new-developers/