I am working on a project which has a complex user management layer with multiple user group levels, team and user heirarchies. So I was thinking about using neo4j. But there is other kind of data also which need to be stored. I'm skeptical about using two separate databases for user management and nonuser data storage.. Is it a good idea to use neo4j itself for both ? The non user data I'm going to store is nosql type data ( json data mostly)
The phrase "general NoSQL data" is essentially meaningless, since there are so many different types of non-relational DBs -- each with its own strengths and weaknesses.
You need to look at the characteristics of your data and your use cases to see whether any particular DB technology is appropriate.
You can store anything you like in neo4j... depends on the size of your data, for smaller datasets you can store everything in neo4j, for bigger one the trend right now ( I think) is to store meta-data in neo4j and jsons(raw data) in mongoDB
Related
I am new to DW . When we should use the term Datamart and when we should use the term Datawarehousing . Please explain with example may be your own example or in terms of Adventureworks .
I'm don't work on MS SQL Server. But here's a generic example with a business use case.
Let me add another term to this. First off, there is a main transactional database which interacts with your application (assuming you have an application to interact with, obviously). The data gets written into the Master database (hopefully, you are using Master-Slave replication), and simultaneously gets copied into the salve. According to the business and reporting requirements, cleansing and ETL is performed on the application data and data is aggregated and stored in a denormalized form to improve reporting performance and reduce the number of joins. Complex pre-calculated data is readily available to the business user for reporting and analysis purposes. This is a dimensional database - which is a denormalized form of the main transactional database (most probably in 3NF).
But, as you may know, all businesses have different supporting systems which also bring in data in the form of spreadsheets, csvs and flatfiles. This data is usually for a single domain, such as, call center, collections so on and so forth. We can call every such separate domain data as data mart. The data from different domains is also operated upon by an ETL tool and is denormalized in its own fashion. When we combine all the datamarts and dimensional databases for solving reporting and analysis problem for business, we get a data warehouse.
Assume that you have a major application, running on a website - which is your main business. You have all primary consumer interaction on that website. That will give you your primary dimensional database. For consumer support, you may have a separate solution, such as Avaya or Genesys implemented in your company - they will provide you the data on the same (or probably different server). You prepare ETLs to load that data onto your own server. You call the resultant data as data marts. And you combine all of these things to get a data warehouse. I know, I am being repetitive but that is on purpose.
I am required to develop a big application,required to know graph database concepts the link http://sparsity-technologies.com/UserManual/API.html#transactions.I am planning to use core data instead of above link frame work. I want answerers for the following questions.
1)What is Graph Database exactly?.Explain with simple general example.which we can not perform with sqlite.
2)Does core data come under relational data base or not ? Explain.
3)Does core data come under Graph Database? But in apple documentation they mentioned that core data is for object graph management.object graph management means Graph Database .If i want to make relation ships ,weighted edge between objects core data is suitable?.
1)What is Graph Database exactly?.Explain with simple general
example.which we can not perform with sqlite.
Well, since this is all Turing complete, you can do it any database operation with any other database, the real question is a matter of efficiency.
In conventional "relational" databases the "relationships" are nothing but pointers to entries in other tables. They don't inherently communicate any information other than, "A is connected to B" To capture and structure anything more complex than that, you have to build a lot of pseudo-structure.
A1-->B1 // e.g. first-name, last-name
Which is fine but the relationship doesn't necessarily have a reciprocal, nor does the data in each table cell have to be names. To make the relationship always make sense, you've got build a lot of logic to put the data into the tables directly. Ditto for getting it out.
In a GraphDB you have "nodes" and "relationships". Nodes are not entries in a table. They can be arbitrarily complex objects, persisted or not, and persisted in a variety of ways. Nodes general model some "real-world" object like a person.
"Relationships" GraphDBs, owing to the previous meaning in SQL et al, really need another term because instead of be simple pointers, they to can be arbitrarily complex objects. In a node of names (way to simple to actually justify it)
Node-Name-A--(comes before)-->Node-Name-B
Node-Name-B--(comes after)-->Node-Name-B
In a sqlite, to find first and last names you query both tables. In a Graph, you grab one of the nodes and follow its relationship to other node.
(Come to think of it, graph theory in math started out as a way to model bridges of Konigsberg connecting the islands that made up the city. So maybe a transportation map would be a better example)
If cities are nodes, the roads are relationships. The road objects/descriptors would just connect the two but would contain their own logic and data such as their direction, length, conditions, traffic, suseptiblity to weather, and so on.
When you wanted to fetch and optimum route between widely separated cities, nodes for any particular time, traffic weather etc between two different nodes, you'd start with the node representing the start city and the follow the relationship/road-descriptors. In a complex model, any two nearby city-nodes might have several roads connecting them each best in certain circumstances.
All you have to do computationally though is compare the relationships between any to nodes. This is called "walking the graph" The huge benefit is that no matter how big the overall DB is, you only have to process the relationships coming out of the first node, say 3, and ignore utterly the the millions of other nodes and relationships that might be in the DB.
Can't do that in sqlite. The more data, the more "relationships" the more you have to process
2)Does core data come under relational data base or not ? Explain.
No, but if you hum a few bars you can fake it. By default, Core Data is an Object graph, which means it does connect object/nodes, but the relationships are themselves not objects but are instead defined by information contained in the class for each Object. E.g. you could have a Core Data of the usual Company, manager and employee.
CompanyClass
set_of_manager_objects
min_managers==1, max_managers==undefined
delete_Company_Object_delete_all_manager_objects
reciprocal_relationship_from_manager_is_company
ManagerClass
one company object
min_companies==1, max_companies==1
delete_manager_object_nullify (remove from set in company class)
recipocal_relationship_from_company_is_manager
So, Core Data a kind of "missing link" in the evolution of GraphDBs. I has relationships but they're not objects of themselves. They're inside the object/node. The relationship properties are hard coded into the classes themselves and just a few, but not all values can be changed. Still, Core Data does have the advantage of walking the graph. To find the Employees of one manager at one company. You just start at the company object, go through a small set of managers to find the right one, then walk down to the employee set. Even if you had hundreds of companies, thousands of managers and tens of thousands of employees. You can find one employee out of tens of thousands with a couple of hops.
But you can fake a GraphDB by creating relationship objects and putting them between any two object/nodes. Because Core Data allows any subclasses of relationship definition to be in the same relationship set e.g. ManagerClass--> LowManager,MidManager,HighManager, you can define a simple relationship in any given class and then populate with objects of arbitrary complexity as long as they are subclasses. These are usually termed "linking classes" or "linking relationships"
The normal pattern is to have the linking class have a relationship to the two or more classes it might have to link (which can be generic as well, I've started class trees with a base class with nothing but relationship properties, although their is a performance penalty if you get huge.)
If you give each node/object several relationships all defined on separate base linking classes, you can link the same nodes together in multiple ways.
3)Does core data come under Graph Database?
No, because the fundamental task of a database is persistence, saving the data. The fundamental task of Core Data is modeling the logic of the data inside the app.
Two different things. For example, when I start building a Core Data model, I start with an in-memory store, usually with test. The model graph is built from scratch every run, in memory, never touches the disk. As it progresses, I will shift to an XML store on disk, so I can examine it if necessary. The XML and binary stores are written out once entire and read in the same way. Only, at the end do I change the store to MySQL or something custom.
In a GraphDB, the nodes, relationships and the general graph are tied to the persistence systems innately AFAK and can't be altered. When you walk the graph, you walk the persistence, every time (except for caching.)
The usual question people ask is when to use Core Data and when to use SQL in the Apple Ecosystem.
The answer is pretty simple:
Core Data handles complexity inside the running app. The more complex the data model interactions, the more you get free with Core Data.
SQL derived solutions handle volumes of simple data. If the data model inside the app has little or no logic and there's a lot of it.
If your app is displaying something that would fit on a bunch of index cards, library book records, baseball cards etc, the an SQL solution is best because of the logic is just getting particular cards in and out of persistence.
If your app is complex vector drawing app, where every document will be different and of arbitrary complexity, or you're modeling an V8 engine, then most of the logic in the active data model while the app is running while persistence is trivial, then Core Data is the better choice.
Graph Databases are catching on because our data is getting 1) really, really big and 2) increasing complex. We need to model the complexity in the node-relationship graph in persistence so we don't have chew through the entire DB to find the data and then have to add an additional layer of logic
Core data is nothing but Data Model Layer, core data is NOT a datatbase and far away from being a graph database.
Core data only helps you to
Create Tables (Entities)
Columns in a table (Attribute)
Relationship (such as primary key, foreign key, one to one, one to many)
Core Data uses sqlite to store data and make queries.
Core Data is used in iOS mobile apps, I believe what you want is a backend solution for database.
I realize this may be common sense for a lot of people, so apologies if this seems like a stupid question.
I am trying to learn core data for iOS programming, and I have repeatedly read and heard it said that Core Data (CD) is not a relational database. But very little else is said about this, or why exactly it is important to know beyond an academic sense. I mean functionally at least, it seems you can use CD as though it were a database for most things - storing and fetching data, runnings queries etc. From my very rudimentary understanding of it, I don't really see how it differs from a database.
I am not questioning the fact that the distinction is important. I believe that a lot of smart people would not be wasting their time on this point if it weren't useful to understand. But I would like someone please to explain - ideally with examples - how CD not being a relational database affects how we use it? Or perhaps, if I were not told that CD isn't a relational database, how would this adversely impact my performance as an Objective-C/Swift programmer?
Are there things that one might try to do incorrectly if they treated CD as a relational database? Or, are there things which a relational database cannot do or does less well that CD is designed to do?
Thank you all for your collective wisdom.
People stress the "not a relational database" angle because people with some database experience are prone to specific errors with Core Data that result from trying to apply their experience too directly. Some examples:
Creating entities that are essentially SQL junction tables. This is almost never necessary and usually makes things more complex and error prone. Core Data supports many-to-many relationships directly.
Creating a unique ID field in an entity because they think they need one to ensure uniqueness and to create relationships. Sometimes creating custom unique IDs is useful, usually not.
Setting up relationships between objects based on these unique IDs instead of using Core Data relationships-- i.e. saving the unique ID of a related object instead of using ObjC/Swift semantics to relate the objects.
Core Data can and often does serve as a database, but thinking of it in terms of other relational databases is a good way to screw it up.
Core Data is a technology with many powerful features and tools such as:
Change tracking (undo/redo)
Faulting (not having to load entire objects which can save memory)
Persistence
The list goes on..
The persistence part of Core Data is backed by SQLite, which is a relational database.
One of the reasons I think people stress that Core Data is not a relational database is because is it so much more than just persistence, and can be taken advantage of without using persistence at all.
By treating Core Data as a relational database, I assume you mean that relationships between objects are mapped by ids, i.e. a Customer has a customerId and a product has a productId.
This would certainly be incorrect because Core Data let's you define powerful relationships between object models that make things easy to manage.
For example, if you want to have your customer have multiple products and want to delete them all when you delete the customer, Core Data gives you the ability to do that without having to manage customerIds/productIds and figuring out how to format complex SQL queries to match your in-memory model. With Core Data, you simply update your model and save your context, the SQL is done for you under the hood. (In fact you can even turn on debugging to print out the SQL Core Data is performing for you by passing '-com.apple.CoreData.SQLDebug 1' as a launch argument.
In terms of performance, Core Data does some serious optimizations under the hood that make accessing data much easier without having to dive deep into SQL, concurrency, or validation.
I THINK the point is that it is different from a relational database and that trying to apply relational techniques will lead the developer astray as others have mentioned. It actually operates at a higher level by abstracting the functionality of the relational database out of your code.
A key difference, from a programming standpoint, is that you don't need unique identifiers because core data just handles that. If you tried to create your own, you will come to see that they are redundant and a whole lot of extra trouble.
From the programmer's perspective, whenever you access an entity "record", you will have a pointer to any relationship -- perhaps a single pointer for a "to-one" relationship, or a set of pointers to the records in a "to-many" relationship. Core Data handles the retrieval of the actual "records" when you use one of the pointers.
Because Core Data efficiently handles faults (where the "record" (object) referenced by a pointer is not in memory) you do not have to concern yourself with their retrieval generally. When they're needed by your program Core Data will make them available.
At the end of the day, it provides similar functionality but under the hood it is different. It does require some different thinking in that ordinary SQL doesn't make sense in the context of Core Data as the SQL (in the case of a sqlite store) is handled for you.
The main adjustments for me in transitioning to Core Data were as noted -- getting rid of the concept of unique identifiers. They're going on behind the scenes but you never have to worry about them and should not try to define your own. The second adjustment for me was that whenever you need an object that is related to yours, you just grab it by using the appropriate pointer(s) in the entity object you already have.
I know that there is Core Data to store information onto your phone, but what if you are making a social media application such as Instagram. Where did they manage to store all the information for every user? My main question is when there is a large amount of data to be stored for an application where do application designers store this information? Thanks
The data is stored on their servers in the form of a database.
http://en.wikipedia.org/wiki/Database
When the application needs to retrieve some of the user's information such as likes, photos etc... a HTTP Request is made to the server to request the data from the database.
http://en.wikipedia.org/wiki/Http_request#Request_message
PHP and MySQL are standard and low-end languages and databases to use and quite easy to learn and become familiar with. More advanced databases tend to use caching layers and other types of layers over the database which prevent data loss, increase read and write speeds, cache the request data, and more. Furthermore, other companies may chose to use NoSQL databases.
There is a comparison between NoSQL and SQL databases here: http://metadata-standards.org/Document-library/Documents-by-number/WG2-N1501-N1550/WG2_N1537_SQL_Standard_and_NoSQL_Databases%202011-05.pdf
There is a comparison chart in more detail that can be found here: http://en.wikipedia.org/wiki/Comparison_of_relational_database_management_systems
One the data has been requested and delivered onto your iOS Device, it can be manipulated to be stored in many different ways.
Core Data and SQLite are the generally used methods of storing the data as they derive from eachother and it is quite familiar to the majority of SQL databases in terms of the schema.
You can use coredata to store the data. Images can be stored in document folder.
I am developing a web-based application using Rails. I am debating between using a Graph Database, such as InfoGrid, or a Document Database, such as MongoDB.
My application will need to store both small sets of data, such as a URL, and very large sets of data, such as Virtual Machines. This data will be tied to a single user.
I am interested in learning about peoples experiences with either Graph or Document databases and why they would use either of the options.
Thank you
I don't feel enough experienced with both worlds to properly and fully answer your question, however I'm using a document database for some time and here are some personal hints.
The document databases are based on a concept of key,value, and static views and are pretty cool for finding a set of documents that have a particular value.
They don't conceptualize the relations between documents.
So if your software have to provide advanced "queries" where selection criteria act on several 'types of document' or if you simply need to perform a selection using several elements, the [key,value] concept is not appropriate.
There are also a number of other cases where document databases are inappropriate : presenting large datasets in "paged" tables, sortable on several columns is one of the cases where the performances are low and disk space usage is huge.
So in many cases you'll have to perform "server side" processing in order to pick up the pieces, and with rails, or any other ruby based framework, you might run into performance issues.
The graph database are based on the concept of tripplestore, meaning that they also conceptualize the relations between the entities.
The graph can be traversed using the relations (and entity roles), and might be more convenient when performing searches across relation-structured data.
As I have no experience with graph database, I'm not aware if the graph database can be easily queried/traversed with several criterias, however if an advised reader has such an information I'd really appreciate any examples of such queries/traversals.
I'm currently reading about InfoGrid and trying to figure if such databases could by handy in order to perform complex requests on a very large set of data, relations included ....
From what I can read, the InfoGrah should be considered as a "data federator" able to search/mine the data from several sources (Stores) wich can also be a NoSQL database such as Mongo.
Wich means that you could use a mongo store for updates and InfoGraph for data searching, and maybe spare a lot of cpu and disk when it comes to complex searches inside a nosql database.
Of course it might seem a little "overkill" if your app simply stores a large set of huge binary files in a database and all you need is to perform simple key queries and to retrieve the result. In that cas a nosql database such as mongo or couch would probably be handy.
Hope some of this helps ;)
When connecting related documents by edges, will you get a shallow or a deep graph? I think the answer to that question is important when deciding between graphdbs and documentdbs. See Square Pegs and Round Holes in the NOSQL World by Jim Webber for thoughts along these lines.