I've been tasked with coming up with a recommendation of how to proceed with a EDW and am looking for clarification on what I'm seeing. Everything that I am learning about states that Kimball's approach will bring value quicker to business vs Inmon's. I get that Kimball's approach is a dimensional model from the getgo and different data marts (star schema) are integrated through conformed dimensions... thus the theory is I can simply come up with my immediate DM to solve business need and go on from there.
What I'm learning states that Inmon's model suggests that I have a EDW designed in 3NF. The EDW is not defined by source system but instead the structure of the business, Corporate Factory (Orders, HR, etc.). So data from disparate systems map into this structure. Once the data is in this form, ETLs are then created to produce DMs.
Personally I feel Inmon's approach is a better way. I believe this way is going to ensure that data is going to be consistent and it feels like you can do more with this data. What holds me back with this approach though is everything I'm reading says it's going to take much more time to deliver something but I'm not seeing how that is true. From my narrow view, it feels like no matter what the end result is we need a DM. Regardless of using Kimball's or Inmon's approach the end result is the same.
So then the question becomes how do we get there? In Kimballs approach we will create ETLs to some staging location and generally from there create a DM. In Inmon's approach I feel we just add in another layer... that is from the staging area we load this data into another database in 3NF organized by function. What I'm missing is how this step adds so much time.
I feel I can look at the end DM that needs to be made. Map those back to a DW in 3NF and then as more DMs are requested keep building up the DW in 3NF with more and more data. However if I create a DM in Kimballs model that DM is going to be built around the level of grain decided for that DM and what if the next DM requested wants reporting at even a deeper grain (to me it feels like Kimballs methodology would take more work) and with Inmon's it doesn't matter. I have everything at the transnational level so DMs of varying grains are requested, well I have the data, just ETL it to a DM and all DMs will report the same since they are sourced from the same data.
I dunno... just looking for others views. Everything I read says Kimball's is quicker... I say sure maybe a little bit but there is certainly a cost attributed by going to quicker route. And for sake of argument... let's say it takes a week to get a DM up and running through Kimballs methodology... to me it feels like it should only take 10% maybe 20% longer utilizing Inmon's.
If anyone has any real world experience with the different models and if one really takes so much longer then the other... please share. Or if I have this so backwards tell me that too!
For context; I look after a 3 billion record data warehouse, for a large multi-national. Our data makes its way from the various source systems through staging and into a 3NF db. From here our ELT processes move the data into a dimensionally modelled, star schema db.
If I could start again I would definitely drop the 3NF step. When I first built that layer I thought it would add real value. I felt sure that normalisation would protect the integrity of my data. I was equally confident the 3NF db would be the best place to run large/complex queries.
But in practice, it has slowed our development. Most changes require an update to the stage, 3NF and star schema db.
The extra layer also increases the amount of time it takes to publish our data. The additional transformations, checks and reconciliations all add up.
The promised improvement in integrity never materialised. I realise now that because I control the ETL, and the validation processes within, I can ensure my data is both denormalised and accurate. In reporting data we control every cell in every table. The more I think about that, the more I see it as a real opportunity.
Large and complex queries was another myth that has been busted by experience. I now see the need to write complex reporting queries as a failing of my star db. When this occurs I always ask myself: why isn't this question easy to answer? The answer is most often bad table design. The heavy lifting is best carried out when transforming the data.
Running a 3NF and star also creates an opportunity for the two systems to disagree. When this happens it is often a very subtle difference. Neither is wrong, per se. Instead, it is possible the 3NF and star query are asking slightly different questions, and therefore returning different results. Although technically correct, this can be hard to explain. Even minor and explainable differences can erode confidence, over time.
In defence of our 3NF db, it does make loading into the star easier. But I would happily trade more complex SSIS packages for one less layer.
Having said all of this; it is very hard to recommend an approach to anyone without a deep understanding of their systems, requirements, culture, skills, etc. Having read your question I am sure you have wrestled with all these issues, and many more no doubt! In the end, only you can decide what the best approach for your situation is. Once you've made your mind up, stick with it. Consistency, clarity and a well-defined methodology are more important that anything else.
Dimensions and measures are a well proven method for presenting and simplifying data to end users.
If you present a schema based on the source system (3nf) to an end user, vs a dimensionally modelled star schema (Kimball) to an end user, they will be able to make much more sense of the dimensionally modelled one
I've never really looked into an Inmon decision support system but to me it seems to be just the ODS portion of a full datawarehouse.
You are right in saying "The EDW is not defined by source system but instead the structure of the business". A star schema reflects this but an ODS (a copy of the source system) doesn't
A star schema takes longer to build than just an ODS but gives many benefits including
Slowly changing dimensions can track changes over time
Denormalisation simplifies joins and improves performance
Surrogate keys allow you to disconnect from source systems
Conformed dimensions let you report across business units (i.e. Profit per headcount)
If your Inmon 3NF database is not just an ODS (replica of source systems), but some kind of actual business model then you have two layers to model: the 3NF layer and the star schema layer.
It's difficult nowadays to sell the benefit of even one layer of data modelling when everyone thinks they can just do it all in a 'self service' tool! (which I believe is a fallacy). Your system should be no more complicated than it needs to be because all that complexity adds up to maintenance and that's the real issue - introducing changes 12 months into the build when you have to change many layers
To paraphrase #destination-data: your source system to star schema transformation (and seperation) is already achieved through ETL so the 3nf seems redundant to me. You design your star schema to be independent from source systems by correctly implementing surrogate keys and business keys, and modelling it on the business, not on the source system
With ETL and back-end data wrangling taking up about 70% of the project time for this kind of endeavour, an extra layer makes a big difference. Its an extra layer of transforming from source to target, to agree with the business and to test. It all adds up.
Whilst I'm not saying that dimensional models (the Kimball kind) are always easy to change, you've got a whole lot more inflexibility should you have to always change lots of layers when you want to change your BI.
In fact, where I've been consulting in places that have data warehouses that are considered to be inflexible and expensive to develop for, and not keeping pace with changes to the business, they have without exception included the 3NF layer prior to the DMs. As Nick mentioned, it is hard nowadays to sell the idea of a 'proper' data warehouse as opposed to a Data Discovery Bi tool- and the appeal of these is often driven by DWs being seen to be slow and expensive to develop.
Kimball isn't against having a 3NF layer prior to his DW if it makes sense for a situation, he just doesn't agree with Inmon that there's a point.
One common misunderstanding is that Kimball proposes distinct data marts, so that you'd have to change it each time there is a different reporting request. Instead, Kimball's DMs are based on real life business processes and modelled accordingly. Although its true you will then try and make them suitable for reporting, you try and make them so they can answer forseaable queries. You don't aggregate and store just the aggregates: you work with the transactional data in a Kimball dimensional model.
So no need to be reluctant from that perspective.
If an ODS works for you, then go for it- but a Kimball DW will meet the majority of requirements.
I am interested in what the best practice is for a model that has a lot of data attached to it. Most of my app revolve around one model (SKU), and it seems to have more and more things associated with it.
For example, my SKU model has multiple prices, dimensions, weight, recommended prices for multiple price levels, title, description, shelf life, etc. Would it make sense to break all the pricing info to another table? Or break up the SKU into different uses of the SKU and associate them? For example, WebSKU, StockSKU, etc.
As mentioned in the answer linked by Tom, if all your attributes really belong to that model there is no reason to break it up. However, if you have columns like price1, price2, price3 or dimension_x_1, dimension_y_1, dimension_x_2, dimension_y_2, etc, then it usually means you should be creating another table to contain those.
For example, you could set it up so that you have the following models
Sku
has_many :prices
has_many :dimensions
Price
belongs_to :sku
Dimension
belongs_to :sku
As everyone else said, the design of a database should respond to the logic behind it. Why? Mainly, because it will be easier to maintain and understand.
I was also going to drive attention to normalization rules, as #sawa did.
Generally, is a good approach to normalize your database, as it provides several advantages. You should read this wikipedia link (at least as a starting point).
Following normal rules will help you to design your database taking into account the logic behind your data.
But denormalization also has it's advantages. The first (always considered) being optimizing read performance. This basically means having data on one table that you would have had in different tables when following normal rules, and generally makes sense when that data has some logic relation.
You have to aim to achieve a balance depending on the problem you are facing.
On the other side, for the tags on your post I can see you are using ruby on rails, that uses the active record pattern. One consequence of the database model you are presenting, is that you will probably have a domain model just as complex. I mean, very large. I don't know every detail about your project, but I guess that it will quickly grow to be a god object, making your code hard to maintain, extend and understand.
Database should be designed not according to how many columns it has, but according to logic, particularly following Codd's normal forms. If there is systematic redundancy in your database, then that is a sign for splitting it into multiple tables. If not, keep it as is.
I think it is good to design data model, taking into account how DB engine works with files and memory. The first bottleneck of PostgreSQL is file IO. Memory consumption is also an important part. When PostgreSQL reads some table data (FYI: table data is not read at Index-Only-Scans) it reads 8 KB (compile time parameter) pages. More tuples in such a page, - less file IO, less memory consumption, better cache using (more often hits, fast prewarming, etc.), better performance.
So, if one have a really high-loaded project, it can be useful to think about separation of often used data to isolated tables (as a next step - place this tables into a separate tablespace on SDD or powerful RAID).
I.e. there should be some balance between a logic simplicity and performance tweaks.
I have the following models..
Provider
Patient
Clinic
Like an eco-system, all models should have many-to-many relationships with one another. It's very important that I am able to query data from all directions.
After intensive research on Active Record associations, I find many blogs warning against has_and_belongs_to_many and using has_many :through. Only issue is that requires a table to act as the "middle-man" for lack of better words but I'm unsure how that would work with a 3 models.
The other option is a polymorphic association but I'm unsure if I should invest the time in understanding that method if it's inapplicable for this particular situation.
Any advice on how to create these relationships for maximum flexibility and efficiency?
I have the following models
Tables, if they were modelled for a Relational database, files otherwise.
It's very important that I am able to query data from all directions.
Understood. That is simple and easy in a Relational Database.
Like an eco-system, all models should have many-to-many relationships with one another.
That is not correct.
If each of the three can be related to the other two, then your data is not modelled. There are basic dependencies that you have not identified. Eg: I can imagine that:
Providers provide services at Clinics
Clinics provide services to Patients
Patients visit Clinics to obtain services
Therefore, any relationship that a Patient may have with a Provider is via a Clinic, and not privately, without a Clinic.
Straighten those rules and dependencies out first, that will result in less than three Associative tables second. Something like this:
Clinic Provider Patient Table Relation Diagram
Response to Comment
Any advice on how to create these relationships for maximum flexibility and efficiency?
Well, Dr E F Codd's Relational Model is strongly established as the model, the method for organising data, such that it has (a) complete integrity (objects can't) (b) maximum flexibility (c) maximum speed. In the 45 years since its advent, there have been no other contenders. That is the model I am using. The principles that underpin that model are the principles that I rely upon, when I make my proposals.
All that has been, of course, confirmed and reinforced during my 34 years of database implementations. As well as thousands of other high-end implementations.
Data Independence
I've taken your post with great consideration and the difficulty is that there are many unique nuances with my app that won't allow me to simplify it to mirror the real world.
It is the other way around. The fact is, you are writing an app+db that deals with the real world. Therefore the db has to reflect that real world (limited to the scope of the enterprise) that you wish to engage with. Thousands of modellers have done that successfully (millions have done it incorrectly).
To the extent that you have "nuances" and complexities in the app, you have not modelled the data, as data, and the result will be a complex app that engages with an incorrectly modelled database, or worse, a non-database. All those "nuances" and complexities are in fact data; facts about data; rules about the data; and relationships between data. But you have an established view that the "nuances" and complexities are in the app, in your "models", not in the data.
Therefore your notion is false.
Every rule, constraint, control, nuance, complexity re the data, must be implemented in the data. That is simply data definition as per the RM. Otherwise you have no data independence, no database (just a bunch of files for data storage), no integrity, no relational power, no speed. And worse, you will be forever fixing up the "nuances" and complexities in the app layers, the object stack.
Data Definition
Let me start from first principles.
a databse is a collections of Facts, including relationships between those Facts
Assuming you understand that, there is an important next point.
There is no Fact that cannot be declared.
This is First order Logic, which the RM is based on. Therefore there is no such thing, as a "model" that is too complex (has "nuances" or too many "nuances") to be declared in terms of FOL. The scientific exercise that is called for, is to reduce that complexity to FOL Facts.
To the extent that those Facts are Facts about the real world, and they reflect the truth, your database will be isolated from the effect of change (you can extend the database and the app easily). Eg. Provider, Service, Specilisation are separate Facts.
To the extent that those "facts" are "facts" that you have chosen to store (as being relevant, from the perspective of your app, object design, etc), eg. Provider, Service and Specialisation as a single complex "fact", not discrete Facts, and not a reflection of the real world, your database and objects (a) will be hard to change, and (b) will keep changing, forever, until they are elevated, such that they do reflect the real world. You will have to "re-factor" the "database" every quarter.
Confidentiality
The data is very confidential so I'm reluctant to get to far into the matter.
We have been working with confidential data and/or systems for over 45 years without breaching confidentiality.
There is nothing new under the Sun.
We are dealing with structure, not content, it is not reasonable to suggest that your structure is so new and unique that it cannot be (i) discussed or (ii) modelled.
But most important, if you cannot describe it (in FOL terms), you cannot model it. If you cannot model it, you cannot write an app to engage with it (you can try, but as evidenced here, you will be stuck in that unresolved position).
Noting that the OO/ORM literature teaches people to obsess about the data content, in order to avoid dealing with the structure, meaning, relevance, etc, please note that I do not want to know. and the exercise does not need to know, the content. Describe only the data in terms of meaning, relevance, relationships, and we will model the required structure.
My question was more about how to create a cycle of many-to-many relationships with 3 models than whether my models "should" have them or not.
I think I understand that. That would be adding complexity to an article of which the complexity has been determined to be the fundamental problem. If you ask me to build an airplane without wings, and I tell you that your approach is incorrect, that you need wings, there is no point in telling me that you are seeking someone who can tell you HOW to build an airplane without wings, you have missed the point.
Reasoning
I would love to hear your reasoning if you believe there should never be a situation like this in any database.
Again, you have that the wrong way around. It is not that there should never be a situation like this in any database, it is that if there is a situation like this, it raises a red flag (to qualified and experienced modellers) that the data is not yet Normalised, not yet organised into discrete Facts. That means you need to take a step back and deal with the complexity in your "models", first. Then the relationships that are your current focus will be simplified.
Then, yes, there will not be a situation like this in the database.
My reasoning is Codd's RM, and the principles behind it. It has been the subject of many papers. (As well as many "papers" that are non-relational or anti-relational, such as those that support the OO/ORM "model".)
Specifically, here, that if you have a n-ary relation (technical term for the three-way relationship that you are seeking) that that can be, and should be, resolved into [multiple] binary relations (two-way relationships). Eg. the TRD I have suggested.
OO/ORM Mythology
In the context of "love to hear your reasoning", there are two sides, I have given the what you should be doing side, above. This is the what you should not be doing or why your method is broken side. Where do I start.
The OO/ORM model is that the "database" is merely a storage location to make the objects "persistent", a slave of the objects, and that constructing a monolith, layers of object classifiers, complexity, is the way to solve any problem.
The OO/ORM model is a total, abject failure. It has no scientific basis whatsoever.
(Noting that due to the destruction of the education system, these days "mathematicians" and "theoreticians" write "papers" that "prove" complete and utter nonsense. It is nonsense because it contradicts established science. They are not the class of mathematicians and theoreticians of the old school, who reject contradiction; non-science; nonsensical proofs. The only way that they can write such absurd "proofs", in to maintain a state of ignorance, a state of pathological denial, of other sciences.)
Specifically, they are in denial of the Relational Model (whilst referring to it, to give their papers some credibility, which is a fraud); its prescriptions (such as Data Independence, FOL); its prohibitions, they are in denial of Relational data models (UML cannot model data like IDEF1X can), thus they produce non-relational files, which have no Relational integrity, power, or speed.
They employ the Hammer phylosophy (ie. if one only knows a hammer, then every problem looks like a nai), in staggering denial that Maslow destroyed it, scientifically, over a century ago. Which leads them to pile more layers, more complexity, into the monolith. The converse is of course, to use the right tool for the job, which means define data according to the standard for data, the RM, and separately, objects according to whatever OO philosophy you choose.
They attempt to do everything in objects, to model everything in UML (which is not a standard by any means; nor adequate [one symbol plus a million notations], it has no decomposition, it is in fact a free-for-all in which everyone does their own thing).
The model exists is denial of the fact that since 1980's, in the software industry, we architect, write, and deploy components. Database components in the database, program components in the objects. Not monolithic Towers of Babel, that is pre-1970's technology.
Since 1984, we have had Client/Server and Open Architecture Standards. We have had OLTP Transaction Standards since 1960, restated in the C/S context in 1984. The OO/ORM crowd are in flagrant rebellion of each and every one of those Standards, they build monolithic object stacks sans architecture, sans components, sans Transactions, sans everything. (Apologies to the Bard.)
You might consider what everyone (even cartoonists) knows about the OO/ORM stack, the monolith, the non-architecture and compare it with the deployment of components in the Open Architecture diagram, given above.
Further, they are in denial that every implementation of the "model" is a massive failure, they deny the evidenced facts, and keep adding complexity to the already complex and unmanageable object layers. A "model" that has failed due to its non-architecture, part of which is precisely that failed complexity.
In case their evidenced pathological denial of the reality does not stand, in and of itself, as evidence of insanity, there is more, much more. The Twelve Steppers have an interesting definition of insanity: doing the same thing over and over again, being aware that it produces the same result, every time, but expecting a different result the next time. That doesn't stop them from adding more complexity to the complex model, or from marketing their pre-1970's technology, as "modern" "science".
But that doesn't stop them from writing yet more books and marketing their failed "model".
The OO/ORM crowd exists in isolation from, in pathological denial of, reality
Put another way, it is insane, in 2015, to implement software, in a un-architected monolith, that "does everything", rather than to architect; design; build; and deploy software components in the correct architectural position.
OO/ORM "Model" as Data
The fact that you keep calling your tables "model" is another red flag. That confirms that you have way too much complexity in them. A database consists of simple tables, each reflecting a discrete Fact, not models. To the extent that you consider them "models", you have (usually due to fixed notions re complex objects and classifiers) objects+data+complexity combined, not discrete data and discrete objects. That is the precise problem that will cause the app+db to fail.
So the next step is to (a) shelve the current focus re HOW to relate un-normalised complex "models", and to (b) normalise those "models" such that they are defined in terms of a Relational Database, such that they are discrete Facts. Following which, (c) the relating of the then Normalised tables will be straight-forward.
In one instance, each model acts as a type of 'User' but in another, they don't.
That is exactly the type of mashed-up concept that has to be rationalised, Normalised, such that it is (a) absolutely clear [when it is an user, and when it isn't] (b) defined in data, in FOL terms, as discrete Facts (c) such that you can confidently write code against it, build objects from it. Conversely, the absence of that clarity, the retention of complexity in the object layers, will result in complex objects that fail, and more important, data stored in files that have no integrity, power, or speed.
Self-Contradiction
Consider this. Since you are seeking Relational integrity, power, and speed, you cannot at the same time be
seeking to retain unresolved complexity that is well-known to destroy integrity, power, and speed, or
refusing to implement the requirements the host integrity, power, and speed, that you seek.
It is a massive, and double, contradiction, on your side. It is a philosophical and reasoning issue, that you have to consider and resolve for yourself. The OO/ORM seduces people into believing in magicke, into such crazy self-contradictions.
Regardless, I was very impressed with your answer and really appreciate the time you took to make the diagram.
Thank you. You are most welcome.
That took me all of five minutes. Because I have clarity. Because I follow scientific principles and standards. Because we have had a science and a methodology since 1970. Because we have had a modelling methodology and full notation for modelling Relational Data since 1987 (as a standard, IDEF1X, since 1993). The point is, it is nothing special, the sad fact is, it is not common, and it should be. The second sad fact is, it is unknown in the OO/ORM world.
Further Reading
You may be interested in this Question and Answer. The Answer covers many aspects of Relational Databases, that you will most certainly have to deal with, if not now, then at some point in the future. The minimum reading I draw your attention to right now is:
- Response to Update 3, pages 1, 2 and 6,
- specifically including the embedded link to Predicates
That might give you an idea of the reasoning; the depth of data definition that the RM affords; that all Facts can be declared in terms of FOL Predicates, that the OO/ORM crowd is totally ignorant of.
You may choose to add an Update to your question and ask how to declare one or the other "model", in terms of the RM, as discrete Facts, or open a new Question (and ping me).
Conversely, if you choose to stick to your position, the original question, then I think I have answered it (but the answer raises issues that you must address and resolve).
Please study carefully and comment or ask questions.
I'm working on financial data mart structure.
And I'm having some doubts on whats the better approach to do so.
The source system database,Dynamics AX 2009, has three tables for customer transaction.
One table for open transactions, where the Customer still needs to pay for service/product;
One table for settle transactions, where it holds what the customer have already paid;
Finally a table that have all customers transactions, holds transactions from open to settle and also others transactions as customer to bank or ledger accounts.
I thought in two options, first I will maintain a fact table representing the three table, fact for open transactions, fact for any customer transaction and fact for settle transaction.
Second is to create a single fact to hold all transactions, to do so I would have to do a full join on three tables.
I'm not sure on both approaches, as the first seems to copy tables from production and create the proper dimension.
On the Second one I would create a massive fact table, that where data would constantly change, as open transaction are delete on source system when they are settle.
Another doubt, should i create a fact with scd(slowly changing dimension) structure to maintain history data?(star date, end date , flag)
It's hard to say from the information given whether this needs to be one or more Fact tables. However, the key point which you should use to decide is whether all of the information is at the same granularity. Consider the grain of your intended Fact table(s) and you should find an answer for whether you need one table or multiple tables.
If all of the information sits at the same grain - i.e. all of the same dimensions apply to all of the measures you are considering putting into the same Fact table - then they can probably all live in the same Fact table. If you're finding that some of the Dimensions wouldn't apply to some of the measures then you probably need to re-think your design. Either you might need multiple Fact tables, or you might need to take all of your measures down to the lowest grain and combine hierarchies into single Dimensions if you currently have them split across multiple Dimensions.
While it's been mentioned that having measures in separate cubes could make it difficult to compare things, keep in mind that you don't need one cube per Fact table. You can have multiple Fact tables in a single cube, and sometimes this is very helpful when you need to be able to compare measures which share some Dimensions but not others. This is far, far better than forcing data which does not have the same grain into one Fact table.
Also, it sounds like what you're trying to model is the sales ledger of an organisation. I'd suggest having a dig around via Google as you may well be able to find materials discussing dimensional data warehouse design for sales ledger structures, rather than reinventing the wheel. If you don't have a decent understanding of the accounting concepts you're trying to model I would especially recommend looking for a reference schema to work from, or failing that doing some reading up on accountancy concepts (and sales ledgers specifically). Understanding the account structure should help you understand what the grain of your Fact table(s) needs to be, how to model the Dimensions, and so on.
This is a really helpful abridged version of Kimball's modelling techniques which discusses grain, and the different types of Fact table, amongst many other topics:
http://www.kimballgroup.com/wp-content/uploads/2013/08/2013.09-Kimball-Dimensional-Modeling-Techniques11.pdf
I think you should just use one fact table (one cube) and use a dimension to differentiate between open/settled/etc. transactions. That's what dimensions are for: They help you to categorize your measures and get a specific view on them. This approach will also open much more possibilities to create knowledge with your cube. With separate cubes for open/settled/etc. transactions, it will be harder or not possible to set this data into contrast.
Since the data is changing constantly, you should consider to update your fact table in a given time and rebuild your cube if it needs to.
If you use scd or not really depends on the data you process and what it is used for. Is there a business case claiming it? Is there a technical use?
I think this is something you have to decide on your own.