I am developing an ASP.NET MVC4 web application. It uses the entity framework for data access. Many of the pages contain grids. These need to support paging, sorting, filtering and grouping. For performance the grid filtering, sorting, paging etc needs to occur on the database (i.e. the entity framework needs to generate a suitable SQL query). One complication is that the view model to represent the grid rows is built by combining the data from multiple business entities (tables). This could be simply getting the data from an entity a couple of levels down or by calculating it based on the values of related business entities. What approach is recommended to handle this scenario? Does anyone know of a good example on the web? Most have a simple mapping between the view model and business domain model.
Update 28/11 - To further clarify the initial display of the grid and paging works performs well. (See comment below) The problem is how do you handle sorting/ordering (and filtering) when the column that the user clicked on does not map directly to a column on the underlying business table. I am looking for a general solution to achieving this as the system will have approx 100 grids with a number of columns each and trying to handle this on a per column basis will not be maintainable.
If you want to be able to order a calculated field that isn't pre calculated in the database or do any Database Operation against it, then you are going to have to precalulate the value and store it in the database. I don't know anyway around that.
The only other solution is to move the paging and sorting etc to the web server, I am sure you don't really want to do that as you will have to calculate ALL the values to find what order they go in.
So if you want to achieve what you want - I think you will have to do the following, I would love to hear alternate solutions though:
Database Level Changes:
Add a Nullable Column for each calculated field you have in your View Model.
Write a SQL Script the calculates these values.
Set the column to Not Null if necessary
App Level Changes:
In your Add and Edit Pages you will have to calculate these values and Commit them with the rest of the data
You can now query against these at a Database level and use Queryable as you wanted.
Related
I have setup a time-series / events database using the AWS Firehose -> S3/Glue -> Athena stack. It is being used to track various user actions - session started, action performed etc. across a number of our products. My question is about how best to store different types of IDs in this system.
The existing schema is one big 'fact table' with a bunch of different columns. Two of the most important columns are event_type_id and object_id. To use StackOverflow as an example, two events might be:
question_asked - in this case I would be storing the question id in the object_id column.
tag_created - in this case I would be storing the tag id in the object_id column.
My question is - is storing multiple different types of IDs in the same column bad practice? It's working OK for us at the moment, but it does require the person/system performing queries to know what type of object the object_id column refers to, based on the event they are querying.
If bad practice, what other approaches might be better? Multiple columns where they are NULL if not relevant for the event in that row? Or is this where dimension tables would be a better fit?
This isn't necessarily bad practice, depending on how you use it.
It sounds like you're aware of the potential pitfalls of such an approach (i.e. users of the data have to be aware of the context - in this case "event type" - to use the values correctly), so as you're using Athena you could mitigate that by creating views over source table for different event types, inserting a WHERE clause filter on event type and possibly renaming object_id to something more context specific e.g. question_id.
This makes it easier for users to work with the data and understand exactly what the values are they're working with.
In a big data environment I wouldn't recommend creating dimension tables if it can be avoided as JOINs between tables start to get expensive. Having multiple columns for different ids is possible but then you create new problems for users such as having to account for NULL values in an Id column, and this also potentially makes it harder to add new event types and ids as you have to change the schema to accommodate them.
I wonder if it's possible to create a logic that automatically creates a denormalized table and it's data (and maintains it) by a specific SQL-like query.
Given a system where the user can maintain his data model and data. All data are stored in "relational" tables, but those tables are only used for the user to maintain his data. If he wants to display data on a webpage he has to write a query (SQL) which will automatically turn into a denormalized table and also be kept up-to-date when updating/deleting the relational data.
Let's say I got a query like this:
select t1.a, t1.b from t1 where t1.c = 1
The logic will automatically create a denormalized table with a copy of the needed data according to the query. It's mostly like a view (I wonder if views will be more performant than my approach). Whenever this query (give it a name) is needed by some business logic it will be replaced by a simple query on that new table.
Any update in t1 will search for all queries where t1 is involved and update the denormalized data automatically, but for performance win it will only update the rows infected (in this example just one row). That's the point where I'm not sure if it's achievable in an automatic way. The example query is simple, but what if there are queries with joins, aggregation or even sub queries?
Does an approach like this exist in the NoSQL world and maybe can somebody share his experience with it?
I would also like to know whether creating one table per query does conflict with any best practises when using NoSQL databases.
I have an idea how to solve simple queries just by finding the involved entity by its primary key when updating data and run the query on that specific entity again (so that joins will be updated, too). But with aggregation and sub queries I don't really know how to determine which denormalized table's entity is involved.
I'll try to be short and clear with this question.
We have an asp.net mvc app that uses entity framework 4.
Our business model is relatively straightforward:
We have an object (which corresponds to a table) called Photo(s).
That photos table has a handful of columns that match up to properties on the object.
Description,Title,Date etc.
It also has a number columns that reference foreign keys for other tables:
AuthorId,LicenseId etc...
The author and license tables are complex in their own right, with multiple fields (Title,Summary,Date etc.)
I have multiple clients using this application to view their photos. I would like each client to dictate what fields they see when viewing the photos, as well as what fields they see when editing those fields.
My thought is to have tables setup saying client-a should see Field1,Field2 and Field3 when viewing their photos - and client-b should see Field1,Field4 and Field5. But some of these fields are not simply columns in the main photos table, they may be fields in a child table. so Field1 might be: Table.Photos.Title -> which corresponds to an object as: Objects.Photo.title...
but Field3 might be: Table.Licenses.LicenseSummary -> which corresponds to an object as: Objects.Photo.License.LicenseSummary
I'm trying to figure out the methodology that we would use to have a very data driven environment so in the DB I can say, display this object/property (for viewing or editing) and then it would know how to map to whatever table it needs to pull that information. also, during editing... give it some way to pull a list of available values if it is that type of property, and not just a text field.
I'm looking for an example of what this might involve, our model is actually more complex than this, but this is just an idea of what we are trying to accomplish. I don't know if what I'm trying to do is normal, perhaps it involves reflection? This is a new area for me.
If the clients are defining their own custom fields, I would simply give them a Key/Value pairs table.
PhotoID FK
Key string
Value string
Display bool
Note that this essentially amounts to EAV, which comes with its own set of difficulties.
If it's just about permissions on existing fields, you need to capture that information:
PhotoID FK
ClientID FK
FieldName string
Display Bool
You can use this information to inhibit the display of fields in the View. The easiest way to do that would be to use a loop in the View itself, writing the field to the output only if Display is set to true.
I'm starting to write a project and its my first time using ASP.NET MVC and Entity Framework (since now I've used PHP for around 5 years).
I got categories and posts, each category has to have its own unique filters that can be strings or Booleans (I'll get them via textboxes or checkboxes). I'm starting to get confused when designing the entities.
I'm using code-first approach but don't know how to set-up custom fields in Entity Framework. If I'd design that in PHP and pure SQL, I think that in order to keep it perform good, I'd create extra columns on the fly (e.g. "filter_1", "filter_2") and then create mapping table that contains the description of the field, the type, etc. and I didn't figure out if this kind of implementation is possible in EF.
I've thought about some options:
- I can create the filter using many-to-many relationships when creating a filters and filters-values tables and when creating a post add the filter values into that table. The main con about it is he performance - I'll most likely have 40k+ rows and more than 20 custom filters for sure... so searching and data fetching will be too slow...
- I thought about serializing the filtered values into some form of content, for example into bytes array and then only desterilize it - but the problem is that I won't be able to search within that...
- I can use the traditional ADO.NET way in order to create my initial idea (that I've described when talking about PHP - using the "filter_N" columns) but that'd create too much mess with EF...
There's any other way to achieve my goal (to create custom filters)? any way to create custom fields using EF?
Thanks for your help!
I don't know of a good way offhand to implement this in the Entity Framework. If you really wanted to use the Entity Framework I believe you could design your database tables in such a way that you wouldn't have to create the extra columns on the fly. You could have separate tables to hold the filters and relate them back to categories.
In our ASP.Net project we've got a view with 3 (or even more) cascaded combo-boxes,
The question is how should we design our (NHibernate-based) entities given that the relationship between the combo-boxes is relevant only for the sake of building the view, and given that the entire catesian product of the 3 entity types is around 2000 entities.
So the questions are actually:
Should we design the entities as a heirarchy even though the collections has no additional business value (thus making them an overhead for any other scenario)
Should the data be sent as flat rows? keeping the entities more "clean", but requires additional
coding on the view's side
Seperate to consequent requests (e.g only after combo A is selected, combo B's data is fetched from the server)? making the entire process maybe more efficient but at the cost of performance and perhaps caching?
Any thoughts?
First of all, you should not design your model (entities) by one intended UI. Today is it 3 dropdown, tommorow it can be autocomplete nad in a week some new fancy "wow" component, but your model should stay solid - it describes actual, business relations between entities, not your current UI. From this, i think it is quite obvious that you should use combination of 2 and 3.