Let's say I have some collection called someentity that has connections to other entities via a verb subscribedto. For a particular entity with id some_id, I'd like to pull all the entities that are connected to this entity via the subscribedto verb. I can do that via this GET request:
/org/app/someentity/some_id/subscribedto
However, I'm not sure how I stream or page through this data if there are thousands or more results. I'm not getting back a cursor or anything. Is having a huge number of connections to a particular entity something that's just not a smart thing to do with Usergrid?
This is currently a bug. I did not phrase my original question well, as I did not make it clear that this is a bug for INCOMING connections. See here: https://issues.apache.org/jira/browse/USERGRID-1310
Related
Scenario:
(with an ASP.NET web app - Core or MVC)
I have a database with Users and Items for each user.
That means the UserId is a foreign key in the Items table.
From the browser I login as a User. I get my Items as a list of ItemViewModels, which are mapped (AutoMapper) to ItemViewModels via a simple api GET request.
I want to update one of the items (which should belong to me - the logged in user) via a simple API call. So I send the modified item back to the server via a PUT request as an ItemViewModel.
First approach:
The simplest approach would be to include the Item's database ID, ItemId, in the ItemViewModel - so when I receive the item to be updated as an ItemViewModel, I can map it back to the existing item in the database.
This however sounds pretty unsafe to me, as anyone could modify the PUT request with any ItemId and affect items which don't belong to the user who executed the request. Is there anything I'm missing about this approach?
Second approach:
Don't pass the database PK ItemId in the ItemViewModel.
Instead use an additional form of identification: let's say that user X has 10 items. And they are numbered from 1 to 10 using a property named UserItemId(which also exists in the database).
I can then pass this UserItemId in the ItemViewModel and when I get it back I can map it to an existing Item in the database (if all was ok with the request) or discard it and reject the request if the UserItemId didn't match anything from the logged in user's items.
Is anyone using this approach?
Pros:
The user only has access to it's own items and can't affect anyone else's since it doesn't know the actual Item ID (primary key), and any modifications are restricted to it's items.
Cons:
A great deal of extra management must be implemented on the server side for this approach to work.
Any other approaches ?
Please consider that the case mentioned above applies to all entities in the database which a client side implementation can CRUD, so it's not just the simple case described above.
The proposed solution should work for the entire app data.
I know this question has been asked here and here but the first one doesn't have a satisfying answer and I don't think the second one really applies to my situation, since it just deals with the UserId.
Thanks.
EDIT
Please consider the Item above as an aggregate root which contains multiple complex subItems each with a table in the db. And the question applies for them as much as for the main Item. That means that each subItem is passed as a ViewModel to the client.
I should mention that regarding further securing the update request:
For the first approach I can easily check if the user is allowed to change the item. But I should do this for all subItems too.
For the second approach I can check if the user can update the Item as follows: I get the userItemId of the incoming ViewModel -> I get all the logged in user's items from the database and try to find a match with the same userItemId, if I get a hit then I proceed with the update.
I think your application is not secure, if you only hide the Id.
You must check, before changing the database entity, if the user is allowed to change the entity.
In your case you should check, if your Id from the authenticated user is the UserId in your item.
If your ViewModel ist similar or identical for your API you could use a FilterAttribute in your controller.
In the comment of this post (Problem with WCF Data Service (OData), SetEntitySetPageSize and custom operations), it mentioned “OData stable paging”.
I am quite new to OData, and trying to get stable paging in OData query.
By "stable paging", I mean there is an OData database in which new entries are constantly being created, I wanted the paging are not interrupted by the new data creation in the database.
For example, if I get the first page by $top=100, then get second page by $top=100&$skip=100, then the third page by $top=100&$skip=200, in which page size is 100. However, because the database is changing, the data I have skipped may not necessarily be the same data I retrieved from previous requests.
Just like the graph example shows data sorting by entry creation time, the first request and the second request:
I wonder what's the best way to do "Stable Paging"?
Thanks!
My current solution follows this post's recommendation - "Paging Through Results" by Markus Winand:
http://use-the-index-luke.com/sql/partial-results/fetch-next-page
It discussed general solutions to stable paging, although it's not OData specific, it works for my scenario.
Right now I have a server that formats my data exactly how restkit wants it, and restkit just takes it and directly maps it to coredata.
This works fine, but when I start to accumulate a lot of data it becomes slow.
For example, I have one object called "stories" and each story contains an array of "posts". each time a new "post" gets added, I regenerate the "story" object to which the new post belongs to, and return the story object to the user for restkit to map. As a story starts to accumulate many posts, this process becomes very slow for restkit. I would prefer a way to just send back new posts, and then tell restkit "hey, add this post to the array of posts on this story", which is in contrast to what I do now which is more like "replace this story with this one I just returned, which includes all posts including any new or updated ones".
Is this possible within restkit? Am I better served just manipulating core data myself to support updates?
Yes, it's possible.
You can look at 'foreign key mapping' to connect your new posts to the existing story. The most important part is to set the relationship assignment type to Union because the default is replace.
If I use breeze to load a partial entity:
var query = EntityQuery.from('material')
.select('Id, MaterialName, MaterialType, MaterialSubType')
.orderBy(orderBy.material);
return manager.executeQuery(query)
.then(querySucceeded)
.fail(queryFailed);
function querySucceeded(data) {
var list = partialMapper.mapDtosToEntities(
manager, data.results, entityNames.material, 'id');
if (materialsObservable) {
materialsObservable(list);
}
log('Retrieved Materials from remote data source',
data, true);
}
...and I also want to have another slightly different partial query from the same entity (maybe a few other fields for example) then I'm assuming that I need to do another separate query as those fields weren't retrieved in the first query?
OK, so what if I want to use the same fields retrieved in the first query (Id, Materialname, MaterialType, MaterialSubType) but I want to call those fields different names in the second query (Materialname becomes just "name", MaterialType becomes "masterType" and so on) then is it possible to clone the partial entity I already have in memory (assuming it is in memory?) and rename the fields or do I still need to do a completely separate query?
I think I would "union" the two cases into one projection if I could afford to do so. That would simplify things dramatically. But it's really important to understand the following point:
You do not need to turn query projection results into entities!
Backgound: the CCJS example
You probably learned about the projection-into-entities technique from the CCJS example in John Papa's superb PluralSight course "Single Page Apps JumpStart". CCJS uses this technique for a very specific reason: to simplify list update without making a trip to the server.
Consider the CCJS "Sessions List" which is populated by a projection query. John didn't have to turn the query results into entities. He could have bound directly to the projected results. Remember that Knockout happily binds to raw data values. The user never edits the sessions on that list directly. If displayed session values can't change, turning them into observable properties is a waste of CPU.
When you tap on a Session, you go to a Session view/edit screen with access to almost every property of the complete session entity. CCJS needs the full entity there so it looks for the full (not partial) session in cache and, if not found, loads the entity from the server. Even to this point there is no particular value in having previously converted the original projection results into (partial) session entities.
Now edit the Session - change the title - and save it. Return to the "Sessions List"
Question
How do you make sure that the updated title appears in the Sessions List?
If we bound the Sessions List HTML to the projection data objects, those objects are not entities. They're just objects. The entity you edited in the session view is not an object in the collection displayed in the Sessions List. Yes, there is a corresponding object in the list - one that has the same session id. But it is not the same object.
Choices
#1: Refresh the list from the server by reissuing the projection query. Bind directly to the projection data. Note that the data consist of raw JavaScript objects, not entities; they are not in the Breeze cache.
#2: Publish an event after saving the real session entity; the subscribing "Sessions List" ViewModel hears the event, extracts the changes, and updates its copy of the session in the list.
#3: Use the projection-into-entity technique so that you can use a session entity everywhere.
Pros and Cons
#1 is easy to implement. But it requires a server trip every time you enter the Sessions List view.
One of the CCJS design goals was that, once loaded, it should be able to operate entirely offline with zero access to the server. It should work crisply when connectivity is intermittent and poor.
CCJS is your always-ready guide to the conference. It tells you instantly what sessions are available, when and where so you can find the session you want, as you're walking the halls, and get there. If you've been to a tech conference or hotel you know that the wifi is generally awful and the app is almost useless if it only works when it has direct access to the server.
#1 is not well suited to the intended operating environment for CCJS.
The CCJS Jumpstart is part way down that "server independent" path; you'll see something much closer to a full offline implementation soon.
You'll also lose the ability to navigate to related entities. The Sessions List displays each session's track, timeslot and room. That's repetitive information found in the "lookup" reference entities. You'll either have to expand the projection to include this information in a "flattened" view of the session (fatter payload) or get clever on the client-side and patch in the track, timeslot and room data by hand (complexity).
#2 helps with offline/intermittent connectivity scenarios. Of course you'll have to set up the messaging system, establish a protocol about saved entities and teach the Sessions List to find and update the affected session projection object. That's not super difficult - the Breeze EntityManager publishes an event that may be sufficient - but it would take even more code.
#3 is good for "server independence", has a small projection payload, is super-easy, and is a cool demonstration of breeze. You have to manage the isPartial flag so you always know whether the session in cache is complete. That's not hard.
It could get more complicated if you needed multiple flavors of "partial entity" ... which seems to be where you are going. That was not an issue in CCJS.
John chose #3 for CCJS because it fit the application objectives.
That doesn't make it the right choice for every application. It may not be the right choice for you.
For example, if you always have a fast, low latency connection, then #1 may be your best choice. I really don't know.
I like the cast-to-entity approach myself because it is easy and works so well most of the time. I do think carefully about that choice before I make it.
Summary
You do not have to turn projection query results into entities
You can bind to projected data directly, without Knockout observable properties, if they are read-only
Make sure you have a good reason to convert projected data into (partial) entities.
CCJS has a good reason to convert projected query data into entities. Do you?
I have something like a Customer object with up to 50000 order in an ICollection<Orders>.
Assume the Custome being in the local cache, the orders not. How can i delete the Cutomer and all of its related orders without loading all of the Customer orders into the cache and marking them with setDeleted()?
What is the best practice here. I assume extending the public SaveResult SaveChanges(JObject saveBundle) method is the best way. Any other possibilities here on the client side like a flag delete_all_navigation_too()?
Thanks
I must suppose that you do not have and do not want cascade delete on your database. Personally, I'm "terrified" of deletes in general and try to avoid them. I prefer a soft delete (marking a record as inactive). But not everyone agrees or can follow suit
I would consider adding a Web API method (say "DeleteCustomerAndOrders") to your controller to do it. You can call any API method from your client, not just a Breeze method.
In recommending this, I'm assuming that this kind of thing is a relative rarity in your app. You don't need a general purpose deleter, a deleter that takes an array of parent object IDs, a deleter that will delete some child objects and not others, ... etc., etc.
Follow this path and you will have moved the problem from the client to the server. That's good: you didn't have to load the orders on the client. Now you have to get rid of them on the server. If you're using Entity Framework, you face the same challenge of deleting the orders without loading them. Check out Alex James' solution: Bulk-deleting in LINQ to Entities.
Simplest approach that I can come up with is to create a cascade delete constraint on the database so that when a customer is deleted all of its orders get deleted as well. Then simply delete the customer on the client and call 'SaveChanges'. In addition, since Breeze does not yet support client side 'cascaded' deletes ( we are considering this one), you will need to iterate over any client side orders that are already loaded and 'detach' them.