$orderby on One To Many Navigation Property in OData - odata

Question

Looking for way to do $orderby on a one to many navigation property using OData v4 under WebApi2. As an example, a classic Contact and Address scenario where there is a one to many.
public class Contact
{
public int ContactId { get; set; }
public String FirstName { get; set; }
public virtual ICollection<ContactAddress> Addresses { get; set; }
}
public class ContactAddress
{
public int ContactAddressId { get; set; }
public int ContactId { get; set; }
public virtual Contact Contact { get; set; }
public String AddressOne { get; set; }
public String StateCode { get; set; }
public bool IsPreferred { get; set; }
public String Type { get; set; }
public bool IsActive { get; set; }
}
Where we want to support composing queries that might include ordering by the preferred address StateCode, etc.
I realize that the following won't work as it doesn't terminate in a single value / object:
http://localhost:49380/odata/ODataContacts?$filter=LastName eq 'Smith'&$expand=Addresses&$orderby=Addresses/StateCode
But what I am trying to understand is what the options are to deal with this. It is such a fundamental scenario. In straight LINQ world it is pretty easy to build an expression that will do the appropriate sub filtering on Addresses to allow this. But would like to avoid special case as really trying to just support simple query composition through the OData URL / filter and orderby conventions.
What would be ideal would be something like:
http://localhost:49380/odata/ODataContacts?$filter=LastName eq 'Smith'&$expand=Addresses&$orderby=Addresses($filter=IsPreferred eq true)/StateCode
But this doesn't seem to be supported, at least in OData4 as it is in WebApi.
I have pondered a custom function, and per spec this should be supported in an $orderby expression. But have stumbled on how to implement this. The assumption would be something like:
http://localhost:49380/odata/ODataContacts?$filter=LastName eq 'Smith'&$expand=Addresses&$orderby=ContactService.PreferredAddress(ContactId=$it.ContactId)/StateCode
I have not been successful with this, and it strikes me this will all be "too late" to build the query expression for server side execution regardless, as it is really just a call to another endpoint.
I also stumbled upon some of the ModelBinding functions, many of which take lambda expressions, which seems promising. So for example:
builder.EntitySet<Contact>("ODataContacts")
.HasOptionalBinding<ContactAddress>(c => c.Addresses.FirstOrDefault(a => a.IsPreferred), "PreferredAddress");
Only I am not clear on what I then need to do on actual Entity or Controller to make this all work. I am getting a runtime exception as PreferredAddress doesn't really exist anywhere. But it would seem this is the approach that might work the best?
I can probably get things to work with a DTO/Queryable approach but this seems really unfortunate and a lot of extra work.
Is there a practical approach to this problem???