I work with a Web Service API that can pump through a generic type of Results that all offer certain basic information, most notably a unique ID. That unique ID tends to be--but is not required to be--a UUID defined by the sender, which is not always the same person (but IDs are unique across the system).
Fundamentally, the API results in something along the lines of this (written in Java, but the language should be irrelevant), where only the base interface represents common details:
interface Result
{
String getId();
}
class Result1 implements Result
{
public String getId() { return uniqueValueForInstance; }
public OtherType1 getField1() { /* ... */ }
public OtherType2 getField2() { /* ... */ }
}
class Result2 implements Result
{
public String getId() { return uniqueValueForInstance; }
public OtherType3 getField3() { /* ... */ }
}
It's important to note that each Result type may represent a completely different kind of information. Some of it cannot be correlated with other Results, and some of it can, whether or not they have identical types (e.g., Result1 may be able to be correlated with Result2, and therefore vice versa, but some ResultX may exist that cannot be correlated because it represents different information).
We are currently implementing a system that receives some of those Results and correlates them where possible, which generates a different Result object that is a container of what it correlated together:
class ContainerResult implements Result
{
public String getId() { return uniqueValueForInstance; }
public Collection<Result> getResults() { return containedResultsList; }
public OtherType4 getField4() { /* ... */ }
}
class IdContainerResult implements Result
{
public String getId() { return uniqueValueForInstance; }
public Collection<String> getIds() { return containedIdsList; }
public OtherType4 getField4() { /* ... */ }
}
These are two containers, which present different use cases. The first, ContainerResult, allows someone to receive the correlated details as well as the actual complete, correlated data. The second, IdContainerResult, sacrifices the complete listing in favor of bandwidth by only sending the associated IDs. The system doing the correlating is not necessarily the same as the client, and the client can receive Results that those IDs would represent, which is intended to allow them to show correlations on their system by simply receiving the IDs.
Now, my problem may be non-obvious to some, and it may be obvious to others: if I send only the ID as part of the IdContainerResult, then how does the client know how to match the Result on their end if they do not have a single ID-store? The types of data that are actually represented by each Result implementation lend themselves to being segregated when they cannot be correlated, which means that a single ID-store is unlikely in most situations without forcing a memory or storage burden.
The current solution that we have come up with entails creating a new type of ID, we'll call it TypedId, which combines the XML Namespace and XML Name from each Result with the Result's ID.
My main problem with that solution is that it requires either maintaining a mutable collection of types that is updated as they are discovered, or prior knowledge of all types so that the ID can be properly associated on any client's system. Unfortunately, I cannot come up with a better solution, but the current solution feels wrong.
Has anyone faced a similar situation where they want associate generic Results with its original type, particularly with the limitations of WSDLs in mind, and solved it in a cleaner way?
Here's my suggestion:
You want to have "the client know how to match the Result on their end". So include in your response an extra discriminator field called "RequestType", a String.
You want to avoid "maintaining a mutable collection of types that is updated as they are discovered, or prior knowledge of all types so that the ID can be properly associated on any client's system". Obviously, each client request call DOES know what area of processing the Result will relate to. So you can have the client pass the "RequestType" string in as part of the request. As long as the RequestType is a unique string for each different type of client request, your service can process and correlate it without hard-coding any knowledge.
Here's one possible example of java classes for request and response messages (i.e. not the actual service endpoint):
interface Request {
String getId();
String getRequestType();
// anything else ...
}
interface Result {
String getId();
String getRequestType();
}
class Result1 implements Result {
public String getId() { return uniqueValueForInstance; }
public OtherType1 getField1() { /* ... */ }
public OtherType2 getField2() { /* ... */ }
}
class Result2 implements Result {
public String getId() { return uniqueValueForInstance; }
public OtherType3 getField3() { /* ... */ }
}
Here's the gotcha. (2) and (3) above do not give a completely dynamic solution. You want your service to be able to return a flexible record structure relating to each different request. You have the following options:
4A) In XSD, declare Result as a singular strongly-typed variant record type, and in WSDL return Result from a single service endpoint and single operation. The XSD will still need to hardcode the values for the discriminator element when declaring variant record structure.
4B) In XSD, declare multiple strongly-typed unique types Result1, Result2, etc for each possible client request. In WSDL, have a multiple uniquely named operations to return each one of these. These operations can be across one or many service endpoints - or even across multiple WSDLs. While this avoids hard coding the request type as a specific field per se, it is not actually a generic client-independent solution because you are still explicitly hard-coding to discriminate each request type by creating a uniquely name for each result type and each operation. So any apparent dynamism is a mirage.
4C) In XSD, define a flexible generic data structure that is not variant, but has plenty of generally named fields that could be able to handle all possible results required. Example fields could be "stringField1", "stringField2", "integerField1", "dateField1058", etc. i.e. use extremely weak typing and put the burden on the client to magically know what data is in each field. This option may be very generic, but it is usually considered terrible practice. It is inelegant, pretty unreadable, error prone and has limitations/assumptions built in anyway - how do you know you have enough generic fields included? In your case, (4A) is probably the best option.
4D) Use flexible XSD schema design tactics - type substitutability and use of "any" element. See http://www.xfront.com/ExtensibleContentModels.html.
4E) Use the #Produces #SomeQualifier annotations against your own factory class method which creates a high level type. This tells CDI to always use this method to construct the specificied bean type & qualifier. Your factory method can have fancy logic to decide which specific low-level type to construct upon each call. #SomeQualifier can have additional parameters to give guidance towards selecting the type. This potentially reducing the number of qualifiers to just one.
If you use (4D) you will have a flexible service endpoint design that can deal with changing requirements quite effectively. BUT your service implementation still needs to implement the flexible behaviour to decide which results fields to return for each request. Fact is, if you have a logical requirement for varying data structures, your code must know how to process these data structures for each separate request, so must depend on some form of RequestType / unique operation names to discriminate. Any goal of completely dynamic processing (without adapting to each client's needs for results data) is over-ambitious.
Related
I'm inserting data into Azure CosmosDB via FSharp.ComosDb. Here is the record type that I write in the DB:
[<CLIMutable>]
type DbType =
{ id: Guid
Question: string
Answer: int }
The persistence layer works fine but I face an inelegant redundancy. The record I'm inserting originates from the Data Transfer Object (DTO) with the following shape:
type DataType =
{ QuestionId: Guid
Question: string
Answer: int }
CosmosDb accepts only records with a lowercase id. Is there any way to derive the DbType from DataType or I have to define DbType from scratch?
Is there anything a la copy and update record expression record2 = { record1 with id = record1.QuestionId } but at the type level?
There's no type-level way of deriving one record type from another the way you describe, you can however get reasonably close with the addition of anonymous records in F# 4.6.
type DataType =
{ QuestionId: Guid
Question: string
Answer: int }
let example =
{ QuestionId = Guid.NewGuid()
Question = "The meaning of life etc."
Answer = 42 }
let extended =
{| example with id = example.QuestionId |}
This gives you a value of an anonymous record type with an added field, and may be well suited to your scenario, however it's unwieldy to write code against such type once it leaves the scope of the function you create it in.
If all you care is how this single field is named - serialization libraries usually have ways of providing aliases for field names (like Newtonsoft.Json's JsonProperty attribute). Note that this might be obscured from you by the CosmosDb library you're using, which I'm not familiar with.
Another more involved approach is to use generic envelope types so that the records you persist have a uniform data store specific header across your application:
type Envelope<'record> =
{
id: string
// <other fields as needed>
payload: 'record
}
In that case the envelope contains the fields that your datastore expects to be there to fulfill the contract (+ any application specific metadata you might find useful, like timestamps, event types, versions, whatnot) and spares you the effort of defining data store specific versions of each type you want to persist.
Note that it is still a good idea in general to decouple the internal domain types from the representation you use for storage for maintainability reasons.
I have a number of logs like this:
Log.Information("Submitting order {#order}", order);
This log goes through RabbitMq -> LogStash -> Elastic and ends up generating a lot of fields (I assume one field for each propery). Eventually I have thousands and thousands of fields in Elastic which brings all kinds of problems.
If I specify the whole object as a parameter, it usually means I don't care much about having all its fields being parsed, I would be more than happy if it was stored as a single string object (but still serlialised as json). Is there a way to customise it in Serilog?
#flaxel's answer works well if you're happy to change the ToString() representation of your object. If you have already overriden ToString() or you don't want it to return a JSON string then consider one of the following options.
If you don't want to log the type as JSON all the time, consider just serializing the object when you log the message. This is the most explicit approach, and allows you to pick and choose which messages have the serialized form and which have the destructured form, but it might make your log statements quite verbose:
// Using Newtonsoft.Json to serialize.
Log.Information("Submitting order {Order}", JsonConvert.SerializeObject(order));
If you always want to serialize a type to JSON, you could register a destructuring policy for that specific type. This keeps your log statements concise and ensures that type is always serialized in the same way:
// When configuring your logger.
Log.Logger = new LoggerConfiguration()
.Destructure.ByTransforming<Order>(order => JsonConvert.SerializeObject(order))
// ...
// This will use the destructurer registered above, so will convert to a JSON string.
Log.Information("Submitting order {#Order}", order);
// These will still use the ToString() method.
Log.Information("Submitting order {Order}", order);
Log.Information("Submitting order {$Order}", order);
Another advantage of this approach is that if you want to change the way you're representing objects of that type, or if you want to revert to the default destructuring approach, you just have to change the policy used when configuring the logger (i.e. the lambda in the snippet above).
If your serialization approach is too complicated to fit in a lambda, or you want to use the same serialization approach for a large number of types, you could define your own IDestructuringPolicy and then register it in a similar way:
class SerializationPolicy : IDestructuringPolicy
{
public bool TryDestructure(object value, ILogEventPropertyValueFactory propertyValueFactory, out LogEventPropertyValue result)
{
// Check type of `value` and serialize if required.
}
}
// When configuring your logger.
Log.Logger = new LoggerConfiguration()
.Destructure.With<SerializationPolicy>()
// ...
I think you can force stringification with the $ operator. And I think you can then override the ToString method to adjust the output. There is also a short example in the documentation of serilog how to force stringification:
var unknown = new[] { 1, 2, 3 }
Log.Information("Received {$Data}", unknown);
And this is the output of the logging function:
Received "System.Int32[]"
I am using DB4o and want to make sure I get one unique object back when I use code like this:
public List<Object> getListOfObjects(final Object o){
List<Object> result = db.query(new Predicate<Object>(){
#Override
public boolean match (Object arg0){
if(arg0.equals(o)){
return true;
}
else{
return false;
}
});
return result;
}
The List object "result" desirably should have no more than 1 element. However, isn't it possible that Java creates objects with the same identities during different runtimes (different JVMs)? If that could occur then it would mess up my database.
Surely there must be an answer to whether or not Java objects can have the same identities across JVMs.
-Alex
If you overwrite the .equals() method of your object, then its easy to have multiple instances of a object which is equals. The whole purpose of the equal() method is to compare two objects about the 'semantic/content" equality. It does not quarantine any uniqueness.
Now if you do not override any equal method, then the object identity is compared. (like using the == operator). The identity is unique in a JVM and there are never two objects with the same identity.
Btw/Offtopic: If you store thousands of object in db4o and use your query, it will be quite slow. More about that here.
I'm working on an application at the moment in ASP.NET MVC which has a number of look-up tables, all of the form
LookUp {
Id
Text
}
As you can see, this just maps the Id to a textual value. These are used for things such as Colours. I now have a number of these, currently 6 and probably soon to be more.
I'm trying to put together an API that can be used via AJAX to allow the user to add/list/remove values from these lookup tables, so for example I could have something like:
http://example.com/Attributes/Colours/[List/Add/Delete]
My current problem is that clearly, regardless of which lookup table I'm using, everything else happens exactly the same. So really there should be no repetition of code whatsoever.
I currently have a custom route which points to an 'AttributeController', which figures out the attribute/look-up table in question based upon the URL (ie http://example.com/Attributes/Colours/List would want the 'Colours' table). I pass the attribute (Colours - a string) and the operation (List/Add/Delete), as well as any other parameters required (say "Red" if I want to add red to the list) back to my repository where the actual work is performed.
Things start getting messy here, as at the moment I've resorted to doing a switch/case on the attribute string, which can then grab the Linq-to-Sql entity corresponding to the particular lookup table. I find this pretty dirty though as I find myself having to write the same operations on each of the look-up entities, ugh!
What I'd really like to do is have some sort of mapping, which I could simply pass in the attribute name and get out some form of generic lookup object, which I could perform the desired operations on without having to care about type.
Is there some way to do this to my Linq-To-Sql entities? I've tried making them implement a basic interface (IAttribute), which simply specifies the Id/Text properties, however doing things like this fails:
System.Data.Linq.Table<IAttribute> table = GetAttribute("Colours");
As I cannot convert System.Data.Linq.Table<Colour> to System.Data.Linq.Table<IAttribute>.
Is there a way to make these look-up tables 'generic'?
Apologies that this is a bit of a brain-dump. There's surely imformation missing here, so just let me know if you'd like any further details. Cheers!
You have 2 options.
Use Expression Trees to dynamically create your lambda expression
Use Dynamic LINQ as detailed on Scott Gu's blog
I've looked at both options and have successfully implemented Expression Trees as my preferred approach.
Here's an example function that i created: (NOT TESTED)
private static bool ValueExists<T>(String Value) where T : class
{
ParameterExpression pe = Expression.Parameter(typeof(T), "p");
Expression value = Expression.Equal(Expression.Property(pe, "ColumnName"), Expression.Constant(Value));
Expression<Func<T, bool>> predicate = Expression.Lambda<Func<T, bool>>(value, pe);
return MyDataContext.GetTable<T>().Where(predicate).Count() > 0;
}
Instead of using a switch statement, you can use a lookup dictionary. This is psuedocode-ish, but this is one way to get your table in question. You'll have to manually maintain the dictionary, but it should be much easier than a switch.
It looks like the DataContext.GetTable() method could be the answer to your problem. You can get a table if you know the type of the linq entity that you want to operate upon.
Dictionary<string, Type> lookupDict = new Dictionary<string, Type>
{
"Colour", typeof(MatchingLinqEntity)
...
}
Type entityType = lookupDict[AttributeFromRouteValue];
YourDataContext db = new YourDataContext();
var entityTable = db.GetTable(entityType);
var entity = entityTable.Single(x => x.Id == IdFromRouteValue);
// or whatever operations you need
db.SubmitChanges()
The Suteki Shop project has some very slick work in it. You could look into their implementation of IRepository<T> and IRepositoryResolver for a generic repository pattern. This really works well with an IoC container, but you could create them manually with reflection if the performance is acceptable. I'd use this route if you have or can add an IoC container to the project. You need to make sure your IoC container supports open generics if you go this route, but I'm pretty sure all the major players do.
I've got a repository using LINQ for modelling the data that has a whole bunch of functions for getting data out. A very common way of getting data out is for things such as drop down lists. These drop down lists can vary. If we're creating something we usually have a drop down list with all entries of a certain type, which means I need a function available which filters by the type of entity. We also have pages to filter data, the drop down lists only contain entries that currently are used, so I need a filter that requires used entries. This means there are six different queries to get the same type of data out.
The problem with defining a function for each of these is that there'd be six functions at least for every type of output, all in one repository. It gets very large, very quick. Here's something like I was planning to do:
public IEnumerable<Supplier> ListSuppliers(bool areInUse, bool includeAllOption, int contractTypeID)
{
if (areInUse && includeAllOption)
{
}
else if (areInUse)
{
}
else if (includeAllOption)
{
}
}
Although "areInUse" doesn't seem very English friendly, I'm not brilliant with naming. As you can see, logic resides in my data access layer (repository) which isn't friendly. I could define separate functions but as I say, it grows quite quick.
Could anyone recommend a good solution?
NOTE: I use LINQ for entities only, I don't use it to query. Please don't ask, it's a constraint on the system not specified by me. If I had the choice, I'd use LINQ, but I don't unfortunately.
Have your method take a Func<Supplier,bool> which can be used in Where clause so that you can pass it in any type of filter than you would like to construct. You can use a PredicateBuilder to construct arbitrarily complex functions based on boolean operations.
public IEnumerable<Supplier> ListSuppliers( Func<Supplier,bool> filter )
{
return this.DataContext.Suppliers.Where( filter );
}
var filter = PredicateBuilder.False<Supplier>();
filter = filter.Or( s => s.IsInUse ).Or( s => s.ContractTypeID == 3 );
var suppliers = repository.ListSuppliers( filter );
You can implement
IEnumerable<Supplier> GetAllSuppliers() { ... }
end then use LINQ on the returned collection. This will retrieve all suppliers from the database that are then filtered using LINQ.
Assuming you are using LINQ to SQL you can also implement
IQueryable<Supplier> GetAllSuppliers() { ... }
end then use LINQ on the returned collection. This will only retrieve the necessary suppliers from the database when the collection is enumerated. This is very powerful and there are also some limits to the LINQ you can use. However, the biggest problem is that you are able to drill right through your data-access layer and into the database using LINQ.
A query like
var query = from supplier in repository.GetAllSuppliers()
where suppliers.Name.StartsWith("Foo") select supplier;
will map into SQL similar to this when it is enumerated
SELECT ... WHERE Name LIKE 'Foo%'