I have a number of logs like this:
Log.Information("Submitting order {#order}", order);
This log goes through RabbitMq -> LogStash -> Elastic and ends up generating a lot of fields (I assume one field for each propery). Eventually I have thousands and thousands of fields in Elastic which brings all kinds of problems.
If I specify the whole object as a parameter, it usually means I don't care much about having all its fields being parsed, I would be more than happy if it was stored as a single string object (but still serlialised as json). Is there a way to customise it in Serilog?
#flaxel's answer works well if you're happy to change the ToString() representation of your object. If you have already overriden ToString() or you don't want it to return a JSON string then consider one of the following options.
If you don't want to log the type as JSON all the time, consider just serializing the object when you log the message. This is the most explicit approach, and allows you to pick and choose which messages have the serialized form and which have the destructured form, but it might make your log statements quite verbose:
// Using Newtonsoft.Json to serialize.
Log.Information("Submitting order {Order}", JsonConvert.SerializeObject(order));
If you always want to serialize a type to JSON, you could register a destructuring policy for that specific type. This keeps your log statements concise and ensures that type is always serialized in the same way:
// When configuring your logger.
Log.Logger = new LoggerConfiguration()
.Destructure.ByTransforming<Order>(order => JsonConvert.SerializeObject(order))
// ...
// This will use the destructurer registered above, so will convert to a JSON string.
Log.Information("Submitting order {#Order}", order);
// These will still use the ToString() method.
Log.Information("Submitting order {Order}", order);
Log.Information("Submitting order {$Order}", order);
Another advantage of this approach is that if you want to change the way you're representing objects of that type, or if you want to revert to the default destructuring approach, you just have to change the policy used when configuring the logger (i.e. the lambda in the snippet above).
If your serialization approach is too complicated to fit in a lambda, or you want to use the same serialization approach for a large number of types, you could define your own IDestructuringPolicy and then register it in a similar way:
class SerializationPolicy : IDestructuringPolicy
{
public bool TryDestructure(object value, ILogEventPropertyValueFactory propertyValueFactory, out LogEventPropertyValue result)
{
// Check type of `value` and serialize if required.
}
}
// When configuring your logger.
Log.Logger = new LoggerConfiguration()
.Destructure.With<SerializationPolicy>()
// ...
I think you can force stringification with the $ operator. And I think you can then override the ToString method to adjust the output. There is also a short example in the documentation of serilog how to force stringification:
var unknown = new[] { 1, 2, 3 }
Log.Information("Received {$Data}", unknown);
And this is the output of the logging function:
Received "System.Int32[]"
Related
We have a Set<String> that we are printing to the page in Thymeleaf:
<h5>Unique Students names</h5>
<ol>
<div th:each="uniqueName: ${course.uniqueStudentsNames}">
<li th:text="${uniqueName}"></li>
</div>
</ol>
Since it is a Set<String> there is no order of the items in the set.
We like to convert the non-ordered Set<String> to an ordred List<String> so we will be able to print it by ABC. Since it is a UI issue, we don't like the backend to change its data structure.
An option we thought about is to create such a list in the Model:
model.addAttribute("course", course);
model.addAttribute("orderedList", /* convert Set to ordred List */);
But this seems very strange that the Model needs to handle such a UI issue.
Is there a way to do it in Thymeleaf?
Instantiate it as a TreeSet (you are probably using hashset?) and you have a set with the ordering behaviour you want. Set<String> (Set<anything> really) guarantees there won't be repeated items, but it does not imply anything about the order/unorder.
On the other side, thymeleaf, as a template engine, won't do such things as adding ordering behaviour to a collection that does not implement it. The most it will do is allow you to access methods of the collection; in the case of set: set utility methods.
If you no matter what want to use a non ordered set, you'd have to use something like javascript to manipulate the items on the client to order them.
Use the Thymeleaf utility method to convert the object to a List:
${#lists.toList(object)}.
Then perform a utility operation to sort as needed from there, optionally using your own Comparator.
The current Thymeleaf source code shows that the toList() method will check if the target object is an instance of Iterable. In your case, Set implements the interface.
To reference the external link:
if (target instanceof Iterable<?>) {
final List<Object> elements = new ArrayList<Object>(10);
for (final Object element : (Iterable<?>)target) {
elements.add(element);
}
return elements;
}
I want to build a wrapper around log4j2 to do the below:
1) There are around 6 mandatory fields like event_name, action, desc etc
2) Some fields, i want to make them use only certain values, like enum
3) log should be created in key value pairs for Splunk.
Below is my approach:
1) Created a class called CustomLogger accepeting the mandatory fields, logger and variable fields as key value
2) Users can call methods like below:
CustomLogger.info(logger, transactionId, app_name, event_name,
"inside the loop", "inside the loop of the sample app",
CustomLogger.Result.success, "looped in", "loop_count",
String.valueOf(i));
Method definition:
public static String log(LogLevel logLevel, Logger logger,
String transactionId, String app_name, String event_name,
String action, String desc, Result result, String reason,
String... addtnlFields)
Issues with the approach:
1) Not extending the log4j, not sure if this is the right way
2) need to pass the logger from every class. If that can be avoided
3) method and line number is lost since it is getting called from a different method
This will be widely used across my internal applications, so want to do it right. Is this approach ok or is there a better approach?
Take a look at the code generator attached to this Jira: https://issues.apache.org/jira/browse/LOG4J2-519
Perhaps you can use that as a base class? Gives you a slightly nicer API.
(I still need to update this to reflect some API changes in log4j-2.0-rc2...)
UPDATE
A different approach is to have a custom implementation of the Message interface defined in the log4j2 api module. Your custom message would have a constructor with all fields you define as required, and the toString method (and perhaps some other methods too) would format these fields as you require into key-value pairs.
I have an array of bytes in my model
public byte[] created_dt { get; set; }
It represents a timestamp value in the database.
In my view , I am referring it as #model.created_dt
but it is coming as system.byte[]
How to resolve this?
Try as this is just an example to achieve the functionality
#foreach(var a in model.created_dt){
<label>#a</label>
}
Judging by the behavior, this looks like ASP.NET and you are simply seeing the output of an implicit call to ToString() on the array (the default way of displaying anything that does not have a template defined). You will have to do something with the raw byte data and present it to the user.
Since you refer to "timestamp" and the property name might suggest a record creation time, you may want to write a helper method to translate this raw data to a DateTime which you could then format accordingly.
However, one of the following is most likely true:
It strikes me as odd that you are using raw binary to store what should otherwise be a datetime2 column. (Or it is a datetime in your database but you're doing something unorthodox to retrieve the value.)
Your property/column name of "created_dt" is a misnomer and it is really a timestamp (i.e. rowversion) column. In which case I don't think this is something a user would know what to do with, and it probably doesn't belong on the UI.
I work with a Web Service API that can pump through a generic type of Results that all offer certain basic information, most notably a unique ID. That unique ID tends to be--but is not required to be--a UUID defined by the sender, which is not always the same person (but IDs are unique across the system).
Fundamentally, the API results in something along the lines of this (written in Java, but the language should be irrelevant), where only the base interface represents common details:
interface Result
{
String getId();
}
class Result1 implements Result
{
public String getId() { return uniqueValueForInstance; }
public OtherType1 getField1() { /* ... */ }
public OtherType2 getField2() { /* ... */ }
}
class Result2 implements Result
{
public String getId() { return uniqueValueForInstance; }
public OtherType3 getField3() { /* ... */ }
}
It's important to note that each Result type may represent a completely different kind of information. Some of it cannot be correlated with other Results, and some of it can, whether or not they have identical types (e.g., Result1 may be able to be correlated with Result2, and therefore vice versa, but some ResultX may exist that cannot be correlated because it represents different information).
We are currently implementing a system that receives some of those Results and correlates them where possible, which generates a different Result object that is a container of what it correlated together:
class ContainerResult implements Result
{
public String getId() { return uniqueValueForInstance; }
public Collection<Result> getResults() { return containedResultsList; }
public OtherType4 getField4() { /* ... */ }
}
class IdContainerResult implements Result
{
public String getId() { return uniqueValueForInstance; }
public Collection<String> getIds() { return containedIdsList; }
public OtherType4 getField4() { /* ... */ }
}
These are two containers, which present different use cases. The first, ContainerResult, allows someone to receive the correlated details as well as the actual complete, correlated data. The second, IdContainerResult, sacrifices the complete listing in favor of bandwidth by only sending the associated IDs. The system doing the correlating is not necessarily the same as the client, and the client can receive Results that those IDs would represent, which is intended to allow them to show correlations on their system by simply receiving the IDs.
Now, my problem may be non-obvious to some, and it may be obvious to others: if I send only the ID as part of the IdContainerResult, then how does the client know how to match the Result on their end if they do not have a single ID-store? The types of data that are actually represented by each Result implementation lend themselves to being segregated when they cannot be correlated, which means that a single ID-store is unlikely in most situations without forcing a memory or storage burden.
The current solution that we have come up with entails creating a new type of ID, we'll call it TypedId, which combines the XML Namespace and XML Name from each Result with the Result's ID.
My main problem with that solution is that it requires either maintaining a mutable collection of types that is updated as they are discovered, or prior knowledge of all types so that the ID can be properly associated on any client's system. Unfortunately, I cannot come up with a better solution, but the current solution feels wrong.
Has anyone faced a similar situation where they want associate generic Results with its original type, particularly with the limitations of WSDLs in mind, and solved it in a cleaner way?
Here's my suggestion:
You want to have "the client know how to match the Result on their end". So include in your response an extra discriminator field called "RequestType", a String.
You want to avoid "maintaining a mutable collection of types that is updated as they are discovered, or prior knowledge of all types so that the ID can be properly associated on any client's system". Obviously, each client request call DOES know what area of processing the Result will relate to. So you can have the client pass the "RequestType" string in as part of the request. As long as the RequestType is a unique string for each different type of client request, your service can process and correlate it without hard-coding any knowledge.
Here's one possible example of java classes for request and response messages (i.e. not the actual service endpoint):
interface Request {
String getId();
String getRequestType();
// anything else ...
}
interface Result {
String getId();
String getRequestType();
}
class Result1 implements Result {
public String getId() { return uniqueValueForInstance; }
public OtherType1 getField1() { /* ... */ }
public OtherType2 getField2() { /* ... */ }
}
class Result2 implements Result {
public String getId() { return uniqueValueForInstance; }
public OtherType3 getField3() { /* ... */ }
}
Here's the gotcha. (2) and (3) above do not give a completely dynamic solution. You want your service to be able to return a flexible record structure relating to each different request. You have the following options:
4A) In XSD, declare Result as a singular strongly-typed variant record type, and in WSDL return Result from a single service endpoint and single operation. The XSD will still need to hardcode the values for the discriminator element when declaring variant record structure.
4B) In XSD, declare multiple strongly-typed unique types Result1, Result2, etc for each possible client request. In WSDL, have a multiple uniquely named operations to return each one of these. These operations can be across one or many service endpoints - or even across multiple WSDLs. While this avoids hard coding the request type as a specific field per se, it is not actually a generic client-independent solution because you are still explicitly hard-coding to discriminate each request type by creating a uniquely name for each result type and each operation. So any apparent dynamism is a mirage.
4C) In XSD, define a flexible generic data structure that is not variant, but has plenty of generally named fields that could be able to handle all possible results required. Example fields could be "stringField1", "stringField2", "integerField1", "dateField1058", etc. i.e. use extremely weak typing and put the burden on the client to magically know what data is in each field. This option may be very generic, but it is usually considered terrible practice. It is inelegant, pretty unreadable, error prone and has limitations/assumptions built in anyway - how do you know you have enough generic fields included? In your case, (4A) is probably the best option.
4D) Use flexible XSD schema design tactics - type substitutability and use of "any" element. See http://www.xfront.com/ExtensibleContentModels.html.
4E) Use the #Produces #SomeQualifier annotations against your own factory class method which creates a high level type. This tells CDI to always use this method to construct the specificied bean type & qualifier. Your factory method can have fancy logic to decide which specific low-level type to construct upon each call. #SomeQualifier can have additional parameters to give guidance towards selecting the type. This potentially reducing the number of qualifiers to just one.
If you use (4D) you will have a flexible service endpoint design that can deal with changing requirements quite effectively. BUT your service implementation still needs to implement the flexible behaviour to decide which results fields to return for each request. Fact is, if you have a logical requirement for varying data structures, your code must know how to process these data structures for each separate request, so must depend on some form of RequestType / unique operation names to discriminate. Any goal of completely dynamic processing (without adapting to each client's needs for results data) is over-ambitious.
I'm working on an application at the moment in ASP.NET MVC which has a number of look-up tables, all of the form
LookUp {
Id
Text
}
As you can see, this just maps the Id to a textual value. These are used for things such as Colours. I now have a number of these, currently 6 and probably soon to be more.
I'm trying to put together an API that can be used via AJAX to allow the user to add/list/remove values from these lookup tables, so for example I could have something like:
http://example.com/Attributes/Colours/[List/Add/Delete]
My current problem is that clearly, regardless of which lookup table I'm using, everything else happens exactly the same. So really there should be no repetition of code whatsoever.
I currently have a custom route which points to an 'AttributeController', which figures out the attribute/look-up table in question based upon the URL (ie http://example.com/Attributes/Colours/List would want the 'Colours' table). I pass the attribute (Colours - a string) and the operation (List/Add/Delete), as well as any other parameters required (say "Red" if I want to add red to the list) back to my repository where the actual work is performed.
Things start getting messy here, as at the moment I've resorted to doing a switch/case on the attribute string, which can then grab the Linq-to-Sql entity corresponding to the particular lookup table. I find this pretty dirty though as I find myself having to write the same operations on each of the look-up entities, ugh!
What I'd really like to do is have some sort of mapping, which I could simply pass in the attribute name and get out some form of generic lookup object, which I could perform the desired operations on without having to care about type.
Is there some way to do this to my Linq-To-Sql entities? I've tried making them implement a basic interface (IAttribute), which simply specifies the Id/Text properties, however doing things like this fails:
System.Data.Linq.Table<IAttribute> table = GetAttribute("Colours");
As I cannot convert System.Data.Linq.Table<Colour> to System.Data.Linq.Table<IAttribute>.
Is there a way to make these look-up tables 'generic'?
Apologies that this is a bit of a brain-dump. There's surely imformation missing here, so just let me know if you'd like any further details. Cheers!
You have 2 options.
Use Expression Trees to dynamically create your lambda expression
Use Dynamic LINQ as detailed on Scott Gu's blog
I've looked at both options and have successfully implemented Expression Trees as my preferred approach.
Here's an example function that i created: (NOT TESTED)
private static bool ValueExists<T>(String Value) where T : class
{
ParameterExpression pe = Expression.Parameter(typeof(T), "p");
Expression value = Expression.Equal(Expression.Property(pe, "ColumnName"), Expression.Constant(Value));
Expression<Func<T, bool>> predicate = Expression.Lambda<Func<T, bool>>(value, pe);
return MyDataContext.GetTable<T>().Where(predicate).Count() > 0;
}
Instead of using a switch statement, you can use a lookup dictionary. This is psuedocode-ish, but this is one way to get your table in question. You'll have to manually maintain the dictionary, but it should be much easier than a switch.
It looks like the DataContext.GetTable() method could be the answer to your problem. You can get a table if you know the type of the linq entity that you want to operate upon.
Dictionary<string, Type> lookupDict = new Dictionary<string, Type>
{
"Colour", typeof(MatchingLinqEntity)
...
}
Type entityType = lookupDict[AttributeFromRouteValue];
YourDataContext db = new YourDataContext();
var entityTable = db.GetTable(entityType);
var entity = entityTable.Single(x => x.Id == IdFromRouteValue);
// or whatever operations you need
db.SubmitChanges()
The Suteki Shop project has some very slick work in it. You could look into their implementation of IRepository<T> and IRepositoryResolver for a generic repository pattern. This really works well with an IoC container, but you could create them manually with reflection if the performance is acceptable. I'd use this route if you have or can add an IoC container to the project. You need to make sure your IoC container supports open generics if you go this route, but I'm pretty sure all the major players do.