Golang XML Unmarshal and time.Time fields - xml-parsing

I have XML data I am retrieving via a REST API that I am unmarshal-ing into a GO struct. One of the fields is a date field, however the date format returned by the API does not match the default time.Time parse format and thus the unmarshal fails.
Is there any way to specify to the unmarshal function which date format to use in the time.Time parsing? I'd like to use properly defined types and using a string to hold a datetime field feels wrong.
Sample struct:
type Transaction struct {
Id int64 `xml:"sequencenumber"`
ReferenceNumber string `xml:"ourref"`
Description string `xml:"description"`
Type string `xml:"type"`
CustomerID string `xml:"namecode"`
DateEntered time.Time `xml:"enterdate"` //this is the field in question
Gross float64 `xml:"gross"`
Container TransactionDetailContainer `xml:"subfile"`
}
The date format returned is "yyyymmdd".

I had the same problem.
time.Time doesn't satisfy the xml.Unmarshaler interface. And you can not specify a date fomat.
If you don't want to handle the parsing afterward and you prefer to let the xml.encoding do it, one solution is to create a struct with an anonymous time.Time field and implement your own UnmarshalXML with your custom date format.
type Transaction struct {
//...
DateEntered customTime `xml:"enterdate"` // use your own type that satisfies UnmarshalXML
//...
}
type customTime struct {
time.Time
}
func (c *customTime) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
const shortForm = "20060102" // yyyymmdd date format
var v string
d.DecodeElement(&v, &start)
parse, err := time.Parse(shortForm, v)
if err != nil {
return err
}
*c = customTime{parse}
return nil
}
If your XML element uses an attribut as a date, you have to implement UnmarshalXMLAttr the same way.
See http://play.golang.org/p/EFXZNsjE4a

From what I have read the encoding/xml has some known issues that have been put off until a later date...
To get around this issue, instead of using the type time.Time use string and handle the parsing afterwards.
I had quite a bit of trouble getting time.Parse to work with dates in the following format: "Fri, 09 Aug 2013 19:39:39 GMT"
Oddly enough I found that "net/http" has a ParseTime function that takes a string that worked perfectly...
http://golang.org/pkg/net/http/#ParseTime

I've implemented a xml dateTime format conforming a spec, you can find it on GitHub: https://github.com/datainq/xml-date-time
You can find XML dateTime in W3C spec

A niche technique to override the JSON marshaling/unmarshaling of select keys in a struct is also applicable to XML marshaling/unmarshaling, with minimal modifications. The idea remains the same: Alias the original struct to hide the Unmarshal method, and embed the alias in another struct where individual fields may be overridden to allow the default unmarshaling method to apply.
In your example, following the idea above, I'd do it like this:
type TransactionA Transaction
type TransactionS struct {
*TransactionA
DateEntered string `xml:"enterdate"`
}
func (t *Transaction) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
const shortForm = "20060102" // yyyymmdd date format
var s = &TransactionS{TransactionA: (*TransactionA)(t)}
d.DecodeElement(s, &start)
t.DateEntered, _ = time.Parse(shortForm, s.DateEntered)
return nil
}
The smartest point of this method is, by sending TransactionS into DecodeElement, all fields of t gets populated through the embedded alias type *TransactionA, except for those overridden in the S-type. You can then process overridden struct members as you wish.
This approach scales very well if you have multiple fields of different types that you want to handle in a custom way, plus the benefit that you don't introduce an otherwise useless customType that you have to convert over and over again.

const shortForm = "20060102" // yyyymmdd date format
It is unreadable. But it is right in Go. You can read the source in http://golang.org/src/time/format.go

Related

Flutter/Dart DateTime parsing UTC and converting to local

I am trying to parse a UTC Date string to DateTime and then parse it to local, however I am having troubles with converting it to the local time. In the UK it should be plus one, however when I print .isUtc it returns as false.
This is what I have now:
print(widget.asset.purchaseDate);
DateTime temp = DateTime.parse(widget.asset.purchaseDate);
print(temp.toLocal());
I/flutter (5434): 2020-05-07 21:29:00
I/flutter (5434): 2020-05-07 21:29:00.000
You need to indicate a timezone to DateTime.parse, otherwise it assumes local time. From the dartdoc:
An optional time-zone offset part, possibly separated from the
previous by a space. The time zone is either 'z' or 'Z', or it is a
signed two digit hour part and an optional two digit minute part.
Since you know your string represents UTC, you can tell the parser by adding the Z suffix.
var temp = DateTime.parse(widget.asset.purchaseDate + 'Z');
print(temp.isUtc); // prints true
While #Richard Heap's answer stands I'd like DateTime.parseUtc() to exist.
If there will be a day when dart allows static extension methods here is an implementation that would work:
extension DateTimeExtension on DateTime {
static DateTime parseUtc(String formattedDate) => DateTime.parse('${formattedDate}z');
static DateTime? tryParseUtc(String? formattedDate) {
if (formattedDate != null) {
return DateTime.tryParse('${formattedDate}z');
}
return null;
}
}
Currently you'd have to use it with the extension classes name, like so:
DateTimeExtension.parseUtc(someDate)
or
DateTimeExtension.tryParseUtc(someOtherDate)
I think the latter is more useful where someOtherDate is nullable

Attempting to map dates to index in ElasticSearch

I am using ElasticSearch and attempting to create an index with a class that has a dynamic type property in it. This property may have strings or dates in it. When indexing, I have been using the code below:
dynamic instance = MyObject.GetDynamicJson();
var indexResponse = client.Index((object) instance, i=>i
.Index("myIndex")
.Type("MyObject")
);
Here's the code for GetDynamicJson().
MyObject has only Name and Value as properties. (apologies, I've had issues in the past with Elastic choking on json without all the quotes, which I have escaped with \ characters):
String json = "{ \"Name\":\" + Name + "\",\"DateValue\":\"";
try {
var date = DateTime.parse(Value);
json += DateTime.ToString("yyyy/MM/dd HH:mm:ss Z") + "\", \"Value\":\"\"}";
} catch { //If the DateTime parse fails, DateValue is empty and I'll have text in Value
json += "\",\"Value\":\"" + Value + "\"}";
}
return json;
For some reason it doesn't seem to like the string in DateValue and I definitely don't know why it's leaving out that property entirely in the error:
For whatever reason, ElasticSearch is completely dumping the DateValue property, doesn't seem to see the DateValue property at all.
I'm getting the error:
{"name":"CreatedDate","value":"2017-11-07T13:37:11.4340238-06:00"}
[indices:data/write/bulk[s][p]]"}],"type":"class_cast_exception","reason":"org.elasticsearch.index.mapper.TextFieldMapper cannot be cast to org.elasticsearch.index.mapper.DateFieldMapper"},"status":500
Later note: I have changed the index creator method to update the mapping. I added a third field to the Object, so now it has properties: Name, Value, DateValue:
public static void CreateRecordsIndex(ElasticClient client)
{
client.CreateIndex("myIndex", i => i
.Settings(s => s
.NumberOfShards(2)
.NumberOfReplicas(0)
)
.Mappings(x => x
.Map<MyObject>(m => m.AutoMap())));
}
Now, it is successfully mapping and creating a property each time, but it still seems to drop the property I am sending it in the json. It just sets them all to the default datetime: "dateValue": "0001-01-01T00:00:00". This is strange, because when making the dynamic instance I send to Elastic, I use only the MyObject.GetDynamicJson() method to build it. I no longer get the mapping error, but Elastic still seems oblivious to "dateValue":"some date here" in the object when it is set.
OK, I got rid of the dynamic object type (ultimately I wasn't actually getting data from the json method, I had a typo and Elastic was getting the original object directly - it's a wonder it was still handling it). So I let Elastic do the parse using its mapping. In order to do that, I first updated MyObject to include multiple properties, one for each type the incoming property could be (I am only handling text and dates in this case). For the DateValue property of MyObject, I have this:
public DateTime DateValue {
get
{
try
{
return DateTime.Parse(Value);
} catch
{
return new DateTime();
}
}
set
{
try {
DateValue = value;
} catch
{
DateValue = new DateTime();
}
}
}
Now, if Value is a date, my DateValue field will be set. Otherwise it'll have the default date (a very early date "0001-01-01T00:00:00"). This way, I can later search both for text against that dynamic field, or if a date is set, I can do date and date range queries against it (technically they end up in two different fields, but coming from the same injested data).
Key to this is having the index mapping setup, as you can see in this method from the question:
public static void CreateRecordsIndex(ElasticClient client)
{
client.CreateIndex("myIndex", i => i
.Settings(s => s
.NumberOfShards(2)
.NumberOfReplicas(0)
)
.Mappings(x => x
.Map<MyObject>(m => m.AutoMap())));
}
In order to recreate the index with the updated MyObject, I did this:
if (client.IndexExists("myIndex").Exists)
{
client.DeleteIndex("myIndex");
CreateRecordsIndex(client); //Goes to the method above
}

How to parse a String of "hh:mm:ss.SSS" into a DateTime

The format of "new DateTime.now()" will print following output:
2015-05-20 07:34:43.018
By having only the time as a String in the correct format ("07:34:43.018"), how do I parse the time to a DateTime object? The usage of the intl package does not support the mentioned format AFAIK.
By prefixing the Date in front of the time, DateTime would be able to parse the given String.
DateTime parse(String timeValue) {
var prefix = '0000-01-01T';
return DateTime.parse(prefix + timeValue);
}
If you also only want to display the time afterwards, format your DateTime variable accordingly:
String format(DateTime value) {
return "${value.hour}:${value.minute}:${value.second}.${value.millisecond}";
}
I've mainly had to parse the String for calculations (difference) with other time values.

datetime utc handling with DateTimeZoneHandling

i am trying to figure out why my dates on the client are different than my dates on the server... down below i have a breezeconfig class that i thought would allow my server side "unspecified" dates to be serialized as "utc"... however a date of "2011-08-11" is still being serialized as "2011-08-11T00:00:00.000" which i believe is missing the "Z" at the end in order to signify utc... any ideas? thank you
public class BreezeConfig : Breeze.WebApi.BreezeConfig
{
protected override Newtonsoft.Json.JsonSerializerSettings CreateJsonSerializerSettings()
{
var ret = base.CreateJsonSerializerSettings();
ret.DateTimeZoneHandling = Newtonsoft.Json.DateTimeZoneHandling.Utc;
return ret;
}
}
Setting ret.DateTimeZoneHandling = Newtonsoft.Json.DateTimeZoneHandling.Utc; doesn't work as expected because ret created by the base Breeze.WebApi.BreezeConfig.CreateJsonSerializerSettings() adds IsoDateTimeConverter to the ret.Converters list. By default IsoDateTimeConverter will not add 'Z' to the output string if DateTime.Kind is DateTimeKind.Unspecified. Like Jay Trabant answered -
breeze.js fixes this client-side.
With this knowledge it's ease to make your server return all dates as UTC:
var isoDateTimeConverter = ret.Converters.OfType<Newtonsoft.Json.Converters.IsoDateTimeConverter>().Single();
isoDateTimeConverter.DateTimeStyles = System.Globalization.DateTimeStyles.AssumeUniversal;
Because there is a bug in IsoDateTimeConverter - Issue with DateTimeStyles, it's better to completely remove IsoDateTimeConverter from the Converters collection and set DateTimeZoneHandling to Utc:
public class CustomBreezeConfig : Breeze.ContextProvider.BreezeConfig
{
protected override Newtonsoft.Json.JsonSerializerSettings CreateJsonSerializerSettings()
{
var settings = base.CreateJsonSerializerSettings();
settings.DateTimeZoneHandling = Newtonsoft.Json.DateTimeZoneHandling.Utc;
var isoDateTimeConverter = settings.Converters.OfType<Newtonsoft.Json.Converters.IsoDateTimeConverter>().Single();
settings.Converters.Remove(isoDateTimeConverter);
return settings;
}
}
My guess is that your server datatype is a DateTime ( not a DateTime2 or DateTimeOffset), and as such has no explicit timezone information.
Breeze does not manipulate the datetimes going to and from the server in any way EXCEPT to add a UTZ timezone specifier to any dates returned from the server that do not already have one. This is only done because different browsers interpret dates without a timezone specifier differently and we want consistency between browsers.
This is discussed in more detail in the answer posted here. breezejs: date is not set to the right time

Multiple-types decoder in golang

I have an XML document. Some fields have custom format. Example:
<document>
<title>hello world</title>
<lines>
line 1
line 2
line 3
</lines>
</document>
I want to import it into structure like:
type Document struct {
Title string `xml:"title"`
Lines []string `xml:"lines"`
}
Is there some way how to implement custom decoder, which will split lines string into array of lines (["line 1", "line 2", "line 3"])?
Its possible to make Lines field a string type and make split after xml import, but it doesn't seems to be very elegant solution. Is there any way i can define custom decoder for line spliting and combine it with xml decoder?
You can achieve this by defining a new type that conforms to the xml.Unmarshaler interface. So rather than making Lines a []string, declare a new type with an appropriate UnmarshalXML method. For instance:
type Lines []string
func (l *Lines) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
var content string
if err := d.DecodeElement(&content, &start); err != nil {
return err
}
*l = strings.Split(content, "\n")
return nil
}
You can see a full example here: http://play.golang.org/p/3SBu3bOGjR
If you want to support encoding this type too, you can implement the MarshalXML method in a similar fashion (construct the string content you want and pass that to the encoder).
Here is a spelled out example of what CSE is suggesting:
type Document struct {
Title string `xml:"title"`
LineData string `xml:"lines"`
}
func (d *Document)Lines() []string {
return strings.Split(d.LineData, '\n')
}
This is similar to what net/http Request does: read the data into a struct, and then provide accessors to interpret that struct.
If you really don't want to do that, then another approach that I have used is to create two structs. Read the raw data into the first and then use that to construct the second.
If you are planning on shipping this out as JSON or some other wire format, the second struct could just be a map.
func (d *Document) Map() map[string]interface{} {
m := make(map[string]interface{})
m["lines"] = strings.Split(d.LineData, '\n')
m["title"] = d.Title
return m
}

Resources