I'm trying to use the XMLProvider to work with a file that uses the AEOI XML format for exchange of tax information.
I'm generating the type from the .xsd file and the relevant bit is:
<xs:element name="XMLTimeStamp" type="xs:dateTime" id="S1.2">
<xs:annotation>
<xs:documentation>Date and time that the XML was originally created.</xs:documentation>
</xs:annotation>
</xs:element>
And then in the XML file I have:
<XMLTimeStamp>2022-04-28T10:09:17</XMLTimeStamp>
The provider gets created and used like this:
type CrsReport = XmlProvider<Schema="X:\\x\\uk_aeoi_submission_v2.0.xsd",
ResolutionFolder="X:\\x">
let sample = CrsReport.Load("X:\\x\\9999999999SAMPLE_FILE.xml")
let ts = sample.MessageData.XmlTimeStamp
However when I try and access the element in question I get this error:
Installed Packages
Fsharp.Data, 4.2.8
Error: System.Exception: Expecting DateTimeOffset in Value, got 2022-04-28T10:09:17
at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThenFail#1439.Invoke(String message) in D:\workspace\_work\1\s\src\fsharp\FSharp.Core\printf.fs:line 1439
at System.Runtime.CompilerServices.RuntimeHelpers.DispatchTailCalls(IntPtr callersRetAddrSlot, IntPtr callTarget, IntPtr retVal)
at FSharp.Data.Runtime.TextRuntime.GetNonOptionalValue[T](String name, FSharpOption`1 opt, FSharpOption`1 originalValue)
at <StartupCode$FSI_0029>.$FSI_0029.main#()
So it looks as if type provider is expecting a DateTimeOffset instead of a DateTime and if I use a sample of XML to generate the provider it correctly identifies the element of the file as a DateTime.
Is this a bug with the provider or am I missing something about how to access or specify these elements?
Having spent some time reading the FSharp.Data code it looks as if it is a deliberate decision to use DateTimeOffset for xs:dateTime values. The value given should parse to either a DateTime or a DateTimeOffset so something must be going on in the conversion process.
This looks like a bug in the AsDateTimeOffset function in the TextConversions.fs.
The code is currently as shown below:
let ParseISO8601FormattedDateTime text cultureInfo =
match DateTime.TryParse(text, cultureInfo, dateTimeStyles) with
| true, d -> d |> Some
| false, _ -> None
match ParseISO8601FormattedDateTime text cultureInfo with
| Some d when d.Kind <> DateTimeKind.Unspecified ->
match DateTimeOffset.TryParse(text, cultureInfo, dateTimeStyles) with
| true, dto -> dto |> Some
| false, _ -> None
| _ -> None
So what is happening is that when the date is 2022-04-28T10:09:17 them d.Kind is set to DateTimeKind.Unspecified and the code returns None. Changing the date 2022-04-28T10:09:17Z causes the d.Kind to be set and the date gets returned correctly. (So I have a work around for now)
I think that the current code might be over complicating things, it parses the date string twice and I don't see why it needs to worry about d.Kind being set or not. My suggestion for this part of the code would be:
match ParseISO8601FormattedDateTime text cultureInfo with
| Some d -> new DateTimeOffset(d) |> Some
| _ -> None
I did consider doing something like the xs:date conversion does which is to set the Kind to local.
// Parse ISO 8601 format, fixing time zone if needed
match ParseISO8601FormattedDateTime text cultureInfo with
| Some d when d.Kind = DateTimeKind.Unspecified -> new DateTime(d.Ticks, DateTimeKind.Local) |> Some
| x -> x
So for asDateTimeOffset this would then be
match ParseISO8601FormattedDateTime text cultureInfo with
| Some d when d.Kind = DateTimeKind.Unspecified ->
let d1 = new DateTime(d.Ticks,DateTimeKind.Local)
new DateTimeOffset(d1) |> Some
| Some d -> new DateTimeOffset(d) |> Some
| _ -> None
This doesn't set the Kind in the DateTime internals of the DateTimeOffset, but then DateTimeOffset is designed to:
defines the difference between the current DateTimeOffset instance's
date and time and Coordinated Universal Time (UTC)
So the offset encodes this information.
Related
Based on a condition, i want to assign either an anonymous record or null to a variable.
For example
let cryptoParameters =
match status with
| Some cryptoParameters ->
{| SecretSalt = cryptoParameters.SecretSalt
PublicIdentityKey = cryptoParameters.PublicIdentityKey
PrivateIdentityKey = cryptoParameters.PrivateIdentityKey |}
| _ -> null
I keep getting the following error:
error FS0043: The type '{| PrivateIdentityKey: string; SecretSalt: string; PublicIdentityKey: string |}' does not have 'null' as a proper value
You can get null values by using Unchecked.defaultof. In the case of an anonymous type. You must provide the whole anonymous definition as an pamarameter like:
let x = Unchecked.defaultof<{|Name:string; Age:int|}>
in your case. The compiler can infer the type, and you can use underscore instead.
let isEven x = x % 2 = 0
let person =
match isEven 3 with
| true -> {Name="David"; Age=100}
| false -> Unchecked.defaultof<_>
But maybe you want to use the Option type instead of using null. That's the way how F# handles null. Because null is usally prohibited by F#.
If you have a problem because Option is a reference type, and it causes some kind of performance problems, then maybe you want to use the ValueOption type instead.
See additionally information on Option like:
Microsoft - Null Values in F#
Microsoft - Options
Microsoft - Value Options
F# for Fun and Profit - The Option Type
I have written several helper functions in F# that enable me to deal with the dynamic nature of Excel over the COM/PIA interface. However when I go to use these functions in an Excel-DNA UDF they do not work as expected as Excel-DNA is pre-processing the values in the array from excel.
e.g. null is turned into ExcelDna.Integration.ExcelEmpty
This interferes with my own validation code that was anticipating a null. I am able to work around this by adding an additional case to my pattern matching:
let (|XlEmpty|_|) (x: obj) =
match x with
| null -> Some XlEmpty
| :? ExcelDna.Integration.ExcelEmpty -> Some XlEmpty
| _ -> None
However it feels like a waste to convert and then convert again. Is there a way to tell Excel-DNA not to do additional processing of the range values in a UDF and supply them equivalent to the COM/PIA interface? i.e. Range.Value XlRangeValueDataType.xlRangeValueDefault
EDIT:
I declare my arguments as obj like this:
[<ExcelFunction(Description = "Validates a Test Table Row")>]
let isTestRow (headings: obj) (row: obj) =
let validator = TestTable.validator
let headingsList = TestTable.testHeadings
validateRow validator headingsList headings row
I have done some more digging and #Jim Foye's suggested question also confirms this. For UDF's, Excel-DNA works over the C API rather than COM and therefore has to do its own marshaling. The possible values are shown in this file:
https://github.com/Excel-DNA/ExcelDna/blob/2aa1bd9afaf76084c1d59e2330584edddb888eb1/Distribution/Reference.txt
The reason to use ExcelEmpty (the user supplied an empty cell) is that for a UDF, the argument can also be ExcelMissing (the user supplied no argument) which might both be reasonably null and there is a need to disambiguate.
I will adjust my pattern matching to be compatible with both the COM marshaling and the ExcelDNA marshaling.
I have a Deedle DataFrame of type Frame<int,string> that contains some missing values. I would like to convert the missing values into empty strings "". I tried to use the valueOr function but that did not help. Is there a way to do this?
Here is my DataFrame:
let s1 = Series.ofOptionalObservations [ 1 => Some("A"); 2 => None ]
let s2 = Series.ofOptionalObservations [ 1 => Some("B"); 2 => Some("C") ]
let df = Frame.ofColumns ["A", s1; "BC", s2]
Typing df;; in FSI yields some information including
ColumnTypes = seq [System.String; System.String];. So the values of df are of type string and not string option.
This is the function valueOr:
let valueOr (someDefault: 'a) (xo: 'a option) : 'a =
match xo with
| Some v -> v
| None -> someDefault
I defined an auxiliary function emptyFoo as:
let emptyFoo = valueOr ""
The signature of emptyFoo is string option -> string. This means emptyFoo should not be acceptable to the compiler in the following command:
let df' = Frame.mapValues emptyFoo df
This is because the values of df are of type string and not string option.
Still, the compiler does not complain and the code runs. However, df' still has a missing value.
Is there a way to transform the missing value into the empty string?
The Deedle documentation for Frame.mapValues:
Builds a new data frame whose values are the results of applying the specified function on these values, but only for those columns which can be converted to the appropriate type for input to the mapping function
So the mapping does nothing because strings are found, rather than string options.
I noticed another function that seems to do exactly what you want.
let df' = Frame.fillMissingWith "" df
The key thing I noticed was that Deedle shows those missing values as <missing>, suggesting that it uses it's own representation (as opposed to option for example). With that knowledge I guessed that the library would provide some way of manipulating missing values, so I explored the API by doing Frame. in my IDE and browsing the list of available functions and their documentation.
I've been writing some F# now for about 6 months and I've come across some behavior that I can't explain. I have some boiled down code below. (value names have been changed to protect the innocent!)
I have a hierarchy defined using record types rec1 and rec2, and also a dicriminated union type with possible values CaseA and CaseB. I'm calling a function ('mynewfunc') that takes a du_rec option type. Internally this function defines a recursive function that processes the hierarchy .
I'm kicking off the processing by passing the None option value to represent the root of the hierarchy (In reality, this function is deserializing the hierarchy from a file).
When I run the code below I hit the "failwith "invalid parent"" line of code. I can not understand why this is, because the None value that is passed down should match the outer pattern matching's None case.
The code works if I delete either of the sets of comments. This is not a showstopper for me - I just feel a bit uncomfortable not knowing why this is happening (I thought I was understanding f#)
Thanks in advance for any replies
James
type rec2 =
{
name : string
child : rec1 option
}
and rec1 =
{
name : string ;
child : rec2 option
}
and du_rec =
| Case1 of rec1
| Case2 of rec2
let mynewfunc (arg:du_rec option) =
let rec funca (parent:du_rec option) =
match parent with
| Some(node) ->
match node with
| Case2(nd) ->
printfn "hello"
(* | Case1(nd) ->
printfn "bye bye" *)
| _ ->
failwith "invalid parent"
| None ->
// printfn "case3"
()
funcb( None )
and funcb (parent: du_rec option) =
printfn "this made no difference"
let node = funca(arg)
()
let rootAnnot = mynewfunc(None)
Based on the comments, this is just a bad experience in the debugger (where the highlighting suggests that the control flow is going places it is not); the code does what you expect.
(There are a number of places where the F# compiler could improve its sequence-points generated into the pdbs, to improve the debugging experience; I think we'll be looking at this in a future release.)
I have a function that will create a select where clause, but right now everything has to be a string.
I would like to look at the variable passed in and determine what type it is and then treat it properly.
For example, numeric values don't have single quotes around them, option type will either be null or have some value and boolean will actually be zero or one.
member self.BuildSelectWhereQuery (oldUser:'a) = //'
let properties = List.zip oldUser.ToSqlValuesList sqlColumnList
let init = false, new StringBuilder()
let anyChange, (formatted:StringBuilder) =
properties |> Seq.fold (fun (anyChange, sb) (oldVal, name) ->
match(anyChange) with
| true -> true, sb.AppendFormat(" AND {0} = '{1}'", name, oldVal)
| _ -> true, sb.AppendFormat("{0} = '{1}'", name, oldVal)
) init
formatted.ToString()
Here is one entity:
type CityType() =
inherit BaseType()
let mutable name = ""
let mutable stateId = 0
member this.Name with get() = name and set restnameval=name <- restnameval
member this.StateId with get() = stateId and set stateidval=stateId <- stateidval
override this.ToSqlValuesList = [this.Name; this.StateId.ToString()]
So, if name was some other value besides a string, or stateId can be optional, then I have two changes to make:
How do I modify ToSqlValuesList to
have the variable so I can tell the
variable type?
How do I change my select function
to handle this?
I am thinking that I need a new function does the processing, but what is the best FP way to do this, rather than using something like typeof?
You can use a type test pattern in a match. Would this meet your needs?
let f (x : obj) =
match x with
| :? int -> "int"
| :? string -> "string"
| :? bool -> "bool"
| _ -> "who knows?"
I think that one clear functional approach would be to define a data type that represents the various (more complicated situations) that you need to handle. You mentioned that a value may be optional and that you need to distinguish numeric and textual values (for the encoding to SQL).
You could define a discriminated union (if there are other cases that you'd like to handle, the definition may be a bit more complicated):
type SqlValue =
| Missing
| Numeric of string
| Textual of string
Note that the Textual case also carries string, because I assume that the client who produces the value takes care of converting it to string - this is only information for your SQL query generator (so that it knows whether it needs to add quotes).
Your ToSqlValuesList member would return a list of values string & SqlValue, so for example, a sample product could be represented using the following list:
columns = [ "Name"; "Price"; "Description" ]
values = [ Textual("Tea"); Numeric(10); Missing ]
In the code that generates the SQL query, you'd use pattern matching to handle all the different cases (most importantly, encode string to avoid SQL injection in case the value is Textual :-)).
EDIT You'd need to implement the conversion from the specific data types to the SqlValue representation in every client. However, this can be simplified by writing a utility type (using the fact that members can be overloaded):
type SqlValue with
static member From(a:int) = Numeric(a.ToString())
static member From(a:int option) =
match a with None -> Missing | Some(n) -> SqlValue.From(n)
// ... similarly for other types
In the implementation of ToSqlValuesList, you would write SqlValue.From(description) and it would deal with the details autoamtically.
A more sophisticated approach would be to annotate public members of the types representing your data entities with .NET attributes and use Reflection to extract the values (and their types) at runtime. This is more advanced, but quite elegant (there is a nice exmaple of this technique in Don Syme's Expert F# book)