Env: OS-X / Xamarin Studio 6.1 / F# Interactive for F# 4.0
Compiled: The following works fine as compiled:
type Movie = {
Name : string
Year: int
}
[<EntryPoint>]
let main argv =
let movies = [
{ Name = "Bad Boys"; Year = 1995 }
]
let json = Newtonsoft.Json.JsonConvert.SerializeObject(movies)
System.Console.WriteLine json
System.Console.ReadKey() |> ignore
0 // return an integer exit code
Interactive: In interactive mode Newtonsoft.Json' JsonConvert.SerializeObject fails
type Movie = {
Name : string
Year: int
}
let movies = [
{ Name = "Bad Boys"; Year = 1995 }
]
let json = Newtonsoft.Json.JsonConvert.SerializeObject(movies)
Output:
val movies : Movie list = [{Name = "Bad Boys";
Year = 1995;}; {Name = "Bad Boys 2";
Year = 2003;}]
System.BadImageFormatException: Method has zero rva
File name: 'System.Net.Http.Formatting'
at <StartupCode$FSI_0015>.$FSI_0015.main# () [0x00006] in <filename unknown>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in <filename unknown>:0
[MVID] 7747cf446af449e194e4b4e70d85e773 2
[MVID] eddc6e27796e462ba5a0f4fbcf15e179 0
I am missing some setup or does Newtonsoft.Json not work in F#'s Interactive mode.
Note: I am using Xamarin Studio to add all references automatically to the interactive session (152 of them)
Update:
Using Xamarin Studio:
Edit / Send references to F# Interactive:
Which includes:
--> Referenced '/Users/sushi/code/sushi/GeneticGraphDatabase/packages/Newtonsoft.Json.9.0.1/lib/net45/Newtonsoft.Json.dll' (file may be locked by F# Interactive process)
JsonConvert.SerializeObject Produces:
System.BadImageFormatException: Method has zero rva
Referencing it 'manually' with the same assembly path:
#r "/Users/sushi/code/sushi/GeneticGraphDatabase/packages/Newtonsoft.Json.9.0.1/lib/net45/Newtonsoft.Json.dll";;
--> Referenced '/Users/sushi/code/sushi/GeneticGraphDatabase/packages/Newtonsoft.Json.9.0.1/lib/net45/Newtonsoft.Json.dll' (file may be locked by F# Interactive process)
Works as expected:
val json : string = "[{"Name":"Bad Boys","Year":1995}]"
Referencing Json.Net 'manually' using the same assembly path that Xamarin Studio is using when using the menu item: "Edit / Send references to F# Interactive"
#r "/Users/sushi/code/sushi/GeneticGraphDatabase/packages/Newtonsoft.Json.9.0.1/lib/net45/Newtonsoft.Json.dll";;
--> Referenced '/Users/sushi/code/sushi/GeneticGraphDatabase/packages/Newtonsoft.Json.9.0.1/lib/net45/Newtonsoft.Json.dll' (file may be locked by F# Interactive process)
Works as expected:
val json : string = "[{"Name":"Bad Boys","Year":1995}]"
Bugzilla: https://bugzilla.xamarin.com/show_bug.cgi?id=43307
Related
I have a csv file with 6 columns and 678,552 rows. Unfortunately I cannot share any data sample but the types are straightforward: int64, int64, date, date, string, string and there are no missing values.
Time to load this data in a dataframe in R using read.table: ~ 3 seconds.
Time to load this data using CsvFile.Load in F#: ~ 3 seconds.
Time to load this data in a Deedle dataframe in F#: ~ 7 seconds.
Adding inferTypes=falseand providing a schema to Deedle's Frame.ReadCsv reduces the time to ~ 3 seconds
Time to load this data using CsvProvider in F#: ~ 5 minutes.
And this 5 minutes is even after I define the types in the Schema parameter, presumably eliminating the time F# would use to infer them.
I understand that the type provider needs to do a lot more than R or CsvFile.Load in order to parse the data into the correct data type but I am surprised by the x100 speed penalty. Even more confusing is the time Deedle takes to load the data since it also needs to infer types and cast appropriately, organize in Series, etc. I would actually have expected Deedle to take longer than CsvProvider.
In this issue the bad performance of CsvProvider was caused by a large number of columns which is not my case.
I am wondering if I am doing something wrong or if there is any way to speed things up a bit.
Just to clarify: creating the provider is almost instantaneous. It is when I force the generated sequence to be realized by Seq.length df.Rows that it takes ~ 5 minutes for the fsharpi prompt to return.
I'm on a linux system, F# v4.1 on mono v4.6.1.
Here is the code for the CsvProvider
let [<Literal>] SEP = "|"
let [<Literal>] CULTURE = "sv-SE"
let [<Literal>] DATAFILE = dataroot + "all_diagnoses.csv"
type DiagnosesProvider = CsvProvider<DATAFILE, Separators=SEP, Culture=CULTURE>
let diagnoses = DiagnosesProvider()
EDIT1:
I added the time Deedle takes to load the data into a frame.
EDIT2:
Added the time Deedle takes if inferTypes=false and a schema is provided.
Also, supplying CacheRows=false in the CsvProvider as suggested in the comments has no perceptible effect in the time to load.
EDIT3:
Ok, we are getting somewhere. For some peculiar reason it seems that Culture is the culprit. If I omit this argument, CsvProvider loads the data in ~ 7 seconds. I am unsure what could be causing this. My system's locale is en_US. The data however come from an SQL Server in swedish locale where decimal digits are separated by ',' instead of '.'. This particular dataset does not have any decimals, so I can omit Culture altogether. Another set however has 2 decimal columns and more than 1,000,000 rows. My next task is to test this on a Windows system which I don't have available at the moment.
EDIT4:
Problem seems solved but I still don't understand what causes it. If I change the culture "globally" by doing:
System.Globalization.CultureInfo.DefaultThreadCurrentCulture = CultureInfo("sv-SE")
System.Threading.Thread.CurrentThread.CurrentCulture = CultureInfo("sv-SE")
and then remove the Culture="sv-SE" argument from the CsvProvider the load time is reduced to ~ 6 seconds and the decimals are parsed correctly. I'm leaving this open in case anyone can give an explanation for this behavior.
I am trying to reproduce the problem you are seeing, since you can't share the data I tried generating some test data. However, on my machine (.NET 4.6.2 F#4.1) I don't see it taking minutes, it takes seconds.
Perhaps you can try to see how my sample application performs in your setup and we can work from that?
open System
open System.Diagnostics
open System.IO
let clock =
let sw = Stopwatch ()
sw.Start ()
fun () ->
sw.ElapsedMilliseconds
let time a =
let before = clock ()
let v = a ()
let after = clock ()
after - before, v
let generateDataSet () =
let random = Random 19740531
let firstDate = DateTime(1970, 1, 1)
let randomInt () = random.Next () |> int64 |> (+) 10000000000L |> string
let randomDate () = (firstDate + (random.Next () |> float |> TimeSpan.FromSeconds)).ToString("s")
let randomString () =
let inline valid ch =
match ch with
| '"'
| '\\' -> ' '
| _ -> ch
let c = random.Next () % 16
let g i =
if i = 0 || i = c + 1 then '"'
else 32 + random.Next() % (127 - 32) |> char |> valid
Array.init (c + 2) g |> String
let columns =
[|
"Id" , randomInt
"ForeignId" , randomInt
"BirthDate" , randomDate
"OtherDate" , randomDate
"FirstName" , randomString
"LastName" , randomString
|]
use sw = new StreamWriter ("perf.csv")
let headers = columns |> Array.map fst |> String.concat ";"
sw.WriteLine headers
for i = 0 to 700000 do
let values = columns |> Array.map (fun (_, f) -> f ()) |> String.concat ";"
sw.WriteLine values
open FSharp.Data
[<Literal>]
let sample = """Id;ForeignId;BirthDate;OtherDate;FirstName;LastName
11795679844;10287417237;2028-09-14T20:33:17;1993-07-21T17:03:25;",xS# %aY)N*})Z";"ZP~;"
11127366946;11466785219;2028-02-22T08:39:57;2026-01-24T05:07:53;"H-/QA(";"g8}J?k~"
"""
type PerfFile = CsvProvider<sample, ";">
let readDataWithTp () =
use streamReader = new StreamReader ("perf.csv")
let csvFile = PerfFile.Load streamReader
let length = csvFile.Rows |> Seq.length
printfn "%A" length
[<EntryPoint>]
let main argv =
Environment.CurrentDirectory <- AppDomain.CurrentDomain.BaseDirectory
printfn "Generating dataset..."
let ms, _ = time generateDataSet
printfn " took %d ms" ms
printfn "Reading dataset..."
let ms, _ = time readDataWithTp
printfn " took %d ms" ms
0
The performance numbers (.NET462 on my desktop):
Generating dataset...
took 2162 ms
Reading dataset...
took 6156 ms
The performance numbers (Mono 4.6.2 on my Macbook Pro):
Generating dataset...
took 4432 ms
Reading dataset...
took 8304 ms
Update
It turns out that specifying Culture to CsvProvider explicitly seems to degrade performance alot. It can be any culture, not just sv-SE but why?
If one checks the code the provider generates for the fast and the slow cases one notice a difference:
Fast
internal sealed class csvFile#78
{
internal System.Tuple<long, long, System.DateTime, System.DateTime, string, string> Invoke(object arg1, string[] arg2)
{
Microsoft.FSharp.Core.FSharpOption<string> fSharpOption = TextConversions.AsString(arg2[0]);
long arg_C9_0 = TextRuntime.GetNonOptionalValue<long>("Id", TextRuntime.ConvertInteger64("", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[1]);
long arg_C9_1 = TextRuntime.GetNonOptionalValue<long>("ForeignId", TextRuntime.ConvertInteger64("", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[2]);
System.DateTime arg_C9_2 = TextRuntime.GetNonOptionalValue<System.DateTime>("BirthDate", TextRuntime.ConvertDateTime("", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[3]);
System.DateTime arg_C9_3 = TextRuntime.GetNonOptionalValue<System.DateTime>("OtherDate", TextRuntime.ConvertDateTime("", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[4]);
string arg_C9_4 = TextRuntime.GetNonOptionalValue<string>("FirstName", TextRuntime.ConvertString(fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[5]);
return new System.Tuple<long, long, System.DateTime, System.DateTime, string, string>(arg_C9_0, arg_C9_1, arg_C9_2, arg_C9_3, arg_C9_4, TextRuntime.GetNonOptionalValue<string>("LastName", TextRuntime.ConvertString(fSharpOption), fSharpOption));
}
}
Slow
internal sealed class csvFile#78
{
internal System.Tuple<long, long, System.DateTime, System.DateTime, string, string> Invoke(object arg1, string[] arg2)
{
Microsoft.FSharp.Core.FSharpOption<string> fSharpOption = TextConversions.AsString(arg2[0]);
long arg_C9_0 = TextRuntime.GetNonOptionalValue<long>("Id", TextRuntime.ConvertInteger64("sv-SE", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[1]);
long arg_C9_1 = TextRuntime.GetNonOptionalValue<long>("ForeignId", TextRuntime.ConvertInteger64("sv-SE", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[2]);
System.DateTime arg_C9_2 = TextRuntime.GetNonOptionalValue<System.DateTime>("BirthDate", TextRuntime.ConvertDateTime("sv-SE", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[3]);
System.DateTime arg_C9_3 = TextRuntime.GetNonOptionalValue<System.DateTime>("OtherDate", TextRuntime.ConvertDateTime("sv-SE", fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[4]);
string arg_C9_4 = TextRuntime.GetNonOptionalValue<string>("FirstName", TextRuntime.ConvertString(fSharpOption), fSharpOption);
fSharpOption = TextConversions.AsString(arg2[5]);
return new System.Tuple<long, long, System.DateTime, System.DateTime, string, string>(arg_C9_0, arg_C9_1, arg_C9_2, arg_C9_3, arg_C9_4, TextRuntime.GetNonOptionalValue<string>("LastName", TextRuntime.ConvertString(fSharpOption), fSharpOption));
}
}
More specific this is the difference:
// Fast
TextRuntime.ConvertDateTime("", fSharpOption), fSharpOption)
// Slow
TextRuntime.ConvertDateTime("sv-SE", fSharpOption), fSharpOption)
When we specify a culture this is passed to ConvertDateTime which forwards it to GetCulture
static member GetCulture(cultureStr) =
if String.IsNullOrWhiteSpace cultureStr
then CultureInfo.InvariantCulture
else CultureInfo cultureStr
This means that for the default case we use the CultureInfo.InvariantCulture but for any other case for each field and row we are creating a CultureInfo object. Caching could be done but it's not. The creation process itself doesn't seem to take too much time but something happens when we are parsing with a new CultureInfo object each time.
Parsing DateTime in FSharp.Data essentially is this
let dateTimeStyles = DateTimeStyles.AllowWhiteSpaces ||| DateTimeStyles.RoundtripKind
match DateTime.TryParse(text, cultureInfo, dateTimeStyles) with
So let's make a performance test where we use a cached CultureInfo object and another one where we create one each time.
open System
open System.Diagnostics
open System.Globalization
let clock =
let sw = Stopwatch ()
sw.Start ()
fun () ->
sw.ElapsedMilliseconds
let time a =
let before = clock ()
let v = a ()
let after = clock ()
after - before, v
let perfTest c cf () =
let dateTimeStyles = DateTimeStyles.AllowWhiteSpaces ||| DateTimeStyles.RoundtripKind
let text = DateTime.Now.ToString ("", cf ())
for i = 1 to c do
let culture = cf ()
DateTime.TryParse(text, culture, dateTimeStyles) |> ignore
[<EntryPoint>]
let main argv =
Environment.CurrentDirectory <- AppDomain.CurrentDomain.BaseDirectory
let ct = "sv-SE"
let cct = CultureInfo ct
let count = 10000
printfn "Using cached CultureInfo object..."
let ms, _ = time (perfTest count (fun () -> cct))
printfn " took %d ms" ms
printfn "Using fresh CultureInfo object..."
let ms, _ = time (perfTest count (fun () -> CultureInfo ct))
printfn " took %d ms" ms
0
Performance numbers on .NET 4.6.2 F#4.1:
Using cached CultureInfo object...
took 16 ms
Using fresh CultureInfo object...
took 5328 ms
So it seems caching the CultureInfo object in FSharp.Data should improve CsvProvider performance significantly when culture is specified.
The problem was caused by CsvProvider not memoizing the explicitly set Culture. The problem was addressed by this pull request.
In my previous question Kurt pointed me to this code of FsCheck about setting the Arbitrary type.
I have the following Arbitrary (disclaimer: I have no idea what I am doing..., still finding FsCheck notoriously hard to understand but I'm dead set on getting it to work), which in itself is a simplified version of something I created earlier:
type MyArb() =
inherit Arbitrary<DoNotSize<int64>>()
override x.Generator = Arb.Default.DoNotSizeInt64().Generator
And I use it as instructed:
[<Property(Verbose = true, Arbitrary= [| typeof<MyArb> |])>]
static member MultiplyIdentity (x: int64) = x * 1L = x
This gives me a (somewhat hopeful) error message that I'm missing something:
System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
----> System.Exception : No instances found on type Tests.Arithmetic.MyArb. Check that the type is public and has public static members with the right signature.
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at FsCheck.Runner.checkMethod(Config config, MethodInfo m, FSharpOption`1 target) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck\Runner.fs:line 318
at FsCheck.NUnit.Addin.FsCheckTestMethod.runTestMethod(TestResult testResult) in C:\Users\Kurt\Projects\FsCheck\FsCheck\src\FsCheck.NUnit.Addin\FsCheckTestMethod.fs:line 100
Looking back at that Github code I see two Atrbitrary classes but neither with any inheritance and they both have different static members.
How can I create a random-number generator and assign it as an Arbitrary statically to my NUnit tests?
The type you provide in the Property.Arbitrary parameter should have static members (possibly several) of type Arb. As in the code you linked:
type TestArbitrary2 =
static member NegativeDouble() =
Arb.Default.Float()
|> Arb.mapFilter (abs >> ((-) 0.0)) (fun t -> t <= 0.0)
Applying this to your code, it should look like this:
type MyArb() =
static member m() = Arb.Default.DoNotSizeInt64()
The meaning of the Property.Arbitrary parameter is not "an implementation of Arbitrary", but rather "a bucket of typeclass implementations".
You see, the original Haskell implementation of QuickCheck relies on typeclasses to provide values of different types. In order for a particular type to be "quick-checkable", there needs to be an instance of the 'Arbitrary' class defined for that type (for example, here are instances for all basic types).
Since F# doesn't support type classes as such, FsCheck has to fake it, and this is the scheme used there: each type class instance is represented by a static member that returns the function table. For example, if we wanted to simulate the Eq typeclass, we'd define it something like this:
type Eq<'a> = { eq: 'a -> 'a -> bool; neq: 'a -> 'a -> bool }
type EqInstances() =
static member ForInt() : Eq<int> =
{ eq = (=); neq = (<>) }
static member ForMyCustomType() : Eq<MyCustomType> =
{ eq = fun a b -> a.CompareTo(b) = 0
neq = fun a b -> a.CompareTo(b) <> 0 }
But because you can't just scan all static member in all loaded assemblies (that would be prohibitively expensive), there is this little inconvenience of providing the type explicitly (as a bonus, it allows to control the visibility of "instances").
This question demonstrates clearly, IMO, why the Reflection-based API for FsCheck is less than ideal. I tend to avoid that API completely, so I'd instead write the OP property like this:
open FsCheck
open FsCheck.Xunit
[<Property>]
let MultiplyIdentity () =
Arb.Default.DoNotSizeInt64 () |> Prop.forAll <| fun (DoNotSize x) -> x * 1L = x
As the open directives suggest, this uses FsCheck.Xunit instead of FsCheck.NUnit, but AFAIK, there's no difference in the way the API works.
The advantage of this approach is that it's type-safe and more lightweight, because you don't have to implement static classes every time you need to tweak FsCheck.
If you prefer the approach described by Mark Seemann, then you may also consider using plain-FsCheck and get rid of FsCheck.Xunit entirely:
module Tests
open FsCheck
let [<Xunit.Fact>] ``Multiply Identity (passing)`` () =
Arb.Default.DoNotSizeInt64 ()
|> Prop.forAll
<| fun (DoNotSize x) ->
x * 1L = x
|> Check.QuickThrowOnFailure
let [<Xunit.Fact>] ``Multiply Identity (failing)`` () =
Arb.Default.DoNotSizeInt64 ()
|> Prop.forAll
<| fun (DoNotSize x) ->
x * 1L = -1L |# sprintf "(%A should equal %A)" (x * 1L) x
|> Check.QuickThrowOnFailure
xUnit.net testrunner output:
------ Test started: Assembly: Library1.dll ------
Test 'Tests.Multiply Identity (failing)' failed: System.Exception:
Falsifiable, after 1 test (2 shrinks) (StdGen (2100552947,296238694)):
Label of failing property: (0L should equal 0L)
Original:
DoNotSize -23143L
Shrunk:
DoNotSize 0L
at <StartupCode$FsCheck>.$Runner.get_throwingRunner#365-1.Invoke(String me..
at <StartupCode$FsCheck>.$Runner.get_throwingRunner#355.FsCheck-IRunner-On..
at FsCheck.Runner.check[a](Config config, a p)
at FsCheck.Check.QuickThrowOnFailure[Testable](Testable property)
C:\Users\Nikos\Desktop\Library1\Library1\Library1.fs(15,0): at Tests.Multi..
1 passed, 1 failed, 0 skipped, took 0.82 seconds (xUnit.net 2.1.0 build 3179).
I've been trying to write a JSON deserialiser for a while now, but haven't been able to find my error. Why is Newtonsoft telling me, Unexpected token when deserializing object: StartObject, after deserialising this?
type ThisFails =
{ a : string * string
m : Map<string, string> }
type ThisWorks =
{ y : Map<string, string>
z : string * string }
testCase "failing test - array before object" <| fun _ ->
let res = deserialise<ThisFails> Serialisation.converters
"""{"a":["xyz","zyx"],"m":{}}"""
Assert.Equal("should be eq to res", { a = "xyz", "zyx"; m = Map.empty }, res)
testCase "passing test - array after object" <| fun _ ->
let res = deserialise<ThisWorks> Serialisation.converters
"""{"y":{},"z":["xyz","zyx"]}"""
Assert.Equal("should be eq to res", { y = Map.empty; z = "xyz", "zyx" }, res)
The subject is the TupleArrayConverter.
The trace of that converter is:
reading json [Newtonsoft.Json.FSharp.TupleArrayConverter]
type => System.Tuple`2[System.String,System.String]
value token, pre-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
path => "a[0]"
token_type => String
value token, post-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
path => "a[1]"
token_type => String
value token, pre-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
path => "a[1]"
token_type => String
value token, post-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
path => "a"
token_type => EndArray
after EndArray token, returning [Newtonsoft.Json.FSharp.TupleArrayConverter]
path => "m"
token_type => PropertyName
In the converter, I'm consuming the last token, the end array, as you can see in the terminating case:
match reader.TokenType with
| JsonToken.EndArray ->
read JsonToken.EndArray |> req |> ignore
acc
And I'm consuming the StartArray token in the beginning...
So: why isn't this code working? (Newtonsoft.Json 6.0.8)
This is the error:
map tests/failing test - array before object: Exception: Newtonsoft.Json.JsonSerializationException: Unexpected token when deserializing object: StartObject. Path 'm', line 1, position 24.
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues (Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.JsonReader reader, System.Type objectType, IDictionary`2& extensionData) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.Serialization.ObjectConstructor`1 creator, System.String id) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract objectContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id, System.Boolean& createdFromNonDefaultCreator) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType, Boolean checkAdditionalContent) [0x00000] in <filename unknown>:0 (00:00:00.1500996)
1 tests run: 0 passed, 0 ignored, 0 failed, 1 errored (00:00:00.2482623)
Been debugging your code together with JSON.Net.
It turns out that after you consumed the EndArray token in your failing case the reader then points to a PropertyName token, that is all good.
Then after your converter has completed execution JSON.Net executes this.
} while (!exit && reader.Read());
Read() then moves the reader to the next token which in your failing case is StartObject causing the deserializer to fail.
So, I am not an expert at JSON.Net but thinking about providing a provider for a string value in JSON.Net I probably wouldn't advance the reader after conversion meaning the reader still points to the string value. Along the same line of thinking it makes sense when consuming an array to leave the reader at last token of the array value ie the EndArray token.
So my suggestion is merely this:
match reader.TokenType with
| JsonToken.EndArray ->
// read JsonToken.EndArray |> req |> ignore
Logger.debug logger <| fun _ ->
LogLine.sprintf
[ "path", reader.Path |> box
"token_type", reader.TokenType |> box ]
"after EndArray token, returning"
acc
This makes my test program:
[<EntryPoint>]
let main argv =
let works = deserialize<ThisWorks> """{"y":{},"z":["xyz","zyx"]}"""
printfn "%A" works
let fails = deserialize<ThisFails> """{"a":["xyz","zyx"],"m":{}}"""
printfn "%A" fails
0
Print
{y = map [];
z = ("xyz", "zyx");}
{a = ("xyz", "zyx");
m = map [];}
Hopefully this helps you resolve this error (you might already have done so)
I'm playing f# and making a server program for fun.
I think I can solve this question by using f# to access c# data class but I want to try the f# syntax.
I got f# record with [<CLIMutable>] works but f# class got error
my test code :
open System
open System.IO
open ProtoBuf
[<ProtoContract; Serializable>]
type Point (m_x : int, m_y : int) =
[<ProtoMember(1)>]
member this.x = m_x
[<ProtoMember(2)>]
member this.y = m_y
[<EntryPoint>]
let main argv =
let p : Point = new Point(10, 10)
let out = Console.OpenStandardOutput()
Serializer.Serialize<Point>(out, p)
printfn "finish"
0
and I got following output:
Unhandled Exception:
System.InvalidOperationException: Cannot apply changes to property Program+Point.x
at ProtoBuf.Serializers.PropertyDecorator.SanityCheck (ProtoBuf.Meta.TypeModel model, System.Reflection.PropertyInfo property, IProtoSerializer tail, System.Boolean& writeValue, Boolean nonPublic, Boolean allowInternal) [0x00000] in <filename unknown>:0
at ProtoBuf.Serializers.PropertyDecorator..ctor (ProtoBuf.Meta.TypeModel model, System.Type forType, System.Reflection.PropertyInfo property, IProtoSerializer tail) [0x00000] in <filename unknown>:0
...
Although I don't know much about Protobuf I suspect that its serialization cannot assign a new value to class Point fields because in F# they are immutable by default. You could change your class definition to use mutable fields/properties:
open System
open System.IO
open ProtoBuf
[<ProtoContract; Serializable>]
type Point (m_x : int, m_y : int) =
let mutable vx = 0
let mutable vy = 0
do
vy <- m_y
vx <- m_x
[<ProtoMember(1)>]
member this.x with get() = vx and set v = vx <- v
[<ProtoMember(1)>]
member this.y with get() = vy and set v = vy <- v
Maybe it'll help.
Also if F# record with [] works then maybe it is easier to use records? Records in F# support members as well as classes.
I'm trying to get F# async working, and I just can't figure out what I'm doing wrong. Here's my sorta syncronous code that runs:
open System.Net
open System.Runtime.Serialization
open System.Threading.Tasks
[<DataContract>]
type Person = {
[<field: DataMember(Name = "name")>]
Name : string
[<field: DataMember(Name = "phone")>]
Phone : int
}
let url = "http://localhost:5000/app/plugins/anon/CCure"
let js = Json.DataContractJsonSerializer(typeof<Person>)
let main x =
let client = new WebClient()
let url = url + "/" + x
let reader = client.OpenRead(url)
let person = js.ReadObject(reader) :?> Person
printfn "Name: %s, Phone number: %d" person.Name person.Phone
printfn "starting x"
let x = Task.Factory.StartNew(fun () -> main "x")
printfn "starting y"
let y = Task.Factory.StartNew(fun () -> main "y")
Task.WaitAll(x, y)
I was thinking that to run it asyncronously this would work, but it doesn't:
open System.Net
open System.Runtime.Serialization
open System.Threading.Tasks
[<DataContract>]
type Person = {
[<field: DataMember(Name = "name")>]
Name : string
[<field: DataMember(Name = "phone")>]
Phone : int
}
let url = "http://localhost:5000/app/plugins/anon/CCure"
let js = Json.DataContractJsonSerializer(typeof<Person>)
let main x = async {
let client = new WebClient()
let url = url + "/" + x
let! reader = client.OpenReadAsync(url)
let person = js.ReadObject(reader) :?> Person
printfn "Name: %s, Phone number: %d" person.Name person.Phone }
printfn "starting x"
let x = Task.Factory.StartNew(fun () -> main "x")
printfn "starting y"
let y = Task.Factory.StartNew(fun () -> main "y")
Task.WaitAll(x, y)
$ fsharpc -r System.Runtime.Serialization foo.fs && ./foo.exe F#
Compiler for F# 3.1 (Open Source Edition) Freely distributed under the
Apache 2.0 Open Source License
/home/frew/code/foo.fs(19,18): error FS0001: This expression was
expected to have type
Async<'a> but here has type
unit
/home/frew/code/foo.fs(20,17): error FS0041: A unique overload for
method 'ReadObject' could not be determined based on type information
prior to this program point. A type annotation may be needed.
Candidates: XmlObjectSerializer.ReadObject(reader:
System.Xml.XmlDictionaryReader) : obj,
XmlObjectSerializer.ReadObject(reader: System.Xml.XmlReader) : obj,
XmlObjectSerializer.ReadObject(stream: System.IO.Stream) : obj
/home/frew/code/foo.fs(20,17): error FS0008: This runtime coercion or
type test from type
'a to
Person involves an indeterminate type based on information prior to this program point. Runtime type tests are not allowed on
some types. Further type annotations are needed.
What am I missing here?
OpenReadAsync is part of the .NET BCL and therefore wasn't designed with F# async in mind. You'll notice it returns unit, rather than Async<Stream>, so it won't work with let!.
The API is designed to be used with events (i.e. you have to wire up client.OpenReadCompleted).
You have a couple of options here.
There are some nice helper methods in FSharp.Core that can help
you to convert the API into a more F# friendly one (see
Async.AwaitEvent).
Use AsyncDownloadString, an extension method for WebClient that can be found in Microsoft.FSharp.Control.WebExtensions. This is easier so I've done it below although it does mean holding the whole stream in memory as a string so if you have a huge amount of Json this may not be the best idea.
It's also more idiomatic F# to use async instead of tasks for running things in parallel.
open System.Net
open System.Runtime.Serialization
open System.Threading.Tasks
open Microsoft.FSharp.Control.WebExtensions
open System.Runtime.Serialization.Json
[<DataContract>]
type Person = {
[<field: DataMember(Name = "name")>]
Name : string
[<field: DataMember(Name = "phone")>]
Phone : int
}
let url = "http://localhost:5000/app/plugins/anon/CCure"
let js = Json.DataContractJsonSerializer(typeof<Person>)
let main x = async {
printfn "Starting %s" x
let client = new WebClient()
let url = url + "/" + x
let! json = client.AsyncDownloadString(System.Uri(url))
let bytes = System.Text.Encoding.UTF8.GetBytes(json)
let st = new System.IO.MemoryStream(bytes)
let person = js.ReadObject(st) :?> Person
printfn "Name: %s, Phone number: %d" person.Name person.Phone }
let x = main "x"
let y = main "y"
[x;y] |> Async.Parallel |> Async.RunSynchronously |> ignore<unit[]>