How do I make this function correctly asyncronous? - f#

I'm trying to get F# async working, and I just can't figure out what I'm doing wrong. Here's my sorta syncronous code that runs:
open System.Net
open System.Runtime.Serialization
open System.Threading.Tasks
[<DataContract>]
type Person = {
[<field: DataMember(Name = "name")>]
Name : string
[<field: DataMember(Name = "phone")>]
Phone : int
}
let url = "http://localhost:5000/app/plugins/anon/CCure"
let js = Json.DataContractJsonSerializer(typeof<Person>)
let main x =
let client = new WebClient()
let url = url + "/" + x
let reader = client.OpenRead(url)
let person = js.ReadObject(reader) :?> Person
printfn "Name: %s, Phone number: %d" person.Name person.Phone
printfn "starting x"
let x = Task.Factory.StartNew(fun () -> main "x")
printfn "starting y"
let y = Task.Factory.StartNew(fun () -> main "y")
Task.WaitAll(x, y)
I was thinking that to run it asyncronously this would work, but it doesn't:
open System.Net
open System.Runtime.Serialization
open System.Threading.Tasks
[<DataContract>]
type Person = {
[<field: DataMember(Name = "name")>]
Name : string
[<field: DataMember(Name = "phone")>]
Phone : int
}
let url = "http://localhost:5000/app/plugins/anon/CCure"
let js = Json.DataContractJsonSerializer(typeof<Person>)
let main x = async {
let client = new WebClient()
let url = url + "/" + x
let! reader = client.OpenReadAsync(url)
let person = js.ReadObject(reader) :?> Person
printfn "Name: %s, Phone number: %d" person.Name person.Phone }
printfn "starting x"
let x = Task.Factory.StartNew(fun () -> main "x")
printfn "starting y"
let y = Task.Factory.StartNew(fun () -> main "y")
Task.WaitAll(x, y)
$ fsharpc -r System.Runtime.Serialization foo.fs && ./foo.exe F#
Compiler for F# 3.1 (Open Source Edition) Freely distributed under the
Apache 2.0 Open Source License
/home/frew/code/foo.fs(19,18): error FS0001: This expression was
expected to have type
Async<'a> but here has type
unit
/home/frew/code/foo.fs(20,17): error FS0041: A unique overload for
method 'ReadObject' could not be determined based on type information
prior to this program point. A type annotation may be needed.
Candidates: XmlObjectSerializer.ReadObject(reader:
System.Xml.XmlDictionaryReader) : obj,
XmlObjectSerializer.ReadObject(reader: System.Xml.XmlReader) : obj,
XmlObjectSerializer.ReadObject(stream: System.IO.Stream) : obj
/home/frew/code/foo.fs(20,17): error FS0008: This runtime coercion or
type test from type
'a to
Person involves an indeterminate type based on information prior to this program point. Runtime type tests are not allowed on
some types. Further type annotations are needed.
What am I missing here?

OpenReadAsync is part of the .NET BCL and therefore wasn't designed with F# async in mind. You'll notice it returns unit, rather than Async<Stream>, so it won't work with let!.
The API is designed to be used with events (i.e. you have to wire up client.OpenReadCompleted).
You have a couple of options here.
There are some nice helper methods in FSharp.Core that can help
you to convert the API into a more F# friendly one (see
Async.AwaitEvent).
Use AsyncDownloadString, an extension method for WebClient that can be found in Microsoft.FSharp.Control.WebExtensions. This is easier so I've done it below although it does mean holding the whole stream in memory as a string so if you have a huge amount of Json this may not be the best idea.
It's also more idiomatic F# to use async instead of tasks for running things in parallel.
open System.Net
open System.Runtime.Serialization
open System.Threading.Tasks
open Microsoft.FSharp.Control.WebExtensions
open System.Runtime.Serialization.Json
[<DataContract>]
type Person = {
[<field: DataMember(Name = "name")>]
Name : string
[<field: DataMember(Name = "phone")>]
Phone : int
}
let url = "http://localhost:5000/app/plugins/anon/CCure"
let js = Json.DataContractJsonSerializer(typeof<Person>)
let main x = async {
printfn "Starting %s" x
let client = new WebClient()
let url = url + "/" + x
let! json = client.AsyncDownloadString(System.Uri(url))
let bytes = System.Text.Encoding.UTF8.GetBytes(json)
let st = new System.IO.MemoryStream(bytes)
let person = js.ReadObject(st) :?> Person
printfn "Name: %s, Phone number: %d" person.Name person.Phone }
let x = main "x"
let y = main "y"
[x;y] |> Async.Parallel |> Async.RunSynchronously |> ignore<unit[]>

Related

How do I condense this repetitive F# code?

EDIT: possible solutions at bottom
I'm doing some data work where I need to be very careful about string lengths that will eventually be sent in fixed width text output, stored in limited size nvarchar fields, etc. I want to have good strict typing for these rather than naked System.String types.
Suppose I've got some code like this to represent these, with a few useful module functions that play nicely with Result.map, Option.map, etc.
module String40 =
let private MaxLength = 40
type T = private T of string
let create (s:string) = checkStringLength MaxLength s |> Result.map T
let trustCreate (s:string) = checkStringLength MaxLength s |> Result.okVal |> T
let truncateCreate (s:string) = truncateStringToLength MaxLength s |> T
let toString (T s) = s
type T with
member this.AsString = this |> toString
module String100 =
let private MaxLength = 100
type T = private T of string
let create (s:string) = checkStringLength MaxLength s |> Result.map T
let trustCreate (s:string) = checkStringLength MaxLength s |> Result.okVal |> T
let truncateCreate (s:string) = truncateStringToLength MaxLength s |> T
let toString (T s) = s
type T with
member this.AsString = this |> toString
Obviously these are almost entirely repetitive with only the module name and max length different in each block.
What options are available to try and cut down on the repetitiveness here? I would love to have something like this:
type String40 = LengthLimitedString<40>
type String100 = LengthLimitedString<100>
tryToRetrieveString () // returns Result<string, ERRType>
|> Result.bind String40.create
T4 code generation doesn't seem to be an option for F# projects
Type providers seem like overkill for this kind of simple templating, and as best as I can tell they can only produce classes, not modules.
I'm aware of Scott Wlaschin's constrained strings page, but I end up with roughly the same level of repetitive code in the 'Create a type', 'Implement IWrappedString', 'create a public constructor' steps he lists.
These code blocks are fairly short and it wouldn't be the end of the world to just copy/paste a dozen times for the different field lengths. But I feel like I'm missing a simpler way to do this.
UPDATE:
One other note is that it's important that records using these types give information about what type they're carrying:
type MyRecord =
{
FirstName: String40;
LastName: String100;
}
and not be something like
type MyRecord =
{
FirstName: LimitedString;
LastName: LimitedString;
}
Tomas' answer, using the Depended Type Provider nuget library is a pretty good one, and would be a good solution for a lot of people who are fine with its behavior as-is. I felt like it would be a little tricky to extend and customize unless I wanted to maintain my own copy of a type provider which I was hoping to avoid.
Marcelo's suggestion of static parameter constraints was a fairly productive path of research. They give me basically what I was looking for -- a generic argument that is basically an 'interface' for a static methods. However the kicker is that they require inline functions to operate and I don't have the time to evaluate how much that would or would not matter in my code base.
But I took that and altered it to use regular generic constraints. It's a bit goofy to have to instantiate an object to get a max-length value, and fsharp type/generic code is just gross to look at, but from the module users's perspective it's clean, and I can easily extend this however I want.
type IMaxLengthProvider = abstract member GetMaxLength: unit -> int
type MaxLength3 () = interface IMaxLengthProvider with member this.GetMaxLength () = 3
type MaxLength4 () = interface IMaxLengthProvider with member this.GetMaxLength () = 4
module LimitedString =
type T< 'a when 'a :> IMaxLengthProvider> = private T of string
let create< 't when 't :> IMaxLengthProvider and 't : (new:unit -> 't)> (s:string) =
let len = (new 't()).GetMaxLength()
match checkStringLength len s with
| Ok s ->
let x : T< 't> = s |> T
x |> Ok
| Error e -> Error e
let trustCreate< 't when 't :> IMaxLengthProvider and 't : (new:unit -> 't)> (s:string) =
let len = (new 't()).GetMaxLength()
match checkStringLength len s with
| Ok s ->
let x : T< 't> = s |> T
x
| Error e ->
let msg = e |> formErrorMessage
failwith msg
let truncateCreate< 't when 't :> IMaxLengthProvider and 't : (new:unit -> 't)> (s:string) =
let len = (new 't()).GetMaxLength()
let s = truncateStringToLength len s
let x : T< 't> = s |> T
x
let toString (T s) = s
type T< 'a when 'a :> IMaxLengthProvider> with
member this.AsString = this |> toString
module test =
let dotest () =
let getString () = "asd" |> Ok
let mystr =
getString ()
|> Result.bind LimitedString.create<MaxLength3>
|> Result.okVal
|> LimitedString.toString
sprintf "it is %s" mystr
I think the BoundedString type provider from the Dependent type provider project lets you do exactly what you need. Using the example from the project documentation, you can do e.g.:
type ProductDescription = BoundedString<10, 2000>
type ProductName = BoundedString<5, 50>
type Product = { Name : ProductName; Description : ProductDescription }
let newProduct (name : string) (description : string) : Product option =
match ProductName.TryCreate(name), ProductDescription.TryCreate(description) with
| Some n, Some d -> { Name = n; Description = d }
| _ -> None
I don't know how many people use this project in practice, but it seems quite simple and it does exactly what you were asking for, so it might be worth a try.
Using a touch of careful reflection magic, we can achieve a lot and get some really nice types. How about something like this?
module Strings =
type [<AbstractClass>] Length(value: int) =
member this.Value = value
let getLengthInst<'L when 'L :> Length> : 'L =
downcast typeof<'L>.GetConstructor([||]).Invoke([||])
type LimitedString<'Length when 'Length :> Length> =
private | LimitedString of maxLength: 'Length * value: string
member this.Value =
let (LimitedString(_, value)) = this in value
member this.MaxLength =
let (LimitedString(maxLength, _)) = this in maxLength.Value
module LimitedString =
let checkStringLength<'L when 'L :> Length> (str: string) =
let maxLength = getLengthInst<'L>.Value
if str.Length <= maxLength then Ok str
else Error (sprintf "String of length %d exceeded max length of %d" str.Length maxLength)
let create<'L when 'L :> Length> (str: string) =
checkStringLength<'L> str
|> Result.map (fun str -> LimitedString (getLengthInst<'L>, str))
open Strings
// === Usage ===
type Len5() = inherit Length(5)
type Len1() = inherit Length(1)
// Ok
LimitedString.create<Len5> "Hello"
// Error
LimitedString.create<Len1> "world"
One option might be to use a single module for limited-length strings that uses curried parameters for the length-limit and the string itself, then just partially apply the limit parameter. An implementation might look like this:
module LimitedString =
type T = private T of string
let create length (s:string) = checkStringLength length s |> Result.map T
let trustCreate length (s:string) = checkStringLength length s |> Result.okVal |> T
let truncateCreate length (s:string) = truncateStringToLength length s |> T
let toString (T s) = s
type T with
member this.AsString = this |> toString
Then, your modules for each length would still be required, but wouldn't have all the boilerplate:
module String100 =
let create = LimitedString.create 100
let trustCreate = LimitedString.trustCreate 100
let truncateCreate = LimitedString.truncateCreate 100
EDIT
After reading the comment and the update to the original post, I would change my suggestion a bit. Instead of defining the T type inside each module, I would have a specific struct-type single-case union for each string length at the top level. Then, I would move to toString to the individual string modules. Finally, I would add one more parameter to the LimitedString module to allow us to partially apply both the length and the specific single-case union type:
[<Struct>] type String40 = private String40 of string
[<Struct>] type String100 = private String100 of string
module LimitedString =
let create length ctor (s:string) = checkStringLength length s |> Result.map ctor
let trustCreate length ctor (s:string) = checkStringLength length s |> Result.okVal |> ctor
let truncateCreate length ctor (s:string) = truncateStringToLength length s |> ctor
module String40 =
let create = LimitedString.create 40 String40
let trustCreate = LimitedString.trustCreate 40 String40
let truncateCreate = LimitedString.truncateCreate 40 String40
let toString (String40 s) = s
module String100 =
let create = LimitedString.create 100 String100
let trustCreate = LimitedString.trustCreate 100 String100
let truncateCreate = LimitedString.truncateCreate 100 String100
let toString (String100 s) = s
type MyRecord =
{
FirstName: String40
LastName: String100
}
There's still a fair amount of boilerplate here, but I think this is in the ballpark for a solution using single-case unions and modules. A Type Provider might be possible, but you would have to consider whether the added complexity outweighs the boilerplate.
Can't you use Static Parameters, as shown on the F#.Data package's HtmlProvider example?

Dynamic Lookup in F#

Can somebody help me with article of Tomas Petricek: http://tomasp.net/blog/fsharp-dynamic-lookup.aspx/#dynfslinks?
The problem is that it is severely outdated. I understand that namespaces
open Microsoft.FSharp.Quotations.Typed
open Microsoft.FSharp.Quotations.Raw
are gone. So I removed the openings. But there are still errors. "Typed" is not defined. "RecdGet" is not defined. And I suspect they are not the last. I'm trying to prove to my boss that F# is good to use for database normalization. Dynamic lookup of fields would really helped me to deal with similarly named fields having different prefixes.
There is also post of Tomas on fpish: https://fpish.net/topic/None/57493, which I understand predates the article
Here's a rough equivalent:
open Microsoft.FSharp.Quotations
open Microsoft.FSharp.Quotations.Patterns
type DynamicMember<'t,'u> = Expr<'t -> 'u>
let getValueReader (expr:DynamicMember<'recdT, 'fieldT>) =
// Match the quotation representing the symbol
match expr with
| Lambda(v, PropertyGet (Some (Var v'), pi, [])) when v = v' ->
// It represents reading of the F# record field..
// .. get a function that reads the record field using F# reflection
let rdr = Reflection.FSharpValue.PreComputeRecordFieldReader pi
// we're not adding any additional processing, so we just
// simply add type conversion to the correct types & return it
((box >> rdr >> unbox) : 'recdT -> 'fieldT)
| _ ->
// Quotation doesn't represent symbol - this is an error
failwith "Invalid expression - not reading record field!"
type SampleRec = { Str : string; Num : int }
let readStrField = getValueReader <# fun (r : SampleRec) -> r.Str #>
let readNumField = getValueReader <# fun (r : SampleRec) -> r.Num #>
let rc = { Str = "Hello world!"; Num = 42 }
let s, n = readStrField rc, readNumField rc
printfn "Extracted: %s, %d" s n

Populate list with Types

Im trying to populate list with my own type.
let getUsers =
use connection = openConnection()
let getString = "select * from Accounts"
use sqlCommand = new SqlCommand(getString, connection)
try
let usersList = [||]
use reader = sqlCommand.ExecuteReader()
while reader.Read() do
let floresID = reader.GetString 0
let exName = reader.GetString 1
let exPass = reader.GetString 2
let user = [floresID=floresID; exName=exName; exPass=exPass]
// what here?
()
with
| :? SqlException as e -> printfn "Došlo k chybě úrovni připojení:\n %s" e.Message
| _ -> printfn "Neznámá výjimka."
In C# I would just add new object into userList. How can I add new user into list? Or is it better approach to get some sort of list with data from database?
Easiest way to do this is with a type provider, so you can abstract away the database. You can use SqlDataConnection for SQLServer, SqlProvider for everything (incl. SQLServer), and also SQLClient for SQLServer.
Here is an example with postgres's dvdrental (sample) database for SQLProvider:
#r #"..\packages\SQLProvider.1.0.33\lib\FSharp.Data.SqlProvider.dll"
#r #"..\packages\Npgsql.3.1.8\lib\net451\Npgsql.dll"
open System
open FSharp.Data.Sql
open Npgsql
open NpgsqlTypes
open System.Linq
open System.Xml
open System.IO
open System.Data
let [<Literal>] dbVendor = Common.DatabaseProviderTypes.POSTGRESQL
let [<Literal>] connString1 = #"Server=localhost;Database=dvdrental;User Id=postgres;Password=root"
let [<Literal>] resPath = #"C:\Users\userName\Documents\Visual Studio 2015\Projects\Postgre2\packages\Npgsql.3.1.8\lib\net451"
let [<Literal>] indivAmount = 1000
let [<Literal>] useOptTypes = true
//create the type for the database, based on the connection string, etc. parameters
type sql = SqlDataProvider<dbVendor,connString1,"",resPath,indivAmount,useOptTypes>
//set up the datacontext, ideally you would use `use` here :-)
let ctx = sql.GetDataContext()
let actorTbl = ctx.Public.Actor //alias the table
//set up the type, in this case Records:
type ActorName = {
firstName:string
lastName:string}
//extract the data with a query expression, this gives you type safety and intellisense over SQL (but also see the SqlClient type provider above):
let qry = query {
for row in actorTbl do
select ({firstName=row.FirstName;lastName=row.LastName})
}
//seq is lazy so do all kinds of transformations if necessary then manifest it into a list or array:
qry |> Seq.toArray
The two important parts are defining the Actor record, and then in the query extracting the fields into a sequence of Actor records. You can then manifest into a list or array if necessary.
But you can also stick to your original solution. In that case just wrap the .Read() into a seq:
First define the type:
type User = {
floresID: string
exName: string
exPass: string
}
Then extract the data:
let recs = cmd.ExecuteReader() // execute the SQL Command
//extract the users into a sequence of records:
let users =
seq {
while recs.Read() do
yield {floresID=recs.[0].ToString()
exName=recs.[1].ToString()
exPass=recs.[2].ToString()
}
} |> Seq.toArray
Taking your code, you can use list expression:
let getUsers =
use connection = openConnection()
let getString = "select * from Accounts"
use sqlCommand = new SqlCommand(getString, connection)
try
[
use reader = sqlCommand.ExecuteReader()
while reader.Read() do
let floresID = reader.GetString 0
let exName = reader.GetString 1
let exPass = reader.GetString 2
let user = [floresID=floresID; exName=exName; exPass=exPass]
yield user
]
with
| :? SqlException as e -> failwithf "Došlo k chybě úrovni připojení:\n %s" e.Message
| _ -> failwithf "Neznámá výjimka."
That being said, I'd use FSharp.Data.SqlClient library so all of that boiler plate becomes a single line with added benefit of type safety (if you change the query, the code will have compile time error which are obvious to fix).

How can I pass a parameter to Sql.execReaderF in FsSql?

I am trying out the samples for FsSql and I seem to be stuck on how to properly use the Sql.execReaderF function. The example code uses an int parameter but I have a string. The following code blocks show my attempts. Does FsSql only support int for this function maybe?
Setup code:
module FsSqlTests
open System
open System.Data
open System.Data.SqlClient
open NUnit.Framework
open Swensen.Unquote
let openConn() =
let conn = new SqlConnection(#"Data Source=MYSERVER;Initial Catalog=MYDB;Integrated Security=True")
conn.Open()
conn :> IDbConnection
let connMgr = Sql.withNewConnection openConn
let P = Sql.Parameter.make
let execReader sql = Sql.execReader connMgr sql
let execReaderf sql = Sql.execReaderF connMgr sql
Using Sql.execReader (Test case passes using this one)
let selectSummaryByeFolderName eFolderName =
execReader "select summary from ework.V_DQ_Iccm_Activity_By_Team WHERE efoldername = #eFolderName"
[P("#eFolderName", eFolderName)]
Using Sql.execReaderF (Test case fails using this one)
let selectSummaryByeFolderName =
execReaderf "select summary from ework.V_DQ_Iccm_Activity_By_Team WHERE efoldername = '%s'"
Calling code in the test case:
[<TestCase>]
let ``Gets CM summary given eFolderName``() =
let c = selectSummaryByeFolderName "CM008671"
let r = c
|> Seq.ofDataReader
|> Seq.map(fun dr ->
let s =
match dr?summary with
| None -> "No Summary"
| Some x -> x
s)
|> Seq.length
test <# r > 0 #>
How can I modify my call to execReaderF to make it pass the parameter and run correctly?
UPDATE:
I tried it out with an integer parameter and it works fine. It seems the function may only support integers.
let selectSummaryByCallPriority =
execReaderf "select top 10 summary from ework.V_DQ_Iccm_Activity_By_Team WHERE callpriority = %d"
I had a look at the implementation to try and verify this but it's over my head. Anyway the Sql.execReader function works fine for other datatypes so I can just switch to that function for my string parameters.

Download Windows Phone app reviews using F#

I am learning F# on my own (this is for fun, it is not for work/school) and I am trying to write a simple parser which count the number of reviews across multiple markets for a Windows Phone app. There's no doubt that the code I have so far is ugly, but I am trying to improve it and follow functional programming paradigm. Since I come from the C, C++, C# world, it is pretty hard.
Coming from C world, I like null values. I know that functional programming / F# doesn't encourage the use of null, but I can't figure out a way to not use it. For example, in the function parse there's a null check. How do I not do that?
Right now my code only count the number of reviews on the first page, but it is possible that an app has more than 10 reviews and as a result multiple pages. How do I recursively go through all page (functuion downloadReviews or parse).
How could we extend this code to be entirely async?
Below is the code I have so far. In addition to the questions above, I would really like if someone could help me and give me directions on how to improve the overall structure of my code.
open System
open System.IO
open System.Xml
open System.Xml.Linq
open Printf
type DownloadPageResult = {
Uri: System.Uri;
ErrorOccured: bool;
Source: string;
}
type ReviewData = {
CurrentPageUri: System.Uri;
NextPageUri: System.Uri;
NumberOfReviews: int;
}
module ReviewUrl =
let getBaseUri path =
new Uri(sprintf "http://cdn.marketplaceedgeservice.windowsphone.com/%s" path)
let getUri country locale appId =
getBaseUri(sprintf "/v8/ratings/product/%s/reviews?os=8.0.0.0&cc=%s&oc=&lang=%s&hw=520170499&dm=Test&chunksize=10" appId country locale)
let downloadPage (uri: System.Uri) =
try
use webClient = new System.Net.WebClient()
printfn "%s" (uri.ToString())
webClient.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
webClient.Headers.Add("Accept-Encoding", "zip,deflate,sdch")
webClient.Headers.Add("Accept-Language", "en-US,en;q=0.8,fr;q=0.6")
webClient.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1482.0 Safari/537.36")
{ Uri = uri; Source = webClient.DownloadString(uri); ErrorOccured = false }
with error -> { Uri = uri; Source = String.Empty; ErrorOccured = true }
let downloadReview country locale appId =
let uri = ReviewUrl.getUri country locale appId
downloadPage uri
let parse(pageResult: DownloadPageResult) =
if pageResult.ErrorOccured then { CurrentPageUri = pageResult.Uri; NextPageUri = null; NumberOfReviews = 0 }
else
let reader = new StringReader(pageResult.Source)
let doc = XDocument.Load(reader)
let ns = XNamespace.Get("http://www.w3.org/2005/Atom")
let nextUrl = query { for link in doc.Descendants(ns + "link") do
where (link.Attribute(XName.Get("rel")).Value = "next")
select link.Value
headOrDefault }
if nextUrl = null then
{ CurrentPageUri = pageResult.Uri; NextPageUri = null; NumberOfReviews = doc.Descendants(ns + "entry") |> Seq.length }
else
{ CurrentPageUri = pageResult.Uri; NextPageUri = ReviewUrl.getBaseUri(nextUrl); NumberOfReviews = doc.Descendants(ns + "entry") |> Seq.length }
let downloadReviews(locale: string) =
let appId = "4e08377c-1240-4f80-9c35-0bacde2c66b6"
let country = locale.Substring(3)
let pageResult = downloadReview country locale appId
let parseResult = parse pageResult
parseResult
[<EntryPoint>]
let main argv =
let locales = [| "en-US"; "en-GB"; |]
let results = locales |> Array.map downloadReviews
printfn "%A" results
0
I was playing with this problem a bit more and tried using the XML type provider and other features from F# Data. It is not complete code, but it should be enough to give you the idea (and to show that type providers are really nice :-)):
First, I need some references:
#r "System.Xml.Linq.dll"
#r "FSharp.Data.dll"
open FSharp.Data
open FSharp.Net
Next, I wrote the following code to download one sample page.
let data =
Http.Request
( "http://cdn.marketplaceedgeservice.windowsphone.com//v8/ratings/product/4e08377c-1240-4f80-9c35-0bacde2c66b6/reviews",
query=["os", "8.0.0.0"; "cc", "US"; "lang", "en-US"; "hw", "520170499"; "dm", "Test"; "chunksize", "10" ],
headers=["User-Agent", "F#"])
I saved the sample as D:\temp\appstore.xml and then used the XML type provider to get a nice type for parsing the page:
type PageDocument = XmlProvider< #"D:\temp\appstore.xml" >
Then you can download & parse the page like this (this shows how to get the number of reviews and information about the next link):
let parseAsync (locale:string) appId = async {
let country = locale.Substring(3)
// Make the request (asynchronously) using the parameters specified
let! data =
Http.AsyncRequest
( "http://cdn.marketplaceedgeservice.windowsphone.com//v8/ratings/product/"
+ appId + "/reviews",
query=[ "os", "8.0.0.0"; "cc", country; "lang", locale;
"hw", "520170499"; "dm", "Test"; "chunksize", "10" ],
headers=["User-Agent", "F#"])
// Parse the result using the type-provider generated type
let page = PageDocument.Parse(data)
// Now you can type 'page' followed by '.' and explore the results!
// page.GetLinks() returns all links and page.GetEntries() returns
// review entries. Each link also has 'Rel' and 'Href' properties:
let nextLink =
page.GetLinks()
|> Seq.tryFind (fun link -> link.Rel = "next")
|> Option.map (fun link -> link.Href)
let reviewsCount = page.GetEntries().Length
return (reviewsCount, nextLink) }
The general pattern for making code asynchronous is to find the I/O expensive operation (somewhere down in the call tree) and then go "up" from there and make all code that uses it asynchronous too until you reach a point where you need to block.
In your example, the primitive operation is downloading, so you would start by making downloadPage asynchronous:
let downloadPage (uri: System.Uri) = async {
try
use webClient = new System.Net.WebClient()
printfn "%s" (uri.ToString())
// (Headers omitted)
let! source = webClient.AsyncDownloadString(uri)
return { Uri = uri; Source = source; ErrorOccured = false }
with error ->
return { Uri = uri; Source = String.Empty; ErrorOccured = true } }
You need to wrap code in async { ... }, make call to asynchronous version of DownloadString using let! and return the results using return (in both branches).
Then you need to make functions like downloadReview and downloadReviews (again, wrap them in async block, call other asynchronous operations like downloadPage using let! or using return!).
In the end, if you're writing console application you'll need to block, but you can run downloads for different locales in parallel. Assuming downloadReviews is asynchronous:
let locales = [| "en-US"; "en-GB"; |]
let results =
locales
|> Array.map downloadReviews // Build an array of asynchronous computations
|> Async.Parallel // Compose them into a single, parallel computation
|> Async.RunSynchronously // Run the computation and wait
To answer other questions, I think using null in the example above is probably okay (you are calling LINQ which returns it, so there is no easy way to avoid that). It is actually possible to use option type instead, but it is a bit tricky - see this snippet if you're interested.
Also, you could use the Http.AsyncRequest method from F# Data Library which gives you a bit simpler way to construct complex HTTP requests (but I'm one of the contributors to that library, so I'm biased!)
As Tomas said, it would be more "functional" to create an async-based version of DownloadString (or just use his FSharp.Data library to handle it).
You could also combine FSharp.Data with ExtCore to take advantage of the asyncMaybe or asyncChoice workflows in ExtCore. Those workflows provide very easy-to-use error handling on top of the normal async workflow.
Anyway, I spent a few minutes cleaning up your code. It's not much, but it does simplify your code in a few spots:
open System
open System.IO
open System.Xml
open System.Xml.Linq
open Printf
type DownloadPageResult = {
Uri : System.Uri;
ErrorOccured : bool;
Source : string;
}
type ReviewData = {
CurrentPageUri : System.Uri;
NextPageUri : System.Uri option;
NumberOfReviews : uint32;
}
module ReviewUrl =
let baseUri = Uri ("http://cdn.marketplaceedgeservice.windowsphone.com/", UriKind.Absolute)
let getUri country locale (appId : System.Guid) =
let localUri =
let appIdStr = appId.ToString "D"
sprintf "/v8/ratings/product/%s/reviews?os=8.0.0.0&cc=%s&oc=&lang=%s&hw=520170499&dm=Test&chunksize=10" appIdStr country locale
Uri (baseUri, localUri)
let downloadPage (uri : System.Uri) =
try
use webClient = new System.Net.WebClient()
printfn "%s" (uri.ToString())
webClient.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
webClient.Headers.Add("Accept-Encoding", "zip,deflate,sdch")
webClient.Headers.Add("Accept-Language", "en-US,en;q=0.8,fr;q=0.6")
webClient.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1482.0 Safari/537.36")
{ Uri = uri; Source = webClient.DownloadString uri; ErrorOccured = false }
with error ->
{ Uri = uri; Source = String.Empty; ErrorOccured = true }
let parse (pageResult : DownloadPageResult) =
if pageResult.ErrorOccured then
{ CurrentPageUri = pageResult.Uri; NextPageUri = None; NumberOfReviews = 0u }
else
use reader = new StringReader (pageResult.Source)
let doc = XDocument.Load reader
let ns = XNamespace.Get "http://www.w3.org/2005/Atom"
let nextUrl =
query {
for link in doc.Descendants(ns + "link") do
where (link.Attribute(XName.Get("rel")).Value = "next")
select link.Value
headOrDefault }
{ CurrentPageUri = pageResult.Uri;
NextPageUri =
if System.String.IsNullOrEmpty nextUrl then None
else Some <| Uri (ReviewUrl.baseUri, nextUrl);
NumberOfReviews =
doc.Descendants (ns + "entry") |> Seq.length |> uint32; }
let downloadReviews (locale : string) =
System.Guid "4e08377c-1240-4f80-9c35-0bacde2c66b6"
|> ReviewUrl.getUri (locale.Substring 3) locale
|> downloadPage
|> parse
[<EntryPoint>]
let main argv =
let locales = [| "en-US"; "en-GB"; |]
let results = locales |> Array.map downloadReviews
printfn "%A" results
0

Resources