Construct Query to database dynamically - f#

I am currently using FSharp.Data.SqlClient, but I am interested in any solution of my task.
I have a web client and the backend written in F#. In the web client user can set 10-20 filters and make the GET request to my F# backend. The URL looks like:
http://mybackend.com/get/1/0/34/ ....
Any number in this URL is a filter. There is no filter on the corresponding field if the value of corresponding filter is zero.
Now I need to construct the SQL query in my backend. While the number of possible filters was 3-4 I was using pattern matching:
type GetEObyOrg1Org2AsutpParent = SqlCommandProvider<"SELECT * FROM ITEM WHERE ORGID1 = #org1 AND ORGID2 = #org2 AND ASUTPID = #asutp AND PARENTAUTOINCID = #parent", "name=MobileConnectionProvider", ResultType.Tuples>
type GetEObyOrg1Org2Org3AsutpParent = SqlCommandProvider<"SELECT * FROM ITEM WHERE ORGID1 = #org1 AND ORGID2 = #org2 AND ORGID3 = #org3 AND ASUTPID = #asutp AND PARENTAUTOINCID = #parent", "name=MobileConnectionProvider", ResultType.Tuples>
match (asutpid, orgId1, orgId2, orgId3) with
| "0", _, "0", "0" ->
let cmd = new GetEObyOrg1Org2AsutpParent()
cmd.Execute(org1 = orgId1, parent = parentAid)
| "0", _, _ , "0" ->
let cmd = new GetEObyOrg1Org2Org3AsutpParent()
cmd.Execute(org1 = orgId1, org2 = orgId2, parent = parentAid)
But when the number of filters is greater then 3-4 it is very difficult to write pattern matching for all combinations of the parameters.
I think I need to construct the SQL query dynamically. Each non-zero parameter in the URL must add AND Field = Value expression to SQL statement. But I can not do it:
type GetEObyOrg1AsutpParent = SqlCommandProvider<Query, "name=MobileConnectionProvider", ResultType.Tuples>
Query paramter is this expression must be Literal, and I can not to construct literal expression dinamically:
[<Literal>]
let q1 = "SELECT * FROM ITEM WHERE ORGID1 = #org1 AND ASUTPID = #asutp"
[<Literal>]
let q2 = " AND PARENTAUTOINCID = #parent"
let f a =
match a with
| 1 -> q1 + q2
| _ -> q1
[<Literal>]
let q3 = f()
What is the best way of doing it in my case ?

I think in a case like this the core library's SqlDataConnection type provider might be a better fit. Then you should be able to use standard IQueryable operations to dynamically add additional WHERE clauses, something like:
type MyDb = SqlDataConnection<...>
let items = MyDb.GetContext().Item
let q1 = if asutpid = 0 then items else items.Where(fun i -> i.Asutpid = asutpid)
let q2 = if orgId1 = 0 then q1 else q1.Where(fun i -> i.OrgId1 = orgid1)
...

Related

Cannot Get Value When OleDbParameter Used

It is not good form to create a query like
let fnam_query =
"select * from file_name_info where fnam_origin = 'invoice_cloud'"
But the code block below has two problems. First, the fnam_readOk returns false from the read.
Second, how can the OleDbParameter be disposed? I tried using use, but got a compile-time error saying OleDbType.Char could not be used within a use.
let fnam_query =
"select * from file_name_info where fnam_origin = '?' "
use fnam_cmd = new OleDbCommand(fnam_query, db_con)
let local_params = new OleDbParameter("fnam_origin", OleDbType.Char)
fnam_cmd.Parameters.Add(local_params) |> ignore
let fnam_reader = fnam_cmd.ExecuteReader ()
let fnam_readOK = fnam_reader.Read ()
let ic_lb_fnam =
if fnam_readOK then
fnam_reader.GetString(2)
else
"ic_lockbox.txt"
There are a lot of problems with this code. One of them, is that you used the OleDbParameter overload that passes a name and value. The line new OleDbParameter("fnam_origin", OleDbType.Char) specifies a parameter with the name fnam_origin and an integer value equal to whatever the underlying value of OleDbType.Char is.
The other problem is that you don't use that parameter at all. '?' is just a string that contains ?.
You don't need to quote parameters in a parameterized query. They aren't placeholders for string replacement. They specify actual, strongly typed parameters, just like an F# function parameter.
Your query should be :
let fnam_query =
"select * from file_name_info where fnam_origin = ? "
You should also use the correct parameter type. Char is used only for fixed-length parameters. You should use VarChar or even better, NVarchar.
Finally, you should pass the parameter value that you want. Your code doesn't specify a parameter value at all.
The entire function should look like this :
let fnam_query = "select * from file_name_info where fnam_origin = ? "
use db_con = new OleDbConnection("...")
use fnam_cmd = new OleDbCommand(fnam_query, db_con)
let local_params = new OleDbParameter("origin", SqlDbType.NVarChar,100)
fnam_cmd.Parameters.Add(local_params) |> ignore
local_params.Value <- "GR"
db_con.Open()
let fnam_reader = fnam_cmd.ExecuteReader ()
let fnam_readOK = fnam_reader.Read ()
...
A better implementation though, would be to create the command once and reuse it with different connections and values:
let build_cmd =
let fnam_query = "select * from file_name_info where fnam_origin = ? "
let fnam_cmd = new OleDbCommand(fnam_query)
let local_params = new OleDbParameter("whatever", SqlDbType.NVarChar,100)
fnam_cmd.Parameters.Add(local_params) |> ignore
fnam_cmd
use db_con = new OleDbConnection("...")
build_cmd.Connection <- db_con
build_cmd.Parameters.[0].Value <- "GR"
db_con.Open()
Based on the excellent answer I got from Panagiotis Kanavos, I've altered a different place in my code where I needed parameters that were not embedded in the query string. I chose to use cmd.Parameters.Add rather than have a separate OleDbParameter val.
(* by looking at the xfer_type, really the arg passed to main,
we can determine the report type parameter for the Access
database. *)
let select_report_type xfer_type =
match xfer_type with
| "/al" -> 0
| "/am" -> 1
| "/ap" -> 2
| "/pm" -> 3
| "/pp" -> 4
| _ -> 99
let query = "select count(*) from ProcessStatus where ReportType = ? and ReportDate = ? and ReportFileName = ? "
use cmd = new OleDbCommand(query , db_con)
cmd.Parameters.Add(new OleDbParameter("ReportType",(OleDbType.VarChar,5))) |> ignore
cmd.Parameters.[0].Value <- ((select_report_type xfer_type).ToString())
cmd.Parameters.Add(new OleDbParameter("ReportDate",OleDbType.VarChar, 11)) |> ignore
cmd.Parameters.[1].Value <- report_date
cmd.Parameters.Add(new OleDbParameter("ReportFileName",OleDbType.VarChar, 100)) |> ignore
cmd.Parameters.[2].Value <- fn
let sql_reader = cmd.ExecuteReader ()
if (sql_reader.Read ()) then

Populate list with Types

Im trying to populate list with my own type.
let getUsers =
use connection = openConnection()
let getString = "select * from Accounts"
use sqlCommand = new SqlCommand(getString, connection)
try
let usersList = [||]
use reader = sqlCommand.ExecuteReader()
while reader.Read() do
let floresID = reader.GetString 0
let exName = reader.GetString 1
let exPass = reader.GetString 2
let user = [floresID=floresID; exName=exName; exPass=exPass]
// what here?
()
with
| :? SqlException as e -> printfn "Došlo k chybě úrovni připojení:\n %s" e.Message
| _ -> printfn "Neznámá výjimka."
In C# I would just add new object into userList. How can I add new user into list? Or is it better approach to get some sort of list with data from database?
Easiest way to do this is with a type provider, so you can abstract away the database. You can use SqlDataConnection for SQLServer, SqlProvider for everything (incl. SQLServer), and also SQLClient for SQLServer.
Here is an example with postgres's dvdrental (sample) database for SQLProvider:
#r #"..\packages\SQLProvider.1.0.33\lib\FSharp.Data.SqlProvider.dll"
#r #"..\packages\Npgsql.3.1.8\lib\net451\Npgsql.dll"
open System
open FSharp.Data.Sql
open Npgsql
open NpgsqlTypes
open System.Linq
open System.Xml
open System.IO
open System.Data
let [<Literal>] dbVendor = Common.DatabaseProviderTypes.POSTGRESQL
let [<Literal>] connString1 = #"Server=localhost;Database=dvdrental;User Id=postgres;Password=root"
let [<Literal>] resPath = #"C:\Users\userName\Documents\Visual Studio 2015\Projects\Postgre2\packages\Npgsql.3.1.8\lib\net451"
let [<Literal>] indivAmount = 1000
let [<Literal>] useOptTypes = true
//create the type for the database, based on the connection string, etc. parameters
type sql = SqlDataProvider<dbVendor,connString1,"",resPath,indivAmount,useOptTypes>
//set up the datacontext, ideally you would use `use` here :-)
let ctx = sql.GetDataContext()
let actorTbl = ctx.Public.Actor //alias the table
//set up the type, in this case Records:
type ActorName = {
firstName:string
lastName:string}
//extract the data with a query expression, this gives you type safety and intellisense over SQL (but also see the SqlClient type provider above):
let qry = query {
for row in actorTbl do
select ({firstName=row.FirstName;lastName=row.LastName})
}
//seq is lazy so do all kinds of transformations if necessary then manifest it into a list or array:
qry |> Seq.toArray
The two important parts are defining the Actor record, and then in the query extracting the fields into a sequence of Actor records. You can then manifest into a list or array if necessary.
But you can also stick to your original solution. In that case just wrap the .Read() into a seq:
First define the type:
type User = {
floresID: string
exName: string
exPass: string
}
Then extract the data:
let recs = cmd.ExecuteReader() // execute the SQL Command
//extract the users into a sequence of records:
let users =
seq {
while recs.Read() do
yield {floresID=recs.[0].ToString()
exName=recs.[1].ToString()
exPass=recs.[2].ToString()
}
} |> Seq.toArray
Taking your code, you can use list expression:
let getUsers =
use connection = openConnection()
let getString = "select * from Accounts"
use sqlCommand = new SqlCommand(getString, connection)
try
[
use reader = sqlCommand.ExecuteReader()
while reader.Read() do
let floresID = reader.GetString 0
let exName = reader.GetString 1
let exPass = reader.GetString 2
let user = [floresID=floresID; exName=exName; exPass=exPass]
yield user
]
with
| :? SqlException as e -> failwithf "Došlo k chybě úrovni připojení:\n %s" e.Message
| _ -> failwithf "Neznámá výjimka."
That being said, I'd use FSharp.Data.SqlClient library so all of that boiler plate becomes a single line with added benefit of type safety (if you change the query, the code will have compile time error which are obvious to fix).

F# Read Fixed Width Text File

Hi I'm looking to find the best way to read in a fixed width text file using F#. The file will be plain text, from one to a couple of thousand lines long and around 1000 characters wide. Each line contains around 50 fields, each with varying lengths. My initial thoughts were to have something like the following
type MyRecord = {
Name : string
Address : string
Postcode : string
Tel : string
}
let format = [
(0,10)
(10,50)
(50,7)
(57,20)
]
and read each line one by one, assigning each field by the format tuple(where the first item is the start character and the second is the number of characters wide).
Any pointers would be appreciated.
The hardest part is probably to split a single line according to the column format. It can be done something like this:
let splitLine format (line : string) =
format |> List.map (fun (index, length) -> line.Substring(index, length))
This function has the type (int * int) list -> string -> string list. In other words, format is an (int * int) list. This corresponds exactly to your format list. The line argument is a string, and the function returns a string list.
You can map a list of lines like this:
let result = lines |> List.map (splitLine format)
You can also use Seq.map or Array.map, depending on how lines is defined. Such a result will be a string list list, and you can now map over such a list to produce a MyRecord list.
You can use File.ReadLines to get a lazily evaluated sequence of strings from a file.
Please note that the above is only an outline of a possible solution. I left out boundary checks, error handling, and such. The above code may contain off-by-one errors.
Here's a solution with a focus on custom validation and error handling for each field. This might be overkill for a data file consisting of just numeric data!
First, for these kinds of things, I like to use the parser in Microsoft.VisualBasic.dll as it's already available without using NuGet.
For each row, we can return the array of fields, and the line number (for error reporting)
#r "Microsoft.VisualBasic.dll"
// for each row, return the line number and the fields
let parserReadAllFields fieldWidths textReader =
let parser = new Microsoft.VisualBasic.FileIO.TextFieldParser(reader=textReader)
parser.SetFieldWidths fieldWidths
parser.TextFieldType <- Microsoft.VisualBasic.FileIO.FieldType.FixedWidth
seq {while not parser.EndOfData do
yield parser.LineNumber,parser.ReadFields() }
Next, we need a little error handling library (see http://fsharpforfunandprofit.com/rop/ for more)
type Result<'a> =
| Success of 'a
| Failure of string list
module Result =
let succeedR x =
Success x
let failR err =
Failure [err]
let mapR f xR =
match xR with
| Success a -> Success (f a)
| Failure errs -> Failure errs
let applyR fR xR =
match fR,xR with
| Success f,Success x -> Success (f x)
| Failure errs,Success _ -> Failure errs
| Success _,Failure errs -> Failure errs
| Failure errs1, Failure errs2 -> Failure (errs1 # errs2)
Then define your domain model. In this case, it is the record type with a field for each field in the file.
type MyRecord =
{id:int; name:string; description:string}
And then you can define your domain-specific parsing code. For each field I have created a validation function (validateId, validateName, etc).
Fields that don't need validation can pass through the raw data (validateDescription).
In fieldsToRecord the various fields are combined using applicative style (<!> and <*>).
For more on this, see http://fsharpforfunandprofit.com/posts/elevated-world-3/#validation.
Finally, readRecords maps each input row to the a record Result and chooses the successful ones only. The failed ones are written to a log in handleResult.
module MyFileParser =
open Result
let createRecord id name description =
{id=id; name=name; description=description}
let validateId (lineNo:int64) (fields:string[]) =
let rawId = fields.[0]
match System.Int32.TryParse(rawId) with
| true, id -> succeedR id
| false, _ -> failR (sprintf "[%i] Can't parse id '%s'" lineNo rawId)
let validateName (lineNo:int64) (fields:string[]) =
let rawName = fields.[1]
if System.String.IsNullOrWhiteSpace rawName then
failR (sprintf "[%i] Name cannot be blank" lineNo )
else
succeedR rawName
let validateDescription (lineNo:int64) (fields:string[]) =
let rawDescription = fields.[2]
succeedR rawDescription // no validation
let fieldsToRecord (lineNo,fields) =
let (<!>) = mapR
let (<*>) = applyR
let validatedId = validateId lineNo fields
let validatedName = validateName lineNo fields
let validatedDescription = validateDescription lineNo fields
createRecord <!> validatedId <*> validatedName <*> validatedDescription
/// print any errors and only return good results
let handleResult result =
match result with
| Success record -> Some record
| Failure errs -> printfn "ERRORS %A" errs; None
/// return a sequence of records
let readRecords parserOutput =
parserOutput
|> Seq.map fieldsToRecord
|> Seq.choose handleResult
Here's an example of the parsing in practice:
// Set up some sample text
let text = """01name1description1
02name2description2
xxname3badid-------
yy badidandname
"""
// create a low-level parser
let textReader = new System.IO.StringReader(text)
let fieldWidths = [| 2; 5; 11 |]
let parserOutput = parserReadAllFields fieldWidths textReader
// convert to records in my domain
let records =
parserOutput
|> MyFileParser.readRecords
|> Seq.iter (printfn "RECORD %A") // print each record
The output will look like:
RECORD {id = 1;
name = "name1";
description = "description";}
RECORD {id = 2;
name = "name2";
description = "description";}
ERRORS ["[3] Can't parse id 'xx'"]
ERRORS ["[4] Can't parse id 'yy'"; "[4] Name cannot be blank"]
By no means is this the most efficient way to parse a file (I think there are some CSV parsing libraries available on NuGet that can do validation while parsing) but it does show how you can have complete control over validation and error handling if you need it.
A record of 50 fields is a bit unwieldy, therefore alternate approaches which allow dynamic generation of the data structure may be preferable (eg. System.Data.DataRow).
If it has to be a record anyway, you could spare at least the manual assignment to each record field and populate it with the help of Reflection instead. This trick relies on the field order as they are defined. I am assuming that every column of fixed width represents a record field, so that start indices are implied.
open Microsoft.FSharp.Reflection
type MyRecord = {
Name : string
Address : string
City : string
Postcode : string
Tel : string } with
static member CreateFromFixedWidth format (line : string) =
let fields =
format
|> List.fold (fun (index, acc) length ->
let str = line.[index .. index + length - 1].Trim()
index + length, box str :: acc )
(0, [])
|> snd
|> List.rev
|> List.toArray
FSharpValue.MakeRecord(
typeof<MyRecord>,
fields ) :?> MyRecord
Example data:
"Postman Pat " +
"Farringdon Road " +
"London " +
"EC1A 1BB" +
"+44 20 7946 0813"
|> MyRecord.CreateFromFixedWidth [16; 16; 16; 8; 16]
// val it : MyRecord = {Name = "Postman Pat";
// Address = "Farringdon Road";
// City = "London";
// Postcode = "EC1A 1BB";
// Tel = "+44 20 7946 0813";}

How can I manage an exclusive state in F#?

I am not sure about "exclusive state management" thing in the title, I did my best making it up trying to put the problem concisely.
I am porting some of my C# code to F# trying to do it as idiomatic as I can. I have an entity that requests a number of ID's from a sequence in my database and then dispenses these ID to anyone in need. Once an id is given out it should no longer be available for anybody else. Hence there must be some sort of state associated with that entity that keeps track of the remaining number of IDs. Since using a mutable state is not idiomatic, what I can do is to write something like this:
let createIdManager =
let idToStartWith = 127
let allowed = 10
let givenOut = 0
(idToStartWith, allowed, givenOut)
-
let getNextAvailableId (idToStartWith, allowed, givenOut) =
if givenOut< allowed
then ((idToStartWith, allowed, givenOut+ 1), Some(idToStartWith + givenOut))
else ((idToStartWith, allowed, givenOut), None)
let (idManager, idOpt) = getNextAvailableId createIdManager()
match idOpt with
| Some(id) -> printf "Yay!"
| None -> reloadIdManager idManager |> getNextAvailableId
This approach is idiomatic (as far as I can tell) but extremely vulnerable. There are so many ways to get it messed up. My biggest concern is that once an id is advanced and a newer copy of id manager is made, there is no force that can stop you from using the older copy and get the same id again.
So how do I do exclusive state management, per se, in F#?
If you only need to initialize the set of ids once then you can simply hide a mutable reference to a list inside a local function scope, as in:
let nextId =
let idsRef = ref <| loadIdsFromDatabase()
fun () ->
match idsRef.Value with
| [] ->
None
| id::ids ->
idsRef := ids
Some id
let id1 = nextId ()
let id2 = nextId ()
You could use a state-monad(Computational Expression).
First we declare the state-monad
type State<'s,'a> = State of ('s -> 'a * 's)
type StateBuilder<'s>() =
member x.Return v : State<'s,_> = State(fun s -> v,s)
member x.Bind(State v, f) : State<'s,_> =
State(fun s ->
let (a,s) = v s
let (State v') = f a
v' s)
let withState<'s> = StateBuilder<'s>()
let runState (State f) init = f init
Then we define your 'IdManager' and a function to get the next available id as well as the new state after the execution of the function.
type IdManager = {
IdToStartWith : int
Allowed : int
GivenOut : int
}
let getNextId state =
if state.Allowed > state.GivenOut then
Some (state.IdToStartWith + state.GivenOut), { state with GivenOut = state.GivenOut + 1 }
else
None, state
Finally we define our logic that requests the ids and execute the state-monad.
let idStateProcess =
withState {
let! id1 = State(getNextId)
printfn "Got id %A" id1
let! id2 = State(getNextId)
printfn "Got id %A" id2
//...
return ()
}
let initState = { IdToStartWith = 127; Allowed = 10; GivenOut = 0 }
let (_, postState) =
runState
idStateProcess
initState //This should be loaded from database in your case
Output:
Got id Some 127
Got id Some 128

retrieving all results in RavenDB

I thought I could force to retrieve all results through multiple page and skip, using the statistics function
type Linq.IRavenQueryable<'T>
with member q.getAll() = let mutable stat = Linq.RavenQueryStatistics()
let total = stat.TotalResults
let a = q.Statistics(&stat)
let rec addone n = seq { yield q.Skip(n*1024).Take(1024).ToArray()
if n*1024 < total then
yield! addone (n + 1) }
addone 0 |> Array.concat
It works when you do
let q = session.Query<productypfield>()
let r = q.getAll()
but breaks with
let q = session.Query<productypfield>().Where(System.Func ....)
let r = q.getAll()
As the type Linq.IRavenQueryable is not idempotent through Linq composition : If I use Linq, I get an IEnumerable on which no q.Statistics(&stat) is defined.
I read the doc, and I dont see anyway to keep the type through Linq composition.
IS the only way to loop a fixed (high) amount of times, or set a high servepagesize, and take(a lot of elements) ?
edit : actually, even the code above does not work as apparently, to get a valid count, you need to run the query once. one has to call Take(0) to trigger it.
use session = store.OpenSession()
let q = session.Query<productypfield>()
let mutable s = Linq.RavenQueryStatistics()
let a = q.Statistics(&s)
s.TotalResults = 0 //true
printfn "%A" a //triggers evaluation
s.TotalResults = 0 //false
Can you change your 2nd code sample to this (I'm not familiar with F#):
let q = session.Query<productypfield>().Where(Expression<System.Func<....>>)
let r = q.getAll()
that should let you keep the IQueryable that you need

Resources