I have the following code that uses Sequence objects to read data from a database table.
V1 works correctly but since the Seq.generate function is deprecated I receive compiler warnings.
I tried to rerplace this code with V2 (below) but it does not work correctly -
basically the reader is opened but only the first record is read - IOW - Read Next does not
work correctly.
To avoid compilation errors/warnings I need to convert V1 code to use a sequence expression.
Any ideas on the correct approach here.
(BTW - This code is based on examples in Rob Pickering's Beginning F# book -
(Data Access/ADO.NET Section).
********************** V1 - Sequence Approach - Deprecated ************************
// execute a command using the Seq.generate
let execCommand (connName: string) (cmdString: string) =
Seq.generate
// This function gets called to open a connection and create a reader
(fun () -> openReader connName cmdString)
// This function gets called to read a single item in
// the enumerable for a reader/connection pair
(fun reader -> readRow(reader))
(fun reader -> reader.Dispose())
*********************** V2 Alternative - (Does not work) ***************************
let generateSequence connName cmdString =
seq { // This function gets called to open a connection and
//create a reader
use reader = openReader connName cmdString
// This function gets called to read a single item in
// the enumerable for a reader/connection pair
yield readRow(reader) }
// execute a command using the Seq.generate
let execCommand (connName: string) (cmdString: string) =
generateSequence connName cmdString
***************************** Common Functions **********************************
// Open a db connection
let openReader connName cmdString =
let conn = openSQLConnection(connName)
let cmd = conn.CreateCommand(CommandText=cmdString,
CommandType = CommandType.Text)
let reader = cmd.ExecuteReader(CommandBehavior.CloseConnection)
// read a row from the data reader
let readRow (reader: #DbDataReader) =
if reader.Read() then
let dict = new Dictionary<string, obj>()
for x in [ 0 .. (reader.FieldCount - 1) ] do
dict.Add(reader.GetName(x), reader.[x])
Some(dict)
else
None
You need to move around a few control structures to get this working.
let generateSequence connName cmdString =
seq {
use reader = openReader connName cmdString
while reader.Read () do
yield readRow reader
yield None
}
let readRow (reader:#DbDataReader) =
let dict = new Dictionary<string, obj>()
for x in [0..(reader.FieldCount - 1)] do
dict.Add (reader.GetName(x), reader.[x])
Some dict
The only thing you left out is: while reader.Read() do
let generateSequence connName cmdString =
seq {
use reader = openReader connName cmdString
while reader.Read() do
yield readRow reader
}
Related
I have a bunch of files several MiB in size which are very simple:
They have a size of multiples of 8
They only contain doubles in little endian, so can be read with BinaryReader's ReadDouble() method
When lexicographically sorted, they contain all values in the sequence they need to be.
I can't keep everything in memory as a float list or float array so I need a float seq that goes through the necessary files when actually being accessed. The portion that goes through the sequence actually does it in imperative style using GetEnumerator() because I don't want any resource leaks and want to close all files correctly.
My first functional approach was:
let readFile file =
let rec readReader (maybeReader : BinaryReader option) =
match maybeReader with
| None ->
let openFile() =
printfn "Opening the file"
new BinaryReader(new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read))
|> Some
|> readReader
seq { yield! openFile() }
| Some reader when reader.BaseStream.Position >= reader.BaseStream.Length ->
printfn "Closing the file"
reader.Dispose()
Seq.empty
| Some reader ->
reader.BaseStream.Position |> printfn "Reading from position %d"
let bytesToRead = Math.Min(1048576L, reader.BaseStream.Length - reader.BaseStream.Position) |> int
let bytes = reader.ReadBytes bytesToRead
let doubles = Array.zeroCreate<float> (bytesToRead / 8)
Buffer.BlockCopy(bytes, 0, doubles, 0, bytesToRead)
seq {
yield! doubles
yield! readReader maybeReader
}
readReader None
And then, when I have a string list containing all the files, I can say something like:
let values = files |> Seq.collect readFile
use ve = values.GetEnumerator()
// Do stuff that only gets partial data from one file
However, this only closes the files when the reader reaches its end (which is clear when looking at the function). So as a second approach I implemented the file enumerating imperatively:
type FileEnumerator(file : string) =
let reader = new BinaryReader(new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read))
let mutable _current : float = Double.NaN
do file |> printfn "Enumerator active for %s"
interface IDisposable with
member this.Dispose() =
reader.Dispose()
file |> printfn "Enumerator disposed for %s"
interface IEnumerator with
member this.Current = _current :> obj
member this.Reset() = reader.BaseStream.Position <- 0L
member this.MoveNext() =
let stream = reader.BaseStream
if stream.Position >= stream.Length then false
else
_current <- reader.ReadDouble()
true
interface IEnumerator<float> with
member this.Current = _current
type FileEnumerable(file : string) =
interface IEnumerable with
member this.GetEnumerator() = new FileEnumerator(file) :> IEnumerator
interface IEnumerable<float> with
member this.GetEnumerator() = new FileEnumerator(file) :> IEnumerator<float>
let readFile' file = new FileEnumerable(file) :> float seq
now, when I say
let values = files |> Seq.collect readFile'
use ve = values.GetEnumerator()
// do stuff with the enumerator
disposing the enumerator correctly bubbles through to my imperative enumerator.
While this is a feasible solution for what I want to achieve (I could make it faster by reading it blockwise like the first functional approach but for brevity I didn't do it here) I wonder if there is a truly functional approach for this avoiding the mutable state in the enumerator.
I don't quite get what you mean when you say that using GetEnumerator() will prevent resource leaks and allow to close all files correctly. The below would be my attempt at this (ignoring block copy part for demonstration purposes) and I think it results in the files properly closed.
let eof (br : BinaryReader) =
br.BaseStream.Position = br.BaseStream.Length
let readFileAsFloats filePath =
seq{
use file = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read)
use reader = new BinaryReader(file)
while (not (eof reader)) do
yield reader.ReadDouble()
}
let readFilesAsFloats filePaths =
filePaths |> Seq.collect readFileAsFloats
let floats = readFilesAsFloats ["D:\\floatFile1.txt"; "D:\\floatFile2.txt"]
Is that what you had in mind?
Im trying to populate list with my own type.
let getUsers =
use connection = openConnection()
let getString = "select * from Accounts"
use sqlCommand = new SqlCommand(getString, connection)
try
let usersList = [||]
use reader = sqlCommand.ExecuteReader()
while reader.Read() do
let floresID = reader.GetString 0
let exName = reader.GetString 1
let exPass = reader.GetString 2
let user = [floresID=floresID; exName=exName; exPass=exPass]
// what here?
()
with
| :? SqlException as e -> printfn "Došlo k chybě úrovni připojení:\n %s" e.Message
| _ -> printfn "Neznámá výjimka."
In C# I would just add new object into userList. How can I add new user into list? Or is it better approach to get some sort of list with data from database?
Easiest way to do this is with a type provider, so you can abstract away the database. You can use SqlDataConnection for SQLServer, SqlProvider for everything (incl. SQLServer), and also SQLClient for SQLServer.
Here is an example with postgres's dvdrental (sample) database for SQLProvider:
#r #"..\packages\SQLProvider.1.0.33\lib\FSharp.Data.SqlProvider.dll"
#r #"..\packages\Npgsql.3.1.8\lib\net451\Npgsql.dll"
open System
open FSharp.Data.Sql
open Npgsql
open NpgsqlTypes
open System.Linq
open System.Xml
open System.IO
open System.Data
let [<Literal>] dbVendor = Common.DatabaseProviderTypes.POSTGRESQL
let [<Literal>] connString1 = #"Server=localhost;Database=dvdrental;User Id=postgres;Password=root"
let [<Literal>] resPath = #"C:\Users\userName\Documents\Visual Studio 2015\Projects\Postgre2\packages\Npgsql.3.1.8\lib\net451"
let [<Literal>] indivAmount = 1000
let [<Literal>] useOptTypes = true
//create the type for the database, based on the connection string, etc. parameters
type sql = SqlDataProvider<dbVendor,connString1,"",resPath,indivAmount,useOptTypes>
//set up the datacontext, ideally you would use `use` here :-)
let ctx = sql.GetDataContext()
let actorTbl = ctx.Public.Actor //alias the table
//set up the type, in this case Records:
type ActorName = {
firstName:string
lastName:string}
//extract the data with a query expression, this gives you type safety and intellisense over SQL (but also see the SqlClient type provider above):
let qry = query {
for row in actorTbl do
select ({firstName=row.FirstName;lastName=row.LastName})
}
//seq is lazy so do all kinds of transformations if necessary then manifest it into a list or array:
qry |> Seq.toArray
The two important parts are defining the Actor record, and then in the query extracting the fields into a sequence of Actor records. You can then manifest into a list or array if necessary.
But you can also stick to your original solution. In that case just wrap the .Read() into a seq:
First define the type:
type User = {
floresID: string
exName: string
exPass: string
}
Then extract the data:
let recs = cmd.ExecuteReader() // execute the SQL Command
//extract the users into a sequence of records:
let users =
seq {
while recs.Read() do
yield {floresID=recs.[0].ToString()
exName=recs.[1].ToString()
exPass=recs.[2].ToString()
}
} |> Seq.toArray
Taking your code, you can use list expression:
let getUsers =
use connection = openConnection()
let getString = "select * from Accounts"
use sqlCommand = new SqlCommand(getString, connection)
try
[
use reader = sqlCommand.ExecuteReader()
while reader.Read() do
let floresID = reader.GetString 0
let exName = reader.GetString 1
let exPass = reader.GetString 2
let user = [floresID=floresID; exName=exName; exPass=exPass]
yield user
]
with
| :? SqlException as e -> failwithf "Došlo k chybě úrovni připojení:\n %s" e.Message
| _ -> failwithf "Neznámá výjimka."
That being said, I'd use FSharp.Data.SqlClient library so all of that boiler plate becomes a single line with added benefit of type safety (if you change the query, the code will have compile time error which are obvious to fix).
I'm try to write a tool that compares two db using F#'s SqlDataProvider as the data access. This means excuting the same query on two different databases. The would be easy, if I could pass the data content to a function as a parameter, however, because the data context is a generated type is doesn't seem to have proper name, so I'm unable to pass it as a parameter.
Here an example of what I'd like to able to do:
type MyDb = SqlDataProvider<
#"Server=myServerDatabase=myDatabase;Trusted_Connection=True;",
Common.DatabaseProviderTypes.MSSQLSERVER>
let ctx1 = RfqDb.GetDataContext("Server=myServerDatabase=myDatabase;Trusted_Connection=True;")
let ctx2 = RfqDb.GetDataContext("Server=myServerDatabase=myOtherDatabase;Trusted_Connection=True;")
let getGetData (ctx: ...) = // don't know what to put for ...
query { for ue in ctx.``[dbo].[MyTable]`` do
where (ue.UnderlyingID = "MyId")}
|> Seq.toArray
let grid1 = new EntityViewGrid()
let grid2 = new EntityViewGrid()
grid1.ItemsSource <- getGetData ctx1
grid2.ItemsSource <- getGetData ctx2
It's the line with the // don't know what to put for ... comment that's giving me problems.
Just figured it out, simpler than I thought, just the VS tooltips are a bit misleading. The correct sample looks like:
type MyDb = SqlDataProvider<
#"Server=myServerDatabase=myDatabase;Trusted_Connection=True;",
Common.DatabaseProviderTypes.MSSQLSERVER>
let ctx1 = RfqDb.GetDataContext("Server=myServerDatabase=myDatabase;Trusted_Connection=True;")
let ctx2 = RfqDb.GetDataContext("Server=myServerDatabase=myOtherDatabase;Trusted_Connection=True;")
let getGetData (ctx: MyDb.dataContext) =
query { for ue in ctx.``[dbo].[MyTable]`` do
where (ue.UnderlyingID = "MyId")}
|> Seq.toArray
let grid1 = new EntityViewGrid()
let grid2 = new EntityViewGrid()
grid1.ItemsSource <- getGetData ctx1
grid2.ItemsSource <- getGetData ctx2
I wrote the following code to execute a SQLServer StoredProc in F#
module SqlUtility =
open System
open System.Data
open System.Data.SqlClient
SqlUtility.GetSqlConnection "MyDB"
|> Option.bind (fun con -> SqlUtility.GetSqlCommand "dbo.usp_MyStordProc" con)
|> Option.bind (fun cmd ->
let param1 = new SqlParameter("#User", SqlDbType.NVarChar, 50)
param1.Value <- user
cmd.Parameters.Add(param1) |> ignore
let param2 = new SqlParameter("#PolicyName", SqlDbType.NVarChar, 10)
param2.Value <- policyName
cmd.Parameters.Add(param2) |> ignore
Some(cmd)
)
|> Option.bind (fun cmd -> SqlUtility.ExecuteReader cmd)
|> Option.bind (fun rdr -> ExtractValue rdr)
let GetSqlConnection (conName : string) =
let conStr = ConfigHandler.GetConnectionString conName
try
let con = new SqlConnection(conStr)
con.Open()
Some(con)
with
| :? System.Exception as ex -> printfn "Failed to connect to DB %s with Error %s " conName ex.Message; None
| _ -> printfn "Failed to connect to DB %s" conName; None
let GetSqlCommand (spName : string) (con : SqlConnection) =
let cmd = new SqlCommand()
cmd.Connection <- con
cmd.CommandText <- spName
cmd.CommandType <- CommandType.StoredProcedure
Some(cmd)
let AddParameters (cmd : SqlCommand) (paramList : SqlParameter list) =
paramList |> List.iter (fun p -> cmd.Parameters.Add p |> ignore)
let ExecuteReader (cmd : SqlCommand ) =
try
Some(cmd.ExecuteReader())
with
| :? System.Exception as ex -> printfn "Failed to execute reader with error %s" ex.Message; None
I have multiple problems with this code
First and foremost the repeated use of Option.bind is very irritating... and is adding noise. I need a more clearer way to check if the output was None and if not then proceed.
At the end there should be a cleanupfunction where I should be able to close + dispose the reader, command and connection. But currently at the end of the pipeline all I have is the reader.
The function which is adding parameters... it looks like it is modifying the "state" of the command parameter because the return type is still the same command which was sent it... with some added state. I wonder how a more experienced functional programmer would have done this.
Visual Studio gives me a warning at each of the place where i do exception handling. what's wrong with that" it says
This type test or downcast will always hold
The way I want this code to look is this
let x : MyRecord seq = GetConnection "con" |> GetCommand "cmd" |> AddParameter "#name" SqlDbType.NVarchar 50 |> AddParameter "#policyname" SqlDbType.NVarchar 50 |> ExecuteReader |> FunctionToReadAndGenerateSeq |> CleanEverything
Can you recommend how can I take my code to the desired level and also any other improvement?
I think that using options to represent failed computations is more suitable to purely functional langauges. In F#, it is perfectly fine to use exceptions to denote that a computation has failed.
Your code simply turns exceptions into None values, but it does not really handle this situation - this is left to the caller of your code (who will need to decide what to do with None). You may as well just let them handle the exception. If you want to add more information to the exception, you can define your own exception type and throw that instead of leaving the standard exceptions.
The following defines a new exception type and a simple function to throw it:
exception SqlUtilException of string
// This supports the 'printf' formatting style
let raiseSql fmt =
Printf.kprintf (SqlUtilException >> raise) fmt
Using plain .NET style with a few simplifications using F# features, the code looks a lot simpler:
// Using 'use' the 'Dispose' method is called automatically
let connName = ConfigHandler.GetConnectionString "MyDB"
use conn = new SqlConnection(connName)
// Handle exceptions that happen when opening the connection
try conn.Open()
with ex -> raiseSql "Failed to connect to DB %s with Error %s " connName ex.Message
// Using object initializer, we can nicely set the properties
use cmd =
new SqlCommand( Connection = conn, CommandText = "dbo.usp_MyStordProc",
CommandType = CommandType.StoredProcedure )
// Add parameters
// (BTW: I do not think you need to set the type - this will be infered)
let param1 = new SqlParameter("#User", SqlDbType.NVarChar, 50, Value = user)
let param2 = new SqlParameter("#PolicyName", SqlDbType.NVarChar, 10, Value = policyName)
cmd.Parameters.AddRange [| param1; param2 |]
use reader =
try cmd.ExecuteReader()
with ex -> raiseSql "Failed to execute reader with error %s" ex.Message
// Do more with the reader
()
It looks more like .NET code, but that is perfectly fine. Dealing with databases in F# is going to use imperative style and trying to hide that will only make the code confusing. Now, there is a number of other neat F# features you could use - especially the support for dynamic operators ?, which would give you something like:
let connName = ConfigHandler.GetConnectionString "MyDB"
// A wrapper that provides dynamic access to database
use db = new DynamicDatabase(connName)
// You can call stored procedures using method call syntax
// and pass SQL parameters as standard arguments
let rows = db.Query?usp_MyStordProc(user, policy)
// You can access columns using the '?' syntax again
[ for row in rows -> row?Column1, row?Column2 ]
For more information about this, see the following MSDN series:
How to: Dynamically Invoke a Stored Procedure
Step 1: Create a Database and Show the Poll Options
Step 2: Implement Voting for an Option
This is C# version:
public static IEnumerable<string> ReadLinesEnumerable(string path) {
using ( var reader = new StreamReader(path) ) {
var line = reader.ReadLine();
while ( line != null ) {
yield return line;
line = reader.ReadLine();
}
}
}
But directly translating needs a mutable variable.
If you're using .NET 4.0, you can just use File.ReadLines.
> let readLines filePath = System.IO.File.ReadLines(filePath);;
val readLines : string -> seq<string>
open System.IO
let readLines (filePath:string) = seq {
use sr = new StreamReader (filePath)
while not sr.EndOfStream do
yield sr.ReadLine ()
}
To answer the question whether there is a library function for encapsulating this pattern - there isn't a function exactly for this, but there is a function that allows you to generate sequence from some state called Seq.unfold. You can use it to implement the functionality above like this:
new StreamReader(filePath) |> Seq.unfold (fun sr ->
match sr.ReadLine() with
| null -> sr.Dispose(); None
| str -> Some(str, sr))
The sr value represents the stream reader and is passed as the state. As long as it gives you non-null values, you can return Some containing an element to generate and the state (which could change if you wanted). When it reads null, we dispose it and return None to end the sequence. This isn't a direct equivalent, because it doesn't properly dispose StreamReader when an exception is thrown.
In this case, I would definitely use sequence expression (which is more elegant and more readable in most of the cases), but it's useful to know that it could be also written using a higher-order function.
let lines = File.ReadLines(path)
// To check
lines |> Seq.iter(fun x -> printfn "%s" x)
On .NET 2/3 you can do:
let readLines filePath = File.ReadAllLines(filePath) |> Seq.cast<string>
and on .NET 4:
let readLines filePath = File.ReadLines(filePath);;
In order to avoid the "System.ObjectDisposedException: Cannot read from a closed TextReader." exception, use:
let lines = seq { yield! System.IO.File.ReadLines "/path/to/file.txt" }