I am not very proficient in F#, but I have to find out why one of two pieces of code does not work.
The first one checks, if a file exists and then, if it does streams it, this one works.
The second one only checks, if a file exists with the same function and returns an error, if it doesn't, this one doesn't seem to be working, even though it uses the same function.
File stream working:
let pdfmaybe =
ctx.GetService<IConfiguration>()
|> getBasePath
|> getFileName clId theDoc
|> getDocument
match pdfmaybe with
| Some pdf ->
do! ...
File exists check, not working, always returns 404:
let fileExists =
ctx.GetService<IConfiguration>()
|> getBasePath
|> getFileName clId theDoc
|> doesFileExist
match fileExists with
| true ->
return! setStatusCode 200 next ctx
| false ->
return! setStatusCode 404 next ctx
Code for file exists and getDocument:
let doesFileExist fileName : bool =
fileName
|> File.Exists
let getDocument fileName : Option<FileStream> =
if (fileName |> doesFileExist) then
Some (fileName |> File.OpenRead)
else
None
I am sorry, it turned out we have some very old middleware that changes the formatting for the later request and it is case sensitive.
Related
I need to request data from several URLs and then use the results.
I am using plain Fable 3 with the Fable-Promise and Fable-Fetch libraries.
I have worked out how to fetch from multiple URLs and combine the results into a single Promise that I can then use to update the UI (the multiple results need to be drawn only once).
But if one of the fetch errors then the whole thing falls over. Ideally I'd like to use tryFetch and then propagate the Result<TermData, None | Exception> but nothing I do seems to compile.
How exactly do I use tryFetch and then unwrap the result with a second let! in the CE? (The comments explain more)
module App
open Browser.Dom
open App
open System.Collections.Generic
open System.Text.RegularExpressions
open Fetch
open System
type TermData =
abstract counts : int []
abstract scores : int []
abstract term : string
abstract allWords : bool
type QueryTerm =
{ mutable Term: string
mutable AllWords: bool }
let loadSingleSeries (term: QueryTerm) =
promise {
let url =
$"/api/plot/{term.Term}?allWords={term.AllWords}"
// Works but doesn't handle errors.
let! plotData = fetch url [] // type of plotData: Response
// let plotDataResult = tryFetch url []
// This ideally becomes Promise<Result<TermData, None>>
// let unwrapped = match plotDataResult with
// | Ok res -> Ok (res.json<TermData>()) // type: Promise<TermData>
// | Error err -> ??? // tried Error (Promise.create(fun resolve reject -> resolve None)) among others
let! result = plotData.json<TermData>() // type of result: TermData
return result
}
let dataArrays =
parsed // type Dictionary<int, TermData>
|> Seq.map (fun term -> loadSingleSeries term.Value)
|> Promise.Parallel
|> Promise.map (fun allData -> console.log(allData))
// Here we will handle None when we have it
I don't have much Fable experience, but if I understand your question correctly, I think the following should work:
let loadSingleSeries (term: QueryTerm) =
promise {
let url =
$"/api/plot/{term.Term}?allWords={term.AllWords}"
let! plotDataResult = tryFetch url []
match plotDataResult with
| Ok resp ->
let! termData = resp.json<TermData>()
return Ok termData
| Error ex ->
return Error ex
}
The idea here is that if you get an error, you simply propagate that error in the new Result value. This returns a Promise<Result<TermData, Exception>>, as you requested.
Update: Fixed return type using a second let!.
I haven't run this code but looking at the docs it looks like you need to use Promise.catch
let loadSingleSeries (term: QueryTerm) =
promise {
let url =
$"/api/plot/{term.Term}?allWords={term.AllWords}"
let! plotDataResult =
fetch url []
|> Promise.map Ok // Wraps the "happy path" in a Result.Ok
|> Promise.catch (fun err ->
//handle the error
Error err)
return
match plotDataResult with
| Ok res -> ...
| Error err -> ...
}
I ended up having to use the pipeline rather than CE approach for this as follows:
let loadSingleSeries (term: QueryTerm) =
let url =
$"/api/plot/{term.Term}?allWords={term.AllWords}"
let resultPromise =
fetch url []
|> Promise.bind (fun response ->
let arr = response.json<TermData> ()
arr)
|> Promise.map (Ok)
|> Promise.catch (Error)
resultPromise
The key was using Promise.bind to convert the first promise to get the Response to the promise of Promise<TermData>. The map and catch then convert to a Promise<Result<TermData, exn>>.
I have a text file that contains the following and I need to retrieve the value assigned to taskId, which in this case is AWc34YBAp0N7ZCmVka2u.
projectKey=ProjectName
serverUrl=http://localhost:9090
serverVersion=10.5.32.3
strong text**interfaceUrl=http://localhost:9090/interface?id=ProjectName
taskId=AWc34YBAp0N7ZCmVka2u
taskUrl=http://localhost:9090/api/ce/task?id=AWc34YBAp0N7ZCmVka2u
I have two different ways of reading the file that I've wrote.
let readLines (filePath:string) = seq {
use sr = new StreamReader (filePath)
while not sr.EndOfStream do
yield sr.ReadLine ()
}
readLines (FindFile currentDirectory "../**/sample.txt")
|> Seq.iter (fun line ->
printfn "%s" line
)
and
let readLines (filePath:string) =
(File.ReadAllLines filePath)
readLines (FindFile currentDirectory "../**/sample.txt")
|> Seq.iter (fun line ->
printfn "%s" line
)
At this point, I don't know how to approach getting the value I need. Options that, I think, are on the table are:
use Contains()
Regex
Record type
Active Pattern
How can I get this value returned and fail if it doesn't exist?
I think all the options would be reasonable - it depends on how complex the file will actually be. If there is no escaping then you can probably just look for = in the line and use that to split the line into a key value pair. If the syntax is more complex, this might not always work though.
My preferred method would be to use Split on string - you can then filter to find values with your required key, map to get the value and use Seq.head to get the value:
["foo=bar"]
|> Seq.map (fun line -> line.Split('='))
|> Seq.filter (fun kvp -> kvp.[0] = "foo")
|> Seq.map (fun kvp -> kvp.[1])
|> Seq.head
Using active patterns, you could define a pattern that takes a string and splits it using = into a list:
let (|Split|) (s:string) = s.Split('=') |> List.ofSeq
This then lets you get the value using Seq.pick with a pattern matching that looks for strings where the substring before = is e.g. foo:
["foo=bar"] |> Seq.pick (function
| Split ["foo"; value] -> Some value
| _ -> None)
The active pattern trick is quite neat, but it might be unnecessarily complicating the code if you only need this in one place.
exception NoDirectory of string
let dir = DirectoryInfo "c:\\"
let dirExists (dir: DirectoryInfo): bool = dir.Exists
let rec getFiles (dir: DirectoryInfo) : FileInfo[] =
let subs =
match dirExists dir with
| true -> dir.GetDirectories ""
| false -> raise (NoDirectory("error"))
subs
|> Array.map (fun (dir: DirectoryInfo) -> getFiles dir)
|> Array.reduce (fun acc elem -> acc.append elem) List.empty
So the last line I'm trying reduce FileInfo[][] to FileInfo[] or more accurately FileInfo[][] to List<FileInfo>.
I assumed that because pipe only works on unary functions so that would mean function currying the Array.reduce but I kind of got stuck there, something like:
let unaryReduce (fileInfoArray: FileInfo[]) = Array.reduce (fun acc elem -> acc.append elem) List.empty
Am I right? Getting close... far away...
Edit
So after some modifying I have got this:
let dir = DirectoryInfo "c:\\"
let dirExists (dir: DirectoryInfo): bool = dir.Exists
let getFiles (dir: DirectoryInfo) : FileInfo[] = dir.GetFiles ""
let rec recursiveGetFiles (dir: DirectoryInfo) : FileInfo[] =
let subs =
match dirExists dir with
| true -> dir.GetDirectories ""
| false -> raise (NoDirectory("error"))
subs
|> Array.collect (fun (dir: DirectoryInfo) -> recursiveGetFiles dir)
|> Array.concat<FileInfo[]> getFiles <| dir
I only started looking at F# yesterday, I want to try out using pipes. The error I get is on the last line:
This expression was expected to have type
FileInfo [] -> 'a -> FileInfo []
but here has type
FileInfo [] []
This says to me that collect isn't being used properly to flatten the arrays...
If you want to use pipelining, something like this ought to work:
exception NoDirectory of string
// DirectoryInfo -> bool
let dirExists (dir: DirectoryInfo) = dir.Exists
// DirectoryInfo -> FileInfo []
let getFiles (dir: DirectoryInfo) = dir.GetFiles "."
// DirectoryInfo -> FileInfo []
let rec recursiveGetFiles dir =
match dirExists dir with
| true -> dir.GetDirectories "."
| false -> raise (NoDirectory "error")
|> Array.collect recursiveGetFiles
|> Array.append (getFiles dir)
I wouldn't recommend throwing exceptions like shown here, but I kept that part in order to keep it as close to the OP as possible. Once you're comfortable with the basics of F# and functional programming, however, you should learn about functional error handling using the Either monad. A good place to start is here: http://fsharpforfunandprofit.com/rop
The error you got from your edit comes from the precedence of back piping (<|) a |> f <| b is seen as (a |> f) <| b when what you want is a |> (f <| b) which is just a |> (f b) (as seen in Mark Seemann's answer)
That said you could write something (IMO) more readable that way :
let getAllFiles (dir: DirectoryInfo) =
let rec aux dir = [
for file in getFiles dir -> file
for subDir in getDirs dir do yield! aux subDir
]
if dirExists dir
then aux dir
else raise <| NoDirectory "error"
That way we separate the recursive part from the check part.
The recursive part is a lot simpler, more readable and returns a list instead of an array as wanted initially (I added a getDirs)
The check part is done with an if because a match for true/false doesn't add value here
and done only for the initial argument as there are little chance getDirs return non-existing directories*
*even though it's possible if someone delete one just at the wrong time, in that case you can rewrite it
let rec getAllFiles (dir: DirectoryInfo) = [
if dirExists dir then
for file in getFiles dir -> file
for subDir in getDirs dir do yield! getAllFiles subDir
]
And that way you don't even need to raise an error you just get an empty list for a non-existing directory.
You can also use EnumarateDirectories if you need a seq, although with IO I would be careful using lazy collections. This returns all the 56 files I have in my project (sub)directories:
open System.IO
let dir = DirectoryInfo __SOURCE_DIRECTORY__
dir.GetDirectories("*",SearchOption.AllDirectories) |> Array.collect (fun x -> x.GetFileSystemInfos())
If you replace Array.collect, with Array.map the top level collection will be that of the directories (exactly 5 as that's what I have).
I'm pretty new to F# so it's hard for me to change my mindset after many years of C#/Java OOP.
I have an event handler MyForm.SelectFile(filePath:String) that opens a dialog and let you select the file to read. Once the file is selected, Parser.LoadFile(filePath:String) is called:
static member LoadFile(filePath:String) =
if not <| ZipFile.IsZipFile(filePath) then
failwith "invalid file specified."
use zipFile = new ZipFile(filePath)
if zipFile.Count <> 2 || zipFile |> Seq.exists(fun x -> x.FileName <> "alpha" && x.FileName <> "beta") then
failwith "invalid file specified."
zipFile |> fun x -> Parser.Parse(x.OpenReader())
I'm always expecting the selected file to be a valid zip archive containing 2 files without extension: "alpha" and "beta".
First, is there a better way to sanitize my input?
My if statements are pretty long and I'm sure F# can provide better solutions, but I really can't figure out.
Second, using failwith is forcing me to handle exceptions in my MyForm.SelectFile(filePath:String) method and I think Options could be a better solution.
I can't figure out how to use them if I need to perform two different and consecutive checks (ZipFile.IsZipFile and content) because in between I have to instantiate a ZipFile.
In C# I would just return null whenever a check fails and then checking the return value against null would let me know whether I need to prompt an error or continue.
Current code:
type Parser with
static member isValidZipFile (zipFile:ZipFile) =
(zipFile.Count = 2) && (zipFile |> Seq.forall(fun x -> (x.FileName = "alpha") || (x.FileName = "beta")))
static member LoadFile(filePath:String) =
if not <| ZipFile.IsZipFile(filePath) then
None
else
use zipFile = new ZipFile(filePath)
if not <| Parser.isValidZipFile(zipFile) then
None
else
Some(seq { for zipEntry in zipFile do yield Parser.Parse(zipEntry.OpenReader()) } |> Seq.toArray)
First, the last line of your function could be a bit more elegant if it was written like:
zipFile.OpenReader() |> Parser.Parse
Second, you're on the right track as far as your thinking about using Option. It's really pretty simple in this case:
static member LoadFile(filePath:String) =
if not <| ZipFile.IsZipFile(filePath) then None else
use zipFile = new ZipFile(filePath)
if zipFile.Count <> 2 || zipFile |> Seq.exists(fun x -> x.FileName <> "alpha" && x.FileName <> "beta") then None else
Some (zipFile.OpenReader() |> Parser.Parse)
That last line could also be written as:
zipFile.OpenReader() |> Parser.Parse |> Some
Now, you mentioned that you don't like the long if statement. Let's turn it into a function! And I usually prefer functions with "positive" names, i.e. an isValidInput function is usually more helpful than an isInvalidInput. So let's make a function that checks if a zipfile is actually valid:
let isValid (z:ZipFile) =
z.Count = 2 && z |> Seq.forAll(fun x -> x.FileName = "alpha" || x.FileName = "beta")
Now your LoadFile function can become:
static member LoadFile(filePath:String) =
if not <| ZipFile.IsZipFile(filePath) then None else
use zipFile = new ZipFile(filePath)
if not <| isValid zipFile then None else
zipFile.OpenReader() |> Parser.Parse |> Some
And that looks pretty easy to read, so we can stop refactoring for now.
This piece of code looks weird. Using Sequence expressions for such a simple piece of code is overkill.
Some(seq { for zipEntry in zipFile do yield Parser.Parse(zipEntry.OpenReader()) } |> Seq.toArray)
You could write it better like this
zipFile |> Seq.map (fun ze -> ze.OpenReader () |> Parser.parse) |> Some
Or if you insist in doing it in an array (why?)
zipFile |> Seq.map (fun ze -> ze.OpenReader () |> Parser.parse) |> Seq.toArray |> Some
You'll end up with type signature option<seq<value>>. I am not sure if this is a good idea, but it is not possible to tell without looking at the rest of your code.
First let me apologize for the scale of this problem but I'm really trying to think functionally and this is one of the more challenging problems I have had to work with.
I wanted to get some suggestions on how I might handle a problem I have in a functional manner, particularly in F#. I am writing a program to go through a list of directories and using a list of regex patterns to filter the list of files retrieved from the directories and using a second list of regex patterns to find matches in the text of the retreived files. I want this thing to return the filename, line index, column index, pattern and matched value for each piece of text that matches a given regex pattern. Also, exceptions need to be recorded and there are 3 possible exceptions scenarios: can't open the directory, can't open the file, reading content from the file failed. The final requirement of this is the the volume of files "scanned" for matches could be very large so this whole thing needs to be lazy. I'm not too worried about a "pure" functional solution as much as I'm interested in a "good" solution that reads well and performs well. One final challenge is to make it interop with C# because I would like to use the winform tools to attach this algorithm to a ui. Here is my first attempt and hopefully this will clarify the problem:
open System.Text.RegularExpressions
open System.IO
type Reader<'t, 'a> = 't -> 'a //=M['a], result varies
let returnM x _ = x
let map f m = fun t -> t |> m |> f
let apply f m = fun t -> t |> m |> (t |> f)
let bind f m = fun t -> t |> (t |> m |> f)
let Scanner dirs =
returnM dirs
|> apply (fun dirExHandler ->
Seq.collect (fun directory ->
try
Directory.GetFiles(directory, "*", SearchOption.AllDirectories)
with | e ->
dirExHandler e directory
Array.empty))
|> map (fun filenames ->
returnM filenames
|> apply (fun (filenamepatterns, lineExHandler, fileExHandler) ->
Seq.filter (fun filename ->
filenamepatterns |> Seq.exists (fun pattern ->
let regex = new Regex(pattern)
regex.IsMatch(filename)))
>> Seq.map (fun filename ->
let fileinfo = new FileInfo(filename)
try
use reader = fileinfo.OpenText()
Seq.unfold (fun ((reader : StreamReader), index) ->
if not reader.EndOfStream then
try
let line = reader.ReadLine()
Some((line, index), (reader, index + 1))
with | e ->
lineExHandler e filename index
None
else
None) (reader, 0)
|> (fun lines -> (filename, lines))
with | e ->
fileExHandler e filename
(filename, Seq.empty))
>> (fun files ->
returnM files
|> apply (fun contentpatterns ->
Seq.collect (fun file ->
let filename, lines = file
lines |>
Seq.collect (fun line ->
let content, index = line
contentpatterns
|> Seq.collect (fun pattern ->
let regex = new Regex(pattern)
regex.Matches(content)
|> (Seq.cast<Match>
>> Seq.map (fun contentmatch ->
(filename,
index,
contentmatch.Index,
pattern,
contentmatch.Value))))))))))
Thanks for any input.
Updated -- here is any updated solution based on feedback I received:
open System.Text.RegularExpressions
open System.IO
type ScannerConfiguration = {
FileNamePatterns : seq<string>
ContentPatterns : seq<string>
FileExceptionHandler : exn -> string -> unit
LineExceptionHandler : exn -> string -> int -> unit
DirectoryExceptionHandler : exn -> string -> unit }
let scanner specifiedDirectories (configuration : ScannerConfiguration) = seq {
let ToCachedRegexList = Seq.map (fun pattern -> new Regex(pattern)) >> Seq.cache
let contentRegexes = configuration.ContentPatterns |> ToCachedRegexList
let filenameRegexes = configuration.FileNamePatterns |> ToCachedRegexList
let getLines exHandler reader =
Seq.unfold (fun ((reader : StreamReader), index) ->
if not reader.EndOfStream then
try
let line = reader.ReadLine()
Some((line, index), (reader, index + 1))
with | e -> exHandler e index; None
else
None) (reader, 0)
for specifiedDirectory in specifiedDirectories do
let files =
try Directory.GetFiles(specifiedDirectory, "*", SearchOption.AllDirectories)
with e -> configuration.DirectoryExceptionHandler e specifiedDirectory; [||]
for file in files do
if filenameRegexes |> Seq.exists (fun (regex : Regex) -> regex.IsMatch(file)) then
let lines =
let fileinfo = new FileInfo(file)
try
use reader = fileinfo.OpenText()
reader |> getLines (fun e index -> configuration.LineExceptionHandler e file index)
with | e -> configuration.FileExceptionHandler e file; Seq.empty
for line in lines do
let content, index = line
for contentregex in contentRegexes do
for mmatch in content |> contentregex.Matches do
yield (file, index, mmatch.Index, contentregex.ToString(), mmatch.Value) }
Again, any input is welcome.
I think that the best approach is to start with the simplest solution and then extend it. Your current approach seems to be quite hard to read to me for two reasons:
The code uses a lot of combinators and function compositions in patterns that are not too common in F#. Some of the processing can be more easily written using sequence expressions.
The code is all written as a single function, but it is fairly complex and would be more readable if it was separated into multiple functions.
I would probably start by splitting the code in a function that tests a single file (say fileMatches) and a function that walks over the files and calls fileMatches. The main iteration can be quite nicely written using F# sequence expressions:
// Checks whether a file name matches a filename pattern
// and a content matches a content pattern.
let fileMatches fileNamePatterns contentPatterns
(fileExHandler, lineExHandler) file =
// TODO: This can be imlemented using
// File.ReadLines which returns a sequence.
// Iterates over all the files and calls 'fileMatches'.
let scanner specifiedDirectories fileNamePatterns contentPatterns
(dirExHandler, fileExHandler, lineExHandler) = seq {
// Iterate over all the specified directories.
for specifiedDir in specifiedDirectories do
// Find all files in the directories (and handle exceptions).
let files =
try Directory.GetFiles(specifiedDir, "*", SearchOption.AllDirectories)
with e -> dirExHandler e specifiedDir; [||]
// Iterate over all files and report those that match.
for file in files do
if fileMatches fileNamePatterns contentPatterns
(fileExHandler, lineExHandler) file then
// Matches! Return this file as part of the result.
yield file }
The function is still quite complicated, because you need to pass a lot of parameters around. Wrapping the parameters in a simple type or a record could be a good idea:
type ScannerArguments =
{ FileNamePatterns:string
ContentPatterns:string
FileExceptionHandler:exn -> string -> unit
LineExceptionHandler:exn -> string -> unit
DirectoryExceptionHandler:exn -> string -> unit }
Then you can define both fileMatches and scanner as functions that take just two parameters, which will make your code a lot more readable. Something like:
// Iterates over all the files and calls 'fileMatches'.
let scanner specifiedDirectories (args:ScannerArguments) = seq {
for specifiedDir in specifiedDirectories do
let files =
try Directory.GetFiles(specifiedDir, "*", SearchOption.AllDirectories)
with e -> args.DirectoryExceptionHandler e specifiedDir; [||]
for file in files do
// No need to propagate all arguments explicitly to other functions.
if fileMatches args file then yield file }