I am very new to F# here, I encounter the "Collection was modified" problem in F#. I know this problem is common when we are iterating through a Collection while modifying (adding/removing) it at the same time. And previous threads in stackoverflow also point to this.
But in my case, I am working on 2 different sets:
I have 2 collections:
originalCollection the original collection from which I want to remove stuff
colToRemove a collection containing the objects that I want to remove
Below is the code:
Seq.iter ( fun input -> ignore <| originalCollection.Remove(input)) colToRemove
And I got the following runtime error:
+ $exception {System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
at System.Collections.Generic.List1.Enumerator.MoveNextRare()
at System.Collections.Generic.List1.Enumerator.MoveNext()
at Microsoft.FSharp.Collections.IEnumerator.next#174[T](FSharpFunc2 f, IEnumerator1 e, FSharpRef1 started, Unit unitVar0)
at Microsoft.FSharp.Collections.IEnumerator.filter#169.System-Collections-IEnumerator-MoveNext()
at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc2 action, IEnumerable`1 source)
here is the chunk of code:
match newCollection with
| Some(newCollection) ->
// compare newCollection to originalCollection.
// If there are things that exist in the originalCollection that are not in the newCollection, we want to remove them
let colToRemove = Seq.filter (fun input -> Seq.exists (fun i -> i.id = input.id) newCollection) originalCollection
Seq.iter ( fun input -> ignore <| originalCollection.Remove(input)) colToRemove
| None -> ()
Thanks!
Note: Working on a single-threaded environment here, so there are no multi-threading issues that might result in this exception.
The problem here is that colToRemove is not an independent collection but is a projection of the collection originalCollection. So changing originalCollection changes the projection which is not allowed during the iteration. The C# equivalent of the above code is the following
var colToRemove = originalCollection
.Where(input -> newCollection.Any(i -> i.id == input.id));
foreach (var in input in colToRemove) {
originalCollection.Remove(input);
}
You can fix this by making colToRemove an independent collection via the List.ofSeq method.
let colToRemove =
originalCollection
|> Seq.filter (fun input -> Seq.exists (fun i -> i.id = input.id) newCollection) originalCollection
|> List.ofSeq
I would not try to do a remove, since you are modifying a collection, but instead try to create another collection like so:
let foo () =
let orig = [1;2;3;4]
let torem = [1;2]
let find e =
List.tryFind (fun i-> i = e) torem
|> function
| Some _-> true
| None -> false
List.partition (fun e -> find e) orig
//or
List.filter (fun e-> find e) orig
hth
Related
Is there anyway to contruct a list in reverse order without having to reverse it
Here is an example, I read all lines from stdin
#!/usr/bin/env dotnet fsi
open System
let rec readLines1 () =
let rec helper acc =
match Console.ReadLine() with
| null -> acc
| line ->
helper (line :: acc)
helper [] |> List.rev
readLines1 () |> List.iter (printfn "%s")
Before return from readLines1 I have to List.rev it so that is in right order. Since the result is a slightly linked list it will have to read all trough it and create the reversed version. Is there any way of creating the list in right order?
You can use a sequence instead of accumulating the lines in a list:
open System
let readLines1 () =
let rec helper () =
seq {
match Console.ReadLine() with
| null -> ()
| line ->
yield line
yield! helper ()
}
helper () |> Seq.toList
readLines1 () |> List.iter (printfn "%s")
You cannot create list in reverse order, because that would require mutation. If you read inputs one by one, and want to turn them into a list immediately, the only thing you can do is to create new list, linking to the previous one.
In practice, reversing the list is perfectly fine and that's probably the best way of solving this.
Out of curiosity, you could try defininig a mutable list that has the same structure as immutable F# list:
open System
type MutableList<'T> =
{ mutable List : MutableListBody<'T> }
and MutableListBody<'T> =
| Empty
| Cons of 'T * MutableList<'T>
Now you can implement your function by mutating the list:
let rec readLines () =
let res = { List = Empty }
let rec helper acc =
match Console.ReadLine() with
| null -> res
| line ->
let next = { List = Empty }
acc.List <- Cons(line, next)
helper next
helper res
This may be educational, but it's not very useful and, if you really wanted mutation in F#, you should probably use ResizeArray.
Yet another trick is to work with functions that take the tail of the list:
let rec readLines () =
let rec helper acc =
match Console.ReadLine() with
| null -> acc []
| line -> helper (fun tail -> acc (line :: tail))
helper id
In the line case, this returns a function that takes tail adds line before the tail and then calls whatever function was constructed before to add more things to the front.
This actually creates the list in the right order, but it's probably less efficient than creating a list and reversing it. It may look nice, but you are allocating a new function for each iteration, which is not better than allocating an extra copy of the list. (But it is a nice trick, nevertheless!)
Alternative solution without implementing recursive functions
let lines =
Seq.initInfinite (fun _ -> Console.ReadLine())
|> Seq.takeWhile (not << isNull)
|> Seq.toList
I have a text file that contains the following and I need to retrieve the value assigned to taskId, which in this case is AWc34YBAp0N7ZCmVka2u.
projectKey=ProjectName
serverUrl=http://localhost:9090
serverVersion=10.5.32.3
strong text**interfaceUrl=http://localhost:9090/interface?id=ProjectName
taskId=AWc34YBAp0N7ZCmVka2u
taskUrl=http://localhost:9090/api/ce/task?id=AWc34YBAp0N7ZCmVka2u
I have two different ways of reading the file that I've wrote.
let readLines (filePath:string) = seq {
use sr = new StreamReader (filePath)
while not sr.EndOfStream do
yield sr.ReadLine ()
}
readLines (FindFile currentDirectory "../**/sample.txt")
|> Seq.iter (fun line ->
printfn "%s" line
)
and
let readLines (filePath:string) =
(File.ReadAllLines filePath)
readLines (FindFile currentDirectory "../**/sample.txt")
|> Seq.iter (fun line ->
printfn "%s" line
)
At this point, I don't know how to approach getting the value I need. Options that, I think, are on the table are:
use Contains()
Regex
Record type
Active Pattern
How can I get this value returned and fail if it doesn't exist?
I think all the options would be reasonable - it depends on how complex the file will actually be. If there is no escaping then you can probably just look for = in the line and use that to split the line into a key value pair. If the syntax is more complex, this might not always work though.
My preferred method would be to use Split on string - you can then filter to find values with your required key, map to get the value and use Seq.head to get the value:
["foo=bar"]
|> Seq.map (fun line -> line.Split('='))
|> Seq.filter (fun kvp -> kvp.[0] = "foo")
|> Seq.map (fun kvp -> kvp.[1])
|> Seq.head
Using active patterns, you could define a pattern that takes a string and splits it using = into a list:
let (|Split|) (s:string) = s.Split('=') |> List.ofSeq
This then lets you get the value using Seq.pick with a pattern matching that looks for strings where the substring before = is e.g. foo:
["foo=bar"] |> Seq.pick (function
| Split ["foo"; value] -> Some value
| _ -> None)
The active pattern trick is quite neat, but it might be unnecessarily complicating the code if you only need this in one place.
I am completely at loss why this code doesn't mutate a member variable in a sequence of types:
for p in prescrs do
p.ATC <- "A"
for c in p.Drug.Components do
for s in c.Substances do
s.DoseTotal.Adjust <- adjustKg
s.DoseTotal.Time <- "DAY"
s.DoseTotal.Unit <- s.DrugConcentration.Unit
s.DoseRate.Adjust <- adjustKg
s.DoseRate.Time <- "DAY"
s.DoseRate.Unit <- s.DrugConcentration.Unit
prescrs is a sequence of Prescriptions which is a very simple 'POCO' defined as a type with member values. I don't have clue why this doesn't work.
I tried a simple test case like:
type IterTest () =
member val Name = "" with get, set
member val IterTests = [] |> List.toSeq : IterTest seq with get, set
let iterseq =
[
new IterTest(Name = "Test1")
new IterTest(Name = "Test2")
]
|> List.toSeq
iterseq |> Seq.iter(fun x -> x.IterTests <- iterseq)
iterseq |> Seq.iter(fun x ->
x.IterTests
|> Seq.iter(fun x' -> x'.Name <- "itered"))
But here the result is as expected. So, can't even quite reproduce my problem???
Found a solution (without really understanding the problem above). When I first convert the prescrs sequence to a list like:
let prescrs = prescrs |> Seq.toList
and then do the imperative looping, properties do get mutated.
Try this sample:
type Mutable() =
member val Iterated = false with get, set
let muts = Seq.init 5 (fun _ -> printfn "init"; Mutable())
let muts2 = muts // try again with let muts2 = muts |> List.ofSeq
printfn "Before iter"
for a in muts2 do
printfn "iter"
a.Iterated <- true
printfn "After iter"
muts2 |> List.ofSeq
and check how iter and init are interleaved.
Seqs are lazy, but are not cached once computed. So even if you imperatively try to mutate some of the elements in your prescrs sequence, it all goes away once you pull prescrs again. If you change prescrs into a concrete collection type like list before doing the mutation, you no longer hit the same problem. Note that things might get even trickier if what you have is a seq inside a seq inside a seq.
The best idea would be to avoid mutation in the first place though.
First let me apologize for the scale of this problem but I'm really trying to think functionally and this is one of the more challenging problems I have had to work with.
I wanted to get some suggestions on how I might handle a problem I have in a functional manner, particularly in F#. I am writing a program to go through a list of directories and using a list of regex patterns to filter the list of files retrieved from the directories and using a second list of regex patterns to find matches in the text of the retreived files. I want this thing to return the filename, line index, column index, pattern and matched value for each piece of text that matches a given regex pattern. Also, exceptions need to be recorded and there are 3 possible exceptions scenarios: can't open the directory, can't open the file, reading content from the file failed. The final requirement of this is the the volume of files "scanned" for matches could be very large so this whole thing needs to be lazy. I'm not too worried about a "pure" functional solution as much as I'm interested in a "good" solution that reads well and performs well. One final challenge is to make it interop with C# because I would like to use the winform tools to attach this algorithm to a ui. Here is my first attempt and hopefully this will clarify the problem:
open System.Text.RegularExpressions
open System.IO
type Reader<'t, 'a> = 't -> 'a //=M['a], result varies
let returnM x _ = x
let map f m = fun t -> t |> m |> f
let apply f m = fun t -> t |> m |> (t |> f)
let bind f m = fun t -> t |> (t |> m |> f)
let Scanner dirs =
returnM dirs
|> apply (fun dirExHandler ->
Seq.collect (fun directory ->
try
Directory.GetFiles(directory, "*", SearchOption.AllDirectories)
with | e ->
dirExHandler e directory
Array.empty))
|> map (fun filenames ->
returnM filenames
|> apply (fun (filenamepatterns, lineExHandler, fileExHandler) ->
Seq.filter (fun filename ->
filenamepatterns |> Seq.exists (fun pattern ->
let regex = new Regex(pattern)
regex.IsMatch(filename)))
>> Seq.map (fun filename ->
let fileinfo = new FileInfo(filename)
try
use reader = fileinfo.OpenText()
Seq.unfold (fun ((reader : StreamReader), index) ->
if not reader.EndOfStream then
try
let line = reader.ReadLine()
Some((line, index), (reader, index + 1))
with | e ->
lineExHandler e filename index
None
else
None) (reader, 0)
|> (fun lines -> (filename, lines))
with | e ->
fileExHandler e filename
(filename, Seq.empty))
>> (fun files ->
returnM files
|> apply (fun contentpatterns ->
Seq.collect (fun file ->
let filename, lines = file
lines |>
Seq.collect (fun line ->
let content, index = line
contentpatterns
|> Seq.collect (fun pattern ->
let regex = new Regex(pattern)
regex.Matches(content)
|> (Seq.cast<Match>
>> Seq.map (fun contentmatch ->
(filename,
index,
contentmatch.Index,
pattern,
contentmatch.Value))))))))))
Thanks for any input.
Updated -- here is any updated solution based on feedback I received:
open System.Text.RegularExpressions
open System.IO
type ScannerConfiguration = {
FileNamePatterns : seq<string>
ContentPatterns : seq<string>
FileExceptionHandler : exn -> string -> unit
LineExceptionHandler : exn -> string -> int -> unit
DirectoryExceptionHandler : exn -> string -> unit }
let scanner specifiedDirectories (configuration : ScannerConfiguration) = seq {
let ToCachedRegexList = Seq.map (fun pattern -> new Regex(pattern)) >> Seq.cache
let contentRegexes = configuration.ContentPatterns |> ToCachedRegexList
let filenameRegexes = configuration.FileNamePatterns |> ToCachedRegexList
let getLines exHandler reader =
Seq.unfold (fun ((reader : StreamReader), index) ->
if not reader.EndOfStream then
try
let line = reader.ReadLine()
Some((line, index), (reader, index + 1))
with | e -> exHandler e index; None
else
None) (reader, 0)
for specifiedDirectory in specifiedDirectories do
let files =
try Directory.GetFiles(specifiedDirectory, "*", SearchOption.AllDirectories)
with e -> configuration.DirectoryExceptionHandler e specifiedDirectory; [||]
for file in files do
if filenameRegexes |> Seq.exists (fun (regex : Regex) -> regex.IsMatch(file)) then
let lines =
let fileinfo = new FileInfo(file)
try
use reader = fileinfo.OpenText()
reader |> getLines (fun e index -> configuration.LineExceptionHandler e file index)
with | e -> configuration.FileExceptionHandler e file; Seq.empty
for line in lines do
let content, index = line
for contentregex in contentRegexes do
for mmatch in content |> contentregex.Matches do
yield (file, index, mmatch.Index, contentregex.ToString(), mmatch.Value) }
Again, any input is welcome.
I think that the best approach is to start with the simplest solution and then extend it. Your current approach seems to be quite hard to read to me for two reasons:
The code uses a lot of combinators and function compositions in patterns that are not too common in F#. Some of the processing can be more easily written using sequence expressions.
The code is all written as a single function, but it is fairly complex and would be more readable if it was separated into multiple functions.
I would probably start by splitting the code in a function that tests a single file (say fileMatches) and a function that walks over the files and calls fileMatches. The main iteration can be quite nicely written using F# sequence expressions:
// Checks whether a file name matches a filename pattern
// and a content matches a content pattern.
let fileMatches fileNamePatterns contentPatterns
(fileExHandler, lineExHandler) file =
// TODO: This can be imlemented using
// File.ReadLines which returns a sequence.
// Iterates over all the files and calls 'fileMatches'.
let scanner specifiedDirectories fileNamePatterns contentPatterns
(dirExHandler, fileExHandler, lineExHandler) = seq {
// Iterate over all the specified directories.
for specifiedDir in specifiedDirectories do
// Find all files in the directories (and handle exceptions).
let files =
try Directory.GetFiles(specifiedDir, "*", SearchOption.AllDirectories)
with e -> dirExHandler e specifiedDir; [||]
// Iterate over all files and report those that match.
for file in files do
if fileMatches fileNamePatterns contentPatterns
(fileExHandler, lineExHandler) file then
// Matches! Return this file as part of the result.
yield file }
The function is still quite complicated, because you need to pass a lot of parameters around. Wrapping the parameters in a simple type or a record could be a good idea:
type ScannerArguments =
{ FileNamePatterns:string
ContentPatterns:string
FileExceptionHandler:exn -> string -> unit
LineExceptionHandler:exn -> string -> unit
DirectoryExceptionHandler:exn -> string -> unit }
Then you can define both fileMatches and scanner as functions that take just two parameters, which will make your code a lot more readable. Something like:
// Iterates over all the files and calls 'fileMatches'.
let scanner specifiedDirectories (args:ScannerArguments) = seq {
for specifiedDir in specifiedDirectories do
let files =
try Directory.GetFiles(specifiedDir, "*", SearchOption.AllDirectories)
with e -> args.DirectoryExceptionHandler e specifiedDir; [||]
for file in files do
// No need to propagate all arguments explicitly to other functions.
if fileMatches args file then yield file }
...or, how do I filter a sequence of classes by the interfaces they implement?
Let's say I have a sequence of objects that inherit from Foo, a seq<#Foo>. In other words, my sequence will contain one or more of four different subclasses of Foo.
Each subclass implements a different independent interface that shares nothing with the interfaces implemented by the other subclasses.
Now I need to filter this sequence down to only the items that implement a particular interface.
The C# version is simple:
void MergeFoosIntoList<T>(IEnumerable<Foo> allFoos, IList<T> dest)
where T : class
{
foreach (var foo in allFoos)
{
var castFoo = foo as T;
if (castFoo != null)
{
dest.Add(castFoo);
}
}
}
I could use LINQ from F#:
let mergeFoosIntoList (foos:seq<#Foo>) (dest:IList<'a>) =
System.Linq.Enumerable.OfType<'a>(foos)
|> Seq.iter dest.Add
However, I feel like there should be a more idiomatic way to accomplish it. I thought this would work...
let mergeFoosIntoList (foos:seq<#Foo>) (dest:IList<'a>) =
foos
|> Seq.choose (function | :? 'a as x -> Some(x) | _ -> None)
|> Seq.iter dest.Add
However, the complier complains about :? 'a - telling me:
This runtime coercion or type test from type 'b to 'a involves an indeterminate type based on information prior to this program point. Runtime type tests are not allowed on some types. Further type annotations are needed.
I can't figure out what further type annotations to add. There's no relationship between the interface 'a and #Foo except that one or more subclasses of Foo implement that interface. Also, there's no relationship between the different interfaces that can be passed in as 'a except that they are all implemented by subclasses of Foo.
I eagerly anticipate smacking myself in the head as soon as one of you kind people points out the obvious thing I've been missing.
You can do this:
let foos = candidates |> Seq.filter (fun x -> x :? Foo) |> Seq.cast<Foo>
Typically just adding a 'box' is sufficient (e.g. change function to fun x -> match box x with), but let me try it out...
Yeah; basically you cannot sideways cast from one arbitrary generic type to another, but you can upcast to System.Object (via box) and then downcast to anything you like:
type Animal() = class end
type Dog() = inherit Animal()
type Cat() = inherit Animal()
let pets : Animal list =
[Dog(); Cat(); Dog(); Cat(); Dog()]
printfn "%A" pets
open System.Collections.Generic
let mergeIntoList (pets:seq<#Animal>) (dest:IList<'a>) =
pets
|> Seq.choose (fun p -> match box p with
| :? 'a as x -> Some(x) | _ -> None) //'
|> Seq.iter dest.Add
let l = new List<Dog>()
mergeIntoList pets l
l |> Seq.iter (printfn "%A")
From https://gist.github.com/kos59125/3780229
let ofType<'a> (source : System.Collections.IEnumerable) : seq<'a> =
let resultType = typeof<'a>
seq {
for item in source do
match item with
| null -> ()
| _ ->
if resultType.IsAssignableFrom (item.GetType ())
then
yield (downcast item)
}
Another option for those inclined:
Module Seq =
let ofType<'a> (items: _ seq)= items |> Seq.choose(fun i -> match box i with | :? 'a as a -> Some a |_ -> None)
I have an open source library available on nuget, FSharp.Interop.Compose
That Converts most Linq methods into a idomatic F# form. Including OfType
Test Case:
[<Fact>]
let ofType () =
let list = System.Collections.ArrayList()
list.Add(1) |> ignore
list.Add("2") |> ignore
list.Add(3) |> ignore
list.Add("4") |> ignore
list
|> Enumerable.ofType<int>
|> Seq.toList |> should equal [1;3]