How do you flatten a 2d array to a 1d array by appending each row to the one above?
My problem is not understanding how to use map to do this as other functional languages have a flatmap/(insert similar name here) function to do this.
let colors = Array2D.init 800 480 (fun i j ->
if i % 3 = 0 || j % 3 = 0 then Color.Black
else Color.Red)
let data = colors |> map (fun c -> c)
How woulds I use map such that the return type from the map is changed to a 1d array?
for Sequences use Seq.fold + Seq.append + empty Seq
let seq2d = seq {yield seq{1..3}; yield seq{4..6} }
seq2d |> Seq.fold Seq.append Seq.empty<int>
//[1; 2; 3; 4; 5; 6;]
or even less typing with just Seq.reduce + Seq.append
seq2d |> Seq.reduce Seq.append
//[1; 2; 3; 4; 5; 6;]
for lists (but not Sequences) it's List.reduce + List.append
let list2d = [ [1;2;3]; [4;5] ]
list2d |> List.reduce List.append
//[1; 2; 3; 4; 5]
If you just want to flatten it you can cast it to a seq:
colors |> Seq.cast<Color>
|> Seq.length //val it : int = 384000
There might be something in Array2D that's more convenient but Array2D is really a .NET collection. You can work with ragged arrays or lists and then you can have access to Seq.concat or collect.
Add1
Here it is already in a 1D List:
let colors = [for i in 0..799 do
for j in 0..479 ->
if (i % 3 = 0) || (j % 3 = 0) then Color.Black
else Color.Red]
With Active Patterns
Depending on the actual complexity this might also be a good candidate for active patterns. Below an active recognizer for Black and Red is defined, together with the pattern matching, then the 2D List is generated which is fed to concat, and finally checked against the original Array2D. You don't need to work with Lists of course (e.g. can be seq for laziness or Array for performance).
let (|Black|Red|) input = if fst input % 3 = 0 || snd input % 3 = 0 then Black else Red
let matchColor =
function
|Black -> Color.Black
|Red -> Color.Red
let color3 = List.init 800 (fun i -> List.init 480 (fun j -> matchColor (i,j)))
let color4 = color3 |> List.concat
color4 |> Seq.length
colors
|> Array2D.mapi (fun i j x -> color3.[i].[j] = colors.[i,j])
|> Seq.cast<bool>
|> Seq.filter not
Related
I want a way to get rid of repeating pairs in an array. For my problem, the pairs will be consecutive, and there will be at most one repeating pair.
My current implementation seems too complicated. The elements 3 and 4 form what I'm calling a repeating pair in arr1 below. As a pair, they only appear once in the desired output, arr2. What are some more efficient ways?
let arr1=[|4; 2; 3; 4; 3; 4; 1|]
let n=arr1.Length
let iPlus2IsEqual=Array.map2 (fun x y -> x=y) arr1.[2..] arr1.[..(n-3)]
let consecutive=Array.map2 (fun x y -> x && y) iPlus2IsEqual.[1..] iPlus2IsEqual.[..(n-4)] |> Array.tryFindIndex (fun x -> x)
let dup=if consecutive.IsSome then consecutive.Value+1 else n-1
let arr2=if dup>=n-3 then arr1.[..dup] else Array.append arr1.[..dup] arr1.[(dup+3)..]
>
val arr2 : int [] = [|4; 2; 3; 4; 1|]
We can use recursion like so (it will get multiple repeats for free too)
let rec filterrepeats l =
match l with
|a::b::c::d::t when a=c && b=d -> a::b::(filterrepeats t)
|h::t ->h::(filterrepeats t)
|[] -> []
> filterrepeats [4;2;3;4;3;4;1];;
val it : int list = [4; 2; 3; 4; 1]
This works on lists, so you will need to add a call to Array.toList before you run it.
The above is not tail recursive as the compiler doesn't know what goes on the right hand side of h::(filterrepeats t) until after the function call. You can solve this by using an accumulator like so:
let rec filterrepeats l =
let rec loop l acc =
match l with
|a::b::c::d::t when a=c && b=d ->loop t (b::a::acc)
|h::t ->loop t (h::acc)
|[] -> acc
loop (List.rev l) []
For large arrays this is around 13x faster than your solution:
let inline tryFindDuplicatedPairIndex (xs: _ []) =
let rec loop i x0 x1 x2 =
if i < xs.Length-4 then
let x3 = xs.[i+3]
if x0=x2 && x1=x3 then Some i else
loop (i+1) x1 x2 x3
else None
if xs.Length < 4 then None else
loop 0 xs.[0] xs.[1] xs.[2]
let inline removeDuplicatedPair (xs: _ []) =
match tryFindDuplicatedPairIndex xs with
| None -> Array.copy xs
| Some i ->
let ys = Array.zeroCreate (xs.Length-2)
for j=0 to i-1 do
ys.[j] <- xs.[j]
for j=i+2 to xs.Length-1 do
ys.[j-2] <- xs.[j]
ys
I use inline and test elements individually (i.e. rather than as a tuple: (x0,x1) = (x2,x3)) to try to prevent = from being a generic equality test because that is very slow. I've reused previous array lookups from one iteration to the next. I copy the input array if the output is identical to the input and pre-allocate an array with n-2 elements otherwise. I've hand-rolled the copying to my pre-allocated array to avoid creating any garbage (e.g. instead of Array.append of two slices).
No stack overflow with large list (length >= 100K) and remove all duplicate pairs
let rec distinctPairs list =
List.foldBack (fun x (l,r) -> x::r, l) list ([],[])
|> fun (odds, evens) -> List.zip odds evens
|> Seq.distinct
Not very fast, 1M list take 500ms, anyway faster ?
Only work for list with even length
I'm trying to learn F# by rewriting some C# algorithms I have into idiomatic F#.
One of the first functions I'm trying to rewrite is a batchesOf where:
[1..17] |> batchesOf 5
Which would split the sequence into batches with a max of five in each, i.e:
[[1; 2; 3; 4; 5]; [6; 7; 8; 9; 10]; [11; 12; 13; 14; 15]; [16; 17]]
My first attempt at doing this is kind of ugly where I've resorted to using a mutable ref object after running into errors trying to use mutable type inside the closure. Using ref is particularly unpleasant since to dereference it you have to use the ! operator which when inside a condition expression can be counter intuitive to some devs who will read it as logical not. Another problem I ran into is where Seq.skip and Seq.take are not like their Linq aliases in that they will throw an error if size exceeds the size of the sequence.
let batchesOf size (sequence: _ seq) : _ list seq =
seq {
let s = ref sequence
while not (!s |> Seq.isEmpty) do
yield !s |> Seq.truncate size |> List.ofSeq
s := System.Linq.Enumerable.Skip(!s, size)
}
Anyway what would be the most elegant/idiomatic way to rewrite this in F#? Keeping the original behaviour but preferably without the ref mutable variable.
Implementing this function using the seq<_> type idiomatically is difficult - the type is inherently mutable, so there is no simple nice functional way. Your version is quite inefficient, because it uses Skip repeatedly on the sequence. A better imperative option would be to use GetEnumerator and just iterate over elements using IEnumerator. You can find various imperative options in this snippet: http://fssnip.net/1o
If you're learning F#, then it is better to try writing the function using F# list type. This way, you can use idiomatic functional style. Then you can write batchesOf using pattern matching with recursion and accumulator argument like this:
let batchesOf size input =
// Inner function that does the actual work.
// 'input' is the remaining part of the list, 'num' is the number of elements
// in a current batch, which is stored in 'batch'. Finally, 'acc' is a list of
// batches (in a reverse order)
let rec loop input num batch acc =
match input with
| [] ->
// We've reached the end - add current batch to the list of all
// batches if it is not empty and return batch (in the right order)
if batch <> [] then (List.rev batch)::acc else acc
|> List.rev
| x::xs when num = size - 1 ->
// We've reached the end of the batch - add the last element
// and add batch to the list of batches.
loop xs 0 [] ((List.rev (x::batch))::acc)
| x::xs ->
// Take one element from the input and add it to the current batch
loop xs (num + 1) (x::batch) acc
loop input 0 [] []
As a footnote, the imperative version can be made a bit nicer using computation expression for working with IEnumerator, but that's not standard and it is quite advanced trick (for example, see http://fssnip.net/37).
A friend asked me this a while back. Here's a recycled answer. This works and is pure:
let batchesOf n =
Seq.mapi (fun i v -> i / n, v) >>
Seq.groupBy fst >>
Seq.map snd >>
Seq.map (Seq.map snd)
Or an impure version:
let batchesOf n =
let i = ref -1
Seq.groupBy (fun _ -> i := !i + 1; !i / n) >> Seq.map snd
These produce a seq<seq<'a>>. If you really must have an 'a list list as in your sample then just add ... |> Seq.map (List.ofSeq) |> List.ofSeq as in:
> [1..17] |> batchesOf 5 |> Seq.map (List.ofSeq) |> List.ofSeq;;
val it : int list list = [[1; 2; 3; 4; 5]; [6; 7; 8; 9; 10]; [11; 12; 13; 14; 15]; [16; 17]]
Hope that helps!
This can be done without recursion if you want
[0..20]
|> Seq.mapi (fun i elem -> (i/size),elem)
|> Seq.groupBy (fun (a,_) -> a)
|> Seq.map (fun (_,se) -> se |> Seq.map (snd));;
val it : seq<seq<int>> =
seq
[seq [0; 1; 2; 3; ...]; seq [5; 6; 7; 8; ...]; seq [10; 11; 12; 13; ...];
seq [15; 16; 17; 18; ...]; ...]
Depending on how you think this may be easier to understand. Tomas' solution is probably more idiomatic F# though
Hurray, we can use List.chunkBySize, Seq.chunkBySize and Array.chunkBySize in F# 4, as mentioned by Brad Collins and Scott Wlaschin.
This isn't perhaps idiomatic but it works:
let batchesOf n l =
let _, _, temp', res' = List.fold (fun (i, n, temp, res) hd ->
if i < n then
(i + 1, n, hd :: temp, res)
else
(1, i, [hd], (List.rev temp) :: res))
(0, n, [], []) l
(List.rev temp') :: res' |> List.rev
Here's a simple implementation for sequences:
let chunks size (items:seq<_>) =
use e = items.GetEnumerator()
let rec loop i acc =
seq {
if i = size then
yield (List.rev acc)
yield! loop 0 []
elif e.MoveNext() then
yield! loop (i+1) (e.Current::acc)
else
yield (List.rev acc)
}
if size = 0 then invalidArg "size" "must be greater than zero"
if Seq.isEmpty items then Seq.empty else loop 0 []
let s = Seq.init 10 id
chunks 3 s
//output: seq [[0; 1; 2]; [3; 4; 5]; [6; 7; 8]; [9]]
My method involves converting the list to an array and recursively chunking the array:
let batchesOf (sz:int) lt =
let arr = List.toArray lt
let rec bite curr =
if (curr + sz - 1 ) >= arr.Length then
[Array.toList arr.[ curr .. (arr.Length - 1)]]
else
let curr1 = curr + sz
(Array.toList (arr.[curr .. (curr + sz - 1)])) :: (bite curr1)
bite 0
batchesOf 5 [1 .. 17]
[[1; 2; 3; 4; 5]; [6; 7; 8; 9; 10]; [11; 12; 13; 14; 15]; [16; 17]]
I found this to be a quite terse solution:
let partition n (stream:seq<_>) = seq {
let enum = stream.GetEnumerator()
let rec collect n partition =
if n = 1 || not (enum.MoveNext()) then
partition
else
collect (n-1) (partition # [enum.Current])
while enum.MoveNext() do
yield collect n [enum.Current]
}
It works on a sequence and produces a sequence. The output sequence consists of lists of n elements from the input sequence.
You can solve your task with analog of Clojure partition library function below:
let partition n step coll =
let rec split ss =
seq {
yield(ss |> Seq.truncate n)
if Seq.length(ss |> Seq.truncate (step+1)) > step then
yield! split <| (ss |> Seq.skip step)
}
split coll
Being used as partition 5 5 it will provide you with sought batchesOf 5 functionality:
[1..17] |> partition 5 5;;
val it : seq<seq<int>> =
seq
[seq [1; 2; 3; 4; ...]; seq [6; 7; 8; 9; ...]; seq [11; 12; 13; 14; ...];
seq [16; 17]]
As a premium by playing with n and step you can use it for slicing overlapping batches aka sliding windows, and even apply to infinite sequences, like below:
Seq.initInfinite(fun x -> x) |> partition 4 1;;
val it : seq<seq<int>> =
seq
[seq [0; 1; 2; 3]; seq [1; 2; 3; 4]; seq [2; 3; 4; 5]; seq [3; 4; 5; 6];
...]
Consider it as a prototype only as it does many redundant evaluations on the source sequence and not likely fit for production purposes.
This version passes all my tests I could think of including ones for lazy evaluation and single sequence evaluation:
let batchIn batchLength sequence =
let padding = seq { for i in 1 .. batchLength -> None }
let wrapped = sequence |> Seq.map Some
Seq.concat [wrapped; padding]
|> Seq.windowed batchLength
|> Seq.mapi (fun i el -> (i, el))
|> Seq.filter (fun t -> fst t % batchLength = 0)
|> Seq.map snd
|> Seq.map (Seq.choose id)
|> Seq.filter (fun el -> not (Seq.isEmpty el))
I am still quite new to F# so if I'm missing anything - please do correct me, it will be greatly appreciated.
I have an array of items, from which I'd like to sample.
I was under the impression that a Set would the a good structure to sample from, in a fold where I'd give back the original or a modified set with the retrieved element missing depending if I want replacement of not.
However, there seems to no method to retrieve an element directly from a Set.
Is there something I am missing ? or should I use Set of indices, along with a surrogate function that starts at some random position < Set.count and goes up until it finds a member ?
That is, something along this line
module Seq =
let modulo (n:int) start =
let rec next i = seq { yield (i + 1)%n ; yield! next (i+1)}
next start
module Array =
let Sample (withReplacement:bool) seed (entries:'T array) =
let prng, indexes = new Random(seed), Set(Seq.init (entries |> Array.length) id)
Seq.unfold (fun set -> let N = set |> Set.count
let next = Seq.modulo N (prng.Next(N)) |> Seq.truncate N |> Seq.tryFind(fun i -> set |> Set.exists ((=) i))
if next.IsSome then
Some(entries.[next.Value], if withReplacement then set else Set.remove next.Value set)
else
None)
Edit : Tracking positively what I gave, instead of tracking what I still can give would make it simpler and more efficient.
For sampling without replacement, you could just permute the source seq and take however many elements you want to sample
let sampleWithoutReplacement n s =
let a = Array.ofSeq s
seq { for i = a.Length downto 1 do
let j = rnd.Next i
yield a.[j]
a.[j] <- a.[i - 1] }
|> Seq.take n
To sample with replacement, just pick a random element n times from the source seq
let sampleWithReplacement n s =
let a = Array.ofSeq s
Seq.init n (fun _ -> a.[rnd.Next(a.Length)])
These may not be the most efficient methods with huge data sets however
Continuing our comments...if you want to randomly sample a sequence without slurping the entire thing into memory you could generate a set of random indices the size of your desired sample (not too different from what you already have):
let rand count max =
System.Random()
|> Seq.unfold (fun r -> Some(r.Next(max), r))
|> Seq.distinct
|> Seq.take count
|> set
let takeSample sampleSize inputSize input =
let indices = rand sampleSize inputSize
input
|> Seq.mapi (fun idx x ->
if Set.contains idx indices then Some x else None)
|> Seq.choose id
let inputSize = 100000
let input = Seq.init inputSize id
let sample = takeSample 50 inputSize input
printfn "%A" (Seq.toList sample)
I need to implement a simple dynamic programming algorithm in 2D in F#. For simple 1D cases Seq.unfold seems to be the way to go, see e.g. https://stackoverflow.com/a/7986083/5363
Is there a nice (and efficient) way to achieve a similar result in 2D, e.g. rewrite the following pseudo-code in functional style:
let alpha =
let result = Array2D.zeroCreate N T
for i in 0 .. N-1 do
result.[0, i] <- (initialPi i) * (b i observations.[0])
for t in 1 .. T-1 do
for i in 0 .. N-1 do
let s = row t-1 result |> Seq.mapi (fun j alpha_t_j -> alpha_t_j * initialA.[i, j]) () |> Seq.sum
result.[t, i] <- s * (b i observations.[t])
result
assume that all the missing functions and arrays are defined above.
EDIT: Actually read code, this is at least functional, does have a slightly different return type, although you could avoid that with a conversion
let alpha =
let rec build prev idx max =
match idx with
|0 ->
let r = (Array.init N (fun i -> (initialPi y) * (b i observations.[0]))
r:: (build r 1 max)
|t when t=max -> []
|_ ->
let s = prev |> Seq.mapi (fun j alpha_t_j -> alpha_t_j * initialA.[i, j]) () |> Seq.sum
let r = Array.init N (fun i -> s * (b i observations.[t]))
r:: build r (idx+1 max)
build [] 0 T |> List.toArray
Say I have a sequence of 100 elements. Every 10th element I want a new list of the previous 10 elements. In this case I will end up with a list of 10 sublists.
Seq.take(10) looks promising, how can I repeatedly call it to return a list of lists?
now there's Seq.chunkBySize available:
[1;2;3;4;5] |> Seq.chunkBySize 2 = seq [[|1; 2|]; [|3; 4|]; [|5|]]
This is not bad:
let splitEach n s =
seq {
let r = ResizeArray<_>()
for x in s do
r.Add(x)
if r.Count = n then
yield r.ToArray()
r.Clear()
if r.Count <> 0 then
yield r.ToArray()
}
let s = splitEach 5 [1..17]
for a in s do
printfn "%A" a
(*
[|1; 2; 3; 4; 5|]
[|6; 7; 8; 9; 10|]
[|11; 12; 13; 14; 15|]
[|16; 17|]
*)
I have an evolution of three solutions. None of them preserves the ordering of the input elements, which is hopefully OK.
My first solution is quite ugly (making use of ref cells):
//[[4; 3; 2; 1; 0]; [9; 8; 7; 6; 5]; [14; 13; 12; 11; 10]; [17; 16; 15]]
let solution1 =
let split s n =
let i = ref 0
let lst = ref []
seq {
for item in s do
if !i = n then
yield !lst
lst := [item]
i := 1
else
lst := item::(!lst)
i := !i+1
yield !lst
} |> Seq.toList
split {0..17} 5
My second solution factors out the use of ref cells in the first solution, but consequently forces the use of direct IEnumerator access (push in one side, pop out the other)!
//[[17; 16; 15]; [14; 13; 12; 11; 10]; [9; 8; 7; 6; 5]; [4; 3; 2; 1; 0]]
let solution2 =
let split (s:seq<_>) n =
let e = s.GetEnumerator()
let rec each lstlst lst i =
if e.MoveNext() |> not then
lst::lstlst
elif i = n then
each (lst::lstlst) [e.Current] 1
else
each lstlst ((e.Current)::lst) (i+1)
each [] [] 0
split {0..17} 5
My third solution is based on the second solution except it "cheats" by taking a list as input instead of a seq, which enables the most elegant solution using pattern matching as Tomas points out is lacking with seq (which is why we were forced to use direct IEnumerator access).
//[[17; 16; 15]; [14; 13; 12; 11; 10]; [9; 8; 7; 6; 5]; [4; 3; 2; 1; 0]]
let solution3 =
let split inputList n =
let rec each inputList lstlst lst i =
match inputList with
| [] -> (lst::lstlst)
| cur::inputList ->
if i = n then
each inputList (lst::lstlst) [cur] 1
else
each inputList lstlst (cur::lst) (i+1)
each inputList [] [] 0
split [0..17] 5
If preserving the ordering of the elements is important, you can use List.rev for this purpose. For example, in solution2, change the last line of the split function to:
each [] [] 0 |> List.rev |> List.map List.rev
Out of the top of my head:
let rec split size list =
if List.length list < size then
[list]
else
(list |> Seq.take size |> Seq.toList) :: (list |> Seq.skip size |> Seq.toList |> split size)
Perhaps this simple pure implementation might be useful:
let splitAt n xs = (Seq.truncate n xs, if Seq.length xs < n then Seq.empty else Seq.skip n xs)
let rec chunk n xs =
if Seq.isEmpty xs then Seq.empty
else
let (ys,zs) = splitAt n xs
Seq.append (Seq.singleton ys) (chunk n zs)
For example:
> chunk 10 [1..100];;
val it : seq<seq<int>> =
seq
[seq [1; 2; 3; 4; ...]; seq [11; 12; 13; 14; ...];
seq [21; 22; 23; 24; ...]; seq [31; 32; 33; 34; ...]; ...]
> chunk 5 [1..12];;
val it : seq<seq<int>> =
seq [seq [1; 2; 3; 4; ...]; seq [6; 7; 8; 9; ...]; seq [11; 12]]
If in doubt, use fold.
let split n = let one, append, empty = Seq.singleton, Seq.append, Seq.empty
Seq.fold (fun (m, cur, acc) x ->
if m = n then (1, one x, append acc (one cur))
else (m+1, append cur (one x), acc))
(0, empty, empty)
>> fun (_, cur, acc) -> append acc (one cur)
This has the advantage of being fully functional, yet touch each element of the input sequence only once(*) (as opposed to the Seq.take + Seq.skip solutions proposed above).
(*) Assuming O(1) Seq.append. I should certainly hope so.
I found this to be easily the fastest:
let windowChunk n xs =
let range = [0 .. Seq.length xs]
Seq.windowed n xs |> Seq.zip range
|> Seq.filter (fun d -> (fst d) % n = 0)
|> Seq.map(fun x -> (snd x))
i.e. window the list, zip with a list of integers, remove all the overlapping elements, and then drop the integer portion of the tuple.
I think that the solution from Brian is probably the most reasonable simple option. A probelm with sequences is that they cannot be easily processed with the usual pattern matching (like functional lists). One option to avoid that would be to use LazyList from F# PowerPack.
Another option is to define a computation builder for working with IEnumerator type. I wrote something like that recently - you can get it here. Then you can write something like:
let splitEach chunkSize (s:seq<_>) =
Enumerator.toSeq (fun () ->
let en = s.GetEnumerator()
let rec loop n acc = iter {
let! item = en
match item with
| Some(item) when n = 1 ->
yield item::acc |> List.rev
yield! loop chunkSize []
| Some(item) ->
yield! loop (n - 1) (item::acc)
| None -> yield acc |> List.rev }
loop chunkSize [] )
This enables using some functional patterns for list processing - most notably, you can write this as a usual recursive function (similar to the one you would write for lists/lazy lists), but it is imperative under the cover (the let! constructo of iter takes the next element and modifies the enumerator).