Would like to speed up builds using FAKE. Theoretically if I could run a build sequence in parallel (within a target) that should make it faster.
let buildProject outputDir buildTargets projectName =
let setParams p =
{ p with
Verbosity = Some(Quiet)
Targets = buildTargets
Properties =
[ "DevEnvDir", "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools"
"OutputPath", outputDir
"Optimize", "True"
"DebugSymbols", "True"
"Configuration", buildMode ]
}
build setParams projectName |> DoNothing
Target "BuildLibs" (fun _ ->
!! "**/*.csproj"
-- "**/*.Tests.csproj"
//A way to run in parallel??
|> Seq.iter (buildProject buildOutDir ["ReBuild"])
)
Is there a way to run the sequence iteration in parallel?
The easiest option would be:
Target "BuildLibs" (fun _ ->
!! "**/*.csproj"
-- "**/*.Tests.csproj"
|> Seq.ToArray
|> Array.Parallel.iter (buildProject buildOutDir ["ReBuild"])
)
Another alternative is "F# Parallel Sequences" component.
#r "./PATHTOLIB/lib/net40/FSharp.Collections.ParallelSeq.dll"
open FSharp.Collections.ParallelSeq
Target "BuildLibs" (fun _ ->
!! "**/*.csproj"
-- "**/*.Tests.csproj"
|> PSeq.iter (buildProject buildOutDir ["ReBuild"])
)
Related
I am trying to run unit test cases from fake ,
while i run the script , it throws an error , i.e.
access to the path c:/users/local/application data is denied.
Code :
Target "Test" (fun _ ->
!! (testDir + "/NUnit.Test.*.dll")
|> NUnit (fun p ->
{p with
DisableShadowCopy = true;
OutputFile = testDir + "TestResults.xml" }))
Please Explain above code .
you might to use ## instead of + and might need \ before testresults.xml in OutputFile
try following
Target "Test" (fun _ ->
!! (testDir + "/NUnit.Test.*.dll")
|> NUnit (fun p ->
{p with
DisableShadowCopy = true;
OutputFile = testDir ## "\TestResults.xml" }))
How do I get hold of the projects that are referenced by a solution file?
Here I have a concrete use case. I have stolen this target CopyBinaries from ProjectScaffold. It copies the output of the project builds into a separate folder. It is not very choosy and copies the output of every project it finds.
Target "CopyBinaries" (fun _ ->
!! "src/**/*.??proj"
-- "src/**/*.shproj"
|> Seq.map (fun f ->
((System.IO.Path.GetDirectoryName f) </> "bin/Release",
binDir </> (System.IO.Path.GetFileNameWithoutExtension f)))
|> Seq.iter (fun (fromDir, toDir) ->
CopyDir toDir fromDir (fun _ -> true))
)
What if I want only copy the output of projects which are referenced explicitly in a solution file. I think of something like this:
Target "CopyBinaries" (fun _ ->
!! solutionFile
|> GetProjectFiles
|> Seq.map (fun f ->
((System.IO.Path.GetDirectoryName f) </> "bin/Release",
binDir </> (System.IO.Path.GetFileNameWithoutExtension f)))
|> Seq.iter (fun (fromDir, toDir) ->
CopyDir toDir fromDir (fun _ -> true))
)
The function GetProjectFiles takes a solution file and extracts the referenced project files.
Is there anything like this hypothetical function in FAKE available?
There is nothing that I have found which gives this kind of information about solution files out of the box. With a small number of functions, an alternative is possible:
let root = directoryInfo "."
let solutionReferences baseDir = baseDir |> filesInDirMatching "*.sln" |> Seq.ofArray
let solutionNames paths = paths |> Seq.map (fun (f:System.IO.FileInfo) -> f.FullName)
let projectsInSolution solutions = solutions |> Seq.collect ReadFile |> Seq.filter (fun line -> line.StartsWith("Project"))
let projectNames projects = projects |> Seq.map (fun (line:string) -> (line.Split [|','|]).[1])
Target "SLN" ( fun () -> root |> solutionReferences |> solutionNames |> projectsInSolution |> projectNames |> Seq.iter (printfn "%s"))
This can be consolidated and functions skipped into any grouping you like. You may have already found this out or something else, but it is good that everyone knows there are options. Thank you. Good day.
The following .fsx file is supposed to download and save to disk binary table base files which are posted as links in a html page on the internet, using Fsharp.Data.dll.
What happens, is that the whole thing stalls after a while and way before it is done, not even throwing an exception or alike.
I am pretty sure, I kind of mis-handle the CopyToAsync() thingy in my async workflow. As this is supposed to run while I go for a nap, it would be nice if someone could tell me how it is supposed to be done correctly. (In more general terms - how to handle a System.Threading.Task thingy in an async workflow thingy?)
#r #"E:\R\playground\DataTypeProviderStuff\packages\FSharp.Data.2.2.3\lib\net40\FSharp.Data.dll"
open FSharp.Data
open Microsoft.FSharp.Control.CommonExtensions
let document = HtmlDocument.Load("http://www.olympuschess.com/egtb/gaviota/")
let links =
document.Descendants ["a"] |> Seq.choose (fun x -> x.TryGetAttribute("href") |> Option.map (fun a -> a.Value()))
|> Seq.filter (fun v -> v.EndsWith(".cp4"))
|> List.ofSeq
let targetFolder = #"E:\temp\tablebases\"
let downloadUrls =
links |> List.map (fun name -> "http://www.olympuschess.com/egtb/gaviota/" + name, targetFolder + name )
let awaitTask = Async.AwaitIAsyncResult >> Async.Ignore
let fetchAndSave (s,t) =
async {
printfn "Starting with %s..." s
let! result = Http.AsyncRequestStream(s)
use fileStream = new System.IO.FileStream(t,System.IO.FileMode.Create)
do! awaitTask (result.ResponseStream.CopyToAsync(fileStream))
printfn "Done with %s." s
}
let makeBatches n jobs =
let rec collect i jl acc =
match i,jl with
| 0, _ -> acc,jl
| _, [] -> acc,jl
| _, x::xs -> collect (i-1) (xs) (acc # [x])
let rec loop remaining acc =
match remaining with
| [] -> acc
| x::xs ->
let r,rest = collect n remaining []
loop rest (acc # [r])
loop jobs []
let download () =
downloadUrls
|> List.map fetchAndSave
|> makeBatches 2
|> List.iter (fun l -> l |> Async.Parallel |> Async.RunSynchronously |> ignore )
|> ignore
download()
Note Updated code so it creates batches of 2 downloads at a time and only the first batch works. Also added the awaitTask from the first answer as this seems the right way to do it.
News What is also funny: If I interrupt the stalled script and then #load it again into the same instance of fsi.exe, it stalls right away. I start to think it is a bug in the library I use or something like that.
Thanks, in advance!
Here fetchAndSave has been modified to handle the Task returned from CopyToAsync asynchronously. In your version you are waiting on the Task synchronously. Your script will appear to lock up as you are using Async.RunSynchronously to run the whole workflow. However the files do download as expected in the background.
let awaitTask = Async.AwaitIAsyncResult >> Async.Ignore
let fetchAndSave (s,t) = async {
let! result = Http.AsyncRequestStream(s)
use fileStream = new System.IO.FileStream(t,System.IO.FileMode.Create)
do! awaitTask (result.ResponseStream.CopyToAsync(fileStream))
}
Of course you also need to call
do download()
on the last line of your script to kick things into motion.
I am trying to figure out how to manage multiple lazy sequences from a single function in F#.
For example, in the code below, I am trying to get two sequences - one that returns all files in the directories, and one that returns a sequence of tuples of any directories that could not be accessed (for example due to permissions) with the exception.
While the below code compiles and runs, errorSeq never has any elements when used by other code, even though I know that UnauthorizedAccess exceptions have occurred.
I am using F# 2.0.
#light
open System.IO
open System
let rec allFiles errorSeq dir =
Seq.append
(try
dir |> Directory.GetFiles
with
e -> Seq.append errorSeq [|(dir, e)|]
|> ignore
[||]
)
(try
dir
|> Directory.GetDirectories
|> Seq.map (allFiles errorSeq)
|> Seq.concat
with
e -> Seq.append errorSeq [|(dir, e)|]
|> ignore
Seq.empty
)
[<EntryPoint>]
let main args =
printfn "Arguments passed to function : %A" args
let errorSeq = Seq.empty
allFiles errorSeq args.[0]
|> Seq.filter (fun x -> (Path.GetExtension x).ToLowerInvariant() = ".jpg")
|> Seq.iter Console.WriteLine
errorSeq
|> Seq.iter (fun x ->
Console.WriteLine("Error")
x)
0
If you wanted to take a more functional approach, here's one way to do it:
let rec allFiles (errorSeq, fileSeq) dir =
let files, errs =
try
Seq.append (dir |> Directory.GetFiles) fileSeq, errorSeq
with
e -> fileSeq, Seq.append [dir,e] errorSeq
let subdirs, errs =
try
dir |> Directory.GetDirectories, errs
with
e -> [||], Seq.append [dir,e] errs
Seq.fold allFiles (errs, files) subdirs
Now we pass the sequence of errors and the sequence of files into the function each time and return new sequences created by appending to them within the function. I think that the imperative approach is a bit easier to follow in this case, though.
Seq.append returns a new sequence, so this
Seq.append errorSeq [|(dir, e)|]
|> ignore
[||]
has no effect. Perhaps you want your function to return a tuple of two sequences? Or use some kind of mutable collection to write errors as you encounter them?
I'm trying to make test for this function
let extract_one_rule (rule:Rule.t<'a,'b>) =
let rec expand = function
|PAlt (a,b) -> expand a # expand b
|PSeq (a,b) -> let wrap = List.map (fun x -> (x.rule, fun r -> {x with rule = r})) a
|> List.unzip
in
let rec gen = function
| hd::tl -> [for x in hd -> x :: ( gen tl |> List.concat)]
| [] -> []
in
fst wrap |> List.map expand |> gen
|> List.map (fun x -> PSeq ((List.map2 ( |> ) x (snd wrap)),b))
|PRef _
|PLiteral _
|PToken _ as t -> [t]
| _ -> (System.Console.WriteLine("incorrect tree for alternative expanding!")
; failwith "incorrect tree for alternative expanding!")
in
expand rule.body |> List.map (fun x -> {rule with body = x})
using FsCheck
so i have this
let ExpandAlterTest(t : Rule.t<Source.t,Source.t> ) = convertToMeta t |> List.forall (fun x -> ruleIsAfterEBNF x)
but i'l see exception "incorrect tree for alternative expanding!"
but when i use smth like that
let ExpandAlterTest(t : Rule.t<Source.t,Source.t> ) = (correctForAlExp t.body) ==> lazy ( convertToMeta t |> List.forall (fun x -> ruleIsAfterEBNF x))
NUnit doesn't stop working
Why it can be?
It could be that the precondition you added is very restrictive, so that it takes a long time before a good value (one that actually passes the precondition) is found. FsCheck is hardened against this - by default, it tries to find 100 values but when it has rejected 1000 it gives up and you should see a "Arguments exhausted after x tests" output. But this might take a long time, if generating and checking the value takes a long time.
Could also be that you actually have a bug somewhere, like an infinite loop.
Try changing the FsCheck config to run less tests, doing a verbose run (verboseCheck), and breaking in the debugger when it seems to hang.