F# Process stdout removed \n - f#

I had this code snippet taken from Starting a process synchronously, and "streaming" the output :
type ProcessResult = { exitCode : int; stdout : string; stderr : string }
let executeProcess (exe,cmdline) =
let psi = new System.Diagnostics.ProcessStartInfo(exe,cmdline)
psi.UseShellExecute <- false
psi.RedirectStandardOutput <- true
psi.RedirectStandardError <- true
psi.CreateNoWindow <- true
psi.WorkingDirectory <- "C:\\GIT\\ProjectX"
let p = System.Diagnostics.Process.Start(psi)
let output = new System.Text.StringBuilder()
let error = new System.Text.StringBuilder()
p.OutputDataReceived.Add(fun args -> output.Append(args.Data) |> ignore)
p.ErrorDataReceived.Add(fun args -> error.Append(args.Data) |> ignore)
p.BeginErrorReadLine()
p.BeginOutputReadLine()
p.WaitForExit()
{ exitCode = p.ExitCode; stdout = output.ToString(); stderr = error.ToString() }
It is known that the result of the execution returns one or more lines in linux-like format (lines are separated by \n)
Nevertheless, executing the following code prints -1, so the \n had been removed.
let p = executeProcess ("git","branch -vv")
printfn "%d" (p.stdout.IndexOf('\n'))
This code in c# -which should be equivalent to the f# listed above- works nice:
private static string[] RetrieveOrphanedBranches(string directory)
{
StringBuilder outputStringBuilder = new StringBuilder();
ProcessStartInfo startInfo = new ProcessStartInfo();
Process p = new Process();
startInfo.CreateNoWindow = true;
startInfo.RedirectStandardOutput = true;
startInfo.RedirectStandardInput = true;
startInfo.WorkingDirectory = directory;
startInfo.UseShellExecute = false;
startInfo.Arguments = "branch -vv";
startInfo.FileName = "git";
p.StartInfo = startInfo;
p.OutputDataReceived += (sender, eventArgs) => outputStringBuilder.AppendLine(eventArgs.Data);
p.Start();
p.BeginOutputReadLine();
p.WaitForExit();
var l = outputStringBuilder.ToString().Split(new char[] { '\n' });
return l;
}
Is there a way to prevent f# removing new-line characters from process output?

"equivalent" here is used quite freely.
C# code
p.OutputDataReceived += (sender, eventArgs) => outputStringBuilder.AppendLine(eventArgs.Data);
F# code:
p.OutputDataReceived.Add(fun args -> output.Append(args.Data) |> ignore)
So in C# code line (with \n) is added to the output. In F# it is added without \n. Use AppendLine in both.

Related

F#: How to enumerate through multiple files correctly?

I have a bunch of files several MiB in size which are very simple:
They have a size of multiples of 8
They only contain doubles in little endian, so can be read with BinaryReader's ReadDouble() method
When lexicographically sorted, they contain all values in the sequence they need to be.
I can't keep everything in memory as a float list or float array so I need a float seq that goes through the necessary files when actually being accessed. The portion that goes through the sequence actually does it in imperative style using GetEnumerator() because I don't want any resource leaks and want to close all files correctly.
My first functional approach was:
let readFile file =
let rec readReader (maybeReader : BinaryReader option) =
match maybeReader with
| None ->
let openFile() =
printfn "Opening the file"
new BinaryReader(new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read))
|> Some
|> readReader
seq { yield! openFile() }
| Some reader when reader.BaseStream.Position >= reader.BaseStream.Length ->
printfn "Closing the file"
reader.Dispose()
Seq.empty
| Some reader ->
reader.BaseStream.Position |> printfn "Reading from position %d"
let bytesToRead = Math.Min(1048576L, reader.BaseStream.Length - reader.BaseStream.Position) |> int
let bytes = reader.ReadBytes bytesToRead
let doubles = Array.zeroCreate<float> (bytesToRead / 8)
Buffer.BlockCopy(bytes, 0, doubles, 0, bytesToRead)
seq {
yield! doubles
yield! readReader maybeReader
}
readReader None
And then, when I have a string list containing all the files, I can say something like:
let values = files |> Seq.collect readFile
use ve = values.GetEnumerator()
// Do stuff that only gets partial data from one file
However, this only closes the files when the reader reaches its end (which is clear when looking at the function). So as a second approach I implemented the file enumerating imperatively:
type FileEnumerator(file : string) =
let reader = new BinaryReader(new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read))
let mutable _current : float = Double.NaN
do file |> printfn "Enumerator active for %s"
interface IDisposable with
member this.Dispose() =
reader.Dispose()
file |> printfn "Enumerator disposed for %s"
interface IEnumerator with
member this.Current = _current :> obj
member this.Reset() = reader.BaseStream.Position <- 0L
member this.MoveNext() =
let stream = reader.BaseStream
if stream.Position >= stream.Length then false
else
_current <- reader.ReadDouble()
true
interface IEnumerator<float> with
member this.Current = _current
type FileEnumerable(file : string) =
interface IEnumerable with
member this.GetEnumerator() = new FileEnumerator(file) :> IEnumerator
interface IEnumerable<float> with
member this.GetEnumerator() = new FileEnumerator(file) :> IEnumerator<float>
let readFile' file = new FileEnumerable(file) :> float seq
now, when I say
let values = files |> Seq.collect readFile'
use ve = values.GetEnumerator()
// do stuff with the enumerator
disposing the enumerator correctly bubbles through to my imperative enumerator.
While this is a feasible solution for what I want to achieve (I could make it faster by reading it blockwise like the first functional approach but for brevity I didn't do it here) I wonder if there is a truly functional approach for this avoiding the mutable state in the enumerator.
I don't quite get what you mean when you say that using GetEnumerator() will prevent resource leaks and allow to close all files correctly. The below would be my attempt at this (ignoring block copy part for demonstration purposes) and I think it results in the files properly closed.
let eof (br : BinaryReader) =
br.BaseStream.Position = br.BaseStream.Length
let readFileAsFloats filePath =
seq{
use file = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read)
use reader = new BinaryReader(file)
while (not (eof reader)) do
yield reader.ReadDouble()
}
let readFilesAsFloats filePaths =
filePaths |> Seq.collect readFileAsFloats
let floats = readFilesAsFloats ["D:\\floatFile1.txt"; "D:\\floatFile2.txt"]
Is that what you had in mind?

How to implement Props.Create using F#

I attempted to port a working C# sample to an OOP version of F#.
Remote actors (on a separate process) are not receiving messages.
I receive the following error:
[ERROR][3/23/2017 4:39:10 PM][Thread 0008][[akka://system2/system/endpointManage
r/reliableEndpointWriter-akka.tcp%3A%2F%2Fsystem1%40localhost%3A8090-1/endpointW
riter#1919547364]] AssociationError [akka.tcp://system2#localhost:8080] <- akka.
tcp://system1#localhost:8090: Error [Object reference not set to an instance of
an object.] [ at Akka.Serialization.Serialization.FindSerializerForType(Type o
bjectType)
at Akka.Remote.Serialization.DaemonMsgCreateSerializer.GetArgs(DaemonMsgCreat
eData proto)
at Akka.Remote.Serialization.DaemonMsgCreateSerializer.FromBinary(Byte[] byte
s, Type type)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serialize
rId, String manifest)
Here's the working C# version:
using (var system = ActorSystem.Create("system1", config))
{
var reply = system.ActorOf<ReplyActor>("reply");
//create a remote deployed actor
var remote1 = system.ActorOf(Props.Create(() => new SomeActor()).WithRouter(FromConfig.Instance), "remoteactor1");
var remote2 = system.ActorOf(Props.Create(() => new SomeActor()).WithRouter(FromConfig.Instance), "remoteactor2");
var remote3 = system.ActorOf(Props.Create(() => new SomeActor()).WithRouter(FromConfig.Instance), "remoteactor3");
var hashGroup = system.ActorOf(Props.Empty.WithRouter(new ConsistentHashingGroup(config)));
Task.Delay(500).Wait();
var routee1 = Routee.FromActorRef(remote1);
hashGroup.Tell(new AddRoutee(routee1));
var routee2 = Routee.FromActorRef(remote2);
hashGroup.Tell(new AddRoutee(routee2));
var routee3 = Routee.FromActorRef(remote3);
hashGroup.Tell(new AddRoutee(routee3));
Task.Delay(500).Wait();
for (var i = 0; i < 5; i++)
{
for (var j = 0; j < 7; j++)
{
var message = new SomeMessage(j, $"remote message: {j}");
hashGroup.Tell(message, reply);
}
}
Console.ReadLine();
}
Here's the port to F# using OOP:
use system = ActorSystem.Create("system1", config)
let reply = system.ActorOf<ReplyActor>("reply")
let props1 = Props.Create(fun () -> SomeActor() :> obj)
let props2 = Props.Create(fun () -> SomeActor() :> obj)
let props3 = Props.Create(fun () -> SomeActor() :> obj)
let remote1 = system.ActorOf(props1.WithRouter(FromConfig.Instance), "remoteactor1")
let remote2 = system.ActorOf(props2.WithRouter(FromConfig.Instance), "remoteactor2")
let remote3 = system.ActorOf(props3.WithRouter(FromConfig.Instance), "remoteactor3")
let hashGroup = system.ActorOf(Props.Empty.WithRouter(ConsistentHashingGroup(config)))
Task.Delay(500).Wait();
let routee1 = Routee.FromActorRef(remote1);
hashGroup.Tell(new AddRoutee(routee1));
let routee2 = Routee.FromActorRef(remote2);
hashGroup.Tell(new AddRoutee(routee2));
let routee3 = Routee.FromActorRef(remote3);
hashGroup.Tell(new AddRoutee(routee3));
Task.Delay(500).Wait();
for i = 0 to 5 do
for j = 0 to 7 do
let message = new HashMessage(j, sprintf "remote message: %i" j);
hashGroup.Tell(message, reply);
Console.ReadLine() |> ignore
Question:
Am I suppose to upcast SomeActor to the object type when invoking the Props.Create method?
let props1 = Props.Create(fun () -> SomeActor() :> obj)
let props2 = Props.Create(fun () -> SomeActor() :> obj)
let props3 = Props.Create(fun () -> SomeActor() :> obj)
The code above is the only difference that I'm aware of.
The only other difference is the tcp path.
C#'s TCP:
remote {
dot-netty.tcp {
port = 8090
hostname = localhost
}
F#'s TCP:
remote {
helios.tcp {
port = 8090
hostname = localhost
}
Props object is a descriptor for the creation process of target actor. Moreover, it must be serializable, as sometimes it may get included on messages passed through the network.
In order to work this way Props internally describes actor construction in form of (actor-type, actor-constructor-arguments). Props.Create(() => new Actor()) is only a helper here: what it actually does, is deconstruct the constructor expression into type info with arguments. This is why it works only with new Actor() expressions.
The problem with your F# code is that you're defining actor creation as a F# function, which props deconstructor doesn't know how to handle. You may still want create your actors using:
Props.Create(typeof<Actor>, [| arg1; arg2 |])
but then you need to keep the correctness of the constructor params by yourself. You can also use i.e. Akkling with it's typed version of props.

This expression was expected to have type MemoryStream but here has type StreamWriter?

I'm trying to add a feature to the Http module (https://github.com/fsharp/FSharp.Data/blob/master/src/Library/Http.fs) of FSharp.Data to download and save a file (from a web response stream). I modified the function asyncReadToEnd but got the following error. Whats the best way to define the function asyncReadToEnd so it will write to MemoryStream if streamWriter is not provided.
Error 1 This expression was expected to have type
MemoryStream
but here has type
StreamWriter
Error line. (at Some sw)
use sw = new StreamWriter(fileName)
let! response = Http.InnerRequest(url, true, ?headers=headers, ?query=query, ?meth=meth, ?body=body, ?cookies=cookies, ?cookieContainer=cookieContainer, ?certificate=certificate, ?streamWriter=Some sw)
Code snippet:
/// Read the contents of a stream asynchronously and return it as a string
static let asyncReadToEnd (stream:Stream) isText streamWriter = async {
// Allocate 4kb buffer for downloading dat
let buffer = Array.zeroCreate (4 * 1024)
use output =
match streamWriter with
| Some (sw) -> sw
| None -> new MemoryStream()
let reading = ref true
....
static member private InnerRequest(url:string, forceText, ?query, ?headers, ?meth, ?body, ?cookies, ?cookieContainer, ?certificate, ?streamWriter) = async {
....
return! Http.augmentWebExceptionsWithDetails <| fun () -> async {
use! resp = Async.FromBeginEnd(req.BeginGetResponse, req.EndGetResponse)
use stream = resp.GetResponseStream()
let! respBody = asyncReadToEnd stream (forceText || (isText resp.ContentType)) streamWriter
let cookies = Map.ofList [ for cookie in cookieContainer.GetCookies uri |> Seq.cast<Cookie> -> cookie.Name, cookie.Value ]
let headers = Map.ofList [ for header in resp.Headers.AllKeys -> header, resp.Headers.[header] ]
let statusCode =
match resp with
| :? HttpWebResponse as resp -> int resp.StatusCode
| _ -> 0
return { Body = respBody
Headers = headers
ResponseUrl = resp.ResponseUri.OriginalString
Cookies = cookies
StatusCode = statusCode } }
}
static member AsyncRequestFile(url, (fileName:string), ?query, ?headers, ?meth, ?body, ?cookies, ?cookieContainer, ?certificate) = async {
use sw = new StreamWriter(fileName)
let! response = Http.InnerRequest(url, true, ?headers=headers, ?query=query, ?meth=meth, ?body=body, ?cookies=cookies, ?cookieContainer=cookieContainer, ?certificate=certificate, ?streamWriter=Some sw)
return
match response.Body with
| ResponseBody.Text text -> text
| ResponseBody.Binary binary -> failwithf "Expecting text, but got a binary response (%d bytes)" binary.Length
}
The issue appears to be that in:
use output =
match streamWriter with
| Some (sw) -> sw
| None -> new MemoryStream()
streamWriter is being inferred to be a MemoryStream option and therefore so is the streamWriter parameter of InnerRequest.
When you call it in AsyncRequestFile you are passing a StreamReader option instead, hence the error.

Convert Sequence.Generate to Sequence Expression

I have the following code that uses Sequence objects to read data from a database table.
V1 works correctly but since the Seq.generate function is deprecated I receive compiler warnings.
I tried to rerplace this code with V2 (below) but it does not work correctly -
basically the reader is opened but only the first record is read - IOW - Read Next does not
work correctly.
To avoid compilation errors/warnings I need to convert V1 code to use a sequence expression.
Any ideas on the correct approach here.
(BTW - This code is based on examples in Rob Pickering's Beginning F# book -
(Data Access/ADO.NET Section).
********************** V1 - Sequence Approach - Deprecated ************************
// execute a command using the Seq.generate
let execCommand (connName: string) (cmdString: string) =
Seq.generate
// This function gets called to open a connection and create a reader
(fun () -> openReader connName cmdString)
// This function gets called to read a single item in
// the enumerable for a reader/connection pair
(fun reader -> readRow(reader))
(fun reader -> reader.Dispose())
*********************** V2 Alternative - (Does not work) ***************************
let generateSequence connName cmdString =
seq { // This function gets called to open a connection and
//create a reader
use reader = openReader connName cmdString
// This function gets called to read a single item in
// the enumerable for a reader/connection pair
yield readRow(reader) }
// execute a command using the Seq.generate
let execCommand (connName: string) (cmdString: string) =
generateSequence connName cmdString
***************************** Common Functions **********************************
// Open a db connection
let openReader connName cmdString =
let conn = openSQLConnection(connName)
let cmd = conn.CreateCommand(CommandText=cmdString,
CommandType = CommandType.Text)
let reader = cmd.ExecuteReader(CommandBehavior.CloseConnection)
// read a row from the data reader
let readRow (reader: #DbDataReader) =
if reader.Read() then
let dict = new Dictionary<string, obj>()
for x in [ 0 .. (reader.FieldCount - 1) ] do
dict.Add(reader.GetName(x), reader.[x])
Some(dict)
else
None
You need to move around a few control structures to get this working.
let generateSequence connName cmdString =
seq {
use reader = openReader connName cmdString
while reader.Read () do
yield readRow reader
yield None
}
let readRow (reader:#DbDataReader) =
let dict = new Dictionary<string, obj>()
for x in [0..(reader.FieldCount - 1)] do
dict.Add (reader.GetName(x), reader.[x])
Some dict
The only thing you left out is: while reader.Read() do
let generateSequence connName cmdString =
seq {
use reader = openReader connName cmdString
while reader.Read() do
yield readRow reader
}

How read a file into a seq of lines in F#

This is C# version:
public static IEnumerable<string> ReadLinesEnumerable(string path) {
using ( var reader = new StreamReader(path) ) {
var line = reader.ReadLine();
while ( line != null ) {
yield return line;
line = reader.ReadLine();
}
}
}
But directly translating needs a mutable variable.
If you're using .NET 4.0, you can just use File.ReadLines.
> let readLines filePath = System.IO.File.ReadLines(filePath);;
val readLines : string -> seq<string>
open System.IO
let readLines (filePath:string) = seq {
use sr = new StreamReader (filePath)
while not sr.EndOfStream do
yield sr.ReadLine ()
}
To answer the question whether there is a library function for encapsulating this pattern - there isn't a function exactly for this, but there is a function that allows you to generate sequence from some state called Seq.unfold. You can use it to implement the functionality above like this:
new StreamReader(filePath) |> Seq.unfold (fun sr ->
match sr.ReadLine() with
| null -> sr.Dispose(); None
| str -> Some(str, sr))
The sr value represents the stream reader and is passed as the state. As long as it gives you non-null values, you can return Some containing an element to generate and the state (which could change if you wanted). When it reads null, we dispose it and return None to end the sequence. This isn't a direct equivalent, because it doesn't properly dispose StreamReader when an exception is thrown.
In this case, I would definitely use sequence expression (which is more elegant and more readable in most of the cases), but it's useful to know that it could be also written using a higher-order function.
let lines = File.ReadLines(path)
// To check
lines |> Seq.iter(fun x -> printfn "%s" x)
On .NET 2/3 you can do:
let readLines filePath = File.ReadAllLines(filePath) |> Seq.cast<string>
and on .NET 4:
let readLines filePath = File.ReadLines(filePath);;
In order to avoid the "System.ObjectDisposedException: Cannot read from a closed TextReader." exception, use:
let lines = seq { yield! System.IO.File.ReadLines "/path/to/file.txt" }

Resources