writing binary files with streams - f#

Question

How to download image files via streams in the temp directory, I've following code and I'm stuck and need guidance with the seek and the count part. There are some wrapper approaches but I'm looking specifically for while loop approach for RAM efficiency reasons.
Writing
let tempFileName = Path.GetTempFileName()
let request = WebRequest.CreateHttp "http://example.com/image.png"
use response = request.GetResponse() :?> HttpWebResponse
use stream = response.GetResponseStream()
let buffer = Array.zeroCreate 1024
use reader = new BinaryReader(stream)
use memoryStream = new MemoryStream()
use fileStream = new FileStream(tempFileName, FileMode.Open)
while not (reader.PeekChar() <> -1) do
fileStream.Write(reader.ReadBytes(1024), 0, 1024)
return Ok (tempFileName)

Answer 1

First of all, I notice that although you're creating a buffer array, you're not actually using it. Second, when I look at the BinaryReader documentation, and specifically the documentation for the ReadBytes method, I notice that it takes an int parameter and returns a byte array. This must mean that it's allocating a new array every time, which seems to be the opposite of what you intend (since you mention RAM efficiency, I assume that what you actually want is to re-use the same buffer each time).
And one other observation: the ReadBytes method says that it might return an array smaller than the requested size, if there were fewer bytes available. Your code currently isn't handling that case.
All of these can be fixed, though, by switching to the BinaryReader.Read(byte[], int, int) method instead. With this method, your while loop would look something like the following:
while not (reader.PeekChar() <> -1) do
let bytesRead = reader.Read(buffer, 0, 1024)
fileStream.Write(buffer, 0, bytesRead)
And now that we're keeping track of how many bytes were read by each Read operation, we can get rid of the PeekChar call and save ourselves some time (calling PeekChar on something you're downloading is not without cost since the library has to download the next byte, then save it somewhere so it can be returned the next time you call Read). We can do that by checking how many bytes were read at the previous call: if it was 0, then that means we're at the end of the stream. To do this, we'll move the bytesRead variable out of the loop, which means making it a mutable variable that we'll re-use every time through the loop:
let mutable bytesRead = -1
while not (bytesRead = 0) do
bytesRead <- reader.Read(buffer, 0, 1024)
fileStream.Write(buffer, 0, bytesRead)
Or if you want to be slightly more explicit about the fact that you're skipping Write if bytesRead is 0, you could add an if block:
let mutable bytesRead = -1
while not (bytesRead = 0) do
bytesRead <- reader.Read(buffer, 0, 1024)
if bytesRead > 0 then
fileStream.Write(buffer, 0, bytesRead)
That last if statement isn't strictly necessary, though: FileStream.Write should just return without doing anything if it's asked to write 0 bytes. However, since that's not documented anywhere that I could find, I added the if statement in this last code sample just to be on the safe side.

Answer 2

As of .NET 4.6.2, there is System.IO.Stream#CopyTo method:
namespace FSharpBasics
module ImageCrawler =
open System.Net
open System.IO
open System.Text.RegularExpressions
let private myurl = "https://cdn.pixabay.com/photo/2016/07/06/15/29/math-1500720_960_720.jpg"
let crawler (url: string) =
let fileName = Regex.Match(url, #"\/([^\/]+)$", RegexOptions.RightToLeft).Groups.[1].Value
let request = WebRequest.CreateHttp url
let response = request.GetResponse()
use s = response.GetResponseStream()
use w = File.Create fileName
s.CopyTo w
w.Flush true
[<EntryPoint>]
let main argv =
printfn "JPEG file will be saved"
crawler myurl
printf "Saved"
0