F#: How to identify keys that have multiple values in a HashTable? - f#

Question

I am currently working on a beginner's project to implement my own duplicate file finder. This is my first time working with a .NET language, so I am still extremely unfamiliar with .NET APIs.
Here is the code that I have written so far:
open System
open System.IO
open System.Collections.Generic
let directory = #somePath
let getAllFiles (directory : string) =
Directory.GetFiles(directory)
let getFileInfo (directory : string) =
directory
|> getAllFiles
|> Seq.map (fun eachFile -> (eachFile, new FileInfo(eachFile)))
let getFileLengths (directory: string) =
directory
|> getFileInfo
|> Seq.map (fun (eachFile, eachFileInfo : FileInfo) -> (eachFile, eachFileInfo.Length))
// If two files have the same lengths, they might be duplicates of each other.
let groupByFileLengths (directory: string) =
directory
|> getFileLengths
|> Seq.groupBy snd
|> Seq.map (fun (fileLength, files) -> fileLength, files |> Seq.map fst |> List.ofSeq)
let findGroupsOfTwoOrMore (directory: string) =
directory
|> groupByFileLengths
|> Seq.filter (snd >> List.length >> (<>) 1)
let constructHashtable (someTuple) =
let hashtable = new Hashtable()
someTuple
|> Seq.iter hashtable.Add
hashtable
let readAllBytes (tupleOfFileLengthsAndFiles) =
tupleOfFileLengthsAndFiles
|> snd
|> Seq.map (fun eachFile -> (File.ReadAllBytes eachFile, eachFile))
|> constructHashtable
What I want to do is to construct a hash table with the byte array of each file as the key, and the file name itself as the value. If multiple files with different file names share the same bye array, then they are duplicates, and my goal is to remove the duplicate files.
I have looked through the Hashtable namespace on MSDN, but there is no method for identifying hashtable keys that contain multiple values.
Edit: Here is my attempt at implementing MD5:
let readAllBytesMD5 (tupleOfFileLengthsAndFiles) =
let md5 = MD5.Create()
tupleOfFileLengthsAndFiles
|> snd
|> Seq.map (fun eachFile -> (File.ReadAllBytes eachFile, eachFile))
|> Seq.map (fun (byteArray, eachFile) -> (md5.ComputeHash(byteArray), eachFile))
|> Seq.map (fun (hashCode, eachFile) -> (hashCode.ToString, eachFile))
Please advise on how I may improve and continue, because I am stuck here due to not having a firm grasp of how MD5 works. Thank you.

Answer 1

Hashtable doesn't support multiple values for the same key - you'll get an exception when you try to add a second entry with the same key. It is also untyped, you should almost always prefer a typed mutable System.Collections.Generic.Dictionary or an immutable F# Map.
What you're looking for is a Map<byte array, Set<string>>. Here's my take on it:
let buildMap (paths: string array) =
paths
|> Seq.map (fun eachFile -> (File.ReadAllBytes eachFile, eachFile))
|> Seq.groupBy fst
|> Seq.map (fun (key, items) ->
key, items |> Seq.map snd |> Set.ofSeq)
|> Map.ofSeq
As an aside, unless you're comparing very, very small files, using the entire file contents as a key won't get you very far. You will probably want to look into generating checksums for those files and using them instead.