Python's pandas library allows getting info() on a data frame.
For example.
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 30 entries, 0 to 29
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 30 non-null object
1 PhoneNumber 30 non-null object
2 City 30 non-null object
3 Address 30 non-null object
4 PostalCode 30 non-null object
5 BirthDate 30 non-null object
6 Income 26 non-null float64
7 CreditLimit 30 non-null object
8 MaritalStatus 24 non-null object
dtypes: float64(1), object(8)
memory usage: 2.2+ KB
Is there an equivalent in Deedle's data frame? Something that can get an overview for missing values and the inferred types.
There isn't a single function to do this - it would be a nice addition to the library if you wanted to consider sending a pull-request.
The following gets all the information you would need:
// Prints column names and types, with data preview
df.Print(true)
// Print key range of rows (or key sequence if it is not ordered)
if df.RowIndex.IsOrdered then printfn "%A" df.RowIndex.KeyRange
else printfn "%A" df.RowIndex.Keys
// Get access to the data of the frame so that we can inspect the columns
let dt = df.GetFrameData()
for n, (ty, vec) in Seq.zip dt.ColumnKeys dt.Columns do
// Print name, type of column
printf "%A %A" n ty
// Query the interal data storage to see if it uses
// array of optional values (may have nulls) or not
match vec.Data with
| Vectors.VectorData.DenseList _ -> printfn " (no nulls)"
| _ -> printfn " (nulls)"
Based on Thomas's suggestion (thank you!) I modified it slightly to produce an output similar to pandas:
let info (df: Deedle.Frame<'a,'b>) =
let dt = df.GetFrameData()
let countOptionalValues d =
d
|> Seq.filter (
function
| OptionalValue.Present _ -> true
| _ -> false
)
|> Seq.length
Seq.zip dt.ColumnKeys dt.Columns
|> Seq.map (fun (col, (ty, vec)) ->
{|
Column = col
``Non-Null Count`` =
match vec.Data with
| Vectors.VectorData.DenseList d -> $"%i{d |> Seq.length} non-null"
| Vectors.VectorData.SparseList d -> $"%i{d |> countOptionalValues} non-null"
| Vectors.VectorData.Sequence d -> $"%i{d |> countOptionalValues} non-null"
Dtype = ty
|}
)
Pandas output:
Deedle output:
Related
I'm facing trouble when I try to create missing values in a Frame and later perform operations with them. Here is a "working" sample:
open Deedle
open System.Text.RegularExpressions
do fsi.AddPrinter(fun (printer:Deedle.Internal.IFsiFormattable) -> "\n" + (printer.Format()))
module Frame = let mapAddCol col f frame = frame |> Frame.addCol col (Frame.mapRowValues f frame)
[ {|Desc = "A - 1.50ml"; ``Price ($)`` = 23.|}
{|Desc = "B - 2ml"; ``Price ($)`` = 18.5|}
{|Desc = "C"; ``Price ($)`` = 25.|} ]
|> Frame.ofRecords
(*
Desc Price ($)
0 -> A - 1.50ml 23
1 -> B - 2ml 18.5
2 -> C 25
*)
|> Frame.mapAddCol "Volume (ml)" (fun row ->
match Regex.Match(row.GetAs<string>("Desc"),"[\d\.]+").Value with
| "" -> OptionalValue.Missing
| n -> n |> float |> OptionalValue)
(*
Desc Price ($) Volume (ml)
0 -> A - 1.50ml 23 1.5
1 -> B - 2ml 18.5 2
2 -> C 25 <missing>
*)
|> fun df -> df?``Price ($/ml)`` <- df?``Price ($)`` / df?``Volume (ml)``
//error message: System.InvalidCastException: Object must implement IConvertible.
What is wrong with this approach?
Deedle internally stores a flag whether a value is present or missing. This is typically exposed via the OptionalValue type, but the internal representation is not actually using this type.
When you use a function such as mapRowValues to generate new data, Deedle needs to recognize which data is missing. This happens in only somewhat limited cases only. When you return OptionalValue<float>, Deedle actually produces a series where the type of values is OptionalValue<float> rather than float (the type system does not let it do anything else).
For float values, the solution is just to return nan as your missing value:
|> Frame.mapAddCol "Volume (ml)" (fun row ->
match Regex.Match(row.GetAs<string>("Desc"),"[\d\.]+").Value with
| "" -> nan
| n -> n |> float )
This will create a new series of float values, which you can then access using the ? operator.
I have a csv file containing daily weights as follows:
Date,Name,Weight
11-Sep-2017,Alpha,9-1
13-Sep-2017,Alpha,8-13
15-Sep-2017,Alpha,8-11
Though I can successfully import them using CsvProvider, the weight column defaults to System.DateTime.
// Weight
[<Measure>] type lb
[<Literal>]
let input = "DayWeights.csv"
type Weights = CsvProvider<input, HasHeaders=true>
let data = Weights.GetSample()
for row in data.Rows do
printfn "Output: (%A, %A, %A)" row.Date row.Name row.Weight
Is it possible to create a Unit of Measure (UoM) to define "stlb" with the option to convert to lbs on import and, if so, how?
I don't think you could represent stones-pounds as a single numeric type, and units of measure can only be used on numeric types (although there is some discussion about changing this in future). This is because some of their features only make sense with numeric operations like addition and multiplication. The units themselves are multiplied and divided:
[<Measure>] type lb
2<lb> + 2<lb> // 4<lb>
2<lb> * 2<lb> // 4<lb ^ 2>
2<lb> / 2<lb> // 1
Instead of units of measure, if you want some kind of tag to know that a given value has a type of stones-pounds, you could create a single case discriminated union:
type StonesPounds = StonesPounds of int * int
// StonesPounds -> int<lb>
let convertToLb (StonesPounds (s, p)) = (s * 14 + p) * 1<lb>
StonesPounds (1, 2) |> convertToLb // 16<lb>
The downside of this compared to units of measure is that you have to manually pack and unpack these values in code before you can use the numbers and there is a runtime cost for that too.
I resolved the automatic converting of the input weight column to System.DateTime as follows:
// http://fssnip.net/27
let lazySplit (sep:string) (str:string) =
match sep, str with
| ((null | ""), _) | (_, (null | "")) -> seq [str]
| _ ->
let n, j = str.Length, sep.Length
let rec loop p =
seq {
if p < n then
let i = match str.IndexOf(sep, p) with -1 -> n | i -> i
yield str.Substring(p, i - p)
yield! loop (i + j)
}
loop 0
let weight input =
input
|> (fun x -> lazySplit "/" x |> Seq.take 2 |> String.concat("-"))
let data = Weighings.GetSample()
for row in data.Rows do
let stlbs = weight (string row.Weight)
printfn "Output: (%A, %A, %A)" row.Date row.Name stlbs
// Output: 11-Sep-2017,"Alpha","09-01")
Thanks to one and all for your expert help and guidance.
I want to build a dictionary from a list of items.
An item has the following definition:
type Item =
| A of TotalPrice * Special
| B of TotalPrice * Special
| C of TotalPrice
| D of TotalPrice
I want the keys of the dictionary to map to the case ids:
| A
| B
| C
| D
I would then have the values for the case id be a list.
How do I separate the case ids from the case values?
Example:
let dictionary = items |> List.map (fun item -> item) // uh...
Appendix:
module Checkout
(*Types*)
type UnitPrice = int
type Qty = int
type Special =
| ThreeForOneThirty
| TwoForFourtyFive
type TotalPrice = { UnitPrice:int ; Qty:int }
type Item =
| A of TotalPrice * Special
| B of TotalPrice * Special
| C of TotalPrice
| D of TotalPrice
(*Functions*)
let totalPrice (items:Item list) =
let dictionary = items |> List.map (fun item -> item) // uh...
0
(*Tests*)
open FsUnit
open NUnit.Framework
[<Test>]
let ``buying 2 A units, B unit, A unit = $160`` () =
// Setup
let items = [A ({UnitPrice=50; Qty=2} , ThreeForOneThirty)
B ({UnitPrice=30; Qty=1} , TwoForFourtyFive)
A ({UnitPrice=50; Qty=1} , ThreeForOneThirty)]
items |> totalPrice |> should equal 160
Your data is badly defined for your use case. If you want to refer to the kinds of items by themselves, you need to define them by themselves:
type ItemKind = A | B | C | D
type Item = { Kind: ItemKind; Price: TotalPrice; Special: Special option }
Then you can easily build a dictionary of items:
let dictionary = items |> List.map (fun i -> i.Kind, i) |> dict
Although I must note that such dictionary may not be possible: if the items list contains several items of the same kind, some of them will not be included in the dictionary, because it can't contain multiple identical keys. Perhaps I didn't understand what kind of dictionary you're after.
If you want to create the dictionary with keys like A, B, C and D you will fail because A and B are constructors with type TotalPrice * Special -> Item and C and D are constructors of type TotalPrice -> Item. Dictionary would have to make a decision about type of keys.
Getting DU constructor name should be doable by reflection but is it really necessary for your case?
Maybe different type structure will be more efficient for your case, ie. Fyodor Soikin proposal.
Maybe the following will clarify somewhat why datastructure and code is no good, and as such also clarify that this mainly is not related to FP as indicated in some of the comments et al.
My guess is that the question is related to "how can this be grouped", and lo and behold, there is in fact a groupBy function!
(*Types*)
type UnitPrice = int
type Qty = int
type Special =
| ThreeForOneThirty
| TwoForFourtyFive
type TotalPrice = { UnitPrice:int ; Qty:int }
type Item =
| A of TotalPrice * Special
| B of TotalPrice * Special
| C of TotalPrice
| D of TotalPrice
let items = [A ({UnitPrice=50; Qty=2} , ThreeForOneThirty)
B ({UnitPrice=30; Qty=1} , TwoForFourtyFive)
A ({UnitPrice=50; Qty=1} , ThreeForOneThirty)]
let speciallyStupidTransformation =
function
| ThreeForOneThirty -> 34130
| TwoForFourtyFive -> 2445
let stupidTransformation =
function
| A (t,s) -> "A" + (s |> speciallyStupidTransformation |> string)
| B (t,s) -> "B" + (s |> speciallyStupidTransformation |> string)
| C (t) -> "C"
| D(t) -> "D"
let someGrouping = items |> List.groupBy(stupidTransformation)
val it : (string * Item list) list =
[("A34130",
[A ({UnitPrice = 50;
Qty = 2;},ThreeForOneThirty); A ({UnitPrice = 50;
Qty = 1;},ThreeForOneThirty)]);
("B2445", [B ({UnitPrice = 30;
Qty = 1;},TwoForFourtyFive)])]
Yeah its still a bad idea. But its somewhat grouped uniquely, and may be misused further to aggregate some sums or whatever.
Adding some more code for that, like the following:
let anotherStupidTransformation =
function
| A(t,_) -> (t.UnitPrice, t.Qty)
| B(t,_) -> (t.UnitPrice, t.Qty)
| C(t) -> (t.UnitPrice, t.Qty)
| D(t) -> (t.UnitPrice, t.Qty)
let x4y x y tp q =
if q%x = 0 then y*q/x else tp/q*(q%x)+(q-q%x)/x*y
let ``34130`` = x4y 3 130
let ``2445`` = x4y 2 45
let getRealStupidTotal =
function
| (s, (tp,q)) ->
(s|> List.ofSeq, (tp,q))
|> function
| (h::t, (tp,q)) ->
match t |> List.toArray |> System.String with
| "34130" -> ``34130`` tp q
| "2445" -> ``2445`` tp q
| _ -> tp
let totalPrice =
items
|> List.groupBy(stupidTransformation)
|> List.map(fun (i, l) -> i,
l
|> List.map(anotherStupidTransformation)
|> List.unzip
||> List.fold2(fun acc e1 e2 ->
((fst acc + e1) * e2, snd acc + e2) ) (0,0))
|> List.map(getRealStupidTotal)
|> List.sum
val totalPrice : int = 160
might or might not yield some test cases correct.
For the above testdata as far as I can read the initial code at least is ok. The sum does get to be 160...
Would I use this code anywhere? Nope.
Is it readable? Nope.
Is it fixable? Not without changing the way the data are structured to avoid several of the stupid transformations...
Is there a concise functional way to rename columns of a Deedle data frame f?
f.RenameColumns(...) is usable, but mutates the data frame it is applied to, so it's a bit of a pain to make the renaming operation idempotent. I have something like f.RenameColumns (fun c -> ( if c.IndexOf( "_" ) < 0 then c else c.Substring( 0, c.IndexOf( "_" ) ) ) + "_renamed"), which is ugly.
What would be nice is something that creates a new frame from the input frame, like this: Frame( f |> Frame.cols |> Series.keys |> Seq.map someRenamingFunction, f |> Frame.cols |> Series.values ) but this gets tripped up by the second part -- the type of f |> Frame.cols |> Series.values is not what is required by the Frame constructor.
How can I concisely transform f |> Frame.cols |> Series.values so that its result is edible by the Frame constructor?
You can determine its function when used with RenameColumns:
df.RenameColumns someRenamingFunction
You can also use the function Frame.mapColKeys.
Builds a new data frame whose columns are the results of applying the
specified function on the columns of the input data frame. The
function is called with the column key and object series that
represents the column data.
Source
Example:
type Record = {Name:string; ID:int ; Amount:int}
let data =
[|
{Name = "Joe"; ID = 51; Amount = 50};
{Name = "Tomas"; ID = 52; Amount = 100};
{Name = "Eve"; ID = 65; Amount = 20};
|]
let df = Frame.ofRecords data
let someRenamingFunction s =
sprintf "%s(%i)" s s.Length
df.Format() |> printfn "%s"
let ndf = df |> Frame.mapColKeys someRenamingFunction
ndf.Format() |> printfn "%s"
df.RenameColumns someRenamingFunction
df.Format() |> printfn "%s"
Print:
Name ID Amount
0 -> Joe 51 50
1 -> Tomas 52 100
2 -> Eve 65 20
Name(4) ID(2) Amount(6)
0 -> Joe 51 50
1 -> Tomas 52 100
2 -> Eve 65 20
Name(4) ID(2) Amount(6)
0 -> Joe 51 50
1 -> Tomas 52 100
2 -> Eve 65 20
I have two lists of records with the following types:
type AverageTempType = {Date: System.DateTime; Year: int64; Month: int64; AverageTemp: float}
type DailyTempType = {Date: System.DateTime; Year: int64; Month: int64; Day: int64; DailyTemp: float}
I want to get a new list which is made up of the DailyTempType "joined" with the AverageTempType. Ultimately though for each daily record I want the Daily Temp - Average temp for the matching month.
I think I can do this with loops as per below and massage this into a reasonable output:
let MatchLoop =
for i in DailyData do
for j in AverageData do
if (i.Year = j.Year && i.Month = j.Month)
then printfn "%A %A %A %A %A" i.Year i.Month i.Day i.DailyTemp j.Average
else printfn "NOMATCH"
I have also try to do this with matching but I can't quite get there (I'm not sure how to define the list correctly in the input type and then iterate to get a result. Also I'm not sure sure if this approach even makes sense):
let MatchPattern (x:DailyTempType) (y:AverageTempType) =
match (x,y) with
|(x,y) when (x.Year = y.Year && x.Month = y.Month) ->
printfn "match"
|(_,_) -> printfn "nomatch"
I have looked into Deedle which I think can do this relatively easily but I am keen to understand how to do it a lower level.
What you can do is to create a map of the monthly average data. You can think of a map as a read-only dictionary:
let averageDataMap =
averageData
|> Seq.map (fun x -> ((x.Year, x.Month), x))
|> Map.ofSeq
This particular map is a Map<(int64 * int64), AverageTempType>, which, in plainer words, means that the keys into the map are tuples of year and month, and the value associated with each key is an AverageTempType record.
This enables you to find all the matching month data, based on the daily data:
let matches =
dailyData
|> Seq.map (fun x -> (x, averageDataMap |> Map.tryFind (x.Year, x.Month)))
Here, matches has the data type seq<DailyTempType * AverageTempType option>. Again, in plainer words, this is a sequence of tuples, where the first element of each tuple is the original daily observation, and the second element is the corresponding monthly average, if a match was found, or None if no matching monthly average was found.
If you want to print the values as in the OP, you can do this:
matches
|> Seq.map snd
|> Seq.map (function | Some _ -> "Match" | None -> "No match")
|> Seq.iter (printfn "%s")
This expression starts with the matches; then pulls out the second element of each tuple; then again maps a Some value to the string "Match", and a None value to the string "No match"; and finally prints each string.
I would convert first AverageTempType seq to a Map (reducing cost of join):
let toMap (avg:AverageTempType seq) = avg |> Seq.groupBy(fun a -> a.Year + a.Month) |> Map.ofSeq
Then you can join and return an option, so consuming code can do whatever you want (print, store, error, etc.):
let join (avg:AverageTempType seq) (dly:DailyTempType seq) =
let avgMap = toMap avg
dly |> Seq.map (fun d -> d.Year, d.Month, d.Day, d.DailyTemp, Map.tryFind (d.Year + d.Month) avgMap);;