F# Excel Range.Sort Fails or Rearranges Columns - f#

I have two cases. The preliminary code:
open Microsoft.Office.Interop.Excel
let xl = ApplicationClass()
xl.Workbooks.OpenText(fileName...)
let wb = xl.Workbooks.Item(1)
let ws = wb.ActiveSheet :?> Worksheet
let rows = string ws.UsedRange.Rows.Count
First, I try the following to sort:
ws.Sort.SortFields.Clear()
ws.Sort.SortFields.Add(xl.Range("A8:A" + rows), XlSortOn.xlSortOnValues, XlSortOrder.xlAscending, XlSortDataOption.xlSortNormal) |> ignore
ws.Sort.SortFields.Add(xl.Range("H8:H" + rows), XlSortOn.xlSortOnValues, XlSortOrder.xlAscending, XlSortDataOption.xlSortNormal) |> ignore
ws.Sort.SetRange(xl.Range("A7:I" + rows))
ws.Sort.Header <- XlYesNoGuess.xlYes
ws.Sort.MatchCase <- false
ws.Sort.Orientation <- XlSortOrientation.xlSortRows
ws.Sort.SortMethod <- XlSortMethod.xlPinYin
ws.Sort.Apply()
This results in "The sort reference is not valid. Make sure that it's within the data you want to sort, and the first Sort By box isn't the same or blank.".
Then I try the following to sort:
ws.Range("A7:I" + rows).Sort(xl.Range("A8:A" + rows), XlSortOrder.xlAscending,
xl.Range("H8:H" + rows), "", XlSortOrder.xlAscending,
"", XlSortOrder.xlAscending, XlYesNoGuess.xlYes,
XlSortOrientation.xlSortRows) |> ignore
This results in my columns being rearranged, though I don't see any logic to the rearrangement. And the rows are not sorted.
Please tell me what I'm doing wrong.

I haven't figured out why the first Sort attempt doesn't work. But I checked out the documentation on the second Sort. Using named parameters and dropping all but necessary parameters, I came up with the following:
ws.Range("A8:H" + rows).Sort(Key1=xl.Range("A8:A" + rows), Key2=xl.Range("H8:H" + rows),
Orientation=XlSortOrientation.xlSortColumns) |> ignore
The default Orientation is xlSortRows. I interpret this as sorting the rows, the normal, default, intuitive sort that Excel does when one sorts manually. Oh no. If you want the normal, default, intuitive sort that Excel does when one sorts manually, specify xlSortColumns.

The first Sort doesn't work because you're probably trying to sort a table (judging by property Header set to XlYesNoGuess.xlYes) by columns located on A and H. In this case you can only sort using "Sort top to bottom"(xlSortColumns).
You can also convince yourself by trying this in Excel first and see what options are available:
1) Select the range you want to apply sorting (e.g. in your case "A7:I" + rows)
2) Click on "Data" menu -> "Sort & Filter" task pane -> Sort
3) Add/Delete columns/rows by using "Add Level"/"Delete Level" buttons
4) Then you can select whatever level you want and click "Options..." button and you will see your available "Sort Options".
You will notice that in case of a table you will have only one available option for "Orientation", which is "Sort top to bottom"
xlSortColumns - sorts by columns (meaning rows are sorted based on those columns)
xlSortRows - sort by rows (meaning columns are sorted based on those rows)

Related

Merging Two Files Together and Retaining All Columns

I think this surely must be a simple thing to achieve, but I have tried various appends and merges and can't seem to get it right.
I have two files, one titled 'Previous' and one titled 'Current'. Both show near identical data, like so :
ID Status Date_Changed
1 Closed 10/11/21
2 Open 10/01/21
3 Closed 10/03/21
4 Pending 10/15/21
I'd like to merge both files together, but retain all columns so that it is structured as below. This will allow me to show tables of what has changed etc.
ID Previous.Status Current.Status Previous.Date_Changed Current.Date_Changed
1 Closed Open 10/11/21 10/15/21
2 Open Closed 10/01/21 10/15/21
3 Closed Pending 10/03/21 10/14/21
I am aware this is probably due to my own naivety with PowerBI. I have tried combining the data by connecting to the folder, but that seems to create a new dataset with the data stacked on top (ie with duplicate ID values). I tried using merge queries as new and joiningby ID, but that didn't seem to give me the right output either?
You can start from the Current table and merge in the previous table joining on ID and then expand the columns. Rename and reorder columns as desired.
Here's an example you can paste into the Advanced Editor:
let
CurrentSource = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUfIvSM0DUoYG+oam+kYGRoZKsTrRSkZAIeec/OLUFEw5Y6BQQGpeSmZeOlTSBCFpglMyFgA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, Status = _t, Date_Changed = _t]),
Current = Table.TransformColumnTypes(CurrentSource,{{"ID", Int64.Type}, {"Status", type text}, {"Date_Changed", type date}}),
PreviousSource = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUXLOyS9OTQEyDA30DQ31jQyMDJVidaKVjIBC/gWpeRAZAyQZYzRdBsYIOROgUEBqXkpmXjrUSFOoZCwA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, Status = _t, Date_Changed = _t]),
Previous = Table.TransformColumnTypes(PreviousSource,{{"ID", Int64.Type}, {"Status", type text}, {"Date_Changed", type date}}),
#"Merged Queries" = Table.NestedJoin(Current, {"ID"}, Previous, {"ID"}, "Previous", JoinKind.LeftOuter),
#"Expanded Previous" = Table.ExpandTableColumn(#"Merged Queries", "Previous", {"Status", "Date_Changed"}, {"Previous.Status", "Previous.Date_Changed"}),
#"Renamed Columns" = Table.RenameColumns(#"Expanded Previous",{{"Status", "Current.Status"}, {"Date_Changed", "Current.Date_Changed"}}),
#"Reordered Columns" = Table.ReorderColumns(#"Renamed Columns",{"ID", "Previous.Status", "Current.Status", "Previous.Date_Changed", "Current.Date_Changed"})
in
#"Reordered Columns"
Note: I've defined Previous within the query above so that it's self-contained. Ordinarily, it would be a separate query.
You can try the following steps:
create a new column called "Source Name" in each table with a constant value mentioning if the data if from "Previous" or "Current"
Then append both the table in Power Query. This will still stack both the tables on top of each other.
But now you can use the "Source Name" column to differentiate them in the Matric Visual. You can add a Matrix visual with
"Source Name" in the Column Field
"Date" & "Status" in values field
"ID" in Rows field
This is ideal because you get all the data in a tabular format which will help you in further calculations if necessary
You can check out an example screenshot below:

How do i remove rows based on comma-separated list of values in a Power BI parameter in Power Query?

I have a list of data with a title column (among many other columns) and I have a Power BI parameter that has, for example, a value of "a,b,c". What I want to do is loop through the parameter's values and remove any rows that begin with those characters.
For example:
Title
a
b
c
d
Should become
Title
d
This comma separated list could have one value or it could have twenty. I know that I can turn the parameter into a list by using
parameterList = Text.Split(<parameter-name>,",")
but then I am unsure how to continue to use that to filter on. For one value I would just use
#"Filtered Rows" = Table.SelectRows(#"Table", each Text.StartsWith([key], <value-to-filter-on>))
but that only allows one value.
EDIT: I may have worded my original question poorly. The comma separated values in the parameterList can be any number of characters (e.g.: a,abcd,foo,bar) and I want to see if the value in [key] starts with that string of characters.
Try using List.Contains to check whether the starting character is in the parameter list.
each List.Contains(parameterList, Text.Start([key], 1)
Edit: Since you've changed the requirement, try this:
Table.SelectRows(
#"Table",
(C) => not List.AnyTrue(
List.Transform(
parameterList,
each Text.StartsWith(C[key], _)
)
)
)
For each row, this transforms the parameterList into a list of true/false values by checking if the current key starts with each text string in the list. If any are true, then List.AnyTrue returns true and we choose not to select that row.
Since you want to filter out all the values from the parameter, you can use something like:
= Table.SelectRows(#"Changed Type", each List.Contains(Parameter1,Text.Start([Title],1))=false)
Another way to do this would be to create a custom column in the table, which has the first character of title:
= Table.AddColumn(#"Changed Type", "FirstChar", each Text.Start([Title],1))
and then use this field in the filter step:
= Table.SelectRows(#"Added Custom", each List.Contains(Parameter1,[FirstChar])=false)
I tested this with a small sample set and it seems to be running fine. You can test both and see if it helps with the performance. If you are still facing performance issues, it would probably be easier if you can share the pbix file.
This seems to work fairly well:
= List.Select(Source[Title], each Text.Contains(Parameter1,Text.Start(_,1))=false)
Replace Source with the name of your table and Parameter1 with the name of your Parameter.

What would be the most efficient way to shift objects in a Lua table based on choosing a slot?

I have a table of objects, and a user can choose an object in the table at any given order in the table and place it in another slot in the table. When that happens I need the table to shift from the selected dropped slot and fill the empty slot. Not a swap, that's easy, but a shift at the point of placement.
so if I have this as a highly simplified example of my table
t = {a, b, c, d, e, f}
and the user chooses, say e, and wants to drop it into slot b. how best would I
have e take the b slot
have all the values from "b to d shift right and then also fill
the empty e slot?
how would I handle this shift no matter what one is chosen and where
its moved in the table efficiently no matter what size the table
might be?
Here is an implementation of shift using table.move which is efficient and available in Lua 5.3 as #lhf mentioned:
function shift(t, old, new)
local value = t[old]
if new < old then
table.move(t, new, old - 1, new + 1)
else
table.move(t, old + 1, new, old)
end
t[new] = value
end
If you want to move the item at position old to position new as you describe, you can use this:
table.insert(t, new, table.remove(t,old))
Here is your example:
t = {10,20,30,40,50,60}
print(table.concat(t, ','))
old = 5
new = 2
table.insert(t, new, table.remove(t,old))
print(table.concat(t, ','))
As for efficiency, the code above does shift some elements twice when they could have stayed where they were, but this will probably not matter unless the table is huge.
In Lua 5.3, you can probably do something better with table.move.

F# parent/child lists to Excel

I'm still new to F# so hopefully my question isn't too dumb. I'm creating an Excel file. I've done a lot of Excel with C# so that isn't a problem. I have a list of parent rows and then a list of child rows. What's the best way to spit that into Excel and keep track of the row in Excel that it belongs in.
Assuming my list rowHdr is a list of Row types, I have something like this:
let setCellText (x : int) (y : int) (text : string) =
let range = sprintf "%c%d" (char (x + int 'A')) (y+1)
sheet.Range(range).Value(Missing.Value) <- text
type Row =
{ row_id:string
parent_id:string
text:string }
let printRowHdr (rowIdx:int) (colIdx:int) (rowToPrint:Row) rows =
setCellText colIdx rowIdx rowToPrint.text
List.iteri (fun i x -> printRowHdr (i+1) 0 x rows) <| rowHdr
I still have trouble thinking about what the best functional approach is at times. Somewhere in the printRowHdr function I need to iterate through the child rows for the rows where the parent_id is equal to parent row id. My trouble is knowing what row in Excel it belongs in. Maybe this is totally the wrong approach, but I appreciate any suggestions.
Thanks for any help, I sincerely appreciate it.
Thanks,
Nick
Edited to add:
Tomas - Thanks for the help. Let's say I have two lists, one with US states and another with cities. The cities list also contains the state abbreviation. I would want to loop through the states and then get the cities for each state. So it might look something like this in Excel:
Alabama
Montgomery
California
San Francisco
Nevada
Las Vegas
etc...
Given those two lists could I join them somehow into one list?
I'm not entirely sure if I understand your question - giving a concrete example with some inputs and a screenshot of the Excel sheet that you're trying to get would be quite useful.
However, the idea of using ID to model parent/child relationship (if that's what you're trying to do) does not sound like the best functional approach. I imagine you're trying to represent something like this:
First Row
Foo Bar
Foo Bar
Second Row
More Stuff Here
Some Even More Neste Stuff
This can be represented using a recursive type that contains the list of items in the current row and then a list of children rows (that themselves can contain children rows):
type Row = Row of list<string> * list<Row>
You can then process the structure using recursive function. An example of a value (representing first three lines from the example above) may be:
Row( ["First"; "Row"],
[ Row( ["Foo"; "Bar"], [] )
Row( ["Foo"; "Bar"], [] ) ])
EDIT: The Row type above would be useful if you had arbitrary nesting. If you have just two layers (states and cities), then you can use list of lists. The other list containing state name together with a nested list that contains all cities in that state.
If you start with two lists, then you can use a couple of F# functions to turn the input into a list of lists:
let states = [ ("WA", "Washington"); ("CA", "California") ]
let cities = [ ("WA", "Seattle"); ("WA", "Redmond"); ("CA", "San Francisco") ]
cities
// Group cities by the state
|> Seq.groupBy (fun (id, name) -> id)
|> Seq.map (fun (id, cities) ->
// Find the state name for this group of cities
let _, name = states |> Seq.find (fun (st, _) -> st = id)
// Return state name and list of city names
name, cities |> Seq.map snd)
Then you can recursively iterate over the nested lists (in the above, they are actually sequences, so you can turn them to lists using List.ofSeq) and keep an index of the current row and column.

ISQL Perform instruction: after editadd editupdate of table vs. after add update of table

INFORMIX-SQL 7.3 Perform Screens:
According to documentation, in an "after editadd editupdate of table" control block, its instructions are executed before the row is added or updated to the table, whereas in an "after add update of table" control block, its instructions are executed after the row has been added or updated to the table. Supposedly, this would mean that any instructions which would alter values of field-tags linked to table.columns would not be committed to the table, but field-tags linked to displayonly fields will change?
However, when using "after add update of table", I placed instructions which alter values for field-tags linked to table.columns and their displayed and committed values also changed! I would have thought that an "after add update of table" would only alter displayonly fields.
TABLES
customer
transaction
branch
interest
dates
ATTRIBUTES
[...]
q = transaction.trx_type, INCLUDE=("E","C","V","P","T"), ...;
tb = transaction.trx_int_table,
LOOKUP f1 = ta_days1_f,
t1 = ta_days1_t,
i1 = ta_int1,
[...]
JOINING *interest.int_table, ...;
[...]
INSTRUCTIONS
customer MASTER OF transaction
transaction MASTER OF customer
delimiters ". ";
AFTER QUERY DISPLAY ADD UPDATE OF transaction
if z = "E" then let q = "E"
if z = "C" then let q = "C"
if z = "1" then let q = "E"
[...]
END
Is 'z' a column in the transaction table?
Is the trouble that the value in 'z' is causing a change in the value of 'q' (aka transaction.trx_type), and the modified value is being stored in the database?
Is the value in 'z' part of the transaction table?
Have you verified that the value in the DB is indeed changed - using the Query Language option or a simple (default) form?
It might look as if it is because the instruction is also used AFTER DISPLAY, so when the values are retrieved from the DB, the value displayed in 'q' would be the mapped values corresponding to the value stored in 'z'. You would have to inspect the raw data to hide that mapping.
If this is not the problem, please:
Amend the question to show where 'z' comes from.
Also describe exactly what you do and see.
Confirm that the data in the database, as opposed to on the screen, is amended.
Please can you see whether this table plus form behaves the same for you as it does for me?
Table Transaction
CREATE TABLE TRANSACTION
(
trx_id SERIAL NOT NULL,
trx_type CHAR(1) NOT NULL,
trx_last_type CHAR(1) NOT NULL,
trx_int_table INTEGER NOT NULL
);
Form
DATABASE stores
SCREEN SIZE 24 BY 80
{
trx_id [f000]
trx_type [q]
trx_last_type [z]
trx_int_table [f001 ]
}
END
TABLES
transaction
ATTRIBUTES
f000 = transaction.trx_id;
q = transaction.trx_type, UPSHIFT, AUTONEXT,
INCLUDE=("E","C","V","P","T");
z = transaction.trx_last_type, UPSHIFT, AUTONEXT,
INCLUDE=("E","C","V","P","T","1");
f001 = transaction.trx_int_table;
INSTRUCTIONS
AFTER ADD UPDATE DISPLAY QUERY OF transaction
IF z = "E" THEN LET q = "E"
IF z = "C" THEN LET q = "C"
IF z = "1" THEN LET q = "E"
END
Experiments
[The parenthesized number is automatically generated by IDS/Perform.]
Add a row with data (1), V, E, 23.
Observe that the display is: 1, E, E, 23.
Exit the form.
Observe that the data in the table is: 1, V, E, 23.
Reenter the form and query the data.
Update the data to: (1), T, T, 37.
Observe that the display is: 1, T, T, 37.
Exit the form.
Observe that the data in the table is: 1, T, T, 37.
Reenter the form and query the data.
Update the data to: (1), P, 1, 49
Observe that the display is: 1, E, 1, 49.
Exit the form.
Observe that the data in the table is: 1, P, 1, 49.
Reenter the form and query the data.
Observe that the display is: 1, E, 1, 49.
Choose 'Update', and observe that the display changes to: 1, P, 1, 49.
I did the 'Observe that the data in the table is' steps using:
sqlcmd -d stores -e 'select * from transaction'
This generated lines like these (reflecting different runs):
1|V|E|23
1|P|1|49
That is my SQLCMD program, not Microsoft's upstart of the same name. You can do more or less the same thing with DB-Access, except it is noisier (13 extraneous lines of output) and you would be best off writing the SELECT statement in a file and providing that as an argument:
$ echo "select * from transaction" > check.sql
$ dbaccess stores check
Database selected.
trx_id trx_type trx_last_type trx_int_table
1 P 1 49
1 row(s) retrieved.
Database closed.
$
Conclusions
This is what I observed on Solaris 10 (SPARC) using ISQL 7.50.FC1; it matches what the manual describes, and is also what I suggested in the original part of the answer might be the trouble - what you see on the form is not what is in the database (because of the INSTRUCTIONS section).
Do you see something different? If so, then there could be a bug in ISQL that has been fixed since. Technically, ISQL 7.30 is out of support, I believe. Can you upgrade to a more recent version than that? (I'm not sure whether 7.32 is still supported, but you should really upgrade to 7.50; the current release is 7.50.FC4.)
Transcribing commentary before deleting it:
Up to a point, it is good that you replicate my results. The bad news is that in the bigger form we have different behaviour. I hope that ISQL validates all limits - things like number of columns etc. However, there is a chance that they are not properly validated, given the bug, or maybe there is a separate problem that only shows with the larger form. So, you need to ensure you have a supported version of the product and that the problem reproduces in it. Ideally, you will have a smaller version of the table (or, at least, of the form) that shows the problem, and maybe a still smaller (but not quite as small as my example) version that shows the absence of the problem.
With the test case (table schema and Perform screen that shows the problem) in hand, you can then go to IBM Tech Support with "Look - this works correctly when the form is small; and look, it works incorrectly when the form is large". The bug should then be trackable. You will need to include instructions on how to reproduce the bug similar to those I gave you. And there is no problem with running two forms - one simple and one more complex and displaying the bug - in parallel to show how the data is stored vs displayed. You could describe the steps in terms of 'Form A' and 'Form B', with Form A being Absolutely OK and Form B being Believed to be Buggy. So, add a record with certain values in Form B; show what is displayed in Form B after; show what is stored in the database in Form A after too; show that they are not different when they should be.
Please bear in mind that those who will be fixing the issue have less experience with the product than either you or me - so keep it as simple as possible. Remove as many attributes as you can; leave comments to identify data types etc.

Resources