Why is it that if I create a new CSV type with the CsvProvider<> in F# like this:
type ThisCsv = CsvProvider<Schema = "A (decimal), B (string), C (decimal)", HasHeaders = false>
then create/fill/save the .csv, the resulting file does not contain the headers in the schema I specified? It seems like there should be a way to include headers in the final .csv file, but that's not the case.
Setting HasHeaders = true errors out, because there's no sample provided. The only way for HasHeaders = true to work is to have a sample .csv. It seems to me that there should be a way to specify the schema without a sample and also include the headers in the final file.
Am I missing something when I use [nameOfMyCSV].Save() that can include the headers from the schema or can this not be done?
I'm afraid the headers from the Schema are only used for the property names of the row. To have them in the file you save you have to provide Sample. Though, the sample can contain only the headers. Also, HasHeaders has to be set to true:
type ThisCsv = CsvProvider<
Sample="A, B, C",
Schema = "A(decimal), B, C(decimal)",
HasHeaders = true>
If the sample contains only headers then if you want to specify data types the schema has to be provided as well.
You can see that the schema is used for property only when you rename the Sample headers in the Schema:
type ThisCsv = CsvProvider<
Sample="A, B, C",
Schema = "A->AA(decimal), B->BB, C(decimal)",
HasHeaders = true>
Then the generated row will have properties like AA, B, CC. But the file generated will still have A, B, C. Also, the Headers property of a csv you created using this schema will be Some [|"A"; "B"; "C"|]:
// Run in F# Interactive
let myCsv = new ThisCsv([ThisCsv.Row(1.0m, "a", 2.0m)])
myCsv.Headers
// The last line returns: Some [|"A"; "B"; "C"|]
Also, to get better understanding of what's happening inside the parser worth taking a look at the source code in GitHub: CSV folder in general and CsvRuntime.fs in particular.
Related
I'm trying to replace a very long list of strings (200+) with the str_replace all function. I would like to be able to read in an excel file which has the R code for the replacements as the list of replacements to so that I don't have 200+ lines in my script dedicated to naming strings, but can't seem to get it to work.
What I have:
`REPLACEMENTS = c(
'\\bAFFL\\b'='AFFILIATE',
'\\bACCY\\b'='ACCOUNTANCY',
'\\bACTG\\b'='ACCOUNTING',
'\\bACQUIS\\b'='ACQUISITION',
'\\bADMNR\\b'='ADMINISTRATOR',
)
df$tobereplaced = str_replace_all(df$tobereplaced, REPLACEMENTS)`
What I want:
`REPLACEMENTS = excelfile$replacements
df$tobereplaced = str_replace_all(df$tobereplaced, REPLACEMENTS)`
When I try this I get the error:
Error in fix_replacement(replacement) :
argument "replacement" is missing, with no default
I have a test where after getting a response I would like to validate the entire schema of the response (not individual response node/value comparison).
Sample test:
[<Test>]
let howtoValidateSchema () =
let request = Request.createUrl Post "https://reqres.in/api/users"
|> Request.setHeader (Accept "application/json")
|> Request.bodyString """{"name": "morpheus",
"job": "leader"}"""
|> Request.responseAsString
|> run
Is there a way that I can save my expected Schema somewhere and once I get the response I do the comparison to check that response has same number of nodes (neither less nor more than expected schema)?
I am ok to opt for other libs like FSharp.Data if we there is no direct way in HttpFs.Client. I looked at FSharp.Data (https://fsharp.github.io/FSharp.Data/library/JsonProvider.html) but not able to seek how it meets the requirements where the schema comparison needs to be done with the savedExpectedSchemaJson=ResponseJson.
You can use Newtonsoft.Json.Schemato validate schemas:
open Newtonsoft.Json.Schema
open Newtonsoft.Json.Linq
let schema = JSchema.Parse expectedSchema
let json = JObject.Parse responeJson
let valid = json.IsValid schema
However this assumes you have a schema predefined somewhere. If you don't have such schema is best to use the JsonProvider who can infer it for you.
Run the call manually and save the result in a sample.json file and create a type using the JsonProvider:
type ResponseSchema = JsonProvider<"sample.json">
and you can use this type to parse any new content based on the sample (provided that the sample is a representative.
ResponseSchema.parse response
This won't validate the schema but will try to meet as best as it can given the input.
An Excel file (.xlsx) is uploaded on the frontend which is UI5 Fiori.
The file contents come to SAP ABAP backend via ODATA in XSTRING format.
I need to store that XSTRING into an internal table and then in a DDIC table. Eg: Suppose the Excel has 5 columns then I want to store that data of 5 columns in the corresponding columns in the DDIC table.
I have tried various Function Modules like:
SCMS_XSTRING_TO_BINARY
SCMS_BINARY_TO_STRING
and following Classes & methods:
cl_bcs_convert=>raw_to_string
cl_soap_xml_helper=>xstring_to_string
but none were able to convert the XSTRING to STRING.
Can you please suggest which function module or class/method can be used to solve the problem?
For most comfort, use abap2xlsx.
If you cannot or do not want to use that, you can alternatively parse the Excel file on your own. .xlsx files are basically .zip files with a different file ending. Use cl_abap_zip->load to open the xstring you receive and ->get to extract the individual files from the zip. Afterwards, use XML parsers like cl_ixml or transformations to parse the XML content of the files.
Note that Excel's XML is a complicated file format, with several files that work together to form the worksheets. Refer to Microsoft's File format reference for Word, Excel, and PowerPoint for details. It's non-trivial to interpret this, so you will usually be a lot happier with abap2xlsx.
abap2xlsx is the most powerful and feature-rich way of doing this, as said by Florian, it supports styles, charts, complex tables, however it may not be always available due to the system limitations, restrictions to install custom packages in system or whatever.
Here is the way how to accomplish this with pure standard without using custom frameworks.
Since Netweaver 7.02 SAP supports Open Microsoft formats natively and provides classes for handling them: CL_XLSX_DOCUMENT, CL_DOCX_DOCUMENT and CL_PPTX_DOCUMENT, abap2xlsx is built at these classes too, yes. So let's start a bit of reinventing the wheel.
XLSX file is an OpenXML archive of files, of which the most interesting: sheet1.xml and sharedStrings.xml. Let's build a sample based on MARC table fields
Now you want to transfer this table to internal table with the same structure. The steps would be:
Extract needed files from XLSX archive
Read worksheet structure from sheet1.xml
Read sheet values from sharedStrings.xml
Map them together and write the result to the internal table
Here is the sample class that handles the job, I used the cl_openxml_helper applet to load XLSX, but you can receive XSTRINGed XLSX in whatever way.
CLASS xlsx_reader DEFINITION.
PUBLIC SECTION.
TYPES: BEGIN OF ty_marc,
matnr TYPE char20,
werks TYPE char20,
disls TYPE char20,
ekgrp TYPE char20,
dismm TYPE char20,
END OF ty_marc,
tt_marc TYPE STANDARD TABLE OF ty_marc WITH EMPTY KEY.
METHODS: read RETURNING VALUE(tab) TYPE tt_marc,
extract_xml IMPORTING index TYPE i
xstring TYPE xstring
RETURNING VALUE(rv_xml_data) TYPE xstring.
ENDCLASS.
CLASS xlsx_reader IMPLEMENTATION.
METHOD read.
TYPES: BEGIN OF ty_row,
value TYPE string,
index TYPE abap_bool,
END OF ty_row,
BEGIN OF ty_worksheet,
row_id TYPE i,
row TYPE TABLE OF ty_row WITH EMPTY KEY,
END OF ty_worksheet,
BEGIN OF ty_si,
t TYPE string,
END OF ty_si.
DATA: data TYPE TABLE OF ty_si,
sheet TYPE TABLE OF ty_worksheet.
TRY.
DATA(xstring_xlsx) = cl_openxml_helper=>load_local_file( 'C:\marc.xlsx' ).
CATCH cx_openxml_not_found.
ENDTRY.
"Read the sheet XML
DATA(xml_sheet) = extract_xml( EXPORTING xstring = xstring_xlsx iv_xml_index = 2 ).
"Read the data XML
DATA(xml_data) = extract_xml( EXPORTING xstring = xstring_xlsx iv_xml_index = 3 ).
TRY.
* transforming structure into ABAP
CALL TRANSFORMATION zsheet
SOURCE XML xml_sheet
RESULT root = sheet.
* transforming data into ABAP
CALL TRANSFORMATION zxlsx_data
SOURCE XML xml_data
RESULT root = data.
CATCH cx_xslt_exception.
CATCH cx_st_match_element.
CATCH cx_st_ref_access.
ENDTRY.
* mapping structure and data
LOOP AT sheet ASSIGNING FIELD-SYMBOL(<fs_row>).
APPEND INITIAL LINE TO tab ASSIGNING FIELD-SYMBOL(<line>).
LOOP AT <fs_row>-row ASSIGNING FIELD-SYMBOL(<fs_cell>).
ASSIGN COMPONENT sy-tabix OF STRUCTURE <line> TO FIELD-SYMBOL(<fs_field>).
CHECK sy-subrc = 0.
<fs_field> = COND #( WHEN <fs_cell>-index = abap_false THEN <fs_cell>-value ELSE VALUE #( data[ <fs_cell>-value + 1 ]-t OPTIONAL ) ).
ENDLOOP.
ENDLOOP.
ENDMETHOD.
METHOD extract_xml.
TRY.
DATA(lo_package) = cl_xlsx_document=>load_document( iv_data = xstring ).
DATA(lo_parts) = lo_package->get_parts( ).
CHECK lo_parts IS BOUND AND lo_package IS BOUND.
DATA(lv_uri) = lo_parts->get_part( 2 )->get_parts( )->get_part( index )->get_uri( )->get_uri( ).
DATA(lo_xml_part) = lo_package->get_part_by_uri( cl_openxml_parturi=>create_from_partname( lv_uri ) ).
rv_xml_data = lo_xml_part->get_data( ).
CATCH cx_openxml_format cx_openxml_not_found.
ENDTRY.
ENDMETHOD.
ENDCLASS.
zsheet transformation:
<?sap.transform simple?>
<tt:transform xmlns:tt="http://www.sap.com/transformation-templates" template="main">
<tt:root name="root"/>
<tt:template name="main">
<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:x14ac=
"http://schemas.microsoft.com/office/spreadsheetml/2009/9/ac" xmlns:xr="http://schemas.microsoft.com/office/spreadsheetml/2014/revision" xmlns:xr2="http://schemas.microsoft.com/office/spreadsheetml/2015/revision2" xmlns:xr3=
"http://schemas.microsoft.com/office/spreadsheetml/2016/revision3">
<tt:skip count="4"/>
<sheetData>
<tt:loop name="row" ref="root">
<row>
<tt:attribute name="r" value-ref="row_id"/>
<tt:loop name="cells" ref="$row.ROW">
<c>
<tt:cond><tt:attribute name="t" value-ref="index"/><tt:assign to-ref="index" val="C('X')"/></tt:cond>
<v><tt:value ref="value"/></v>
</c>
</tt:loop>
</row>
</tt:loop>
</sheetData>
<tt:skip count="2"/>
</worksheet>
</tt:template>
</tt:transform>
zxlsx_data transformation
<?sap.transform simple?>
<tt:transform xmlns:tt="http://www.sap.com/transformation-templates" template="main">
<tt:root name="ROOT"/>
<tt:template name="main">
<sst xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">
<tt:loop name="line" ref=".ROOT">
<si>
<t>
<tt:value ref="t"/>
</t>
</si>
</tt:loop>
</sst>
</tt:template>
</tt:transform>
Here is how to call it:
START-OF-SELECTION.
DATA(reader) = NEW xlsx_reader( ).
DATA(marc) = reader->read( ).
The code is pretty self-explanatory, but let's put a couple of notes:
File sheet1.xml contains a special attribute t in each cell which denotes either the value should be treated as a literal or a reference to sharedStrings.xml
I used two simple transformations but XSLT can be used as well, possibly allowing you to reduce all XML stuff to single transformation
I deliberately used generic char20 types to be able to handle headers. If you wanna preserve native types, then you cannot read table header (skip the first line in sheet LOOP), because you'll receive type violation and dump. If you receive table without headers, then it is fine to declare structure with native types
If you don't want to use transformations then sXML is your friend. You can parse XML with classes as well, but ST transformation are considerably faster
With some additional effort you can make this snippet dynamic and parse XLSX with any structure
You can read more about this approach in this doc.
I am having a problem with variables inside tables. this is essential since I use tables as configuration for my program.
so I have tested the following code that works:
> x = "X"
> t = {["ref"]="table with value: "..x}
> print(t["ref"])
table with value: X
> x = "Y"
> t = {["ref"]="table with value: "..x}
> print(t["ref"])
table with value: Y
it however doesn't work without the second > t = ["ref"]="table with value: "..x
now I went to implement this into my main program witch consists of two files, one witch returns the configuration table. And one file with all the functions and stuff. it looks as following
FILE A (main.lua):
testString = "test1"
print(testString)
local CONFIG = require'config'
print(CONIFG[1].test)
testString = "test2"
print(testString)
local CONFIG = require'config'
print(CONIFG[1].test)
FILE B (config.lua):
local CONFIG = {
{["test"]=[[this is a test: ]]..testString}
}
return CONFIG
now when i run file A (a.k.a. main.lua) i get the following output:
test1
this is a test: test1
test2
this is a test: test1
i can't figure out what i am doing wrong here.. i thought it had something to do with that it was a single string so i made testString a table but that gave me the same result...
(that title really seems scary.. sorry)
require, by design, caches the return value. So if you call require with the same string, it will not execute the script again. It will simply return the previously returned value.
require is for loading modules. And modules should not change their return values based on other global state.
The function you're probably looking for is dofile. This will always load and execute the file (but it has none of the path searching properties of require). Alternatively, you can use loadfile to load the file as a function, then execute that function to regenerate the table whenever you want.
Also:
I am having a problem with variables inside tables.
There are no "variables inside tables". Or at least not the way you mean. Expecting a change to a variable to affect some other value is like expecting this:
a = 5
b = a + 5
a = 10
assert(b == 15, "This will never be true.")
When an expression (whether a + 5 or "table with value: " .. x) is evaluated, it results in a value. The resulting value is in no way dependent on any value or variable from the expression that generated it.
That's why you had to regenerate the value; because values don't change just because some variable changes.
The CreateSpreadsheetWorkbook example method from the OpenXml documentation does translate directly to F#. The problem seems to be the Append method of the Sheets object. The code executes without error, but the resulting xlsx file is missing the inner Xml which should have been appended, and the file is unreadable by Excel. I suspect the problem stems from the conversion of functional F# structures into a System.Collections type, but I do not have direct evidence for this.
I have run similar code in C# and VB.NET (i.e. the documentation example) and it executes perfectly and creates a readable, complete xlsx file.
I know that I could deal with the XML directly, but I would like to understand the nature of the mismatch between F# and OpenXml. Any suggestions?
The code is almost directly from the example:
namespace OpenXmlLib
open System
open DocumentFormat
open DocumentFormat.OpenXml
open DocumentFormat.OpenXml.Packaging
open DocumentFormat.OpenXml.Spreadsheet
module OpenXmlXL =
// this function overwrites an existing file without warning!
let CreateSpreadsheetWorkbook (filepath: string) =
// Create a spreadsheet document by supplying the filepath.
// By default, AutoSave = true, Editable = true, and Type = xlsx.
let spreadsheetDocument = SpreadsheetDocument.Create(filepath, SpreadsheetDocumentType.Workbook)
// Add a WorkbookPart to the document.
let workbookpart = spreadsheetDocument.AddWorkbookPart()
workbookpart.Workbook <- new Workbook()
// Add a WorksheetPart to the WorkbookPart.
let worksheetPart = workbookpart.AddNewPart<WorksheetPart>()
worksheetPart.Worksheet <- new Worksheet(new SheetData())
// Add Sheets to the Workbook.
let sheets = spreadsheetDocument.WorkbookPart.Workbook.AppendChild<Sheets>(new Sheets())
// Append a new worksheet and associate it with the workbook.
let sheet = new Sheet()
sheet.Id <- stringValue(spreadsheetDocument.WorkbookPart.GetIdOfPart(worksheetPart))
//Console.WriteLine(sheet.Id.Value)
sheet.SheetId <- UInt32Value(1u)
// Console.WriteLine(sheet.SheetId.Value)
sheet.Name <- StringValue("TestSheet")
//Console.WriteLine(sheet.Name.Value)
sheets.Append (sheet)
// Console.WriteLine("Sheets: {0}", sheets.InnerXml.ToString())
workbookpart.Workbook.Save()
spreadsheetDocument.Close()
The sheet is created, but empty:
sheet.xml:
<?xml version="1.0" encoding="utf-8" ?>
<x:worksheet xmlns:x="http://schemas.openxmlformats.org/spreadsheetml/2006/main" />
workbook.xml:
<?xml version="1.0" encoding="utf-8" ?>
- <x:workbook xmlns:x="http://schemas.openxmlformats.org/spreadsheetml/2006/main">
- <x:sheets>
<x:sheet name="TestSheet" sheetId="1" r:id="R263eb6f245a2497e" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" />
</x:sheets>
</x:workbook>
The problem is very subtle, and is in your calls to the Worksheet constructor and the Sheets.Append method. Both of these methods are overloaded, and can take either a seq<OpenXmlElement> or any number of individual OpenXmlElements (via a [<System.ParamArray>]/params array). The twist is that the OpenXmlElement type itself implements the seq<OpenXmlElement> interface.
In C#, when you call new Worksheet(new SheetData()), the compiler's overload resolution picks the second of the overloads, implicitly creating a one-element array containing the SheetData value. However, in F#, since the SheetData class implements IEnumerable<OpenXmlElement>, the first overload is chosen, which creates a new WorkSheet by enumerating the contents of the SheetData, which is not what you want.
To fix this, you need to set up your calls so that they use the other overload (first example below) or explicitly create a singleton sequence (second example below):
worksheetPart.Worksheet <- new Worksheet(new SheetData() :> OpenXmlElement)
...
sheets.Append([sheet :> OpenXmlElement])