Trying to covert a Db2 query result set to an xml file based on xsl. Can we use the below pattern?
DB2 Connector -> XML_Transformer Stage (imported xsl) - XML _Output Stage.
Thanks...R
There are multiple options assuming you do not have XML already in your Db2 table,
you do not need the XML Transformer.
I strongly suggest you use the modern Hierarchical stage (also known as XML stage depending on the version of DataStage) so I would go for following structure if you want a file or files as a target.
Db2 Connect -> Hierarchical stage -> Sequential File stage
In addition, Db2 offers lots of XML functionality to generate XML by using SQL or XQuery.
Related
As we all know,joern can easily analysis project written by c/c++,and it will generate CPG(code property graph).
Now I hava 2 csv file(nodes.csv and rels.csv),the 2 csv file can build AST(abstract syntax code)by neo4j and it can be shown in neo4j.But AST is not enough for me,what i need is CPG.How could i use the 2 csv file to build a CPG and show it in neo4j.I wanna to know how to use joern to make my ast'csv file become cpg'csv file.Or you hava other ways to make me do this.
I relly need your help,please.Thank you very much.
How can I convert a code property graph(cpg) obtained from joern (https://joern.io/) from .bin format to .json format for feeding it to a graph machine learning library for classification.
Note: CPG = AST + Control Flow Graph + Program Dependency Graph
Task: Machine Learning on Source Code.
You can use scala script 'graph-for-funcs.sc' which is included in the joern scripts directory. However you need to redirect the output in order to store it in file (since the output goes to stdout by default).
I made a custom script to do so.
I am new to Snowflake, but my company has been using it successfully.
Parquet files are currently being written with an existing Avro Schema, using Java parquet-avro v1.10.1.
I have been updating the dependencies in order to use latest Avro, and part of that bumped Parquet to 1.11.0.
The Avro Schema is unchanged. However when using the COPY INTO Snowflake command, I receive a LOAD FAILED with error: Error parsing the parquet file: Logical type Null can not be applied to group node but no other error details :(
The problem is that there are no null columns in the files.
I've cut the Avro schema down, and found that the presence of a MAP type in the Avro schema is causing the issue.
The field is
{
"name": "FeatureAmounts",
"type": {
"type": "map",
"values": "records.MoneyDecimal"
}
}
An example of the Parquet schema using parquet-tools.
message record.ResponseRecord {
required binary GroupId (STRING);
required int64 EntryTime (TIMESTAMP(MILLIS,true));
required int64 HandlingDuration;
required binary Id (STRING);
optional binary ResponseId (STRING);
required binary RequestId (STRING);
optional fixed_len_byte_array(12) CostInUSD (DECIMAL(28,15));
required group FeatureAmounts (MAP) {
repeated group map (MAP_KEY_VALUE) {
required binary key (STRING);
required fixed_len_byte_array(12) value (DECIMAL(28,15));
}
}
}
The 2 files I have, written in parquet 1.10.1 and 1.11.0 output this identical schema.
I have also tried with a bigger schema example, and it appears everything works fine if there is no "map" avro type present in the schema. I have other massive files with huge schemas, many union types that convert to groups in parquet, but all are written and read successfully when they don't contain any "map" types.
But as soon as I add back the "map" type then I get that weird error message from Snowflake when trying to ingest the 1.11.0 version (however 1.10.1 version will load successfully). But parquet-tools with 1.11.0, 1.10.1 etc can still read the files.
I understand that from this comment that there are changes to the Logical Types in Parquet 1.11.0, but that it is supposed to be compatibile still for old versions to read.
But does anyone know what version of Parquet is used by Snowflake to parse these files? Is there something else that could be going on here?
Appreciate any assistance
Logical type Null can not be applied to group node
Looking up the error above, it appears that a version of Apache Arrow's parquet libraries is being used to read the file.
However, looking closer, the real problem lies in the use of legacy types within the Avro based Parquet Writer implementation (the following assumes Java was used to write the files).
The new logicalTypes schema metadata introduced in Parquet defines many types including a singular MAP type. Historically, the former convertedTypes schema field supported use of MAP AND MAP_KEY_VALUE for legacy readers. The new writers that use logicalTypes (1.11.0+) should not be using the legacy map type anymore, but work hasn't been done yet to update the Avro to Parquet schema conversions to drop the MAP_KEY_VALUE types entirely.
As a result, the schema field for MAP_KEY_VALUE gets written out with an UNKNOWN value of logicalType, which trips up Arrow's implementation that only understands logicalType values of MAP and LIST (understandably).
Consider logging this as a bug against the Apache Parquet project to update their Avro writers to stop nesting the legacy MAP_KEY_VALUE type when transforming an Avro schema to a Parquet one. It should've ideally been done as part of PARQUET-1410.
Unfortunately this is hard-coded behaviour and there are no configuration options that influence map-types that can aid in producing a correct file for Apache Arrow (and for Snowflake by extension). You'll need to use an older version of the writer until a proper fix is released by the Apache Parquet developers.
I'm working with IIB v9 mxsd message definitions. I'd like to define one of the XML elements to be of type xsd:anyType. However, in the list of types I can choose from, only anySimpleType and anyUri are possible (besides all other types like string, integer, etc.).
How can I get around this limitation?
The XMLNSC parser supports the entire XML Schema specification, including xs:any and xs:anyType. In IIBv9 you should create a Library and import your xsds into it. Link your Application to the Library and the XMLNSC parser will find and use the model. You do not need to specify the name of the Library in the node properties; the XSD model will be automatically available to the entire application.
You do not need to use a message set at all in IIBv9 and later versions.
The mxsd file format is used only by the MRM (not DFDL) parser.
You shouldn't use an MXSD to model your XML data, use a normal XSD.
MXSD is for modelling data for the DFDL parser, but you should use the XMLNSC parser for XML messages and define them in XSDs, in which you can use anyType.
As far as I know DFDL doesn't support anyType.
I am new to saxon and xslt, we have business in which feeder delivere more than one xml files, xslt generated from altov) create one output xml files , we have selected saxon as transformer.
so far i am able to transform single xml file
do any body have example where xslt takes more than one xmls as input stream, transform using saxon.
Thanks & regards,
Kumar
You haven't told us enough about the requirements, but there are several techniques to be aware of:
You can pass additional documents as stylesheet parameters declared using xsl:param
You can read a document (given its URI) using the doc() or document() functions
You can read a whole collection of documents (e.g. the contents of a directory) using the collection() function