Convert a large XML File (over 3G) to comma delimited file - saxon

Question

I need to convert a large XML file (over 3G) into a comma delimited file. I created an XSL file to convert it. Unfortunately, the file is too large to process using XSLT 1.0. I tried using XSLT 3.0 (Saxon), but I get the error "XTSE3430: Template rule is not streamable".
Script:
java -cp saxon9ee.jar net.sf.saxon.Transform -t -s:costing.xml -xsl:costing.xsl -o:costing.csv
Error Message:
Java version 1.8.0_191
Using license serial number
Stylesheet compilation time: 345.113654ms
Processing file:costing.xml
Streaming file:costing.xml
Using parser com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser
URIResolver.resolve href="" base="file:costing.xsl"
Using parser com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser
Building tree for file:costing.xsl using class net.sf.saxon.tree.tiny.TinyBuilder
Tree built in 5.206935ms
Tree size: 237 nodes, 104 characters, 25 attributes
Error on line 71 of costing.xsl:
XTSE3430: Template rule is not streamable
* Operand {($currNode/element())/element()} of {let $vv:v0 := ...} selects streamed
nodes in a context that allows arbitrary navigation (line 86)
Template rule is not streamable
* Operand {($currNode/element())/element()} of {let $vv:v0 := ...} selects streamed nodes in a context that allows arbitrary navigation (line 86)
XML Structure:
<?xml version="1.0" encoding="UTF-8"?>
<DATA_DS>
<COSTREPORT>
<DR>
<PSU>ABC</PSU>
<TRU>ABC</TRU>
<CA>0</CA>
<DA>0.00</DA>
<UOM>ABC</UOM>
<FN>0</FN>
<RID>0</RID>
<SD>2018-10-25</SD>
<DN>ABC</DN>
<ETD>2018-10-31</ETD>
<DID>0</DID>
<LN>ABC</LN>
<LID>0</LID>
<PN>ABC</PN>
<EN>Jane Doe</EN>
<EID>0</EID>
<ELN>ABC</ELN>
<ELV>ABC</ELV>
<RELA>1234</RELA>
<ETM>A0</ETM>
<ASG>A0</ASG>
<MN>ABC</MN>
<CRY>ABC</CRY
><IVN>ABC</IVN>
<AD>2018-10-31</AD>
<CID>0</CID>
<CCN>ABC</CCN
><BOC>ABC</BOC>
<SG1>0</SG1>
<SG2>0</SG2>
<SG3>0</SG3>
<SG4>0</SG4>
<SG5>0</SG5>
<SG9>0</SG9>
<SG10>0</SG10>
<TRUID>0</TRUID>
</DR>
<DR>
[...]
</DR>
[...]
</COSTREPORT>
</DATA_DS>
XSL file:
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:mode streamable="yes" />
<xsl:output method="text" />
<xsl:variable name="delimiter" select="','" />
<!-- define an array containing the fields we are interested in -->
<xsl:variable name="fieldArray">
<field>PSU</field> <!-- string -->
<field>TRU</field> <!-- string -->
<field>CA</field> <!-- number -->
<field>DA</field> <!-- number -->
<field>UOM</field> <!-- string -->
<field>FN</field> <!-- number -->
<field>RID</field> <!-- number -->
<field>SD</field> <!-- date -->
<field>DN</field> <!-- string -->
<field>ETD</field> <!-- date -->
<field>DID</field> <!-- number -->
<field>LN</field> <!-- string -->
<field>LID</field> <!-- number -->
<field>PN</field> <!-- string -->
<field>EN</field> <!-- string -->
<field>EID</field> <!-- number -->
<field>ELN</field> <!-- string -->
<field>ELV</field> <!-- string -->
<field>RELA</field> <!-- number -->
<field>ETM</field> <!-- string -->
<field>ASG</field> <!-- string -->
<field>MN</field> <!-- string -->
<field>CRY</field> <!-- string -->
<field>IVN</field> <!-- string -->
<field>AD</field> <!-- date -->
<field>CID</field> <!-- number -->
<field>CCN</field> <!-- string -->
<field>BOC</field> <!-- string -->
<field>SG1</field> <!-- number -->
<field>SG2</field> <!-- number -->
<field>SG3</field> <!-- number -->
<field>SG4</field> <!-- number -->
<field>SG5</field> <!-- number -->
<field>SG9</field> <!-- number -->
<field>SG10</field> <!-- number -->
<field>TRUID</field> <!-- number -->
</xsl:variable>
<xsl:param name="fields" select="document('')/*/xsl:variable[#name='fieldArray']/*" />
<!-- HEADER -->
<xsl:template match="/">
<!-- output the header row -->
<xsl:for-each select="$fields">
<xsl:if test="position() != 1">
<xsl:value-of select="$delimiter"/>
</xsl:if>
<xsl:value-of select="." />
</xsl:for-each>
<!-- output newline -->
<xsl:text>
</xsl:text>
<xsl:apply-templates select="DATA_DS/COSTREPORT/DR"/>
</xsl:template>
<!-- BODY -->
<xsl:template match="DR">
<xsl:variable name="currNode" select="." />
<!-- output the data row -->
<!-- loop over the field names and find the value of each one in the xml -->
<xsl:for-each select="$fields">
<xsl:if test="position() != 1">
<xsl:value-of select="$delimiter"/>
</xsl:if>
<xsl:value-of select="$currNode/*/*[name() = current()]" />
</xsl:for-each>
<!-- output newline -->
<xsl:text>
</xsl:text>
</xsl:template>
</xsl:stylesheet>

Answer 1

The problem is the variable:
<xsl:variable name="currNode" select="." />
This binds a variable to a streamed input node, which isn't allowed because there's no way Saxon can ensure that your selections from this input node are done "in the right order"; you select children/descendants of this node by name, and the streamability analysis can't establish that these descendants are selected in the order they appear in the input.
The answer is actually simple: change the variable to
<xsl:variable name="currNode" select="copy-of(.)" />
This way, every time you hit a DR element, Saxon will read the subtree rooted at that element and hold it as a tree in memory. Because the variable is now a regular in-memory node, rather than a streamed node, there are no restrictions on how it is used.
Allow me a couple of other comments on your code.
Firstly, the document('') construct that was popular in XSLT 1.0 is now thoroughly obsolete. It's much better to put your lookup data in a global variable and access it directly, using
<xsl:param name="fields" select="$fieldArray/*"/>
The document('') call will actually fail if you try to compile the stylesheet and execute it somewhere other than the original source code location.
Secondly, the code to output the header row:
<xsl:for-each select="$fields">
<xsl:if test="position() != 1">
<xsl:value-of select="$delimiter"/>
</xsl:if>
<xsl:value-of select="." />
</xsl:for-each>
can be simplified to
<xsl:value-of select="$fields" separator="{$delimiter}"/>
Similarly, the code for the data rows:
<xsl:for-each select="$fields">
<xsl:if test="position() != 1">
<xsl:value-of select="$delimiter"/>
</xsl:if>
<xsl:value-of select="$currNode/*/*[name() = current()]" />
</xsl:for-each>
simplifies to
<xsl:value-of select="for $f in $fields return $currNode/*/*[name()=$f]"
separator="{$delimiter}"/>