Transform portion of ETL using Scriptella? - scriptella

Question

I am trying out Scriptella to see if it will meet my needs. So far, it seems like a great tool. I've spent several hours studying sample scripts, searching forums, and trying to get the hang of nested queries/scripts.
This is an example of my ETL file, slightly cleaned up for brevity. Lines beginning with # are added and not part of the actual ETL file. I am trying to insert/retrieve IDs and then pass them on to later script blocks. The most promising way to do this appears to be using global variables but I'm getting null when trying to retrieve the values. Later, I will be adding code in the scripts blocks that parse and significantly transform fields before adding them into the DB.
There are no errors. I'm just not getting the OS ID and Category IDs that I'd expect. Thank you in advance.
<!DOCTYPE etl SYSTEM "http://scriptella.javaforge.com/dtd/etl.dtd">
<etl>
<connection id="in" driver="csv" url="mycsvfile.csv"/>
<connection id="dest" url="jdbc:mysql://localhost:3306/pvm3" user="user" password="password"/>
<connection id="js" driver="script"/>
<query connection-id="in">
<!-- all columns are selected, notably: OPERATINGSYSTEM, CATEGORY, QID, TITLE -->
<query connection-id="dest">
#Check to see if the OS already exists, and get the ID if it does
select max(os_id) as os_id, count(*) as os_cnt from etl_os where os = ?OPERATINGSYSTEM;
#If it doesnt exist then add it and get the auto_increment value
<script if="os_cnt==0">
insert into etl_os(os) values(?OPERATINGSYSTEM);
<query connection-id="dest">
select last_insert_id() as os_id;
#Store in global so it can be accessed in later script blocks
<script connection-id="js">
etl.globals.put('os_id', os_id);
</script>
</query>
</script>
#Same style select/insert as above for category_id (excluded for brevity)
#See if KB record exists by qid, if not then add it with the OS ID and category ID we got earlier
<query connection-id="dest">
select max(qid) as existing_qid, count(*) as kb_cnt from etl_qids where qid = ?QID
<script if="kb_cnt==0">
insert into etl_qids(qid, category_id, os_id) values (?QID, ?{etl.globals.get('category_id')}, ?{etl.globals.get('os_id')});
</script>
</query>
</query>
</query>
</etl>

Answer 1

Found out how to do it. Essentially, just nest queries to modify the data before passing it to a script. The below is a quick type-up of the solution. I did not understand at first that queries could be immediately nested to transform the row before passing it for processing. My impression was also that only scripts could manipulate the data.
(Query)Raw data -> (Query)manipulate data -> (Script)write new data.
.. in is a CSV file ..
.. js is a driver="script" block ..
<query connection-id="in">
<query connection-id="js">
//transform data as needed here
if (BASE_TYPE == '-') BASE_TYPE = '0';
if (SECONDARY_TYPE == '-') SECONDARY_TYPE = '0';
SIZES = SIZES.toLowerCase();
query.next(); //call nested scripts
<script connection-id="db">
INSERT IGNORE INTO sizes(size) VALUES (?SIZE);
INSERT IGNORE INTO test(base_type,secondary_type) VALUES (?BASE_TYPE, ?SECONDARY_TYPE);
</script>
</query>
</query>