Optimized approach for Snowfake table search for string/list of strings from Java API - stored-procedures

I need to search for string or list of strings Snowflake table using Java API and fetch all the matching rows and display in Angular UI. I am using dynamic SQL (like operator) to generate the query using information schema. I have created this stored procedure and its working. Do we have any better approach or any architectural patterns for this particular usecase.
Based on the search option (starts/ends with) decide the start and end character to be used with Like operator
Get all the varchar,char columns from the table by joining with information schema.
Build dynamic sql with these columns.
Build the json array based on the query result.
CREATE OR REPLACE PROCEDURE SEARCH_DATA(SCHEMA_NAME VARCHAR, TABLE_NAME VARCHAR, SEARCH_OPTION VARCHAR, KEYWORDS ARRAY)
RETURNS VARIANT
LANGUAGE JAVASCRIPT
EXECUTE AS OWNER
AS '
var searchStart = "";
var searchEnd = "";
if (SEARCH_OPTION == "Starts with")
{
searchEnd = "%";
}
else if (SEARCH_OPTION == "Ends with")
{
searchStart = "%";
}
else if (SEARCH_OPTION == "Contains")
{
searchStart = "%";
searchEnd = "%";
}
// Dynamically compose the SQL statement to execute.
var sqlCommand = "select c.column_name from information_schema.COLUMNS c JOIN information_schema.TABLES T ON T.table_name = c.table_name AND T.table_schema = c.table_schema WHERE T.table_schema = ''";
sqlCommand+= SCHEMA_NAME;
sqlCommand+= "'' AND T.table_name= ''";
sqlCommand+= TABLE_NAME;
sqlCommand+= "'' AND c.data_type NOT IN (''TIMESTAMP_TZ'',''BOOLEAN'',''NUMBER'') ORDER BY ordinal_position";
// Prepare statement.
var stmt = snowflake.createStatement({ sqlText: sqlCommand });
// Execute Statement
var rs = stmt.execute();
var columnArray = [];
var columnName = "";
while (rs.next())
{
columnName = rs.getColumnValue(''COLUMN_NAME'');
columnArray.push(columnName);
}
var queryPrefix = "SELECT ''dummy''"
for(var i=0; i< columnArray.length; i++)
{
queryPrefix += "," + columnArray[i];
}
queryPrefix+= " FROM " + SCHEMA_NAME + "." + TABLE_NAME;
var query = "";
for(var j=0; j< KEYWORDS.length; j++)
{
query += queryPrefix;
query += " WHERE (1=0";
for(var i=0; i< columnArray.length; i++)
{
query += " OR "+ columnArray[i] + " LIKE ''" + searchStart + KEYWORDS[j] + searchEnd + "''";
}
query += ")";
if(j < KEYWORDS.length - 1)
query += " UNION ";
}
// Prepare statement.
stmt = snowflake.createStatement({ sqlText: query });
// Execute Statement
rs = stmt.execute();
var resultArray = [];
var row_as_json = {};
while (rs.next())
{
// Put each row in a variable of type JSON.
row_as_json = {};
// For each column in the row...
for (var i=0; i< columnArray.length; i++)
{
row_as_json[columnArray[i]] = rs.getColumnValue(columnArray[i]);
}
// Add the row to the array of rows.
resultArray.push(row_as_json);
}
return resultArray;
';

Related

Joining tables in queries google sheets

Basically I have two google sheets that look something like this:
a table where people can put in their email and select what kind of foo they are using
email
foo
example#email.com
This Foo
and then another table with information about the foo
foo name
foo type
foo boolean1
foo boolean2
This Foo
String
True
True
That Foo
Number
False
True
Other Foo
String
False
False
In a Separate Sheet I'd like to have a dashboard-like view of things wherein I would have counts of various things like number of people, how many of each type of Foo, etc
Where I'm having trouble is figuring out how to pull things like "Number of people who have selected String foos" and such
like, basically i want the google-query equivalent to (in sql)
SELECT COUNT(p.*) FROM people p JOIN info i on p.foo = i.foo_name GROUP BY i.foo_type WHERE i.foo_type = 'String'
What I would be looking for is a table that looks like this:
Data
Count
Active Roster
4
String
3
Number
1
I have also seen many solutions that have complicated formulas using VLOOKUP, INDEX, MATCH, etc.
I decided to write a user function to combine tables, or as I refer to it, de-normalize the database. I wrote the function DENORMALIZE() to support INNER, LEFT, RIGHT and FULL joins. By nesting function calls one can join unlimited tables in theory.
DENORMALIZE(range1, range2, primaryKey, foreignKey, [joinType])
Parameters:
range1, the main table as a named range, a1Notation or an array
range2, the related table as a named range, a1Notation or an array
primaryKey, the unique identifier for the main table, columns start with "1"
foreignKey, the key in the related table to join to the main table, columns start with "1"
joinType, type of join, "Inner", "Left", "Right", "Full", optional and defaults to "Inner", case insensitive
Returns: results as a two dimensional array
Result Set Example:
=QUERY(denormalize("Employees","Orders",1,3), "SELECT * WHERE Col2 = 'Davolio' AND Col8=2", FALSE)
EmpID
LastName
FirstName
OrderID
CustomerID
EmpID
OrderDate
ShipperID
1
Davolio
Nancy
10285
63
1
8/20/1996
2
1
Davolio
Nancy
10292
81
1
8/28/1996
2
1
Davolio
Nancy
10304
80
1
9/12/1996
2
Other Examples:
=denormalize("Employees","Orders",1,3)
=denormalize("Employees","Orders",1,3,"full")
=QUERY(denormalize("Employees","Orders",1,3,"left"), "SELECT * ", FALSE)
=QUERY(denormalize("Employees","Orders",1,3), "SELECT * WHERE Col2 = 'Davolio'", FALSE)
=QUERY(denormalize("Employees","Orders",1,3), "SELECT * WHERE Col2 = 'Davolio' AND Col8=2", FALSE)
=denormalize("Orders","OrderDetails",1,2)
// multiple joins
=denormalize("Employees",denormalize("Orders","OrderDetails",1,2),1,3)
=QUERY(denormalize("Employees",denormalize("Orders","OrderDetails",1,2),1,3), "SELECT *", FALSE)
=denormalize(denormalize("Employees","Orders",1,3),"OrderDetails",1,2)
=QUERY(denormalize("Employees",denormalize("Orders","OrderDetails",1,2),1,3), "SELECT *", FALSE)
=QUERY(denormalize(denormalize("Employees","Orders",1,3),"OrderDetails",4,2), "SELECT *", FALSE)
function denormalize(range1, range2, primaryKey, foreignKey, joinType) {
var i = 0;
var j = 0;
var index = -1;
var lFound = false;
var aDenorm = [];
var hashtable = [];
var aRange1 = "";
var aRange2 = "";
joinType = DefaultTo(joinType, "INNER").toUpperCase();
// the 6 lines below are used for debugging
//range1 = "Employees";
//range1 = "Employees!A2:C12";
//range2 = "Orders";
//primaryKey = 1;
//foreignKey = 3;
//joinType = "LEFT";
// Sheets starts numbering columns starting with "1", arrays are zero-based
primaryKey -= 1;
foreignKey -= 1;
// check if range is not an array
if (typeof range1 !== 'object') {
// Determine if range is a1Notation and load data into an array
if (range1.indexOf(":") !== -1) {
aRange1 = ss.getRange(range1).getValues();
} else {
aRange1 = ss.getRangeByName(range1).getValues();
}
} else {
aRange1 = range1;
}
if (typeof range2 !== 'object') {
if (range2.indexOf(":") !== -1) {
aRange2 = ss.getRange(range2).getValues();
} else {
aRange2 = ss.getRangeByName(range2).getValues();
}
} else {
aRange2 = range2;
}
// make similar structured temp arrays with NULL elements
var tArray1 = MakeArray(aRange1[0].length);
var tArray2 = MakeArray(aRange2[0].length);
var lenRange1 = aRange1.length;
var lenRange2 = aRange2.length;
hashtable = getHT(aRange1, lenRange1, primaryKey);
for(i = 0; i < lenRange2; i++) {
index = hashtable.indexOf(aRange2[i][foreignKey]);
if (index !== -1) {
aDenorm.push(aRange1[index].concat(aRange2[i]));
}
}
// add left and full no matches
if (joinType == "LEFT" || joinType == "FULL") {
for(i = 0; i < lenRange1; i++) {
//index = aDenorm.indexOf(aRange1[i][primaryKey]);
index = aScan(aDenorm, aRange1[i][primaryKey], primaryKey)
if (index == -1) {
aDenorm.push(aRange1[i].concat(tArray2));
}
}
}
// add right and full no matches
if (joinType == "RIGHT" || joinType == "FULL") {
for(i = 0; i < lenRange2; i++) {
index = aScan(aDenorm, aRange2[i][foreignKey], primaryKey)
if (index == -1) {
aDenorm.push(tArray1.concat(aRange2[i]));
}
}
}
return aDenorm;
}
function getHT(aRange, lenRange, key){
var aHashtable = [];
var i = 0;
for (i=0; i < lenRange; i++ ) {
//aHashtable.push([aRange[i][key], i]);
aHashtable.push(aRange[i][key]);
}
return aHashtable;
}
function MakeArray(length) {
var i = 0;
var retArray = [];
for (i=0; i < length; i++) {
retArray.push("");
}
return retArray;
}
function DefaultTo(valueToCheck, valueToDefault) {
return typeof valueToCheck === "undefined" ? valueToDefault : valueToCheck;
}
// Search a multi-dimensional array for a value
function aScan(aValues, searchStr, searchCol) {
var retval = -1;
var i = 0;
var aLen = aValues.length;
for (i = 0; i < aLen; i++) {
if (aValues[i][searchCol] == searchStr) {
retval = i;
break;
}
}
return retval;
}
You can make a copy of the google sheet with data and examples here:
https://docs.google.com/spreadsheets/d/1vziuF8gQcsOxTLEtlcU2cgTAYL1eIaaMTAoIrAS7mnE/edit?usp=sharing

Google Sheets: Exporting to TSV

I would like to create a function that automatically exports the current sheet to a TSV file on Google Sheets. Is it possible to export to a tab separated values file using Google Apps Script, i.e. like convertRangeToCsvFile for CSV?
I found this code for csv files by Michael Derazon and modified it for tsv files. I hope it helps.
/*
* script to export data in all sheets in the current spreadsheet as individual csv files
* files will be named according to the name of the sheet
* author: Michael Derazon
*/
function onOpen() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var tsvMenuEntries = [{name: "export as TSV files", functionName: "saveAstsv"}];
ss.addMenu("tsv", tsvMenuEntries);
};
function saveAstsv() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheets = ss.getSheets();
// create a folder from the name of the spreadsheet
var folder = DriveApp.createFolder(ss.getName().toLowerCase().replace(/ /g,'_') + '_tsv_' + new Date().getTime());
for (var i = 0 ; i < sheets.length ; i++) {
var sheet = sheets[i];
// append ".tsv" extension to the sheet name
fileName = sheet.getName() + ".tsv";
// convert all available sheet data to tsv format
var tsvFile = convertRangeTotsvFile_(fileName, sheet);
// create a file in the Docs List with the given name and the tsv data
folder.createFile(fileName, tsvFile);
}
Browser.msgBox('Files are waiting in a folder named ' + folder.getName());
}
function convertRangeTotsvFile_(tsvFileName, sheet) {
// get available data range in the spreadsheet
var activeRange = sheet.getDataRange();
try {
var data = activeRange.getValues();
var tsvFile = undefined;
// loop through the data in the range and build a string with the tsv data
if (data.length > 1) {
var tsv = "";
for (var row = 0; row < data.length; row++) {
for (var col = 0; col < data[row].length; col++) {
if (data[row][col].toString().indexOf("\t") != -1) {
data[row][col] = "\"" + data[row][col] + "\"";
}
}
// join each row's columns
// add a carriage return to end of each row, except for the last one
if (row < data.length-1) {
tsv += data[row].join("\t") + "\r\n";
}
else {
tsv += data[row].join("\t");
}
}
tsvFile = tsv;
}
return tsvFile;
}
catch(err) {
Logger.log(err);
Browser.msgBox(err);
}
}

Google spreadsheet: Download all the sheets at once

When working with google spreadsheet, how to download all the sheets at once?
I want to use the option:
Comma-separated values
But it only download the current sheet, how to get them all?
For anyone who navigates to this question, trying to download all the tabs in their Google spreadsheets as CSV files at once, even in 2021, there does not seem to be a GUI button to do this. At least I could not see anything. The answer by #Amit Agarwal does well, to get all sheets, but if your file has comma-delimited data in cells, then data could get mangled.
I took Amit's approach https://stackoverflow.com/a/28711961 and combined it with Michael Derazon and Aaron Davis's approach here https://gist.github.com/mrkrndvs/a2c8ff518b16e9188338cb809e06ccf1 to dump all the tabs of a chosen Google spreadsheet into a folder in Google Drive. You can then just download the folder with a single click.
The following is Google script, not exactly a Javascript, and you would have to copy-paste this in https://script.google.com/ login with your Google id, and then create a project and then create a script app, and save and execute this.
// https://stackoverflow.com/a/28711961
function export_sheets_as_csv_to_folder() {
// Sheet id is in URL https://docs.google.com/spreadsheets/d/YOUR_SHEET_ID/edit#gid=IGNORE
var ss = SpreadsheetApp.openById('YOUR_SHEET_ID');
var sheets = ss.getSheets();
if (sheets === undefined || sheets.length === 0) {
return;
}
var passThroughFolder = DriveApp.createFolder('YOUR_PREFERRED_FOLDER_NAME_IN_DRIVE');
for (var s in sheets) {
var csv = convertRangeToCsvFile_(sheets[s])
passThroughFolder.createFile(sheets[s].getName() + ".csv", csv);
}
}
// https://gist.github.com/mrkrndvs/a2c8ff518b16e9188338cb809e06ccf1
function convertRangeToCsvFile_(sheet) {
// get available data range in the spreadsheet
var activeRange = sheet.getDataRange();
try {
var data = activeRange.getValues();
var csvFile = undefined;
// loop through the data in the range and build a string with the csv data
if (data.length > 1) {
var csv = "";
for (var row = 0; row < data.length; row++) {
for (var col = 0; col < data[row].length; col++) {
if (data[row][col].toString().indexOf(",") != -1) {
data[row][col] = "\"" + data[row][col] + "\"";
}
}
// join each row's columns
// add a carriage return to end of each row, except for the last one
if (row < data.length-1) {
csv += data[row].join(",") + "\r\n";
}
else {
csv += data[row];
}
}
csvFile = csv;
}
return csvFile;
}
catch(err) {
Logger.log(err);
Browser.msgBox(err);
}
}
After clicking download > pdf, select export > worksheet (instead of current sheet which is the default)
You can use Google Scripts to save all the sheets of a spreadsheet into separate files.
function myFunction() {
var ss = SpreadsheetApp.openById(SHEET_ID);
var sheets = ss.getSheets();
for (var s in sheets) {
var csv = "";
var data = sheets[s].getDataRange().getValues();
for (d in data) {
csv += data[d].join(",") + "\n";
}
DriveApp.createFile(sheets[s].getName() + ".csv", csv);
}
}
the answer from #Soham works amazingly but it doesn't handle multiline values. It would be an easy fix just to add more checks to character \n along with , but I took the liberty to rewrite the function using map (and string.includes) so it is more concise.
function convertRangeToCsvFile_(sheet) {
return sheet.getDataRange().getValues()
.map(row => row.map(value => value.toString())
.map(value => (value.includes("\n") || value.includes(",")) ? "\"" + value + "\"" : value)
.join(','))
.join('\n')
}
A slight variation on this that uses a zip instead of a folder to contain the sheets and does some modernizing of the great work done by keychera and Soham's answer.
You can use this as a bound script and it will add a menu item to the extensions menu:
// Code.gs
function exportSheetsToDrive() {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const sheets = ss.getSheets();
if (sheets === undefined || sheets.length === 0) {
return;
}
const now = new Date();
const csvBlobs = sheets.map((sheet) => {
const name = sheet.getName();
const csv = convertSheetToCsv(sheet);
Logger.log({ name, length: csv.length });
return Utilities.newBlob(csv, MimeType.CSV, `${name}.csv`)
});
const zipName = `export_${ss.getName()}_${now.toISOString()}.zip`;
const zip = Utilities.zip(csvBlobs, zipName);
DriveApp.createFile(zip);
}
function convertSheetToCsv(sheet) {
return sheet
.getDataRange()
.getValues()
.map((row) =>
row
.map((value) => value.toString())
.map((value) =>
value.includes("\n") || value.includes(",")
? '"' + value + '"'
: value
)
.join(",")
)
.join("\n");
}
and
// Menu.gs
function onOpen(e) {
const menu = SpreadsheetApp.getUi().createAddonMenu();
menu
.addItem('Export all sheets as CSV to Drive', 'exportSheetsToDrive')
.addToUi();
}
function onInstall(e) {
onOpen(e);
}

Open infowindow with sidebar on Google Fusion Table

For the life of me, I can't get this map to open infowindows on the sidebar link clicks: http://web.redding.com/static/redd/asphalt/prod/xmas-lights-2014-complex.html
Here's the fusion table: https://www.google.com/fusiontables/DataSource?docid=1WrvKdTypAmZozAIVeOw4vBX2g1hPInyVyuqn8GUM
Which looks like this (CSV):
Location,Description,Photo,Winner,Name
"1101 Twin View Boulevard, Redding CA",Redding's finest media organization with decades of experience & class.,http://mediaassets.redding.com/photo/2014/03/15/youthcamp17b-01_3471502_ver1.0.jpg,,The Record Searchlight
"1500 Court Street, Redding CA",Shasta Courthouse,,,
"777 Cypress Avenue, Redding CA",City Hall,,,
All I want to do is be able to click on the links in the sidebar and have the associated infowindow open on the map.
I'm new to javascript so I wouldn't be surprised if there's something obvious I'm overlooking.
Code from the linked page:
function createSidebar() {
//set the query using the parameter
var query = new google.visualization.Query(queryText);
var queryText = encodeURIComponent("SELECT 'Name','Description' FROM 1uQLxgNdNR_etBFP8O_0YNDA38PqyZB3NidIJfsgX");
var query = new google.visualization.Query('http://www.google.com/fusiontables/gvizdata?tq=' + queryText);
//set the callback function
query.send(getData);
}
function myFTclick(row) {
var Name = FTresponse.getDataTable().getValue(row,0);
var Description = FTresponse.getDataTable().getValue(row,1);
var Location = FTresponse.getDataTable().getValue(row,2);
var Photo = FTresponse.getDataTable().getValue(row,5);
var Winner = FTresponse.getDataTable().getValue(row,7);
var position = new google.maps.LatLng(lat, lng);
// Set up and create the infowindow
if (!infoWindow) infoWindow = new google.maps.InfoWindow({});
var content = '<div class="FT_infowindow">' + name;
if (Description) content += '<br>'+Description;
if (Location) content += '<br>'+Location;
if (Photo) content += '<br>'+Photo;
if (extraContent) content += "<br>["+extraContent+"]";
content += '<br>'+'zoom in';
content += '</div>';
infoWindow.setOptions({
content: content,
pixelOffset: null,
position: position
});
// Infowindow-opening event handler
infoWindow.open(map);
}
var FTresponse = null;
//define callback function, this is called when the results are returned
function getData(response) {
if (!response) {
alert('no response');
return;
}
if (response.isError()) {
alert('Error in query: ' + response.getMessage() + ' ' + response.getDetailedMessage());
return;
}
FTresponse = response;
//for more information on the response object, see the documentation
//http://code.google.com/apis/visualization/documentation/reference.html#QueryResponse
numRows = response.getDataTable().getNumberOfRows();
numCols = response.getDataTable().getNumberOfColumns();
//concatenate the results into a string, you can build a table here
fusiontabledata = "<table><tr>";
fusiontabledata += "<th>" + response.getDataTable().getColumnLabel(1) + "</th>";
fusiontabledata += "</tr><tr>";
for(i = 0; i < numRows; i++) {
fusiontabledata += "<td><a href='javascript:myFTclick("+i+")'>"+response.getDataTable().getValue(i, 1) + "</a></td>";
fusiontabledata += "</tr><tr>";
}
fusiontabledata += "</table>"
//display the results on the page
document.getElementById('sidebar').innerHTML = fusiontabledata;
}
You have a javascript error in your code, look at the javascript console: Uncaught Error: Invalid column index 2. Should be an integer in the range [0-1].
Your query only includes two columns from the table:
var queryText = encodeURIComponent("SELECT 'Name','Description' FROM 1uQLxgNdNR_etBFP8O_0YNDA38PqyZB3NidIJfsgX");
Which means you can't get any columns beyond 0 and 1, so this will not work:
function myFTclick(row) {
var Name = FTresponse.getDataTable().getValue(row,0);
var Description = FTresponse.getDataTable().getValue(row,1);
var Location = FTresponse.getDataTable().getValue(row,2);
var Photo = FTresponse.getDataTable().getValue(row,5);
var Winner = FTresponse.getDataTable().getValue(row,7);
You need to include those in your query:
var queryText = encodeURIComponent("SELECT 'Name','Description','Location','Photo','Winner' FROM 1uQLxgNdNR_etBFP8O_0YNDA38PqyZB3NidIJfsgX");
Then access them in that order:
function myFTclick(row) {
var Name = FTresponse.getDataTable().getValue(row,0);
var Description = FTresponse.getDataTable().getValue(row,1);
var Location = FTresponse.getDataTable().getValue(row,2);
var Photo = FTresponse.getDataTable().getValue(row,3);
var Winner = FTresponse.getDataTable().getValue(row,4);

Get resultSet keys in phonegap?

With say 5 fields in the DB, I know the columns that can be queried and use:
function getDetails_success (tx, results) {
var len = results.rows.length;
for (var i=0; i<len; i++) {
var content = results.rows.item(i);
buf += '<tr '+ content.key1+'>';
buf += '<tr '+ content.key2+'>';
}
}
and so on.
What if I have 50 fields, of which 5 random fields has to be displayed. Do I get the keys from the resultset? What are the various ways I can approach this?
If 5 random fields are selected by the user to be displayed, put all of them in an array.
var randomFieldsSelected = new Array();
randomFieldsSelected.push(selection1);
randomFieldsSelected.push(selection2); //and so on
Instead of the above for loop, put,
for (var i=0; i<len; i++) {
var content = results.rows.item(i);
buf += '<tr '+ content[randomFieldsSelected[i]]+'>';
}
(The above works,provided, the database column names match with 'selection1' and 'selection2' etc)

Resources