Consider an enterprise that captures sensor data for different production facilities. per facility, we create an aggregation query that averages the values to 5min timeslots. This query exists out of a long list of with-clauses and writes data to a table (called aggregation_table).
Now my problem: currently we have n queries running that exactly run the same logic, the only thing that differs are table names (and sometimes column names but let's ignore that for now).
Instead of managing n different scripts that are basically the same, I would like to put it in a stored procedure that is able to work like this:
CALL aggregation_query(facility_name) -> resolve the different tables for that facility and then use them in the different with clauses
On top of that, instead of having this long set of clauses that give me the end-result, I would like to chunk them up in logical blocks that are parametrizable, So for example, if I call the aforementioned stored_procedure for facility A, I want to be able to pass / use this table name in these different functions, where the output can be re-used in the next statement (like you would do with with clauses).
Another argument of why I want to chunk this up in re-usable blocks is because we have many "derivatives" on this aggregation query, for example to manage historical data, to correct data or to have the sensor data on another aggregation level. As these become overly complex, it is much easier to manage them without having to copy paste and adjust these every time.
In the current set-up, it could be useful to know that I am only entitled to use plain BigQuery, As my team is not allowed to access the CI/CD / scheduling and repository. (meaning that I cannot solve the issue by having CI/CD that deploys the n different versions of the procedure and functions)
So in the end, I would like to end up with something like this using only bigquery:
CREATE OR REPLACE PROCEDURE
`aggregation_function`()
BEGIN
DECLARE
tablename STRING;
DECLARE
active_table_name STRING; ##get list OF tables CREATE TEMP TABLE tableNames AS
SELECT
table_catalog,
table_schema,
table_name
FROM
`catalog.schema.INFORMATION_SCHEMA.TABLES`
WHERE
table_name = tablename;
WHILE
(
SELECT
COUNT(*)
FROM
tableNames) >= 1 DO ##build dataset + TABLE name
SET
active_table_name = CONCAT('`',table_catalog,'.',table_schema,'.' ,table_name,'`'); ##use concat TO build string AND execute
EXECUTE IMMEDIATE '''
INSERT INTO
`aggregation_table_for_facility` (timeslot, sensor_name, AVG_VALUE )
WITH
STEP_1 AS (
SELECT
*
FROM
my_table_function_step_1(active_table_name,
parameter1,
parameter2) ),
STEP_2 AS (
SELECT
*
FROM
my_table_function_step_2(STEP_1,
parameter1,
parameter2) )
SELECT * FROM STEP_2
'''
USING active_table_name as active_table_name;
DELETE
FROM
tableNames
WHERE
table_name = tablename;
END WHILE
;
END
;
I was hoping someone could make a snippet on how I can do this in Standard SQL / Bigquery, so basically:
stored procedure that takes in a string variable and is able to use that as a table (partly solved in the approach above, but not sure if there are better ways)
(table) function that is able to take this table_name parameter as well and return back a table that can be used in the next with clause (or alternatively writes to a temp table)
I think below code snippets should provide you with some insights when dealing with procedures, inserts and execute immediate statements.
Here I'm creating a procedure which will insert values into a table that exists on the information schema. Also, as a value I want to return I use OUT active_table_name to return the value I assigned inside the procedure.
CREATE OR REPLACE PROCEDURE `project-id.dataset`.custom_function(tablename STRING,OUT active_table_name STRING)
BEGIN
DECLARE query STRING;
SET active_table_name= (SELECT CONCAT('`',table_catalog,'.',table_schema,'.' ,table_name,'`')
FROM `project-id.dataset.INFORMATION_SCHEMA.TABLES`
WHERE table_name = tablename);
#multine query can be handled by using ''' or """
Set query =
'''
insert into %s (string_field_0,string_field_1,string_field_2,string_field_3,string_field_4,int64_field_5)
with custom_query as (
select string_field_0,string_field_2,'169 BestCity',string_field_3,string_field_4,55677 from %s limit 1
)
select * from custom_query;
''';
# querys must perform operations and must be the last thing to perform
# pass parameters using format
execute immediate (format(query,active_table_name,active_table_name));
END
You can also use a loop to iterate trough records from a working table so it will execute the procedure and also be able to get the value from the procedure to use somewhere else.ie:A second procedure to perform a delete operation.
DECLARE tablename STRING;
DECLARE out_value STRING;
FOR record IN
(SELECT tablename from `my-project-id.dataset.table`)
DO
SET tablename = record.tablename;
LOOP
call `project-id.dataset`.custom_function(tablename,out_value);
select out_value;
END LOOP;
END FOR;
To recap, there are some restrictions such as the possibility to call procedures inside a execute immediate or to use execute immediate inside an execute immediate, to count a few. I think these snippets should help you dealing with your current situation.
For this sample I use the following documentation:
Data Manipulation Language
Dealing with outputs
Information Schema Tables
Execute Immediate
For...In
Loops
I need to write a procedure in Redshift that will write to a table, but the table name comes from the input string. Then I declare a variable that puts together the table name.
CREATE OR REPLACE PROCEDURE my_schema.data_test(current "varchar")
LANGUAGE plpgsql
AS $$
declare new_table varchar(50) = 'new_tab' || '_' || current;
BEGIN
select 'somestring' as colname into new_table;
commit;
END;
$$
This code runs but it doesn't create a new table, no errors. If I remove the declare statement then it works, creating a table called "new_table". It's just not using the declared variable name.
It's hard to find good examples because Redshift is postgresql and all the postgresql pages say that it only has functions, not procedures. But Redshift procedures were introduced last year and I don't see many examples.
Well, when you are declaring a variable "new_table", and performing a SELECT ..INTO "new_table", the value is getting assigned to the variable "new_table". You will see that if you return your variable using a OUT parameter.
And when you remove the declaration, it simply work as a SELECT INTO syntax of Redshift SQL and creates a table.
Now to the solution:
Create a table using the CREATE TABLE AS...syntax.
Also you need to pass the value of declared variable, so use the EXECUTE command.
CREATE OR REPLACE PROCEDURE public.ct_tab (vname varchar)
AS $$
DECLARE tname VARCHAR(50):='public.swap_'||vname;
BEGIN
execute 'create table ' || tname || ' as select ''name''';
END;
$$ LANGUAGE plpgsql
;
Now if you call the procedure passing 'abc', a table named "swap_abc" will be created in public schema.
call public.ct_tab('abc');
Let me know if it helps :)
I am trying to write a stored procedure in AWS Redshift SQL and one of my parameters needs the possibility to have an integer list (will be using 'IN(0,100,200,...)' inside there WHERE clause). How would I write the input parameter in the header of the procedure so that this is possible (if at all?)
I've tried passing them in as a VARCHAR "integer list" type thing but wasn't sure then how to parse that back into ints.
Update: I found a way to parse the string and loop through it using the SPLIT_PART function and store all of those into a table. Then just use a SELECT * FROM table with the IN() call
What I ended up doing was as follows. I took in the integers that I was expecting as a comma-separated string. I then ran the following on it.
CREATE OR REPLACE PROCEDURE test_string_to_int(VARCHAR)
AS $$
DECLARE
split_me ALIAS FOR $1;
loop_var INT;
BEGIN
DROP TABLE IF EXISTS int_list;
CREATE TEMPORARY TABLE int_list (
integer_to_store INT
);
FOR loop_var IN 1..(REGEXP_COUNT(split_me,',') + 1) LOOP
INSERT INTO int_list VALUES (CAST(SPLIT_PART(split_me,',',loop_var) AS INT));
END LOOP;
END;
$$ LANGUAGE plpgsql;
So I would call the procedure with something like:
CALL test_string_to_int('1,2,3');
and could do a select statement on it to see all the values stored into the table. Then in my queries the need this parameter I ran:
.........................
WHERE num_items IN(SELECT integer_to_store FROM int_list);
Hi I'm a junior developer, I just want to ask Is it possible to add a new column in a stored procedure in db2? what i mean like an alter table for adding a new column but in stored procedure?
Yes, it's possible but you have to use dynamic sql.
--# SET TERMINATOR #
create table test_add_col(a int) in userspace1#
begin
execute immediate 'alter table test_add_col add b int';
end#
select colname
from syscat.columns
where tabschema=user and tabname='TEST_ADD_COL'#
The result is:
COLNAME
--
A
B
I'm trying to define a DB2 Stored Proc that would (ideally), CREATE VIEW, then do a SELECT against that VIEW to build another piece of SQL, then execute that SQL using a CURSOR and return a result set.
I've 2 problems:
DB2 doesn't appear to like the mix of CREATE, SELECT and DECLARE CURSOR within a single SP,
and I can't figure out what syntax to use to declare a cursor based on SQL that is stored as a string in the declared VARCHAR that is the output from the SELECT statement.
Has anyone done anything similar and/or able to give me some syntax examples?
Sure you can do that in DB2. In order to execute a 'create view' you need to use a dynamic SQL. The same for the select on the view. Finally, for the cursor, you define a generic cursor, and execute it dynamically.
DECLARE SENTENCE VARCHAR(256);
DECLARE TABNAME VARCHAR(128);
DECLARE STMT STATEMENT;
DECLARE TABLES_CURSOR CURSOR
FOR TABLES_RS;
SET SENTENCE = 'CREATE VIEW TABS SELECT TABNAME FROM SYSCAT.TABLES';
PREPARE STMT FROM SENTENCE;
EXECUTE STMT;
SET SENTENCE = 'SELECT TABNAME FROM TABS';
PREPARE TABLES_RS FROM SENTENCE;
OPEN TABLES_CURSOR;
FETCH TABLES_CURSOR INTO TABNAME;
You can see real SQL PL examples in my project db2unit https://github.com/angoca/db2unit/blob/master/src/main/sql-pl/04-Body.sql