Morning Guys,
This is a follow on from this question. mysqld.exe keeps crashing.
I have narrowed the crashing down to a single stored procedure which just seems to fall over when this stored procedure gets executed.
I have split it down from its original state to three stored procedures.
The main procedure spProductGroupMenu is as follows:
DELIMITER $$
USE `phclothing`$$
DROP PROCEDURE IF EXISTS `spProductGroupMenu`$$
CREATE DEFINER=`root`#`localhost` PROCEDURE `spProductGroupMenu`(
spGrp VARCHAR(3),
spProductType ENUM('clothing','parts')
)
BEGIN
DECLARE qryType ENUM('group','manufacturer','nothing');
IF spProductType='clothing' THEN
SET qryType = (SELECT
(CASE WHEN
(SELECT
COUNT(DISTINCT(productGroup))
FROM
tblclothingitems
WHERE
productGroup=spGrp)>0
THEN 'group' ELSE
(CASE WHEN
(SELECT
COUNT(DISTINCT(productManufacturer))
FROM
tblclothingitems
WHERE
productManufacturer=spGrp)>0
THEN 'manufacturer' ELSE
'nothing' END) END));
ELSE
SET qryType = (SELECT
(CASE WHEN
(SELECT
COUNT(DISTINCT(productGroup))
FROM
tblpartsitems
WHERE
productGroup=spGrp)>0
THEN 'group' ELSE
(CASE WHEN
(SELECT
COUNT(DISTINCT(productManufacturer))
FROM
tblpartsitems
WHERE
productManufacturer=spGrp)>0
THEN 'manufacturer' ELSE
'nothing' END) END));
END IF;
IF qryType='manufacturer' THEN
CALL spPGMMans(spGrp,spProductType,qryType); ************
ELSEIF qryType='group' THEN
CALL spPGMGrp(spGrp,spProductType,qryType); ************
ELSE
-- This is not going to be used
SELECT
DISTINCT(productgroup) grpCode,
(CASE WHEN spProductType='clothing' THEN
CONCAT('/',grpSEO,'.html')
ELSE
CONCAT(spProductType,'/',grpSEO,'.html') END)seoLink,
pageH1 seoLinkTitle,
'allGroups' entryType,
qryType
FROM
tblclothingitems ci
LEFT JOIN
tblclothinggroups cg
ON
ci.productGroup = cg.grpcode
UNION ALL
SELECT
DISTINCT(productManufacturer) grpCode,
(CASE WHEN spProductType='clothing' THEN
cm.urlExt
ELSE
CONCAT(spProductType,'/',cm.urlExt) END)seoLink,
manTitle seoLinkTitle,
'allMans' entryType,
qryType
FROM
tblclothingitems ci
LEFT JOIN
tblclothingmanufacturers cm
ON
ci.productManufacturer = cm.manCode
ORDER BY entryType, seoLinkTitle ASC;
END IF;
END$$
DELIMITER ;
This used to consist of a series of queries with union all statements depending on what was passed in but this raised the alarm about the mysql server falling over. So where the ************ are is where i have divided the stored procedure into three. Each of these stored procedures runs on the server in its own right but for some reason when this stored procedure gets called the server just falls over and stops running.
I have been checking the logs throughout and it is when this stored procedure is run that the connection to the server goes.
If anybody has any clues as to why this may be happening or if anybody has had experience of this situation then please let me know.
Many thanks in advance.
Graham
Ok well this clearly very strange. Following a hunch I clicked to the end of every line in the stored procedure and removed all white space just in case there were any erroneous non printable characters that were causing an issue, low and behold it seemed to fix it.
To test the theory I then reversed the stored procedure again back to the point when it was causing the server to crash. I ran it on the server and it worked as well, which is clearly very odd.
Needless to say I am completely dumb-founded as to why this error occurred in the first place but can only presume that it was a non-printing character at the end of a line in the stored procedure.
Cheers if you took a look!
Graham
Related
I have a select that return a single column, using as where clause 4 columns, and I created a specific index for this query. When I do this select using Dbeaver, the result return in 30~50 milliseconds.
SELECT
column1
FROM
myTable
WHERE
column2 = a
AND column3 = b
AND (column4 = c AND column5 = d )
FOR READ ONLY WITH UR;
Now I created a procedure with this same simple select. The proc declare a P1, a 'cursor with return', do the select, open the cursor and close the P1. When I use the same dbeaver connection the result is returning between 1.2 ~ 1.6 seconds.
CREATE PROCEDURE myProc (
IN a DECIMAL(10),
IN b DECIMAL(10),
IN c DECIMAL(10),
IN d DECIMAL(10)
)
LANGUAGE SQL
DYNAMIC RESULT SETS 1
SPECIFIC myProc
P1: BEGIN
DECLARE cursor1 CURSOR WITH RETURN for
SELECT
column1
FROM
myTable
WHERE
column2 = a
AND column3 = b
AND (column4 = c AND column5 = d )
FOR READ ONLY WITH UR;
OPEN cursor1;
END P1
Is this huge return difference correct? If wrong, Is there something wrong in my procedure that justifies this return time difference? Or could be something wrong in the DB configuration or the server that justifies this difference, something like few resources in the DB server or something like the proc not using the index( I don't know if procs in the DB2 use the index by default like the queries )?
I am new in DB2 and new in procedures creation. Thank you in advance.
Best regards,
Luis
I don't know if is the best way, but I solved the problem using a solution that I read for SQL Server. The solution is create local variables that will receive the parameters values, and use the variable in the queries, to guarantee always the best execution plan. I didn't knew if this was valid to DB2, but my proc now has almost the same response time compared to query. Worked!
Link of SQL Server post: SQL Server: Query fast, but slow from procedure
In this link above a user called #Jake give this explanation:
"The reason this happens is because the procedures query plan is being cached, along with the parameters that were passed to it. On subsequent calls, this query plan generated will be reused with new parameters. This can cause problems because if the data is unevenly distributed, one parameter can generate a sub-optimal plan vs. another. Using local variables essentially does the same as OPTIMIZE FOR UNKNOWN because local variables cannot be sniffed."
I think that is the same for DB2, because worked. After I change these old procedures to use local variables my execution plan begun to use the indexes recently created
I will try to keep the query as short as possible. This involves 2 tables - lets call them staging_data and audit_data. STAGING_DATA has 3 columns:
user_no with data type number,
update_date_time with data type as date in DD-MON-YYYY HH24:MI:SS format
status_code which is varchar(1).
audit_data table also has the same 3 columns. The ask is to add 3 columns to audit_data table
seq_no (which will be unique to every user),
active_from (date type without the time format)
active_to (date type without the time format).
There is a procedure that inserts data from staging_data to audit_data.
Sample of the table audit_data
That data in audit table should look like :
For the next record for user_no 523(lets assume update_date_time is '23-Nov-2020 10:20') seq_no becomes 3, active_from_date becomes '23-Nov-2020', active_to becomes 31-Dec-99 and the active_to of user_no 523 with seq_no 2 becomes '22-Nov-2020'. So the data should look like this :
Highlighted the 3rd record which will be added later in light green.
So here goes my solution : I suggested to use row_number() over(partition by user_no) analytical function to get seq_no for each user. I wanted to create a view based on that but Boss doesn't want a view. He strictly wants to use a procedure. Procedure should check if the user_no exists (in this example 523). If exists then seq_no increases and active_to of the previous record for 523 changes to latest active_from - 1 date. I will be honest - I have no clue how to achieve this in Procedure. I understand I can create a cursor with the query I had in my mind for the view. But to add seq_no and change active_to date is something that has puzzled me. Can anyone please guide me in right direction/s? Also I apologise in advance if I have left out any other details. Its midnight here now and after 8 hours of racking my brain on this I am very hungry!
edit 11th Mar : here is the code for the procedure I wrote to insert data into the audit table for situation when a particular user_no has no record in audit table :
create or replace procedure test_aud IS
user_found_audit number;
lv_user_no AUDIT_DATA.user_no%TYPE;
cursor member_no is select distinct user_no from STAGING_DATA;
begin
open member_no;
loop
fetch member_no into lv_user_no;
exit when member_no%notfound;
select count(*) into user_found_audit from AUDIT_DATA where user_no = lv_user_no;
if user_found_audit = 0 then
insert into AUDIT_DATA(user_no, update_date_time,status_code, seq_no, last_update_date, active_from, active_to)
select user_no, update_date_time,status_code,row_number() over(partition by user_no order by UPDATE_DATE_TIME) as seqno,
to_char(trunc(update_date_time),'DD-MON-YYYY'),
to_char(trunc(update_date_time),'DD-MON-YYYY'),
lead(to_char(trunc(update_date_time)-1,'DD-MON-YYYY'),1,'31-DEC-99') over(PARTITION BY user_no ORDER BY UPDATE_DATE_TIME) from STAGING_DATA where user_no = lv_user_no;
commit;
else
dbms_output.put_line(lv_user_no||' exists in audit table');
-- to code the block when user_no exists, involves an update and insert
end if;
end loop;
close member_no;
end;
/
Well you need to collect a couple things. The latest stage row and the latest audit row. Then it is just a matter of generating the new audit information and updating the previous latest one. The following makes a couple simplifying assumptions:
Only the latest stage data for a given user_no needs processed as
all prior have been processed, However it does not assume the stage
table has been cleared.
The sequencing of 'Y' and 'N' status_codes are properly order in
that manner. In fact it does not even check the value.
It need not concern itself with the inherent race condition. The
condition is derives from seq_no being generated as Max()+1. This
structure virtually guarantees a duplicate will eventually be
created.
The nested procedure "establish_audit" does all the actual work. The rest are just supporting characters, including a couple just for debug purpose. See fiddle.
create or replace
procedure generate_stage_audit(user_no_in staging_data.user_no%type)
as
k_end_of_time constant date := date '9999-12-31';
l_latest_user_stage staging_data%rowtype;
l_latest_user_audit audit_data%rowtype;
procedure establish_audit
is
begin
insert into audit_data(user_no, update_date_time, status_code
,seq_no, active_from, active_to)
select l_latest_user_stage.user_no
, l_latest_user_stage.update_date_time
, l_latest_user_stage.status_code
, coalesce(l_latest_user_audit.seq_no,0) + 1
, trunc(l_latest_user_stage.update_date_time)
, k_end_of_time
from dual;
update audit_data
set active_to = trunc(l_latest_user_stage.update_date_time - 1)
where user_no = l_latest_user_audit.user_no
and seq_no = l_latest_user_audit.seq_no;
end establish_audit;
procedure retrieve_latest_stage
is
begin
select *
into l_latest_user_stage
from staging_data
where (user_no, update_date_time) =
( select user_no, max(update_date_time)
from staging_data
where user_no = user_no_in
group by user_no
);
end retrieve_latest_stage;
procedure retrieve_latest_audit
is
begin
select *
into l_latest_user_audit
from audit_data
where (user_no, seq_no) =
( select user_no, max(seq_no)
from audit_data
where user_no = user_no_in
group by user_no
);
exception
when no_data_found then
null;
end retrieve_latest_audit;
---- for debugging ---
procedure show_stage
is
begin
dbms_output.put_line('-------- Stage Row -------');
dbms_output.put_line(' user_no==>' || to_char(l_latest_user_stage.user_no));
dbms_output.put_line('update_date_time==>' || to_char(l_latest_user_stage.update_date_time));
dbms_output.put_line(' status_code==>' || to_char(l_latest_user_stage.status_code));
end show_stage;
procedure show_audit
is
begin
dbms_output.put_line('-------- Audit Row -------');
dbms_output.put_line(' user_no==>' || to_char(l_latest_user_audit.user_no));
dbms_output.put_line('update_date_time==>' || to_char(l_latest_user_audit.update_date_time));
dbms_output.put_line(' status_code==>' || to_char(l_latest_user_audit.status_code));
dbms_output.put_line(' seq_no==>' || to_char(l_latest_user_audit.seq_no));
dbms_output.put_line(' active_from==>' || to_char(l_latest_user_audit.active_from));
dbms_output.put_line(' active_to==>' || to_char(l_latest_user_audit.active_to));
end show_audit;
begin -- the main event
retrieve_latest_stage;
show_stage;
retrieve_latest_audit;
show_audit;
establish_audit;
end generate_stage_audit;
A couple warnings:
It seems you may be tempted to use string data type for the audit
columns Active_Form and Active_to as you are trying to declare then
"date type without the time". However there is no such data type in
Oracle; time is part of all dates. Do not do so, store them as
standard dates. (Note Dates are not stored in any format, but an
internal structure. Formats are strictly a visual representation).
Just throwaway the time with the format on the query or by setting
nls_date_format.
You may be tempted to convert call this through a trigger. Do not,
it will likely result in an "ORA-04091: Table is mutating"
exception.
I'm having a problem with a synchronisation issue... I have a source table (mtAllowanceCategory) which I want to update to a copy (qryAllowanceCategory) of it. To make sure records in the copy are deleted if they are no longer present in the source, the copy has a "StillHere" boolean field, which is set to on when the record is added or updated and otherwise stays off. Afterwards, all records with StillHere=false are deleted.
That's the idea, anyway... in practice, the flag fields isn't turned on when posting updates. When I trace the code, the statement is executed; when I look in Access, it stays off. Hence the delete SQL afterwards clears the entire table.
Been trying to figure this for hours now; what am I missing??
mtAllowanceCategory:TFDMemTable (filled from an API call, this works fine)
qryAllowanceCategory:TFDQuery
conn:TFDConnection to a local Access database (also used for qryAllowanceCategory)
conn.ExecSQL('UPDATE AllowanceCategory SET StillHere=false;');
while not mtAllowanceCategory.eof do
begin
if qryAllowanceCategory.locate('WLPid',mtAllowanceCategory.FieldByName('Id').AsString,[loCaseInsensitive]) then
begin
Updating:=true;
qryAllowanceCategory.Edit;
end
else
begin
Updating:=false;
qryAllowanceCategory.Insert;
end;
qryAllowanceCategory.fieldbyname('createdBy').AsString:=mtAllowanceCategory.FieldByName('createdBy').AsString;
qryAllowanceCategory.fieldbyname('createdOn').AsString:=mtAllowanceCategory.FieldByName('createdOn').AsString;
qryAllowanceCategory.fieldbyname('description').AsString:=mtAllowanceCategory.FieldByName('description').AsString;
qryAllowanceCategory.fieldbyname('WLPid').AsString:=mtAllowanceCategory.FieldByName('id').AsString;
qryAllowanceCategory.fieldbyname('isDeleted').Asboolean:=mtAllowanceCategory.FieldByName('isDeleted').Asboolean;
qryAllowanceCategory.fieldbyname('isInUse').Asboolean:=mtAllowanceCategory.FieldByName('isInUse').Asboolean;
qryAllowanceCategory.fieldbyname('modifiedBy').AsString:=mtAllowanceCategory.FieldByName('modifiedBy').AsString;
qryAllowanceCategory.fieldbyname('modifiedOn').AsString:=mtAllowanceCategory.FieldByName('modifiedOn').AsString;
qryAllowanceCategory.fieldbyname('WLPname').AsString:=mtAllowanceCategory.FieldByName('name').AsString;
qryAllowanceCategory.fieldbyname('number').AsInteger:=mtAllowanceCategory.FieldByName('number').AsInteger;
qryAllowanceCategory.fieldbyname('percentage').AsFloat:=mtAllowanceCategory.FieldByName('number').AsFloat;
qryAllowanceCategory.fieldbyname('remark').AsString:=mtAllowanceCategory.FieldByName('remark').AsString;
qryAllowanceCategory.fieldbyname('LocalEdited').AsBoolean:=false;
qryAllowanceCategory.fieldbyname('LocalInserted').AsBoolean:=false;
qryAllowanceCategory.fieldbyname('LocalDeleted').AsBoolean:=false;
qryAllowanceCategory.fieldbyname('StillHere').AsBoolean:=true;
qryAllowanceCategory.Post;
mtAllowanceCategory.next;
end;
conn.commit;
conn.ExecSQL('DELETE FROM AllowanceCategory WHERE StillHere=false;');
When I read your q, I was struck by two thoughts:
One was that I couldn't immediately
see the cause of your problem and the other that you could probably avoid the problem anyway
if you used Sql rather than table traversals in code.
It seemed to me that you might be able to do most
if not all of what you need, in terms of synchronising the two tables, using Access
Sql rather than traversing the qryAllowanceCategory table using a while not EOF loop.
(btw, in the following I'm going to use 'mtAC' and qryAC to reduce typing & typos)
Using Access SQL
Initially, I did not have much luck, as Access rejected my attempts to
refer to both tables in an Update statement against the qryAC one using a Join
or Outer Join, but then I came across a reference that showed that Access does
support an Inner Join syntax. These SQL statements execute successfully by calling
ExecSQL on the FireDAC connection to the database:
update qryAC set qryAC.StillHere = True
where exists(select mtAC.* from mtAC inner join qryAC on mtAC.WLPid = qryAC.WLPid)
and
update qryAC inner join mtAC on mtAC.WLPid = qryAC.WLPid set qryAC.AValue = mtAC.AValue
This first of these obviously provides a way to update the StillHere field to set it to True,
or False with a trivial modification.
The second shows a way to update a set of fields in qryAC from the matching rows in mtAC
and this could, of course, be limited to a subset of rows with a suitable Where clause.
Access Sql also supports checking whether a row in one table exists in the other, as in
select * from qryAC q where exists (select * from mtac m where q.wlpid = m.wlpid)
and for deleting rows in one table which do not exist in the other
delete from qryAC q where not exists (select * from mtac m where q.wlpid = m.wlpid)
Using FireDAC's LocalSQL
I also mentioned LocalSQL in a comment. This supports a far broader range
of Sql statements that native Access Sql and can operate on any TDataSet descendant,
so if you find something that Access Sql syntax doesn't support, it is worth considering
using LocalSQL instead. Its main downside is that it operates on the datasets using
traversals, so in not quite as "instant" as native Sql. It can be a bit tricky to set up,
so here are the settings from the DFM which show how the components need connecting up. You would use it by feeding what you want to FDQuery1.
object AccessConnection: TFDConnection
Params.Strings = (
'Database=D:\Delphi\Code\FireDAC\LocalSQL\Allowance.accdb'
'DriverID=MSAcc')
Connected = True
LoginPrompt = False
end
object mtAC: TFDQuery
AfterOpen = mtACAfterOpen
Connection = AccessConnection
SQL.Strings = (
'select * from mtAC')
end
object qryAC: TFDQuery
Connection = AccessConnection
end
object LocalSqlConnection: TFDConnection
Params.Strings = (
'DriverID=SQLite')
Connected = True
LoginPrompt = False
end
object FDLocalSQL1: TFDLocalSQL
Connection = LocalSqlConnection
DataSets = <
item
DataSet = mtAC
end
item
DataSet = qryAC
end>
end
object FDGUIxWaitCursor1: TFDGUIxWaitCursor
Provider = 'Forms'
end
object FDPhysSQLiteDriverLink1: TFDPhysSQLiteDriverLink
end
object FDQuery1: TFDQuery
Connection = LocalSqlConnection
end
If anyone is interested:
The problem was in not refreshing qryAllowanceCategory after the initial SQL setting StillHere to false. The memory version (qryAllowanceCategory) of the record didn't get that update, so according to him, the flag was still on; after the field updates it appeared there were no changes (all the other fields were unchanged as well) so the post was ignored. In the actual table it was off though, so the final delete SQL removed it.
The problem was solved by adding a refresh after the first UPDATE SQL statement.
I'm writing procedure to transfer data from table in one Db named as 'Dev( or testing)' to another table in fdw db where we'll have repository table. Every month, last business day -2, we'll clean data from dev table and transfer it to repository table.
I'm super new to this technology and testing procedure and getting follwiing error
[Error] ORA-00904 (12: 15):
PL/SQL: ORA-00904: "USB"."BAS2_AGENCY_TO_RISKRATE_REPOS"."AS_OF_DATE: invalid identifier
CREATE OR REPLACE PROCEDURE USB.Basel2_riskrating
AS
BEGIN
INSERT INTO USB.BAS2_AGENCY_TO_RISKRATE_REPOS#OFSADEV --INSERTS DATA IN REPOSITORY TABLE
SELECT *
FROM USB.BAS2_AGENCY_TO_RISKRATING_TRAN M
WHERE USB.BAS2_AGENCY_TO_RISKRATE_REPOS.AS_OF_DATE != M.AS_OF_DATE ; --COMPARES DATE COLUMN TO REMOVE DUPLICACY
COMMIT;
END Basel2_riskrating;
Could you please help me in this. Also, it will be great if one could guide me with sample procedure code in wiping data from USB.BAS2_AGENCY_TO_RISKRATING_TRAN table at each month end.
As suggested in the comments, use NOT EXISTS. It's not celar whether you want to check for duplicates in the remote table usb.bas2_agency_to_riskrate_repos#ofsadev or a local version of the same. Use the appropriate table inside NOT EXISTS to do the comparison.
CREATE OR REPLACE PROCEDURE usb.basel2_riskrating AS
BEGIN
INSERT INTO usb.bas2_agency_to_riskrate_repos#ofsadev
SELECT * FROM usb.bas2_agency_to_riskrating_tran m
WHERE NOT EXISTS ( select 1 from
usb.bas2_agency_to_riskrate_repos e --#ofsadev?
where e.as_of_date = m.as_of_date
);
COMMIT; --Try to avoid commits inside procedure, move it to execution section
END basel2_riskrating;
SELECT * FROM USB.BAS2_AGENCY_TO_RISKRATING_TRAN M WHERE USB.BAS2_AGENCY_TO_RISKRATE_REPOS.AS_OF_DATE != M.AS_OF_DATE ;
is wrong. You can not run such a select against DB, it has no USB.BAS2_AGENCY_TO_RISKRATE_REPOS in from clause
I've created one of many stored procedures as part of an ETL process and one of the queries within a stored procedure isn't executing.
The environment is SQL Server 2012 SP2.
The bizarre thing is this -
Run the select part of the insert (affected query) - returns rows
Run the insert (affected query) - inserts rows
Run the whole stored procedure via SSMS - inserts rows
Run via SSIS - all other queries run barring the one of concern!
There are no conditions in my stored procedure e.g. if x = True the Return and no debug code is in there either e.g. return. There are also no transactions and the table I am reading from is a staging table populated prveiously by a data flow.
The query:
INSERT INTO Person.CustomerLinks
(PersonID, SystemID, CustomerID_bin, CustomerActive)
SELECT i.PersonID
, s.SystemDefID
, i.CustomerID_bin
, 0
FROM Staging.IdentifyOutput i
JOIN Config.SystemDef s ON s.OutputType = i.OutputType
WHERE i.CustomerID_bin IS NOT NULL
AND i.OutputType IN ('L', 'X')
AND i.PersonID > 0
AND i.FileDuplicate = 1
AND i.PreferredRecord = 1
AND NOT EXISTS ( SELECT 1
FROM Person.CustomerLinks cl
WHERE cl.PersonID = i.PersonID
AND cl.CustomerID_bin = i.CustomerID_bin)
The procedure has a Try Catch block and the Catch will raise an error and no error is raised.
The only other non ETL code in the procedure is -
SET NOCOUNT ON
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
But I put this in all of my procedures in this application as I am not concerned about dirty reads as they won't happen.
I placed tsql directly after the insert to write to my audit system and ##RowCount was 0. Yet if I run the select now I get over 1.5 million rows back.
Any suggestions?
If you are using a Execute T-SQL task then please try replacing it with the Execute SQL Task.
I don't know what caused it, but I moved the specific SQL into another stored procedure and it worked. In reality, it warranted being in its own stored procedure by right as it was only semi related to the procedure in question.