How to create a list of objects from 2 tables in KSQLDB - ksqldb

I currently have 2 tables in KSQLDB:
CREATE STREAM "source-mysql-person" (
"_uid" STRING,
"_created" TIMESTAMP,
"_updated" TIMESTAMP,
"_disabled" TIMESTAMP,
"name" STRING,
"age" INT
) WITH (
KAFKA_TOPIC = 'source-mysql-person',
VALUE_FORMAT = 'AVRO'
);
/*
Field | Type
--------------------------------------------
_uid | VARCHAR(STRING) (primary key)
_created | TIMESTAMP
_updated | TIMESTAMP
_disabled | TIMESTAMP
name | VARCHAR(STRING)
age | INTEGER
--------------------------------------------
*/
CREATE TABLE "table-mysql-enriched-person_contact" WITH (
KAFKA_TOPIC = 'table-mysql-enriched-person_contact',
VALUE_FORMAT = 'AVRO'
) AS SELECT
"pc"."_uid" AS "_uid",
"pc"."_created" AS "_created",
"pc"."_updated" AS "_updated",
"pc"."_disabled" AS "_disabled",
"pc"."is_default" AS "is_default",
"pc"."value" AS "value",
"pc"."person_uid" AS "person_uid",
AS_MAP(
ARRAY['_uid', 'value'],
ARRAY["ct"."_uid", "ct"."value"]
) AS "contact_type"
FROM "table-mysql-person_contact" "pc"
INNER JOIN "table-mysql-contact_type" "ct" ON
"ct"."_uid" = "pc"."contact_type_uid"
EMIT CHANGES;
/*
Field | Type
-----------------------------------------------
_uid | VARCHAR(STRING) (primary key)
_created | TIMESTAMP
_updated | TIMESTAMP
_disabled | TIMESTAMP
is_default | INTEGER
value | VARCHAR(STRING)
person_uid | VARCHAR(STRING)
contact_type | MAP<STRING, VARCHAR(STRING)>
-----------------------------------------------
*/
I want to create a table table-mysql-enriched-person that has the data of table-mysql-person and for each "person", a list of "person_contact" related to that "person". For this I am trying to use the following querie:
CREATE TABLE "table-mysql-enriched-person" WITH (
KAFKA_TOPIC = 'table-mysql-enriched-person',
VALUE_FORMAT = 'AVRO'
) AS SELECT
"p"."_uid" AS "_uid",
"p"."_created" AS "_created",
"p"."_updated" AS "_updated",
"p"."_disabled" AS "_disabled",
"p"."name" AS "name",
"p"."age" AS "age",
AS_MAP(
ARRAY[
'_uid',
'_created',
'_updated',
'_disabled',
'is_default',
'value',
'contact_type'
],
ARRAY[
"e"."_uid",
"e"."_created",
"e"."_updated",
"e"."_disabled",
"e"."is_default",
"e"."value",
"e"."contact_type"
]
) AS "list_person_contact"
FROM "table-mysql-enriched-person_contact" "e"
INNER JOIN "table-mysql-person" "p" ON
"p"."_uid" = "e"."person_uid"
GROUP BY
"p"."_uid",
"p"."_created",
"p"."_updated",
"p"."_disabled",
"p"."name",
"p"."age"
EMIT CHANGES;
Theoretically the query is correct because I have primary keys in the 2 queries and, thinking that the table "person_contact" is a child table of "person" and that the field person._uid is represented as person_contact.person_uid, I should manage to create the table but ksqldb is returning me the following message:
Could not determine output schema for query due to error: GROUP BY requires aggregate functions
in either the SELECT or HAVING clause.

Related

FireDac, Master-Detail, set master PK as NEW_id in detail, Cached updates

I have 2 tables:
tbl1(idM int identity(1,1), Name nvarchar(50));
tbl2(idRow int identity(1,1), idM int, Data nvarchar(50));
They are queried via fdQuery1 and fdQuery2 (master:fdQuery1, masterfield:idM).
CachedUpdates = True on both fdQuery.
FDSchemaAdapter is connected to both also.
Also FDUpdateSQL1 and FDUpdateSQL2 connected to fdQuery1 and fdQuery2.
On inserting (fdQuery1.Insert) new row in fdQuery1 it's idM is shown as -2 by default, f.e.
FDUpdateSQL1 insert text:
INSERT INTO tbl1 (Name)
VALUES (:NEW_Name);
SELECT SCOPE_IDENTITY() AS idM
FDUpdateSQL2 insert text:
INSERT INTO tbl2 (idM, Data)
VALUES (:NEW_idM, :NEW_Data);
SELECT SCOPE_IDENTITY() AS idRow
On saving I execute this code:
FDSchemaAdapter.ApplyUpdates;
fdQuery1.CommitUpdates;
fdQuery2.CommitUpdates;
Target: fill form with DB-aware components connected to tbl1 and tbl2 and set tbl2.idM automatically.
How can I do this?
I tried OnUpdateRecord method but it still doesn't return idM from FDUpdateSQL1 insert.
fdQuery1.OnAfterPost field fdQuery1idM.asInteger also returns -2 value.
Current example result in tables:
tbl1
|idM|Name|
|--------|
|5 |Test|
tbl2
|idRow|idM|Data |
|------------------|
|10 |-2 |DataText|
Target tbl2
|idRow|idM|Data |
|------------------|
|10 |5 |DataText|
Thank you!

How to convert the columns to lowercase in join condition in Kusto Database

TableA
| where GuidId == "123"
| where Desc has_any ("processor")
| join kind=leftouter TableB on
$left.SubId == $right.SubId,
$left.ProductName == $right.Name,
$left.GuidId == $right.GuidId
| distinct SubId, PriceTags, ResourceType, ProductName, Name
ProductName is in lower case and Name is in camel case. How to bring ProductName and Name to same case in Join condition.
Thanks
something like:
| extend Name=tolower(Name)
TableA
| where GuidId == "123"
| where Desc has_any ("processor")
| join kind=leftouter (TableB | extend Name=tolower(Name)) on $left.SubId == $right.SubId, $left.ProductName==$right.Name, $left.GuidId==$right.GuidId
|distinct SubId, PriceTags, ResourceType, ProductName, Name
extend Name=tolower(Name)
You'll need to 'normalize' the values before the join.
Ideally you'll do this before ingestion, or at ingestion time (using an update policy).
Given the current non-normalized values, you can do it at query time (performance would be sub-optimal):
TableA
| where GuidId == "123"
| where Desc has "processor"
| join kind=leftouter (
TableB
| extend Name = tolower(Name)
) on
$left.SubId == $right.SubId,
$left.ProductName == $right.Name,
$left.GuidId == $right.GuidId
| distinct SubId, PriceTags, ResourceType, ProductName, Name

How can I include empty strings in HTML text() extracted with XPath?

I have a page which consists of a table with two columns.
header | value
----------------
field1 | 1
field2 |
field3 | 1
field4 |
field5 | 1
When I select the values I need to get the same number as there are fields. I get the right number with:
>s = scrapy.Selector(response)
>values = s.xpath('//tr/td[#class="tdMainBottom"][2]').extract() # get the second column
>len(values)
5
But:
>s = scrapy.Selector(response)
>values = s.xpath('//tr/td[#class="tdMainBottom"][2]/text()').extract() # get the values
>len(values)
3
I can clean the first list up afterwards, but is there a one-shot way of doing this in XPath/Scrapy?
This works but is kind of ugly:
values = [v.xpath('text()').extract()
for v in s.xpath('//tr/td[#class="tdMainBottom"][2]')]

MariaDB store procedure duplicate entry on update?

I've written a stored procedure to update and insert records into a MariaDB version 15.1, distrib 10.0.13-MariaDB for Win64 (x86).
My stored procedure:
exitProc:BEGIN
#--
# procCreateUser
# Parameters:
# biPerson_id, the id of the user, NULL if new
# vcFirstName, the christian name of the user
# vcMiddleName, optional, middle name of the user
# vcSurName, the surname of the user
# vcEmail, the email associated with the user
# biDept, the department ID
# biRole, the role ID
# vcUsername, the login name of the user
# vcPassword, the password for the user
# vcIPorHost, the IP address or Host name of the client
# biUID, the user ID of the user performing this procedure
#--
DECLARE txtAuditEntry TEXT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
GET DIAGNOSTICS CONDITION 1 #sqlstate = RETURNED_SQLSTATE,
#errno = MYSQL_ERRNO, #text = MESSAGE_TEXT;
CALL procLogError(CONCAT("procCreateUser: "
,#errno, " (", #sqlstate, "): ", #text));
END;
#Prepare the parameters
IF (vcFirstName IS NULL) OR (LENGTH(TRIM(vcFirstName)) = 0) THEN
CALL procLogError("vcFirstName must be valid");
LEAVE exitProc;
ELSEIF (vcSurName IS NULL) OR (LENGTH(TRIM(vcSurName)) = 0) THEN
CALL procLogError("vcSurName must be valid");
LEAVE exitProc;
ELSEIF (vcEmail IS NULL) OR (LENGTH(TRIM(vcEmail)) = 0) THEN
CALL procLogError("vcEmail must be valid");
LEAVE exitProc;
ELSEIF (biDept_id IS NULL) OR (biDept_id = 0) THEN
CALL procLogError("biDept_id must be valid");
LEAVE exitProc;
ELSEIF (biRole_id IS NULL) OR (biRole_id = 0) THEN
CALL procLogError("biRole_id must be valid");
LEAVE exitProc;
ELSEIF (vcUsername IS NULL) OR (LENGTH(TRIM(vcUsername)) = 0) THEN
CALL procLogError("vcUsername must be valid");
LEAVE exitProc;
END IF;
#Report parameters
CALL procAuditEntry(CONCAT("biPerson_id: ", biPerson_id), vcIPorHost, biCreator_id);
CALL procAuditEntry(CONCAT("vcFirstName: ", vcFirstName), vcIPorHost, biCreator_id);
CALL procAuditEntry(CONCAT("vcMiddleName: ", vcMiddleName), vcIPorHost, biCreator_id);
CALL procAuditEntry(CONCAT("vcSurName: ", vcSurName), vcIPorHost, biCreator_id);
CALL procAuditEntry(CONCAT("vcEmail: ", vcEmail), vcIPorHost, biCreator_id);
CALL procAuditEntry(CONCAT("biDept_id: ", biDept_id), vcIPorHost, biCreator_id);
CALL procAuditEntry(CONCAT("biRole_id: ", biRole_id), vcIPorHost, biCreator_id);
CALL procAuditEntry(CONCAT("vcUsername: ", vcUsername), vcIPorHost, biCreator_id);
IF (biPerson_id IS NULL) THEN
CALL procAuditEntry("INSERT", vcIPorHost, biCreator_id);
INSERT INTO `tbl_people` (
`vcFirstName`
,`vcMiddleName`
,`vcSurName`
,`vcEmail`
,`biDept_id`
,`biRole_id`
,`vcUserName`
,`vcPassWord`
) VALUES (
vcFirstName
,vcMiddleName
,vcSurName
,vcEmail
,biDept_id
,biRole_id
,vcUsername
,vcPassword
);
#Create audit log entry
SET txtAuditEntry = CONCAT('user \'', vcUsername, '\' created');
CALL procAuditEntry(txtAuditEntry, vcIPorHost, biCreator_id);
ELSE
CALL procAuditEntry("A.UPDATE", vcIPorHost, biCreator_id);
UPDATE `tbl_people` SET
`vcFirstName`=vcFirstName
,`vcMiddleName`=vcMiddleName
,`vcSurName`=vcSurName
,`vcEmail`=vcEmail
,`biDept_id`=biDept_id
,`biRole_id`=biRole_id
,`vcUserName`=vcUsername
WHERE
`biPerson_id`=biPerson_id;
CALL procAuditEntry("B.UPDATE", vcIPorHost, biCreator_id);
IF NOT vcPassWord IS NULL THEN
UPDATE `tbl_people` SET
`vcPassWord`=vcPassword
WHERE
`biPerson_id`=biPerson_id;
END IF;
#Create audit log entry
SET txtAuditEntry = CONCAT('user \'', vcUsername, '\' updated');
CALL procAuditEntry(txtAuditEntry, vcIPorHost, biCreator_id);
END IF;
END
All the calls to 'procAuditEntry' are really just for debugging, when I call this procedure to create a new record I pass the first parameter as null.
I can see from the audit table that it is going into the UPDATE section of the procedure. The issus is, I'm getting a duplicate key error raised by the procedure and I don't understand by because no new entry is being created, only an existing entry being modified. The table definition is as follows:
CREATE TABLE `tbl_people` (
`biPerson_id` BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary key',
`biCompany_id` BIGINT(20) UNSIGNED NULL DEFAULT NULL COMMENT 'Link to companies table',
`biDept_id` BIGINT(20) UNSIGNED NULL DEFAULT NULL COMMENT 'Link to department table',
`biRole_id` BIGINT(20) UNSIGNED NULL DEFAULT NULL COMMENT 'Link to Job title / description',
`tiActive` TINYINT(4) NOT NULL DEFAULT '1' COMMENT '1=active, 0=not',
`dtLastLogin` DATETIME NULL DEFAULT NULL COMMENT 'Date/Time of last login',
`vcFirstName` VARCHAR(48) NOT NULL COMMENT 'First name',
`vcMiddleName` VARCHAR(48) NULL DEFAULT NULL COMMENT 'Middle name',
`vcSurName` VARCHAR(48) NOT NULL COMMENT 'Surname',
`vcEmail` VARCHAR(256) NOT NULL COMMENT 'Email address',
`vcUserName` VARCHAR(48) NOT NULL COMMENT 'User name',
`vcPassWord` VARCHAR(32) NOT NULL COMMENT 'Password',
PRIMARY KEY (`biPerson_id`),
UNIQUE INDEX `Name` (`vcFirstName`, `vcSurName`),
UNIQUE INDEX `userName` (`vcUserName`),
INDEX `active` (`tiActive`),
INDEX `dept` (`biDept_id`),
INDEX `company` (`biCompany_id`),
INDEX `lastLogin` (`dtLastLogin`),
INDEX `jobrole` (`biRole_id`)
)
COMMENT='All timekeeper users'
COLLATE='latin1_swedish_ci'
ENGINE=InnoDB
AUTO_INCREMENT=4;
The error being logged is:
procCreateUser: 1062 (23000): Duplicate entry 'Simon-Platten' for key 'Name'
The really odd thing is if I call the UPDATE outside of the stored procedure it works.
Replacing the UPDATE with a REPLACE statement solves the issue, just not sure why the UPDATE fails.
While I am using a higher version of MariaDB (10.1.x), the problem must also be solved in version 10.0.x.
Try:
> SELECT VERSION();
+---------------------------+
| VERSION() |
+---------------------------+
| 10.1.9-MariaDB-1~wily-log |
+---------------------------+
1 row in set (0.00 sec)
> DROP TABLE IF EXISTS `tbl_people`;
Query OK, 0 rows affected (0.00 sec)
> CREATE TABLE `tbl_people` (
-> `biPerson_id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
-> `vcFirstName` VARCHAR(48) NOT NULL,
-> `vcMiddleName` VARCHAR(48) NULL DEFAULT NULL,
-> `vcSurName` VARCHAR(48) NOT NULL,
-> UNIQUE INDEX `Name` (`vcFirstName`, `vcSurName`)
-> );
Query OK, 0 rows affected (0.00 sec)
> INSERT INTO `tbl_people`
-> (`vcFirstName`, `vcMiddleName`, `vcSurName`)
-> VALUES
-> ('Simon', 'R.', 'Platten'),
-> ('Peter', 'F.', 'Lucas');
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
> SELECT
-> `biPerson_id`,
-> `vcFirstName`,
-> `vcMiddleName`,
-> `vcSurName`
-> FROM
-> `tbl_people`;
+-------------+-------------+--------------+-----------+
| biPerson_id | vcFirstName | vcMiddleName | vcSurName |
+-------------+-------------+--------------+-----------+
| 1 | Simon | R. | Platten |
| 2 | Peter | F. | Lucas |
+-------------+-------------+--------------+-----------+
2 rows in set (0.00 sec)
> DELIMITER //
> DROP PROCEDURE IF EXISTS `sp_test`//
Query OK, 0 rows affected (0.00 sec)
> CREATE PROCEDURE `sp_test`(
-> biPerson_id BIGINT UNSIGNED,
-> vcFirstName VARCHAR(48),
-> vcMiddleName VARCHAR(48),
-> vcSurName VARCHAR(48)
-> )
-> BEGIN
-> UPDATE `tbl_people` SET
-> `vcFirstName` = vcFirstName,
-> `vcMiddleName` = vcMiddleName,
-> `vcSurName` = vcSurName
-> WHERE `biPerson_id` = biPerson_id;
-> END//
Query OK, 0 rows affected (0.00 sec)
> DELIMITER ;
> CALL `sp_test`(1, 'Simon', 'A.', 'Platten');
ERROR 1062 (23000): Duplicate entry 'Simon-Platten' for key 'Name'
> DELIMITER //
> DROP PROCEDURE IF EXISTS `sp_test`//
Query OK, 0 rows affected (0.00 sec)
> CREATE PROCEDURE `sp_test`(
-> `_biPerson_id` BIGINT UNSIGNED,
-> `_vcFirstName` VARCHAR(48),
-> `_vcMiddleName` VARCHAR(48),
-> `_vcSurName` VARCHAR(48)
-> )
-> BEGIN
-> UPDATE `tbl_people` SET
-> `vcFirstName` = `_vcFirstName`,
-> `vcMiddleName` = `_vcMiddleName`,
-> `vcSurName` = `_vcSurName`
-> WHERE `biPerson_id` = `_biPerson_id`;
-> END//
Query OK, 0 rows affected (0.00 sec)
> DELIMITER ;
> CALL `sp_test`(1, 'Simon', 'A.', 'Platten');
Query OK, 1 row affected (0.00 sec)
> SELECT
-> `biPerson_id`,
-> `vcFirstName`,
-> `vcMiddleName`,
-> `vcSurName`
-> FROM
-> `tbl_people`;
+-------------+-------------+--------------+-----------+
| biPerson_id | vcFirstName | vcMiddleName | vcSurName |
+-------------+-------------+--------------+-----------+
| 1 | Simon | A. | Platten |
| 2 | Peter | F. | Lucas |
+-------------+-------------+--------------+-----------+
2 rows in set (0.00 sec)

sqlplus select record between merged dates

table name :document_archive_doc_perms
table desc:
- location char(1), document_id number(5), permission_id number(5),
permission_type char(1), date_time_from date, date_time_to date,
ecode number(5), ecode_s number(2), approved_by number(5),
approved_by_ecode number(2), approved_on date, primary
key(location,document_id,permission_id));
my query :
select permission_id,document_id,date_time_from,date_time_to,approved_on,permission_type from document_archive_doc_perms
where document_id=3 and ecode=1695 and approved_on is not null and (sysdate between date_time_from and date_time_to);
my output is
PERMISSION_ID DOCUMENT_ID DATE_TIME DATE_TIME APPROVED_ P
------------- ----------- --------- --------- --------- -
5 3 01-DEC-14 31-DEC-14 08-DEC-14 V
7 3 09-DEC-14 31-DEC-14 09-DEC-14 P
here my need is the latest permission from records(ie max of permission_id)
how to do this.?
You could use an anaylitic rank() call:
SELECT permission_id,
document_id,
date_time_from,
date_time_to,
approved_on,
permission_type
FROM (SELECT permission_id,
document_id,
date_time_from,
date_time_to,
approved_on,
permission_type,
RANK() OVER (ORDER BY perission_id ASC) AS rk
FROM document_archive_doc_perms
WHERE document_id = 3 AND
ecode = 1695 AND
approved_on IS NOT NULL AND
SYSDATE BETWEEN date_time_from AND date_time_to
)
WHERE rk = 1

Resources