Difference with the previous value. kslqDB - ksqldb

Question

I was trying to make an example with ksqlDB
Based on two properties, in this case "sensor_id" and "user_id".
I calculated the variation of the "value" field, when both attributes match.
// TOPIC KAFKA
{"timestamp": 1000000000, "user_id": "AAAAA", "sensor_id": "sensor_1111", "value": 10000}
{"timestamp": 1555550000, "user_id": "AAAAA", "sensor_id": "sensor_1111", "value": 22000}
{"timestamp": 1666660000, "user_id": "AAAAA", "sensor_id": "sensor_2222", "value": 22000}
{"timestamp": 1777770000, "user_id": "AAAAA", "sensor_id": "sensor_2222", "value": 25000}
{"timestamp": 1666660000, "user_id": "BBBBB", "sensor_id": "sensor_2222", "value": 30000}
{"timestamp": 1777770000, "user_id": "BBBBB", "sensor_id": "sensor_2222", "value": 40000}
When that two values coincide, I calculated the difference with the previous one.
And the elapsed time.
// RESULT
{"timestamp": 1555550000, "last_timestamp": 1000000000, "user_id": "AAAAA", "sensor_id": "sensor_1111", "value": 12000}
{"timestamp": 1777770000, "last_timestamp": 1666660000, "user_id": "AAAAA", "sensor_id": "sensor_2222", "value": 3000}
{"timestamp": 1777770000, "last_timestamp": 1666660000, "user_id": "BBBBB", "sensor_id": "sensor_2222", "value": 10000}

Answer 1

Something like this should be possible using a custom user defined aggregate function or UDAF.
If you were to write a Compare_to_previous UDAF which stores the first value it receives and compares it to the second value, then you could achieve what you want with:
CREATE TABLE DIFF AS SELECT
user_id,
sensor_id,
Compare_to_previous(ROWTIME) as last_timestamp,
Compare_to_previous(value) as value
GROUP BY user_id, sensor_id
HAVING Compare_to_previous(ROWTIME) IS NOT NULL;
Where compare_to_previous might be something like (not tested):
#UdafDescription(
name = "COMPARE_TO_PREVIOUS",
description = CompareToPrevious.DESCRIPTION
)
public final class CompareToPrevious {
static final String DESCRIPTION = "Blah blah";
// UDAF version for BIGINT:
private static Udaf<Long, List<Long>, Long> compareToPrevious() {
return new Udaf<Long, List<Long>, Long>() {
#Override
public List<Long> initialize() {
// Initial value is empty() meaning no previous.
return ImmutableList.of();
}
#Override
public List<Long> aggregate(final Long current, final List<Long> previous) {
if (current == null) {
// Ignore null values? You're choice!
return previous;
}
switch (previous.size()) {
case 0:
// 1st value seen for this key:
return ImmutableList.of(current);
case 1:
// 2nd value seen for this key:
return ImmutableList.of(previous.get(0), current);
default:
// Already seen two values - what to do?
// Do you want to compare pairs
// i.e. compare 2nd with 1st, then 4th with 3rd, etc.
// return ImmutableList.of()
// or
// Or each subsequent row with previous,
// i.e. you'd compare 2nd with 1st, then 3rd with 2nd etc.
return ImmutableList.of(previous.get(0), current);
}
}
#Override
public List<Long> merge(final List<Long> aggOne, final List<Long> aggTwo) {
throw new UnsupportedOperationException(
"Compare_to_previous won't work with session windowed data");
}
// Call to convert the intermediate List<Long> into an output value.
#Override
public Long map(final List<Long> agg) {
if (agg.size() != 2) {
// If we don't have two values to compare we output NULL.
// Use HAVING Compare_To_Previous(COL) IS NOT NULL in the query to exclude these null rows.
return null;
}
// We have two values - calculate the diff:
return ((long) agg.get(1)) - agg.get(0);
}
};
}
}
Such an aggregator won't be able to handle out-of-order data, but it will be suitable for many use cases.