Auto-incrementing column in ksqlDB? - ksqldb

Question

I'm currently using this process (see below) to generate an auto-incrementing column in ksqlDB. But now I'm wondering if there are race conditions or other synchronization problems with this approach. Is this a good way to generate an auto-incrementing column in ksqlDB? If not, is there a better way?
Suppose you want to insert values from one ksqlDB stream into another while auto-incrementing some integer value in the
destination stream.
First, create the two streams:
CREATE STREAM dest (ROWKEY INT KEY, i INT, x INT) WITH (kafka_topic='test_dest', value_format='json', partitions=1);
CREATE STREAM src (x INT) WITH (kafka_topic='test_src', value_format='json', partitions=1);
Next, create a materialized view that will contain the maximum value of the destination stream.
CREATE TABLE dest_maxi AS SELECT MAX(i) AS i FROM dest GROUP BY 1;
We need to be able to join the source stream to the materialized view. To do so, we'll create another intermediate stream
with a dummy one column that's always set to 1, which is what we grouped the materialized view on:
CREATE STREAM src_one AS SELECT x, 1 AS one FROM src;
INSERT INTO dest SELECT COALESCE(dest_maxi.i,0)+1 AS i, src_one.x AS x FROM src_one LEFT JOIN dest_maxi ON src_one.one = dest_maxi.ROWKEY PARTITION BY COALESCE(dest_maxi.i,0)+1 EMIT CHANGES;
Now you can insert values into stream src and watch them come up in stream dest with auto-incrementing IDs.

Answer 1

I don't think your approach will work. kslqDB offers no guarantees in the order in which is processes records across two different queries. In your case that means no ordering guarantees that
CREATE TABLE dest_maxi AS <query>;
Will run and update dest_maxi before
INSERT INTO dest <query>;
Runs. Hence, I think you'll run into issues.
It looks like you're trying to take a stream of numbers, e.g.
1234
24746
24848
4947
34
And add an auto-incrementing id column so that the result looks like:
1, 1234
2, 24746
3, 24848
4, 4947
5, 34
Something like this should give you what you want:
-- source stream of numbers:
CREATE STREAM src (
x INT
) WITH (
kafka_topic='test_src',
value_format='json'
);
-- intermediate 'table' of numbers and current count:
CREATE TABLE with_counter
WITH (partitions = 1) AS
SELECT
1 as k,
LATEST_BY_OFFSET(x) as x,
COUNT(1) AS id
FROM src
GROUP BY 1
-- if you need this back as a stream in ksqlDB you can run:
CREATE STREAM dest (
x INT,
id BIGINT
) WITH (
kafka_topic='WITH_COUNTER',
value_format='json'
);
UDAFs calculate values per-key, hence we group-by a constant, ensuring all input rows are funnelled into a single key (and partition - so this doesn't scale well!).
We use COUNT to count the number of rows seen, so its output auto-incrementing, and we use LATEST_BY_OFFSET to grab the current value of x into our table.
The changelog of the with_counter table will then contain the output you want, only with a constant key of 1:
1 -> 1, 1234
1 -> 2, 24746
1 -> 3, 24848
1 -> 4, 4947
1 -> 5, 34
We re-import this into ksqlDB as the dest stream. Which you can use as normal. If you want a topic without the key you can just run:
CREATE STREAM without_key AS SELECT * FROM dest;