I have produced following data on topic named SENSOR_STATUS_DETAILS in json:
1001
{
"sensorid": 1001,
"status": "CONNECTED",
"lastconnectedtime": "2020-05-31 22:31:54"
}
1002
{
"sensorid": 1002,
"status": "CONNECTED",
"lastconnectedtime": "2020-05-31 22:33:37"
}
I am trying to make a table from it as:
CREATE TABLE STATUS_IB_TABLE (ROWKEY INT KEY,
sensorid INTEGER,
status VARCHAR,
lastconnectedtime STRING)
WITH (TIMESTAMP='lastconnectedtime', TIMESTAMP_FORMAT='yyyy-MM-dd HH:mm:ss', KAFKA_TOPIC='SENSOR_STATUS_DETAILS', VALUE_FORMAT='JSON', KEY='sensorid');
The rowkey ksqlDB is making is as follows:
I want the rowkey to be sensorid....
I don't know whats happening
please help me on this.
thanks in advance!!
PS:
Confluent Platform version: 5.5
The issue here is that the data being produced to the Kafka topic SENSOR_STATUS_DETAILS has a STRING key, not an INT key.
If you take the STRING "1001" it just happens to serialize to the same number of bytes as an INT. If you deserialize those same bytes as an INT you get the number 825241649.
You have two options:
Change how you are producing the data so that you are producing a serialized 32-bit integer as the Kafka message's key and continue to import as ROWKEY INT KEY, or
Change your CREATE TABLE statement to have a STRING key:
CREATE TABLE STATUS_IB_TABLE (
ROWKEY STRING KEY, // <-- string key
sensorid STRING, // <-- matching type here.
status VARCHAR,
lastconnectedtime STRING
)
WITH (<as above>);
The first option would likely result in slightly better performance
Related
I have read https://swagger.io/docs/specification/data-models/dictionaries/, but am still having trouble modeling something like:
{
"top": {
"elem0": {
...
},
"elem1": {
...
},
...
}
}
top is a dictionary, containing an arbitrary number of entries. The trouble is that I want to define a reusable schema for the key and value types.
E.g., instead of saying that the keys are of type string, I say they are of type #/components/schemas/KEY. And instead of saying that the values are of type object, I say they are of type #/components/schemas/VALUE.
Is this possible?
Good day, I am trying to apply named parameter {num phone} to the below example:
main(){
showInfo("abc#mail.com", "Fatima");
}
String showInfo(String email, String name, {num phone}) {
print(email);
print(name);
print(phone);
return "$email : $name : $phone";
}
but I receive the error:
Error: The parameter 'phone' can't have a value of 'null' because of its type 'num', but the implicit
default value is 'null'.
Try adding either an explicit non-'null' default value or the 'required' modifier.
Future<String> showInfo(String email, String name, {num phone}) async {
^^^^^
your help is appreciated.
You marked the parameter as a num, which means it cannot be null. However, the default for named parameters that are not used is null, so you cannot have a named parameter that is optional with a default value of null with a datatype that does not accept null.
One option is to give it a default value other than null:
String showInfo(String email, String name, {num phone = 0})
Another option is to make it a named, but required parameter, so it will never get a default value:
String showInfo(String email, String name, {required num phone})
Another alternative is to actually keep the phone optional:
String showInfo(String email, String name, {num? phone})
Some additional wisdom: phone numbers can start with leading zeroes that are important and should not be deleted on saving it. You cannot use num for a phone number, you will have to use string.
The error is pretty clear, you have to pass a value to the phone parameter or add a default value. This is due to the introduction of Null Safety in Dart 2.12.
You can pass a value like this :
showInfo("abc#mail.com", "Fatima", phone: 0);
Or add a default value to the phone parameter :
String showInfo(String email, String name, {num phone = 0}) {
print(email);
print(name);
print(phone);
return "$email : $name : $phone";
}
If you want, you can use Dart <2.12 to avoid Null Safety, your code should work properly then.
I'm trying to extract data from 2 tables in BigQuery, then join it by CoGroupByKey.
Although the output of BigQuery is PCollection<TableRow>, CoGroupByKey requires PCollection<KV<K,V>>.
How can I convert from PCollection<TableRow> to PCollection<KV<K,V>>?
CoGroupByKey needs to know which key to CoGroup by - this is the K in KV<K, V>, and the V is the value associated with this key in this collection. The result of co-grouping several collections will give you, for each key, all of the values with this key in each collection.
So, you need to convert both of your PCollection<TableRow> to PCollection<KV<YourKey, TableRow>> where YourKey is the type of key on which you want to join them, e.g. in your case perhaps it might be String, or Integer, or something else.
The best transform to do the conversion is probably WithKeys. E.g. here's a code sample converting a PCollection<TableRow> to a PCollection<KV<String, TableRow>> keyed by a hypothetical userId field of type String:
PCollection<TableRow> rows = ...;
PCollection<KV<String, TableRow>> rowsKeyedByUser = rows
.apply(WithKeys.of(new SerializableFunction<TableRow, String>() {
#Override
public String apply(TableRow row) {
return (String)row.get("userId");
}
}));
I have 2 problems. I want to query a dynamodb database on Bool and on Date.
Is this possible as I only see String Number and Binary?
There are two types of expressions.
KeyConditionExpression - Supports only String, Number and Binary
FilterExpression - Can be used for Non-Key attributes. You can query for the Date and BOOL attributes.
There is a BETWEEN operator which can be used for DATE range as well.
Code to filter by Date:-
var params = {
TableName : table,
KeyConditionExpression : 'email = :email',
FilterExpression: 'createdAt = :createdAt',
ExpressionAttributeValues : {
':email' : 'abc#gmail.com',
':createdAt' : "2016-11-07"
}
};
My Item:-
Date is stored as String in database.
Date S (string type). The Date values are stored as ISO-8601 formatted
strings.
I'm working on a legacy project and have found an issue. The code is reading from an sqlite database. It performs a query and iterates over the statement and then for each column it gets the column name and the column value.
To get the value it calls sqlite3_column_type. Then it checks the types and uses sqlite3_column_int, sqlite3_column_double, sqlite3_column_text, etc. to read the value.
The issue is that there is a row in the DB that has a STRING column but the value contains only numbers (the value was set as 12 characters long, something like #"123456789012"). Calling sqlite3_column_type for this row returns SQLITE_INTEGER as it's only looking at the value of the column and not the declared column type. This means that the code will then read the value with sqlite3_column_int and I'll end up with an integer overflow and a number nothing like the original value contained in the DB.
Here is a quick example:
int columnType = sqlite3_column_type(statement, index);
if (columnType == SQLITE_INTEGER) {
//use sqlite3_column_int(statement, index)
}
else if (columnType == SQLITE_FLOAT) {
//use sqlite3_column_double(statement, index)
}
else if (columnType == SQLITE_TEXT) {
//use sqlite3_column_text(statement, index)
}
///etc...
How can I make sure the the value I get out from a STRING column containing only numbers is treated correctly? Can I somehow check the declared column type (other than querying and parsing out the original create table sql)? Is there a better way to do this?
Using column_decltype instead of column_type means that I get that declared column type (STRING) instead of the type of value. Then I can compare the result against STRING, TEXT, etc.to figure out how to read the value.