I create a one master stream with value as array. I would get array fields at one and stream it to ne ksql stream.
At first i create the master stream:
CREATE STREAM runtime_master_stream
(timestamp BIGINT,
opcuaObject VARCHAR,
value array<DOUBLE>)
WITH (KAFKA_TOPIC='runtime_master', VALUE_FORMAT='JSON');
Output is:
{"ROWTIME":1557317077577,"ROWKEY":"\u0000\u0000\u0001j�T�I","timestamp":1557317069589,"opcuaObject":"DatBetrZ.BetrZStdOM","value":[19.737154,512.0,320.18024,423.87027,399.99384,292.1198,450.821]}
Then i create new stream to get the arrayfields at one:
CREATE STREAM runtime_std_om_all_knife_stream
WITH (TIMESTAMP='timestamp',
PARTITIONS=4,
VALUE_FORMAT='JSON') AS
SELECT
timestamp,
opcuaObject,
value[0] AS knife1,
value[1] AS knife02,
value[2] AS knife03,
value[3] AS knife04,
value[4] AS knife05,
value[5] AS knife05,
value[6] AS knife06
FROM RUNTIME_MASTER_STREAM
WHERE opcuaObject ='DatBetrZ.BetrZStdOM';
Output:
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"DatBetrZ.BetrZStdOM","KNIFE1":19.737154,"KNIFE02":512.0,"KNIFE03":320.18024,"KNIFE04":42
3.87027,"KNIFE05":400.02216,"KNIFE06":292.1198,}
What i need is this output in one ksql stream EXAMPLE:
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife1","VALUE":19.737154}
{"ROWTIME":1557317170313,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife02","VALUE":19.737154}
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife03","VALUE":19.737154}
{"ROWTIME":1557317170313,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife04","VALUE":19.737154}
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife05","VALUE":19.737154}
{"ROWTIME":1557317170313,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife06","VALUE":39.737154}
Any idea how to get this output with one stream in kafka ksql?
What you're looking for is an EXPLODE/UNNEST function, which is not currently available but for which there is an existing issue that you can upvote if you think it would be useful.
A workaround for you could be something like this, in which you brute-force populate the target stream with INSERT INTO iterating over all the possible indices of your array:
Populate some test data into a Kafka topic
$ curl "https://api.mockaroo.com/api/440970e0?count=5&key=ff7856d0" | \
kafkacat -P -b localhost -t car_data_01
Examine the data in KSQL:
ksql> PRINT 'car_data_01' FROM BEGINNING;
Format:JSON
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1533200557","car":"Oldsmobile","value":[68.93,53.58]}
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1548442477","car":"Mercury","value":[60.09,69.07,63.77,63.13]}
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1544928225","car":"Volkswagen","value":[59.77,6.94,97.7,30.86,16.9]}
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1545383393","car":"Nissan","value":[13.32]}
{"ROWTIME":1557392065412,"ROWKEY":"null","timestamp":"1552825010","car":"Hyundai","value":[12.92]}
Create a stream over the data:
CREATE STREAM CAR_DATA (timestamp BIGINT, CAR VARCHAR, VALUE ARRAY<DOUBLE>) WITH (KAFKA_TOPIC='car_data_01', VALUE_FORMAT='JSON');
ksql> SELECT TIMESTAMP, CAR, VALUE[0], VALUE[1], VALUE[2] FROM CAR_DATA;
1533200557 | Oldsmobile | 68.93 | 53.58 | null
1548442477 | Mercury | 60.09 | 69.07 | 63.77
1544928225 | Volkswagen | 59.77 | 6.94 | 97.7
1545383393 | Nissan | 13.32 | null | null
1552825010 | Hyundai | 12.92 | null | null
Create the output stream, to start with just containing the zero-index elements of the array:
CREATE STREAM CAR_DATA_EXPLODED AS SELECT TIMESTAMP, CAR, 'Sensor 00' AS SOURCE, VALUE[0] AS VALUE FROM CAR_DATA WHERE VALUE[0] IS NOT NULL;
ksql> SELECT * FROM CAR_DATA_EXPLODED;
1557392065409 | null | 1544928225 | Volkswagen | Sensor 00 | 59.77
1557392065409 | null | 1545383393 | Nissan | Sensor 00 | 13.32
1557392065409 | null | 1533200557 | Oldsmobile | Sensor 00 | 68.93
1557392065412 | null | 1552825010 | Hyundai | Sensor 00 | 12.92
1557392065409 | null | 1548442477 | Mercury | Sensor 00 | 60.09
Insert the remaining array indices to the new stream:
CREATE STREAM CAR_DATA_EXPLODED_00 AS SELECT TIMESTAMP, CAR, 'Sensor 00' AS SOURCE, VALUE[0] AS VALUE FROM CAR_DATA
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 01' AS SOURCE, VALUE[1] AS VALUE FROM CAR_DATA WHERE VALUE[1] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 02' AS SOURCE, VALUE[2] AS VALUE FROM CAR_DATA WHERE VALUE[2] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 03' AS SOURCE, VALUE[3] AS VALUE FROM CAR_DATA WHERE VALUE[3] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 04' AS SOURCE, VALUE[4] AS VALUE FROM CAR_DATA WHERE VALUE[4] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 05' AS SOURCE, VALUE[5] AS VALUE FROM CAR_DATA WHERE VALUE[5] IS NOT NULL;
Examine the exploded data:
ksql> SELECT * FROM CAR_DATA_EXPLODED_00;
1557392065409 | null | 1533200557 | Oldsmobile | Sensor 00 | 68.93
1557392065409 | null | 1545383393 | Nissan | Sensor 00 | 13.32
1557392065409 | null | 1544928225 | Volkswagen | Sensor 00 | 59.77
1557392065409 | null | 1548442477 | Mercury | Sensor 02 | 63.77
1557392065409 | null | 1544928225 | Volkswagen | Sensor 03 | 30.86
1557392065409 | null | 1544928225 | Volkswagen | Sensor 04 | 16.9
1557392065409 | null | 1533200557 | Oldsmobile | Sensor 01 | 53.58
1557392065409 | null | 1544928225 | Volkswagen | Sensor 02 | 97.7
1557392065412 | null | 1552825010 | Hyundai | Sensor 00 | 12.92
1557392065409 | null | 1548442477 | Mercury | Sensor 01 | 69.07
1557392065409 | null | 1548442477 | Mercury | Sensor 00 | 60.09
1557392065409 | null | 1544928225 | Volkswagen | Sensor 01 | 6.94
1557392065409 | null | 1548442477 | Mercury | Sensor 03 | 63.13
ksql> SELECT * FROM CAR_DATA_EXPLODED_00 WHERE SOURCE='Sensor 03';
1557392065409 | null | 1544928225 | Volkswagen | Sensor 03 | 30.86
1557392065409 | null | 1548442477 | Mercury | Sensor 03 | 63.13
Related
I have this simple scope
scope :has_video_link,
-> { where.not(video_link: nil) }
I use it like so in my controller
#videos = Message .featured
.includes(:user, :company, :forum, :topic)
.published
.unremoved
.approved
.has_video_link
.order(created_at: :desc)
.paginate(
page: page,
per_page: limit)
Which I call like so
before_action only: :index
skip_before_action :verify_authenticity_token
def index
self.page_title = 'Home'
fetch_videos
end
That works just fine, and my page loads in under a second on prod no problem. My issue is I have an ajax request that calls this
def videos
fetch_videos
respond_to do |format|
format.js
end
end
On dev this works great but in prod this takes over 60 seconds....
First image is from the first times its called (4ms)
2nd image is the ajax call 69000ms...
Any ideas? How do i fix this?? Thank you!!
edit: here is the explain
EXPLAIN for: SELECT `messages`.* FROM `messages` WHERE `messages`.`is_featured` = 1 AND (messages.created_at<='2021-06-04 12:36:22.241601') AND `messages`.`is_removed` = 0 AND `messages`.`is_approved` = 1 AND (`messages`.`video_link` IS NOT NULL) ORDER BY `messages`.`created_at` DESC LIMIT 5 OFFSET 0
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
| 1 | SIMPLE | messages | NULL | range | index_messages_on_is_removed_and_user_id_and_created_at,index_messages_on_is_removed_and_message_type_and_created_at,index_messages_on_is_removed_company_id_type_created_at,index_messages_on_is_removed_company_id_created_at_id_user_id,index_messages_on_is_removed_and_created_at,index_messages_on_is_removed_company_id_created_at_rating_count,index_messages_on_is_removed_company_id_user_role_created_at,index_messages_on_is_removed_and_company_id_and_rating_total,index_messages_on_is_removed_and_forum_id_and_created_at,index_messages_on_is_approved,index_messages_on_is_removed_and_is_approved | index_messages_on_is_removed_and_created_at | 7 | NULL | 3470 | 9.0 | Using index condition; Using where |
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `users`.* FROM `users` WHERE `users`.`id` = 594568
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
| 1 | SIMPLE | users | NULL | const | PRIMARY | PRIMARY | 4 | const | 1 | 100.0 | NULL |
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `companies`.* FROM `companies` WHERE `companies`.`id` IN (564013, 562440)
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | companies | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.0 | Using where |
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.01 sec)
EXPLAIN for: SELECT `forums`.* FROM `forums` WHERE `forums`.`id` IN (12224, 7759)
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | forums | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.0 | Using where |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `topics`.* FROM `topics` WHERE `topics`.`id` IN (684474, 684473, 684472, 684470, 684467)
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | topics | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 5 | 100.0 | Using where |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.00 sec)
Solved this issue. All I had to do was simply index video_links and that fixed my issue right up!
How can I get the first name and last name from the table User through table Friend which is referenced by the user_id with the condition of user_id 5 and status is pending and place it in a each loop but i also need the contents of friend controller in rails. Thank a lot for the help
Controller
#add = Friend.where(user_id: 5, status: 'Pending')
User Database
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
| first_name | text | YES | | NULL | |
| last_name | text | YES | | NULL | |
| email | text | YES | | NULL | |
| contact_number | varchar | YES | | NULL | |
| birth_date | date | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| updated_at | datetime | NO | | NULL | |
Friend
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
| request_date | date | YES | | NULL | |
| reason | text | YES | | NULL | |
| status | varchar(255) | YES | | NULL | |
| userid | int | YES | MUL | NULL | |
How about this:
#add = Friend.joins(:user).select("users.first_name, users.last_name").where(user_id: 5, status: 'Pending')
Seems like youre already fetching the id of the user, stored in the user_id field. In order to get the associated user, i would do
#user = User.find(#add.id)
This retrieves the user based on the id you pass, and from there you can do stuff like
#user.first_name
to get the values
All you need to do in Friend model belongs_to user and then
#add =Friend.where("user_id = ? AND status = ?", 5,"Pending")
and
<#add.each do |s|>
<%s.user.first_name%>
<%=<%s.user.last_name%>
// and so on..
<%end%>
My problem on getting one row shown with two null values.
FROM tuotemerkki;
tmtunnus | tmnimi | maa
----------+----------+-------------
1 | McCee | Yhdysvallat
2 | KooTek | Italia
3 | Giardino | Italia
(3 rows)
FROM tuote;
ttunnus | tnimi | kuvaus | suositushinta | tmtunnus
---------+-----------------------+--------------------+---------------+----------
111 | Trimmeri TRCee | tehokas 4-tahtinen | 179.00 | 1
112 | Trimmerisiima Cee | laadukas siima | 6.99 | 1
113 | Moottorisaha MSCee RR | robusti ja raskas | 559.00 | 1
114 | Trimmerisiima Y | yleissiima | 3.99 | 2
115 | Lapio L | kevyt yleislapio | 23.95 | 2
(5 rows)
I need to get this selected with the NULL VALUE on giardino from tmnimi.
tmnimi | tnimi | kuvaus
----------+-----------------------+--------------------
McCee | Trimmeri TRCee | tehokas 4-tahtinen
McCee | Trimmerisiima Cee | laadukas siima
McCee | Moottorisaha MSCee RR | robusti ja raskas
KooTek | Trimmerisiima Y | yleissiima
KooTek | Lapio L | kevyt yleislapio
Giardino | |
(6 rows)
I get only this selected
SELECT tmnimi, tnimi, kuvaus
FROM tuote
CROSS JOIN tuotemerkki
WHERE tuote.tmtunnus = tuotemerkki.tmtunnus;
tmnimi | tnimi | kuvaus
--------+-----------------------+--------------------
McCee | Trimmeri TRCee | tehokas 4-tahtinen
McCee | Trimmerisiima Cee | laadukas siima
McCee | Moottorisaha MSCee RR | robusti ja raskas
KooTek | Trimmerisiima Y | yleissiima
KooTek | Lapio L | kevyt yleislapio
(5 rows)
Use a left join between the two tables:
SELECT
t1.tmnimi,
t2.tnimi,
t2.kuvaus
FROM tuotemerkki t1
LEFT JOIN tuote t2
ON t1.tmtunnus = t2.tmtunnus
When you left join from tuotemerkki to tuote, then every record in the former table is guaranteed to appear in the result set. Since the Giardino record does not match to anything in the tuote table, all the columns from that table would have a NULL value for the Giardino record.
Can not understand why aggregateQuery always returns an empty result. Tried to test in aql, the same problem: 0 rows in set.
Indexes are all there.
aql> show indexes
+---------------+-------------+-----------+------------+-------+------------------------------+-------------+------------+-----------+
| ns | bin | indextype | set | state | indexname | path | sync_state | type |
+---------------+-------------+-----------+------------+-------+------------------------------+-------------+------------+-----------+
| "test" | "name" | "NONE" | "profiles" | "RW" | "inx_test_name" | "name" | "synced" | "STRING" |
| "test" | "age" | "NONE" | "profiles" | "RW" | "inx_test_age" | "age" | "synced" | "NUMERIC" |
aql> select * from test.profiles
+---------+-----+
| name | age |
+---------+-----+
| "Sally" | 19 |
| 20 | |
| 22 | |
| 28 | |
| "Ann" | 22 |
| "Bob" | 22 |
| "Tammy" | 22 |
| "Ricky" | 20 |
| 22 | |
| 19 | |
+---------+-----+
10 rows in set (0.026 secs)
aql> AGGREGATE mystream.avg_age() ON test.profiles WHERE age BETWEEN 20 and 29
0 rows in set (0.004 secs)
It seems that you are trying the example here.
There are two problems about the udf script. I paste the code of the lua script :
function avg_age(stream)
local function female(rec)
return rec.gender == "F"
end
local function name_age(rec)
return map{ name=rec.name, age=rec.age }
end
local function eldest(p1, p2)
if p1.age > p2.age then
return p1
else
return p2
end
end
return stream : filter(female) : map(name_age) : reduce(eldest)
end
First, there is no bin named 'gender' in your set, so you got 0 rows after aggregateQuery.
Second, this script isn't doing exactly what the function name 'avg_age' means, it just return the eldest record with name and age.
I paste my code bellow, it just replace the reduce func, and alert the map and filter func to meat the demand. You can just skip the filter process.
function avg_age(stream)
count = 0
sum = 0
local function female(rec)
return true
end
local function name_age(rec)
return rec.age
end
local function avg(p1, p2)
count = count + 1
sum = sum + p2
return sum / count
end
return stream : filter(female) : map(name_age) : reduce(avg)
end
The output looks like bellow :
AGGREGATE mystream.avg_age() ON test.avgage WHERE age BETWEEN 20 and 29
+---------+
| avg_age |
+---------+
| 22 |
+---------+
1 row in set (0.001 secs)
I have a following statement and it generates the mentioned output by averaging data within every 20 minutes of range.
Statement :
SELECT record_no, date_time,
ROUND(AVG(UNIX_TIMESTAMP(date_time))) AS time_value,
ROUND(AVG(ph1_active_power),4) AS p1,
ROUND(AVG(ph2_active_power),4) AS p2,
ROUND(AVG(ph3_active_power),4) AS p3
FROM powerpro1
GROUP BY date_time DIV 2000
Portion of the output
+-----------+---------------------+------------+---------+----------+----------+
| record_no | date_time | time_value | p1 | p2 | p3 |
+-----------+---------------------+------------+---------+----------+----------+
| 1 | 2014-12-01 00:00:00 | 1417372770 | 72.6242 | -68.7428 | -72.6242 |
| 21 | 2014-12-01 00:20:00 | 1417373970 | 71.6624 | -69.7448 | -71.6624 |
| 41 | 2014-12-01 00:40:00 | 1417375170 | 70.6869 | -70.7333 | -70.6869 |
| 61 | 2014-12-01 01:00:00 | 1417376370 | 69.6977 | -71.7082 | -69.6977 |
| 81 | 2014-12-01 01:20:00 | 1417377570 | 68.6952 | -72.6692 | -68.6952 |
| 101 | 2014-12-01 01:40:00 | 1417378770 | 67.6794 | -73.6162 | -67.6794 |
| 121 | 2014-12-01 02:00:00 | 1417379970 | 66.6505 | -74.549 | -66.6505 |
| 141 | 2014-12-01 02:20:00 | 1417381200 | 65.5825 | -75.4901 | -65.5825 |
+-----------+---------------------+------------+---------+----------+----------+
According to the no of records in the table named "powerpro1", the above query selects 1368 records when the executing. (May be increased in the future when receiving new records)
My requirement is to create a highchart using time_value for the x-axis and p1, p1 and p3 for the y-axis. But I needs to limit the no of points in the x-axis.
Can anyone like to help me to show this 1368 points by 1000 points in the chart
Unfortunately have no this kind of apporixmation, only in reverse order (I mean datagrouping, if you have 100 points, return i.e 10). So you need to calcualte it on your own and push to your data all 1000 points.