I'm trying to create a Json format from a postgresql request.
Firstly I have used Rails to request my database in the format.json block of my controller and then used a json.builder file to format the json view. It worked until my requests return hundreds of thousands rows, so I searched how to optimize the json creation, avoiding all the ActiveRecord stack.
To do this I am using Postgresql 9.6 json functions to get directly my data in the right format, which is for example :
SELECT array_to_json('{{1157241840,-1.95},{1157241960,-1.96}}'::float[]);
[[1157241840, -1.95], [1157241960, -1.96]]
But using data from this kind of request :
SELECT date,value FROM measures;
The best I could obtain was something like this :
SELECT array_to_json(array_agg(t)) FROM (SELECT date,value FROM measures) t;
Resulting in :
[
{"date":"1997-06-13T19:12:00","value":1608.4},
{"date":"1997-06-13T19:12:00","value":-0.6}
]
which is quite different ... How would you build this SQL request ?
Thanks for your help !
My measures table look like this :
id | value | created_at | updated_at | parameter_id | quality_id | station_id | date | campain_id | elevation | sensor_id | comment_id
--------+-------+----------------------------+----------------------------+--------------+------------+------------+---------------------+------------+-----------+-----------+------------
799634 | -1.99 | 2017-02-21 09:41:09.062795 | 2017-02-21 09:41:09.118807 | 2 | | 1 | 2006-06-26 23:24:00 | 1 | -5.0 | |
1227314 | -1.59 | 2017-02-21 09:44:12.032576 | 2017-02-21 09:44:12.088311 | 2 | | 1 | 2006-11-30 19:48:00 | 1 | -5.0 | |
1227315 | 26.65 | 2017-02-21 09:44:12.032576 | 2017-02-21 09:44:12.088311 | 3 | | 1 | 2006-11-30 19:48:00 | 1 | -5.0 | |
If you need array of array you need to use json_build_array:
SELECT json_agg(json_build_array(date,value)) FROM measures;
If you want convert timestamp to epoch:
SELECT json_agg(json_build_array(extract(epoch FROM date)::int8, value)) FROM measures;
For test:
WITH measures AS (
SELECT 1157241840 as date, -1.95 as value
UNION SELECT 1157241960, -1.96
UNION SELECT 1157241980, NULL
)
SELECT json_agg(json_build_array(date,value)) FROM measures;
json_agg
----------------------------------------------------------------
[[1157241840, -1.95], [1157241960, -1.96], [1157241980, null]]
create table measures (date timestamp, value float);
insert into measures (date, value) values
(to_timestamp(1157241840),-1.95),
(to_timestamp(1157241960),-1.96);
select array_to_json(array_agg(array[extract(epoch from date), value]::float[]))
from measures
;
array_to_json
-----------------------------------------
[[1157241840,-1.95],[1157241960,-1.96]]
Related
I have three models in my rails project, namely User, Game, Match
user can create many matches on each game
so table structure for matches is like
table name: game_matches
+----+---------+---------+-------------+------------+
| id | user_id | game_id | match_type | match_name |
+----+---------+---------+-------------+------------+
| 1 | 1 | 1 | practice | |
| 2 | 3 | 2 | challenge | |
| 3 | 1 | 1 | practice | |
| 4 | 3 | 2 | challenge | |
| 5 | 1 | 1 | challenge | |
| 6 | 3 | 2 | practice | |
+----+---------+---------+-------------+------------+
i want to generate match_name based on user_id, game_id and match_type values
for example match_name should be create like below
+----+---------+---------+-------------+-------------+
| id | user_id | game_id | match_type | match_name |
+----+---------+---------+-------------+-------------+
| 1 | 1 | 1 | practice | Practice 1 |
| 2 | 3 | 2 | challenge | Challenge 1 |
| 3 | 1 | 1 | practice | Practice 2 |
| 4 | 3 | 2 | challenge | Challenge 2 |
| 5 | 1 | 1 | challenge | Challenge 1 |
| 6 | 3 | 2 | practice | Practice 1 |
+----+---------+---------+-------------+-------------+
How can i achieve this auto incremental value in my rails model during new record creation.
Any help suggestions appreciated.
Thanks in advance.
I see two ways you can solve this:
DB: trigger
Rails: callback
Trigger (assuming Postgres):
DROP TRIGGER IF EXISTS trigger_add_match_name ON customers;
DROP FUNCTION IF EXISTS function_add_match_name();
CREATE FUNCTION function_add_match_name()
RETURNS trigger AS $$
BEGIN
NEW.match_name := (
SELECT
CONCAT(game_matches.match_type, ' ', COALESCE(count(*), 0))
FROM game_matches
WHERE game_matches.user_id = NEW.user_id AND game_matches.match_type = NEW.match_type
);
RETURN NEW;
END
$$ LANGUAGE 'plpgsql';
CREATE TRIGGER trigger_add_match_name
BEFORE INSERT ON game_matches
FOR EACH ROW
EXECUTE PROCEDURE function_add_match_name();
Please note that this is not tested.
Rails
class GameMatch
before_create :assign_match_name
private
def assign_match_name
number = GameMatch.where(user_id: user_id, match_type: match_type).count || 0
name = "#{match_type} #{number + 1}"
self.match_name = name
end
end
Again, untested.
I'd prefer the trigger solution since callbacks can be skipped or ommited altogether when inserting via pure SQL.
Also I'd add "match_number" column instead of the full name and then construct the name within the Model or a Decorator or a view Helper (more flexible, I18n) but the logic behind stays the same.
You should retrieve the last match_name for these user and game, split it, increase the counter and join back with a space. Unfortunately, SQL does not provide SPLIT function, so somewhat like below would be a good start:
SELECT match_name
FROM match_name
WHERE user_id = 3
AND game_id = 2
ORDER BY id DESC
LIMIT 1
I would actually better create a match_number column of type INT to keep the number by type and produce a name by concatenation the type with this number.
I have InfluxDB measurement currently set up with following "schema":
+----+-------------+-----------+
| ts | cost(field) | type(tag) |
+----+-------------+-----------+
| 1 | 10 | 'a' |
| 1 | 20 | 'b' |
| 2 | 12 | 'a' |
| 2 | 18 | 'b' |
| 2 | 22 | 'c' |
+------------------+-----------+
I am trying to write a query that will group my table by timestamp and get a delta between field values of two different tags. If I want to get delta between tag 'a' and tag 'b', it will give me following result (please not that I ignore tag 'c'):
+----+-----------+------------+
| ts | type(tag) | delta_cost |
+----+-----------+------------+
| 1 | 'a' | 10 |
| 2 | 'b' | 6 |
+----+-----------+------------+
Is it something Influx can do or am I using the wrong tool?
Just managed to answer my own question. While one of the obvious ways would be performing self-join, Influx does not support joins anymore. We can, however, use nested selects in a following format:
SELECT MEAN(cost_a) - MEAN(cost_b) as delta_cost
FROM
(SELECT cost as cost_a, tag, tablename where tag='a'),
(SELECT cost as cost_b, tag, tablename where tag='b')
GROUP BY time(60s)
Since I am getting my data every 60 seconds anyway, and I have a guarantee of just one point per tag per 60 seconds, I can use GROUP BY and take MEAN without any problems
Currently, for a recurring search with different parameters, I have this ActiveRecord query built:
current_user.documents.order(:updated_at).reverse_order.includes(:groups,:rules)
Now, usually I tack on a where clause to this to perform this search. However, I now need to do a search through the jsonb field for all rows that have a certain value as in the key:value pair. I've been able to do something a similar to that in my SQL, with this syntax (the data field will only be exactly two levels nested):
SELECT
*
FROM
(SELECT
*
FROM
(SELECT
*
FROM
documents
) A,
jsonb_each(A.data)
) B,
jsonb_each_text(B.value) ASC C
WHERE
C.value = '30';
However, I want to use the current ActiveRecord search to make this query (which includes the groups/rules eager loading).
I'm struggling with the use of the comma, which I understand is an implicit join, which is executed before explicit joins, so when I try something like this:
select * from documents B join (select * from jsonb_each(B.data)) as A on true;
ERROR: invalid reference to FROM-clause entry for table "b"
LINE 1: ...* from documents B join (select * from jsonb_each(B.data)) a...
^
HINT: There is an entry for table "b", but it cannot be referenced from this part of the query.
But I don't understand how to reference the complete "table" the ActiveRecord query I have creates before I make a joins call, as well as make use of the comma syntax for implicit joins to work.
Also, I'm an SQL amateur, so if you see some improvements or other ways to do this, please do tell.
EDIT: Description of documents table:
Table "public.documents"
Column | Type | Modifiers | Storage | Stats target | Description
------------+-----------------------------+--------------------------------------------------------+----------+--------------+-------------
id | integer | not null default nextval('documents_id_seq'::regclass) | plain | |
document_id | character varying | | extended | |
name | character varying | | extended | |
size | integer | | plain | |
last_updated| timestamp without time zone | | plain | |
user_id | integer | | plain | |
created_at | timestamp without time zone | | plain | |
updated_at | timestamp without time zone | | plain | |
kind | character varying | | extended | |
uid | character varying | | extended | |
access_token_id | integer | | plain | |
data | jsonb | not null default '{}'::jsonb | extended | |
Indexes:
"documents_pkey" PRIMARY KEY, btree (id)
```
Sample rows, first would match a search for '30' (data is the last field):
2104 | 24419693037 | LsitHandsBackwards.jpg | | | 1 | 2017-06-25 21:45:49.121686 | 2017-07-01 21:32:37.624184 | box | 221607127 | 15 | {"owner": {"born": "to make history", "price": 30}}
2177 | /all-drive/uml flows/typicaluseractivity.svg | TypicalUserActivity.svg | 12375 | 2014-08-11 02:21:14 | 1 | 2017-07-07 14:00:11.487455 | 2017-07-07 14:00:11.487455 | dropbox | 325694961 | 20 | {"owner": {}}
You can use a query similar to the one you already showed:
SELECT
d.id, d.data
FROM
documents AS d
INNER JOIN json_each(d.data) AS x ON TRUE
INNER JOIN json_each(x.value) AS y ON TRUE
WHERE
cast(y.value as text) = '30';
Assuming your data would be the following one:
INSERT INTO documents
(data)
VALUES
('{"owner": {"born": "to make history", "price": 30}}'),
('{"owner": {}}'),
('{"owner": {"born": "to make history", "price": 50}, "seller": {"worth": 30}}')
;
The result you'd get is:
id | data
-: | :---------------------------------------------------------------------------
1 | {"owner": {"born": "to make history", "price": 30}}
3 | {"owner": {"born": "to make history", "price": 50}, "seller": {"worth": 30}}
You can check it (together with some step-by-step looks at the data) at dbfiddle here
Im trying to find an efficient way to solve the problem:
I need to find all rows in a table where there is another row with an opposite column value.
For example I have transactions with columns id and amount
| id | amount |
|----|--------|
| 1 | 1 |
| 2 | -1 |
| 3 | 2 |
| 4 | -2 |
| 5 | 3 |
| 6 | 4 |
| 7 | 5 |
| 8 | 6 |
The query should return only the first 4 rows:
| id | amount |
|----|--------|
| 1 | 1 |
| 2 | -1 |
| 3 | 2 |
| 4 | -2 |
My current solution is terribly efficient as I am going through 1000's of transactions:
transactions.find_each do |transaction|
unless transactions.where("amount = #{transaction.amount * -1}").count > 0
transactions = transactions.where.not(amount: transaction.amount).order("# amount DESC")
end
end
transactions
Are there any built in Rails or Postgresql functions that could help with this?
Use following query:
SELECT DISTINCT t1.*
FROM transactions t1
INNER JOIN transactions t2 ON t1.amount = t2.amount * -1;
SELECT * FROM the_table t
WHERE EXISTS (
SELECT * FROM the_table x
WHERE x.amount = -1*t.amount
-- AND x.amount > t.amount
);
Consider storing an absolute value indexed column then query for the positive value. Postgres has an absolute value function; but I think the beauty of ActiveRecord is that Arel abstracts away the SQL. DB specific SQL can be a pain if you change later.
There is type called abs which will return irrespective of symobol. From my example data is the table name
SELECT id,amount FROM DATA WHERE id = ABS(amount)
This is the sample test table
Here is the output
I have 1 entity called Project with attribute of:
ProjectNo
ProjectNoRef (reference to ProjectNo attribute)
Month
Example:
===================================
| ProjectNo | ProjectNoRef| Month |
===================================
| Proj01 | NULL | 1 |
| Proj02 | NULL | 1 |
| Proj03 | NULL | 1 |
| Proj04 | NULL | 1 |
| Proj05 | Proj01 | 2 |
| Proj06 | NULL | 2 |
===================================
I want to fetch all projects within certain month that is not referenced to other project.
From above example, it will result 3 for month 1 and 2 for month 2.
What is the best way to do this in Core Data? I don't want to do double fetching as there will be thousand projects.
Your object graph should include an optional relationship between a project and its sub-project(s).
Then it becomes a simple matter to fetch all projects which don't have sub-projects, and group the results by month.