I need create (kibana 4.1) a new dynamique field (string) for use in graph with 'significant terms'
How write equivalent SQL script on kibana interface or E.S ?
Exemple MSSQL:
Select reason_txt, response_txt,
(CASE WHEN reason_txt='none' THEN response_txt else reason_txt end) as newfield
thanks you very much
Related
I have been following along with Felipe Hoffa's blog post "Easy pivot() in BigQuery" (https://towardsdatascience.com/easy-pivot-in-bigquery-one-step-5a1f13c6c710) and I've been able to successfully call his procedure and replicate his example calculations. However, because the data I'm ultimately interested in are hosted in the EU, I can't call his procedure verbatim and have been unsuccessfully trying to create and run a copy of the code into my own personal BigQuery project folder as a result.
As best I can tell, the steps involved are.
Copy the code here https://github.com/fhoffa/code_snippets/blob/5163b921398ee29a8010c164a17af05268ac8639/util/pivot.sql
Update the project ID and dataset (e.g. swap out every instance of "`fhoffa.x." with my BigQuery info "blah.matt.") and create the stored procedure in my own BigQuery account
Run the code, adjusting for the new location.
Something like:
CALL `*blah*.matt.pivot`(
'bigquery-public-data.iowa_liquor_sales.sales' # source table
, 'blah.matt.test2' # destination table
, ['date'] # row_ids
, 'store_number' # pivot_col_name
, 'sale_dollars' # pivot_col_value
, 5 # max_columns
, 'SUM' # aggregation
, '' # optional_limit
);
Unfortunately, when I do this, I encounter the following error.
Invalid EXECUTE IMMEDIATE sql string SELECT STRING_AGG(' SUM(IF('||#pivot_col_name||'="'||x.value||'", '||#pivot_col_value||', null)) e_'||blah.matt.normalize_col_name(x.value)) FROM UNNEST(( SELECT APPROX_TOP_COUNT(store_number, #max_columns) FROM bigquery-public-data.iowa_liquor_sales.sales)) x, Syntax error: Expected ")" but got identifier "matt" at [blah.matt.pivot:5:5]
Can anyone please advise? I'm a novice at Big Query, and can't tell where I've gone off the rails. Apologies if this is a super basic question.
Thank you very much for your time and help!
Regards,
Matt
Most likely your problem is related to not properly referencing your project.dataset or just simply having typo, etc. - to be on safe side do as below (just copy paste from below)
CREATE OR REPLACE FUNCTION
`blah.matt.normalize_col_name`(col_name STRING) AS (
REGEXP_REPLACE(col_name,r'[/+#|]', '_')
);
CREATE OR REPLACE PROCEDURE `blah.matt.pivot`(
table_name STRING
, destination_table STRING
, row_ids ARRAY<STRING>
, pivot_col_name STRING
, pivot_col_value STRING
, max_columns INT64
, aggregation STRING
, optional_limit STRING
)
BEGIN
DECLARE pivotter STRING;
EXECUTE IMMEDIATE (
"SELECT STRING_AGG(' "||aggregation
||"""(IF('||#pivot_col_name||'="'||x.value||'", '||#pivot_col_value||', null)) e_'||`blah.matt.normalize_col_name`(x.value))
FROM UNNEST((
SELECT APPROX_TOP_COUNT("""||pivot_col_name||", #max_columns) FROM `"||table_name||"`)) x"
) INTO pivotter
USING pivot_col_name AS pivot_col_name, pivot_col_value AS pivot_col_value, max_columns AS max_columns;
EXECUTE IMMEDIATE (
'CREATE OR REPLACE TABLE `'||destination_table
||'` AS SELECT '
||(SELECT STRING_AGG(x) FROM UNNEST(row_ids) x)
||', '||pivotter
||' FROM `'||table_name||'` GROUP BY '
|| (SELECT STRING_AGG(''||(i+1)) FROM UNNEST(row_ids) WITH OFFSET i)||' ORDER BY '
|| (SELECT STRING_AGG(''||(i+1)) FROM UNNEST(row_ids) WITH OFFSET i)
||' '||optional_limit
);
END;
So, after above procedures created - below call should now work as expected
CALL `blah.matt.pivot`(
'bigquery-public-data.iowa_liquor_sales.sales' # source table
, 'blah.matt.test2' # destination table
, ['date'] # row_ids
, 'store_number' # pivot_col_name
, 'sale_dollars' # pivot_col_value
, 5 # max_columns
, 'SUM' # aggregation
, '' # optional_limit
);
Note: just simply replace blah and matt with respective proper names of project and dataset
P.S. Obviously you can avoid all this headache and just call Felipe's publicly available proc - fhoffa.x.pivot
But I agree, having those locally will allow to adjust those procs to your potentially specific needs and further improve and code - which I see still quite a room here :o)
MATCH (prs:Issue)-[:REPORTED_BY]-(custs)
MATCH (prs)-[:CLOSED_ON]-(cls:IssueClosedDate)
MATCH (prs)-[:REPORTED_BY]->(custNode:Customer)
MATCH (prs)-[:APP_FUN_CAT]-(afc:AppFunCat)
MATCH (prs)-[:REPORTED_IN]-(release:Release)
WHERE afc.func STARTS WITH 'WEB' AND NOT(cls.closedDate = '' ) AND afc.appName STARTS WITH 'SOCKET'
AND apoc.date.parse(cls.closedDate,'s', 'MM/dd/yyyy') >= apoc.date.parse('01/01/2014','s', 'MM/dd/yyyy')
AND apoc.date.parse(cls.closedDate,'s', 'MM/dd/yyyy') <= apoc.date.parse('06/13/2017','s', 'MM/dd/yyyy')
AND afc.cat IN ["ALL","NEW","SOFTWARE","UNDETERMINED"]
RETURN prs.prId AS prList, custs.customerName AS customer, afc.cat AS category, cls.closedDate AS prClosedDate, release.relName as releaseName `
The above query gives me result shown below:
"prList" "funName" "year" "afc.appName" "afc.cat" "cls.closedDate"
7371322 "WEB" "2015" "SOCKET" "SOFTWARE" "4/27/2015"
8277662 "WEB" "2015" "SOCKET" "SOFTWARE" "9/24/2015"
7513015 "WEB" "2015" "SOCKET" "SOFTWARE" "9/24/2015"
This result is not correct if I check with data base. It should have given more number of list than this.
It is found that if I remove either of the filter
afc.appName STARTS WITH 'SOCKET' or
apoc.date.parse(cls.closedDate,'s', 'MM/dd/yyyy') <= apoc.date.parse('06/13/2017','s', 'MM/dd/yyyy') or
fc.cat IN ["ALL","NEW","SOFTWARE","UNDETERMINED"]
the result I get is correct. So I can say that the database is build properly. Though the above query is showing the three result there are more number of columns.
Is there any limitation from neo4j database that we cannot do this?
Can anybody suggest how can I resolve this problem?
As said by InverseFalcon in comment, version might be the issue. We updated the version as mention below:
Neo4j Browser version: 3.0.1 to 3.2.8
Neo4j Server version: 3.2.0 to 3.4.8 (community)
apoc-3.2.3.5-all.exe to apoc-3.4.0.3-all.exe
Now all the filters are working good. Problem is solved.
I'm using Legacy indexing (now called Manual indexing). After migration from Neo4j 2 to version 3 I have some problems with numeric sorting.
Example of correct statement in Neo4j 2:
queryContext.sort(new Sort(new SortField(AGE, SortField.INT, false)));
This stament should be changed for Neo4j 3 (Lucene 5):
queryContext.sort(new Sort(new SortField(AGE, SortField.Type.INT, false)));
But if you use this sort statement you will get an exception:
java.lang.IllegalStateException: unexpected docvalues type SORTED_SET for field 'firstName' (expected=SORTED). Use UninvertingReader or index with docvalues.
at org.apache.lucene.index.DocValues.checkField(DocValues.java:208)
at org.apache.lucene.index.DocValues.getSorted(DocValues.java:264)
at org.apache.lucene.search.FieldComparator$TermOrdValComparator.getSortedDocValues(FieldComparator.java:762)
at org.apache.lucene.search.FieldComparator$TermOrdValComparator.getLeafComparator(FieldComparator.java:767)
at org.apache.lucene.search.FieldValueHitQueue.getComparators(FieldValueHitQueue.java:183)
at org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector.getLeafCollector(TopFieldCollector.java:164)
at org.neo4j.kernel.api.impl.index.collector.DocValuesCollector.replayTo(DocValuesCollector.java:297)
at org.neo4j.kernel.api.impl.index.collector.DocValuesCollector.getTopDocs(DocValuesCollector.java:275)
at org.neo4j.kernel.api.impl.index.collector.DocValuesCollector.getIndexHits(DocValuesCollector.java:150)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.search(LuceneLegacyIndex.java:346)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.query(LuceneLegacyIndex.java:261)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.query(LuceneLegacyIndex.java:205)
at org.neo4j.index.impl.lucene.legacy.LuceneLegacyIndex.query(LuceneLegacyIndex.java:217)
at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.nodeLegacyIndexQuery(StateHandlingStatementOperations.java:1440)
at org.neo4j.kernel.impl.api.OperationsFacade.nodeLegacyIndexQuery(OperationsFacade.java:1162)
at org.neo4j.kernel.impl.coreapi.LegacyIndexProxy$Type$1.query(LegacyIndexProxy.java:83)
at org.neo4j.kernel.impl.coreapi.LegacyIndexProxy.query(LegacyIndexProxy.java:365)
I think this is caused by new added statement in Neo4j indexer class (Neo4j is indexing field for sorting automatically now?). See in:
org.neo4j.index.impl.lucene.legacy.IndexType CustomType addToDocument( Document document, String key, Object value )
new line:
document.add( instantiateSortField( key, value ) );
and method instantiateSortField is creating SortedSetDocValuesField
So I changed my code to:
queryContext.sort(new Sort(new SortedSetSortField(AGE, false)));
This runs OK but sorting is not working because numbers are sorted as string. I see that "value" parameter is String every time in method "addToDocument". I think the root cause is explained it this old comment:
see comment in class org.neo4j.index.impl.lucene.legacy.IndexType CustomType
// TODO We should honor ValueContext instead of doing value.toString() here.
// if changing it, also change #get to honor ValueContext.
Am I missing some new way how to index, search and sort data in Neo4j 3 or this is really problem that values are indexed as string in Neo4j?
Simple unit test for Neo4j 2 and Neo4j 3 can be downloaded
Solution added by MishaDemianenko at GH issue
I want to create basically an update query on Open Office Base (the same way with Ms ACCESS).
Base does not typically use update queries (but see below). Instead, the easiest way to do an update command is to go to Tools -> SQL. Enter something similar to the following, then press Execute:
UPDATE "Table1" SET "Value" = 'BBB' WHERE ID = 0
The other way is to run the command with a macro. Here is an example using Basic:
Sub UpdateSQL
REM Run an SQL command on a table in LibreOffice Base
Context = CreateUnoService("com.sun.star.sdb.DatabaseContext")
databaseURLOrRegisteredName = "file:///C:/Users/JimStandard/Desktop/New Database.odb"
Db = Context.getByName(databaseURLOrRegisteredName )
Conn = Db.getConnection("","") 'username & password pair - HSQL default blank
Stmt = Conn.createStatement()
'strSQL = "INSERT INTO ""Table1"" (ID,""Value"") VALUES (3,'DDD')"
strSQL = "UPDATE ""Table1"" SET ""Value"" = 'CCC' WHERE ID = 0"
Stmt.executeUpdate(strSQL)
Conn.close()
End Sub
Note that the data can also be modified with a form or by editing the table directly.
Under some circumstances it is possible to create an update query. I couldn't get this to work with the default built-in HSQLDB 1.8 engine, but it worked with MYSQL.
In the Queries section, Create Query in SQL View
Click the toolbar button to Run SQL Command directly.
Enter a command like the following:
update mytable set mycolumn = 'This is some text.' where ID = 59;
Hit F5 to run the query.
It gives an error that The data content could not be loaded, but it still performs the update and changes the data. To get rid of the error, the command needs to return a value. For example, I created this stored procedure in MYSQL:
DELIMITER $$
CREATE PROCEDURE update_val
(
IN id_in INT,
IN newval_in VARCHAR(100)
)
BEGIN
UPDATE test_table SET value = newval_in WHERE id = id_in;
SELECT id, value FROM test_table WHERE id = id_in;
END
$$
DELIMITER ;
Then this query in LibreOffice Base modifies the data without giving any errors:
CALL update_val(2,'HHH')
See also:
https://forum.openoffice.org/en/forum/viewtopic.php?f=5&t=75763
https://forum.openoffice.org/en/forum/viewtopic.php?f=61&t=6655
https://ask.libreoffice.org/en/question/32700/how-to-create-an-update-query-in-base-sql/
Modifying table entries from LibreOffice Base, possible?
I have prepared this query in access and it is working ok.
SELECT Kod, Unvan, CepTel, Telefon, (SELECT Sum(Borc.Tutar) from borc
where carikod=kod) AS BorcTutar, (SELECT Sum(Alacak.Tutar) from Alacak
where carikod=kod) AS AlacakTutar
FROM cari;
But when want to use query in delphi i have to use serverside corsorlocation and static cursor.
Then delphi throw an exception and says "e-fail" , "invalid argument" and diffrent kind of messages when i change the cursor locations and types..
It is working wit clientside cursor but return 0 for sum()..
How to use correctly this query...
I have converted the "amount" fields the currency, then delphi choose correct type for them (tbcdfield beside tintegerfield)....
report is working now