we have a strange issue, where esper query is only partial executed...
select cseShutDownAlarm.INSTANCEID as INSTANCEID, swtDownAlarm.source
from Alarm(severity.getValue()=5, eventType.getValue() = 'SWT_SWITCH_DOWN').win:time(60 sec) as swtDownAlarm,
sql:stormdb['select id as INSTANCEID, source as SOURCE
from Alarm where EVENTTYPE_VALUE =\'cseShutDownNotify\' and source = ${swtDownAlarm.source} and severity !=5'] as cseShutDownAlarm,
sql:stormdb['insert into wcsdba.dyinggasp (parameter, value) VALUES (\'cseShutDownAlarm.SOURCE\', ${cseShutDownAlarm.SOURCE}) '],
sql:stormdb['insert into wcsdba.dyinggasp (parameter, value) VALUES (\'swtDownAlarm.source\', ${swtDownAlarm.source}) '] where swtDownAlarm.source = cseShutDownAlarm.SOURCE
and in the DB, we see that:
SQL> /
PARAMETER VALUE
cseShutDownAlarm.SOURCE 172.16.148.48
cseShutDownAlarm.SOURCE 172.16.148.48
SQL>
but second source (swtDownAlarm.source) is not printed...
If I switch the order then the other one will be inserted only.
any reason why it does not execute both inserts? also the rest of the condition is not checked... as the source on both are identical, but condition not fulfilled.
Thanks,
fcbman
Joins are meant to be the place for event stream joins against SQL database for purpose of querying. The "insert into" isn't a query and would never return rows. And if an inner join doesn't return rows it ends early (outer join would be right then)
We are currently running a number of hand-crafted and optimized OData queries on Exact Online using Python. This runs on several thousand of divisions. However, I want to migrate them to Invantive SQL for ease of maintenance.
But some of the optimizations like explicit orderby in the OData query are not forwarded to Exact Online by Invantive SQL; they just retrieve all data or the top x and then do an orderby.
Especially for maximum value determination that can be a lot slower.
Simple sample on small table:
https://start.exactonline.nl/api/v1/<<division>>/financial/Journals?$select=BankAccountIBAN,BankAccountDescription&$orderby=BankAccountIBAN desc&$top=5
Is there an alternative to optimize the actual OData queries executed by Invantive SQL?
You can either use the Data Replicator or send the hand-craft OData query through a native platform request, such as:
insert into NativePlatformScalarRequests
( url
, orig_system_group
)
select replace('https://start.exactonline.nl/api/v1/{division}/financial/Journals?$select=BankAccountIBAN,BankAccountDescription&$orderby=BankAccountIBAN desc&$top=5', '{division}', code)
, 'MYSTUFF-' || code
from systempartitions#datadictionary
limit 100 /* First 100 divisions. */
create or replace table exact_online_download_journal_top5#inmemorystorage
as
select jte.*
from ( select npt.result
from NativePlatformScalarRequests npt
where npt.orig_system_group like 'MYSTUFF-%'
and npt.result is not null
) npt
join jsontable
( null
passing npt.result
columns BankAccountDescription varchar2 path 'd[0].BankAccountDescription'
, BankAccountIBAN varchar2 path 'd[0].BankAccountIBAN'
) jte
From here on you can use the in memory table, such as:
select * from exact_online_download_journal_top5#inmemorystorage
But of course you can also 'insert into sqlserver'.
I am using EF6.1.2, database-first.
In one situation, this linq to sql query:
var data = from dl in ctx.ObjectContext.DocumentLines.Where(drow => drow.DocumentId == documentId)
join di in ctx.ObjectContext.DocumentsInfoes on dl.DocumentId equals di.Id
join ba in ctx.ObjectContext.Addresses on dl.DeliveryAddressId equals ba.Id into tba
from xba in tba.DefaultIfEmpty()
join sc in ctx.ObjectContext.SalesCategories on dl.SalesCategoryId equals sc.Id into tsc
from xsc in tsc.DefaultIfEmpty()
select new { Line = dl, DocumentInfo = di, DeliveryAddress = xba, SalesCategory = xsc };
var abc = data.ToList();
is not working.
The first documentline is correctly bound, but subsequent rows have some NULL or zero properties. If I run the generated SQL query, all columns are present and correct.
Is this a bug in Entity Framework? I have downloaded the EF source and I want to step into it to attempt to understand what's happening but don't know where to start looking.
The documentline table has a timestamp Version column for concurrency purposes. I have read a few similar issues where problems with a timestamp column, or other columns, (see Entity Framework does not pull through data from some columns) could cause subsequent columns to fail.
EDIT
On further investigation, it seems that, in the one call to this code that exhibits the problem, I can step over var abc = data.ToList(); without hitting any of the breakpoints in the setters of the properties of the DocumentLine entity, in the data model. Bizarre. All other calls to the code hit the breakpoints. Why isn't it loading the entities?
I am testing a Ext JS application (Client Side) and Play Framework (Service Side).
I am using a grid in Ext JS with pagination.
The pagination part requires to send URL Query Parameters to my Play! server. This is no big deal, but how to process these parameters in the SQL Statement??
Example:
First request:
http://myDomain:9000/GetUsers?_dc=123456789&page=1&start=0&limit=25
Second reqeust:
http://myDomain:9000/GetUsers?_dc=123456789&page=2&start=25&limit=25
My thoughts:
Normally in SQL you can set the TOP results:
SELECT TOP 25 FROM USERS
But how to translate the second request into a Sql query?
Thank you for taking time to help me out!
======>>
EDIT: I am developing on SQL Server 2008, but I want this working on Sql Server 2005 or higher and Oracle 9 and higher :-)
Since you're using the Play! framework, what you should do is have a proper model, with entities representing your SQL tables. Then receiving a range of results is built in:
// 25 max users start at 25
List<User> users = User.all().from(25).fetch(25);
You should also look at the pagination module. I haven't tested it, but it looks like exactly what you want.
You could try something like:
WITH Query_1 AS (
SELECT
Field1, Field2, etc
ROW_NUMBER() OVER (ORDER BY Field1, Field2, etc) AS RowID
FROM Table
WHERE x=y
)
SELECT * FROM Query_1 WHERE RowID >= #start
AND RowID < #start + #limit
Of course ROW_NUMBER didn't exist back in SQL 2000 but since you've not told us which SQL you're working with I'm assuming something newer.
EDIT:
I've narrowed my mysql wait timeout down to this line:
IF #resultsFound > 0 THEN
INSERT INTO product_search_query (QueryText, CategoryId) VALUES (keywords, topLevelCategoryId);
END IF;
Any idea why this would cause a problem? I can't work it out!
I've written a stored proc to search for products in certain categories, due to certain constraints I came across, I was unable to do what I wanted (limiting, but whilst still returning the total number of rows found, with sorting, etc..)
It's meant splits up a string of category Ids, from 1,2,3 in to a temporary table, then builds the full-text search query based on sorting options and limits, executes the query string and then selects out the total number of results.
Now, I know I'm no MySQL guru, very far from it, I've got it working, but I keep getting time outs with product searches etc. So I'm thinking this may be causing some kind of problem?
Does anyone have any ideas how I can tidy this up, or even do it in a much better way that I probably don't know about?
Thanks.
DELIMITER $$
DROP PROCEDURE IF EXISTS `product_search` $$
CREATE DEFINER=`root`#`localhost` PROCEDURE `product_search`(keywords text, categories text, topLevelCategoryId int, sortOrder int, startOffset int, itemsToReturn int)
BEGIN
declare foundPos tinyint unsigned;
declare tmpTxt text;
declare delimLen tinyint unsigned;
declare element text;
declare resultingNum int unsigned;
drop temporary table if exists categoryIds;
create temporary table categoryIds
(
`CategoryId` int
) engine = memory;
set tmpTxt = categories;
set foundPos = instr(tmpTxt, ',');
while foundPos <> 0 do
set element = substring(tmpTxt, 1, foundPos-1);
set tmpTxt = substring(tmpTxt, foundPos+1);
set resultingNum = cast(trim(element) as unsigned);
insert into categoryIds (`CategoryId`) values (resultingNum);
set foundPos = instr(tmpTxt,',');
end while;
if tmpTxt <> '' then
insert into categoryIds (`CategoryId`) values (tmpTxt);
end if;
CASE
WHEN sortOrder = 0 THEN
SET #sortString = "ProductResult_Relevance DESC";
WHEN sortOrder = 1 THEN
SET #sortString = "ProductResult_Price ASC";
WHEN sortOrder = 2 THEN
SET #sortString = "ProductResult_Price DESC";
WHEN sortOrder = 3 THEN
SET #sortString = "ProductResult_StockStatus ASC";
END CASE;
SET #theSelect = CONCAT(CONCAT("
SELECT SQL_CALC_FOUND_ROWS
supplier.SupplierId as Supplier_SupplierId,
supplier.Name as Supplier_Name,
supplier.ImageName as Supplier_ImageName,
product_result.ProductId as ProductResult_ProductId,
product_result.SupplierId as ProductResult_SupplierId,
product_result.Name as ProductResult_Name,
product_result.Description as ProductResult_Description,
product_result.ThumbnailUrl as ProductResult_ThumbnailUrl,
product_result.Price as ProductResult_Price,
product_result.DeliveryPrice as ProductResult_DeliveryPrice,
product_result.StockStatus as ProductResult_StockStatus,
product_result.TrackUrl as ProductResult_TrackUrl,
product_result.LastUpdated as ProductResult_LastUpdated,
MATCH(product_result.Name) AGAINST(?) AS ProductResult_Relevance
FROM
product_latest_state product_result
JOIN
supplier ON product_result.SupplierId = supplier.SupplierId
JOIN
category_product ON product_result.ProductId = category_product.ProductId
WHERE
MATCH(product_result.Name) AGAINST (?)
AND
category_product.CategoryId IN (select CategoryId from categoryIds)
ORDER BY
", #sortString), "
LIMIT ?, ?;
");
set #keywords = keywords;
set #startOffset = startOffset;
set #itemsToReturn = itemsToReturn;
PREPARE TheSelect FROM #theSelect;
EXECUTE TheSelect USING #keywords, #keywords, #startOffset, #itemsToReturn;
SET #resultsFound = FOUND_ROWS();
SELECT #resultsFound as 'TotalResults';
IF #resultsFound > 0 THEN
INSERT INTO product_search_query (QueryText, CategoryId) VALUES (keywords, topLevelCategoryId);
END IF;
END $$
DELIMITER ;
Any help is very very much appreciated!
There is little you can do with this query.
Try this:
Create a PRIMARY KEY on categoryIds (categoryId)
Make sure that supplier (supplied_id) is a PRIMARY KEY
Make sure that category_product (ProductID, CategoryID) (in this order) is a PRIMARY KEY, or you have an index with ProductID leading.
Update:
If it's INSERT that causes the problem and product_search_query in a MyISAM table the issue can be with MyISAM locking.
MyISAM locks the whole table if it decides to insert a row into a free block in the middle of the table which can cause the timeouts.
Try using INSERT DELAYED instead:
IF #resultsFound > 0 THEN
INSERT DELAYED INTO product_search_query (QueryText, CategoryId) VALUES (keywords, topLevelCategoryId);
END IF;
This will put the records into the insertion queue and return immediately. The record will be added later asynchronously.
Note that you may lose information if the server dies after the command is issued but before the records are actually inserted.
Update:
Since your table is InnoDB, it may be an issue with table locking. INSERT DELAYED is not supported on InnoDB.
Depending on the nature of the query, DML queries on InnoDB table may place gap locks which will lock the inserts.
For instance:
CREATE TABLE t_lock (id INT NOT NULL PRIMARY KEY, val INT NOT NULL) ENGINE=InnoDB;
INSERT
INTO t_lock
VALUES
(1, 1),
(2, 2);
This query performs ref scans and places the locks on individual records:
-- Session 1
START TRANSACTION;
UPDATE t_lock
SET val = 3
WHERE id IN (1, 2)
-- Session 2
START TRANSACTION;
INSERT
INTO t_lock
VALUES (3, 3)
-- Success
This query, while doing the same, performs a range scan and places a gap lock after key value 2, which will not let insert key value 3:
-- Session 1
START TRANSACTION;
UPDATE t_lock
SET val = 3
WHERE id BETWEEN 1 AND 2
-- Session 2
START TRANSACTION;
INSERT
INTO t_lock
VALUES (3, 3)
-- Locks
Try wrapping your EXECUTE with the following:
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;
EXECUTE TheSelect USING #keywords, #keywords, #startOffset, #itemsToReturn;
SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ ;
I do something similiar in TSQL for all report stored proc and searches where repeatable reads aren't important to reduce locking/blocking issues with other processes running on the database.
Turn on slow queries, that will give you an idea of what is taking so long to execute that there is a timeout.
http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
Pick the slowest query and optimise that. then run for a while and repeat.
There is some excellent information and tools here http://hackmysql.com/nontech
DC
UPDATE:
Either you have a network problem causing the timeout, if you are using a local mysql instance then that is unlikely, OR something is locking a table for far too long causing a timeout. the process that is locking the table or tables for far too long will be listed in the slow log as a slow query. you can also get the slow log query to display any queries that fail to use an index resulting in an inefficient query.
If you can get the problem to occur while you are present then you can also use a tool like phpmyadmin or the commandline to run "SHOW PROCESSLIST\G" this will give you a list of what queries are running while the problem is occurring.
You think the problem is in your insert statement, therefore something is locking that table. therefore you need to find what is locking that table, therefore you need to find what is running so slow its locking the table for far too long. Slow queries is one way to do that.
Other things to look at
CPU - is it idle or running at full pelt
IO - is io causing holdups
RAM - are you swapping all the time (will cause excessive io)
Does the table product_search_query use an index?
What is the primary key?
If your index uses strings that are too long? you may build a huge index file that causes very slow inserts (slow query log will also show that)
And yes the problem may be elsewhere, but you must start somewhere mustn't you.
DC