Dbix::Class slow response - catalyst

I have a DBIx::Class query that's taking too long to complete.
All SQL below were generated by DBIx::Class.
First scenario (DBIx Simple select):
SELECT me.pf_id, me.origin_id, me.event_time, me.proto_id FROM pf me ORDER BY event_time DESC LIMIT 10;
DBIx query time: 0.390221s (ok)
Second scenario (DBIx Simple select using where):
SELECT me.pf_id, me.origin_id, me.event_time, me.proto_id FROM pf me WHERE ( proto_id = 7 ) ORDER BY event_time DESC LIMIT 10;
DBIx query time: 29.27025s!! :(
Third scenario (Using pgadmin3 to run the query above):
SELECT me.pf_id, me.origin_id, me.event_time, me.proto_id FROM pf me WHERE ( proto_id = 7 ) ORDER BY event_time DESC LIMIT 10;
Pgadmin query time: 25ms (ok)
The same query is pretty fast using pgdamin.
Some info:
Catalyst 5.90091
DBIx::Class 0.082820 (latest)
Postgres 9.1
I did all tests on localhost, using Catalyst internal server.
I have no problems with any other table/column combination, it's specific with proto_id.
Database Schema automatically generated by DBIx::Class::Schema::Loader
proto_id definition:
"proto_id",
{ data_type => "smallint", is_foreign_key => 1, is_nullable => 0 },
Anybody have a clue why DBIx is taking so long to run this simple query?
Edit 1: Column is using index (btree).
Edit 2: This is a partitioned table, I'm checking if all the sub-tables have all the indexes, but still doesn't explain why the same query is slower on DBIx::Class.
Edit 3: I did a simple DBIx::Class script and I got the same results, just to make sure the problem is not the Catalyst Framework.
Edit 4: Using tcpdump I noticed postgres is taking too long to respond, still trying...
Edit 5: Using DBI with SQL seems pretty fast, I'm almost convinced this is a DBIx::Class problem.

After some tests, I found the problem:
When I do the query using DBI bind_param() (As DBIx::Class does) for some reason it became very slow.
SELECT me.pf_id, me.origin_id, me.event_time, me.proto_id FROM pf me WHERE ( proto_id = ? ) ORDER BY event_time DESC LIMIT ?;
my $sth = $dbh->prepare($sql);
$sth->bind_param(1, 80, { TYPE => SQL_INTEGER });
$sth->bind_param(2, 10, { TYPE => SQL_INTEGER });
$sth->execute();
So after some time searching CPAN I've noticed that my DBD::Pg was outdated (My bad). I downloaded the source from CPAN, compiled and the problem is gone. Must be some bug from older versions.
TL;DR: If you're having problems with DBI or DBIx::Class make sure your DBI database driver is updated.

Related

How Can I execute SPARQL1.1 queries containing '+', '?' character in property path using ISQL on Virtuoso?

I need to run some SPARQL 1.1 queries, but I have problems with some characters on property paths.
SELECT ?item ?other
WHERE {
?item wdt:P279+ wd:Q37144 . # cartridges
?item wdt:P144 ?other . # if the cartridge is based on another
}
It does not give an error, but it does not deliver results. The problem is the character +, because changing wdt:P279+ to wdt:P279/wdt:P279* works correctly. Similarly, the symbol ?. In this case it does deliver results, but it logs error More than 0 parameters, ignoring all the rest of the statement SPARQL
I have tried with \+, \\+, \002B, \053, etc.
I have just tested your query with my own local command-line iSQL 07.20.3217 built 2017-09-06 and installed with Virtuoso 8.1 (Enterprise Edition).
iSQL had no complaints about the + nor *, and delivered good results for both, but I do see a complaint about the ? (which does not occur in the browser-based iSQL).
Thus, I think that an updated iSQL will get you past the + issue.
It would be helpful if you could log an issue on the github project about the ?.

py2neo Error returning data from Cypher query

My simple code is retrieving attributes from nodes in neo4j.
results = graph.cypher.execute("MATCH (m)-[:AB]->(a) "
"RETURN m.searchField as origin, a.searchField as destination "
"LIMIT {limit}", {"limit": 100})
nodes = []
rels = []
i = 0
for r in results:
print (r)
ent1 = {"title": r.origin, "label": "entity"}
but the server returns "NameError("global name 'searchField' is not defined",)" Certainly I missed something, but I'm puzzled that the searchField inside the Cypher query is the object of the error.
This is still with py2neo 2.0.8.
Thanks for any pointer, hj
Later editing:
Thanks for taking the time to look at this question. Two elements further puzzle me in this error:
1. The query in cypher is fine, and returns the result I expect in neo4j-shell without problem
2. This code seems to work fine when I run bottle as standalone (run(port=8080) in main), but fails when I run it as wsgi under an apache server. I am wondering if it is a problem of running user, or of context in some part of the code.
Do you have a property called searchField on the node(s)?
If not, the query will fail.
BTW, it is easier to use a string for the query like so:
query = '''
MATCH (m)-[:AB]->(a)
RETURN m.searchField as origin, a.searchField as destination
LIMIT {limit}
'''
result = graph.cypher.execute(query, limit='foo')
Got it to work! It was unrelated to code, but I did not know that any refresh of a new python code served through wsgi requires an apache reload at least.
sudo service apache2 reload
With that I obtain the same (and correct) behavior as with the direct server. The error was the result of an old version of the code... newbie mistake!
Thanks and sorry for the hassle, hj

Neo4j - too many connections resets - from Ruby on Rails console

There is one and only one connection and user here.
d = l.descriptions.first
Language#descriptions 1200270ms MATCH language137, language137-[rel1:`DESCRIBED_IN`]->(result_descriptions:`Description`) WHERE (ID(language137) = {ID_language137}) RETURN result_descriptions | {:ID_language137=>137}
Faraday::TimeoutError: too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 70156828873380, last used 1438628883.105085 seconds ago
After that no other connection is allowed until server is restarted.
What is wrong here?
Here is in more detail what I am trying to do: selecting a language i.e. English. Getting the count of descriptions in English. Searching the first description in English. This never returns or delivers the connection error. During the long run of the last one no other connections can be open to the database.
irb(main):001:0> l = Language.find_by(iso_639_2_code: 'eng')
CYPHER 316ms MATCH (n:`Language`) WHERE (n.iso_639_2_code = {n_iso_639_2_code}) RETURN n LIMIT {limit_1} | {:n_iso_639_2_code=>"eng", :limit_1=>1}
=> #<Language uuid: nil, english_name_of_language: "English", french_name_of_language: "anglais", german_name_of_language: "Englisch", iso_639_1_code: "en", iso_639_2_code: "eng", spoken_in: "English, a West Germanic language is the first language for about 309–400 million people. See: Countries by Languages - English Speaking Countries.">
irb(main):002:0>
irb(main):005:0* n = l.descriptions.count
Language#descriptions 17749ms MATCH language137, language137-[rel1:`DESCRIBED_IN`]->(result_descriptions:`Description`) WHERE (ID(language137) = {ID_language137}) RETURN count(result_descriptions) AS result_descriptions | {:ID_language137=>137}
=> 2107041
irb(main):006:0> d = l.descriptions.first
I think that we fixed this is version 5.0 of the gems. Could you try upgrading?
The issue was moved to the neo4jrb git repo and solved by the neo4j maintainers by recommending to upgrade to the gem core version 5.0.11 from 5.0.9
You might see this error if your Neo4j database server becomes unresponsive. To fix this, restart your Neo4j database server.
bundle exec rake neo4j:stop
bundle exec rake neo4j:start

Informix select first 250000 then the last 250000 records in a table

I work on a CFML script to backup some data in a CSV file from Informix database. The problem is the table has many records 286906 and my scripts timeouts (even I set it not to), the best I could successfully was 260000 with:
SELECT FIRST 260000
APE1, APE2, CALLE, CODPOSTAL, DNI, FCADU, FENACI, LOCALIDAD, NOMBRE, NSS, PROV, TELEFONO
FROM
mytable WHERE FCADU IS NOT NULL AND FENACI IS NOT NULL
is there any way to select the rest of 260000 and then the rest?
I tried with:
SELECT SKIP 260000 FIRST 520000
APE1, APE2, CALLE, CODPOSTAL, DNI, FCADU, FENACI, LOCALIDAD, NOMBRE, NSS, PROV, TELEFONO
FROM
mytable WHERE FCADU IS NOT NULL AND FENACI IS NOT NULL
but I get Error Executing Database Query. A syntax error has occurred.
You can use the Unload statement for creating a file from database:
UNLOAD TO 'mytable.txt' SELECT * FROM mytable;
Maybe that this not works in CFML environment. So you can create a stored procedure which unloads your data.
See unload statement in stored procedure
Is it your script timing out or your database connection? From your question it sou ds to me like its not the coldfusion template that is timing out but the cfquery connection to the database. There is a timeout attribute for the cfquery tag. However apparently it is not reliable a better option is to configure the timout in the advanced section of the datasource within the coldfusion administrator.
Charlie Arehart blogged about this feature here:
http://www.carehart.org/blog/client/index.cfm/2010/7/14/hidden_gem_in_cf9_admin_querytimeout

What is the correct syntax to use when trying to create a Data Source View to a linked server?

I have tried several statements but this one at least returns data.. but I get the error message: Deferred prepare could not be prepared. Incorrect syntax near')'. Incorrect syntax near the keyword 'DECLARE'. The following statement executed when creating namedquery:
SELECT[vwStatistics].*
FROM
(
***THIS IS MY QUERY***
DECLARE #SQL1 VARCHAR(500)
SET #SQL1 = 'SELECT *
FROM OPENQUERY(PORTAL, ''SELECT DeviceID, Date, Count
FROM printer_stats.Statistics
GROUP BY DeviceID'')'
EXEC (#SQL1)
***END OF MY QUERY***
)
AS[vwStatistics] (Microsoft.AnalysisServices.Controls)
I am new to linked servers and to SSAS. This is our company's first Cube from a linked server. My query does run in Management Studio and creates a SSRS report but it is slow.
Any suggestions would be helpful. Not much info on syntax for this situation on web. I have been looking for any such situation and have not found much help other than trying changes on server. EX: Make sure openrowset is on and reinstall OWC component.. I do not have that capability.
This is what we found to work:
SELECT DeviceID, CAST(statsdt AS CHAR) AS sdt, Count FROM OPENQUERY (
PORTAL, 'select * from (select DeviceID,CAST( Date AS CHAR) statsdt, Count from printer_stats.Statistics) as pstats')

Resources