Grails execute multiple native sql queries asynchronously - grails

I have a service that executes multiple native sql statements. Is it possible to run those sql statements asynchronously? My query looks something like this with multiple queries:
def sessionFactory
sessionFactory = ctx.sessionFactory // this only necessary if your are working with the Grails console/shell
def session = sessionFactory.currentSession
def query = session.createSQLQuery("select f.* from fee f where f.id = :filter)) order by f.name");

You can use the new grails Promises API
import static grails.async.Promises.*
List queriesToExecute = []
someList.each {
//create task to execute query
def t = task {
//create and execute query here
}
queriesToExecute << t
}
def result = waitAll(queriesToExecute) //if you want to wait till queries are executed
//or if you dont want to wait, use this - onComplete(queriesToExecute) { List results -> }​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

Related

Having clause in GORM

I have a domain class named Keyword and I need to run a query to get all the duplicate keywords in the system, base on some criteria.
The query in MySQL is somethine like this:
select keywordid, md5, match_type, count(md5)
from keyword
where site_id = 'MLU'
and customer_id = 1075613440
group by md5
having count(md5) > 1;
Now I need to "translate" this query to Grails but the requirement from my boss is to aviod HQL.
Is it possible to do it?
For now all I have is this:
def dupKws = criteria.list {
eq('siteId', siteId)
eq('customerId', account.customerId)
projections {
groupProperty('md5')
groupProperty('matchType')
rowCount()
}
}
You can run this query in grails with:
import groovy.sql.Sql
def query = """ select keywordid, md5, match_type, count(md5)\
from keyword\
where site_id = 'MLU'\
and customer_id = 1075613440\
group by md5\
having count(md5) > 1 """
Sql sql = new Sql(datasource)
sql.execute(query)

Insert 10,000,000+ rows in grails

I've read a lot of articles recently about populating a grails table from huge data, but seem to have hit a ceiling. My code is as follows:
class LoadingService {
def sessionFactory
def dataSource
def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP
def insertFile(fileName) {
InputStream inputFile = getClass().classLoader.getResourceAsStream(fileName)
def pCounter = 1
def mCounter = 1
Sql sql = new Sql(dataSource)
inputFile.splitEachLine(/\n|\r|,/) { line ->
line.each { value ->
if(value.equalsIgnoreCase('0') {
pCounter++
return
}
sql.executeInsert("insert into Patient_MRNA (patient_id, mrna_id, value) values (${pCounter}, ${mCounter}, ${value.toFloat()})")
pCounter++
}
if(mCounter % 100 == 0) {
cleanUpGorm()
}
pCounter = 1
mCounter++
}
}
def cleanUpGorm() {
session.currentSession.clear()
propertyInstanceMap.get().clear()
}
}
I have disabled secondary cache, I'm using assigned ids, and I am explicitly handling this many to many relationship through a domain, not the hasMany and belongsTo.
My speed has increased monumentally after applying these methods, but after a while the inserts slow down to the point of almost stopping compared to about 623,000 per minute at the beginning.
Is there some other memory leak that I should be aware of or have I just hit the ceiling in terms of batch inserts in grails?
To be clear it takes about 2 minutes to insert 1.2 million rows, but then they start to slow down.
Try doing batch inserts, it's much more efficient
def updateCounts = sql.withBatch { stmt ->
stmt.addBatch("insert into TABLENAME ...")
stmt.addBatch("insert into TABLENAME ...")
stmt.addBatch("insert into TABLENAME ...")
...
}
I have fought with this in earlier versions of Grails. Back then I resorted to either simply run the batch manually in proper chunks or use another tool for the batch import, such as Pentaho Data Integration (or other ETL tool or DIY).

Concurrency Issue Grails

I am using this code for updating a row.
SequenceNumber.withNewSession {
def hibSession = sessionFactory.getCurrentSession()
Sql sql = new Sql(hibSession.connection())
def rows = sql.rows("select for update query");
}
in this query I am updating the number initially sequenceNumber is 1200.
and every time this code run then it will b increamented by 1.
and I am running this code 5 times in loop.
but this is not flushing the hibernate session so that every time I am getting the same number 1201.
I have also used
hibSession.clear()
hibSession.flush()
but no success.
If I use following code then its works fine.
SequenceNumber.withNewSession {
Sql sql = new Sql(dataSource)
def rows = sql.rows("select for update query")
}
every time I am getting a new number.
Can anybody tell me what's wrong with above code
Try with Transaction, + flush it on the end, like:
SequenceNumber.withTransaction { TransactionStatus status ->
...
status.flush()
}

Why sql.rows Groovy method is so slow

I tried to fetch some data with the sql.rows() Groovy method and it took a very long time to return the values.
So I tried the "standard" way and it's much much faster (150 times faster).
What am I missing ?
Look at the code below : the first method returns results in about 2500ms and the second in 15 ms !
class MyService {
javax.sql.DataSource dataSource
def SQL_QUERY = "select M_FIRSTNAME as firstname, M_LASTNAME as lastname, M_NATIONALITY as country from CT_PLAYER order by M_ID asc";
def getPlayers1(int offset, int maxRows)
{
def t = System.currentTimeMillis()
def sql = new Sql(dataSource)
def rows = sql.rows(SQL_QUERY, offset, maxRows)
println "time1 : ${System.currentTimeMillis()-t}"
return rows
}
def getPlayers2(int offset, int maxRows)
{
def t = System.currentTimeMillis();
Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement();
statement.setMaxRows(offset + maxRows -1);
ResultSet resultSet = statement.executeQuery(SQL_QUERY);
def l_list =[];
if(resultSet.absolute(offset)) {
while (true) {
l_list << [
'firstname':resultSet.getString('firstname'),
'lastname' :resultSet.getString('lastname'),
'country' :resultSet.getString('country')
];
if(!resultSet.next()) break;
}
}
resultSet.close()
statement.close()
connection.close()
println "time2 : ${System.currentTimeMillis()-t}"
return l_list
}
When you call sql.rows, groovy eventually calls SqlGroovyMethods.toRowResult for each row returned by the resultSet.
This method interrogates the ResultSetMetaData for the resultSet each time to find the column names, and then fetches the data for each of these columns from the resultSet into a Map which it adds to the returned List.
In your second example, you directly get the columns required by name (as you know what they are), and avoid having to do this lookup every row.
I think I found the reason this method is so slow : statement.setMaxRows() is never called !
That means that a lot of useless data is sent by the database (when you want to see the first pages of a large datagrid)
I wonder how your tests would turn out if you try with setFetchSize instead of setMaxRows. A lot of this has to the underlying JDBC Driver's default behavior.

Result set mapping in Grails / GORM

I want to map the result of a native SQL query to a simple bean in grails, similar to what the #SqlResultSetMapping annotation does.
For example, given a query
select x.foo, y.bar, z.baz from //etc...
map the result to
class FooBarBaz {
String foo
String bar
String baz
}
Can anyone provide an example of how to do this in grails?
Thanks in advance.
I tested this successfully in the Grails console
import groovy.sql.Sql
class FooBarBaz {
String foo
String bar
String baz
}
// Initialising the Sql object like this ensures that the SQL statement
// will participate in the current transaction (if one exists)
// 'ctx' refers to the Spring ApplicationContext when using the Grails console
def sessionFactory = ctx.getBean('sessionFactory')
Sql sql = new Sql(sessionFactory.currentSession.connection())
def query = 'select email, user_real_name, phone from user'
def results = []
sql.eachRow query, {row -> results << new FooBarBaz(foo: row.email, bar: row.user_real_name, baz: row.phone)}

Resources