Grails Connections behaving very differently in Integration test - grails

We have a custom data source that extends BasicDataSource. We have overridden the getConnection method which does a couple things inside of it. When we run the webapp outside of testing, when we call a service from a controller it will grab a new connection and use that connection until the service is done. All is well. However, inside an integration test, the connection appears to be grabbed before the test even calls the controller. Flow below
Regular Run:
call controller -> controller calls service method -> connection is grabbed -> service method is run and returns to controller
Integration Test:
connection is grabbed -> call controller from test -> controller calls service method -> service method is run and returns to controller
Needless to say, this is giving us problems as having the correct connection is very important for our app. Thoughts?
Edit: Still getting significant issues with this. We've reached a point where we have to avoid creating integration tests, or do some manual connection switching (which defeats half the point of the tests)
DataSource.groovy
dataSource {
pooled = true
dialect="org.hibernate.dialect.OracleDialect"
properties {
maxActive = 50
maxIdle = 10
initialSize = 10
minEvictableIdleTimeMillis = 1800000
timeBetweenEvictionRunsMillis = 1800000
maxWait = 10000
testWhileIdle = true
numTestsPerEvictionRun = 3
testOnBorrow = true
}
}
hibernate {
cache.use_second_level_cache = true
cache.use_query_cache = true
cache.provider_class = 'net.sf.ehcache.hibernate.EhCacheProvider'
}

This is not a final Answer, however I believe this is an explanation of what is going on:
Running as Web app: your Service class has a transactionManager which has a sessionFactory, which gets the connection! So in this case, assuming that you Service is 'transactional=true' all methods that you call in your services will have a 'Session.beginTransaction()' in the beginning of the method(there is a Grails`s Proxy to do that, when you set 'transactional=true'), which will call all that stack until getConnection().
Running as Integration Test: as Grails doesnt commit your DB changes, it always rollback them! I believe that when you are starting your Integration test, grails is creating a transaction right away! so it will be able to rollback it afterwards!(which make totally sense right!), you can confirm that taking a look at the class org.codehaus.groovy.grails.test.support.GrailsTestInterceptor. The method init() is called before your services in your integration test. So that`s why getConnection() is being called before everything!
Suggestion:
You can try setting your integration test class as 'transaction=false' and see if getConnection() doesn't get call in the beginning!
Go to Transactions section in here to see more!
Just dont forget that in your test you will have to rollback your transaction! if your set transaction=false.

Related

Pooled LdapTemplate stalls for minutes during context validation

I'm trying to use Spring-ldap's LdapTemplate to retrieve information from an LDAP source during a Rest call service implementation and, while I think I have a working configuration, we're noticing stalls of up to 15 minutes intermittently when the service is hit. Logging statements have determined the stall happens during the ldapTemplate.search() call.
My beans:
contextSourceTarget(org.springframework.ldap.core.support.LdapContextSource) {
urls = ["https://someldapsource.com"]
userDn = 'uid=someaccount,ou=xxx,cn=users,dc=org,dc=com'
password = 'somepassword'
pooled = true
}
dirContextValidator(org.springframework.ldap.pool2.validation.DefaultDirContextValidator)
poolConfig( org.springframework.ldap.pool2.factory.PoolConfig ) {
testOnBorrow = true
testWhileIdle = true
}
ldapContextSource(org.springframework.ldap.pool2.factory.PooledContextSource, ref('poolConfig')) {
contextSource = ref('contextSourceTarget')
dirContextValidator = ref('dirContextValidator')
}
ldapTemplate(LdapTemplate, ref('ldapContextSource')) {}
I expect this application could be hitting LDAP several times concurrently (via concurrent rest calls to this app) for retrieving data from different users. Here's the code that makes that call:
List attrs =['uid', 'otherattr1', 'otherattr2']
// this just returns a Map containing the key value pairs of the attrs passed in here.
LdapNamedContextMapper mapper = new LdapNamedContextMapper( attrs )
log.debug( "getLdapUser:preLdapSearch")
List<Map> results = ldapTemplate.search(
'cn=grouproot,cn=Groups,dc=org,dc=com',
'uniquemember=userNameImsearchingfor',
SearchControls.SUBTREE_SCOPE,
attrs as String[], mapper )
log.debug( "getLdapUser:postLdapSearch" )
Unfortunately, at random times it seems, the timestamp difference between the preLdapSearch and postLdapSearch logs is upwards of 15 minutes. Obviously, this is bad, and it would seem to be a pool management issue.
So I turned on debug logging for packages org.springframework.ldap and org.apache.commons.pool2
And now when this happens I get the following in the logs:
2018-09-20 20:18:46.251 DEBUG appEvent="getLdapUser:preLdapSearch"
2018-09-20 20:35:03.246 DEBUG A class javax.naming.ServiceUnavailableException - not explicitly configured to be a non-transient exception - encountered; ignoring.
2018-09-20 20:35:03.249 DEBUG DirContext 'javax.naming.ldap.InitialLdapContext#1f4f37b4' failed validation with an exception.
javax.naming.ServiceUnavailableException: my.ldaphost.com:636; socket closed
at com.sun.jndi.ldap.Connection.readReply(Connection.java:454)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:638)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:638)
at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:561)
at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985)
at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1844)
at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1769)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:392)
(LOTS OF STACK TRACE REMOVED)
2018-09-20 20:35:03.249 DEBUG Closing READ_ONLY DirContext='javax.naming.ldap.InitialLdapContext#1f4f37b4'
2018-09-20 20:35:03.249 DEBUG Closed READ_ONLY DirContext='javax.naming.ldap.InitialLdapContext#1f4f37b4'
2018-09-20 20:35:03.249 DEBUG Creating a new READ_ONLY DirContext
2018-09-20 20:35:03.787 DEBUG Created new READ_ONLY DirContext='javax.naming.ldap.InitialLdapContext#5239386d'
2018-09-20 20:35:03.838 DEBUG DirContext 'javax.naming.ldap.InitialLdapContext#5239386d' passed validation.
2018-09-20 20:35:03.890 DEBUG appEvent="getLdapUser:postLdapSearch"
Questions:
How can I find out more? I've got debug logging turned on for org.springframework.ldap and org.apache.commons.pool2
Why is it seeming to take 15+minutes to determine that a connection is stale/unusable? How can I configure to make that much shorter?
There is a good chance that the underlying LDAP system is having connection issues.
You could try adding timeouts in the connection pool settings:
max-wait - default is -1
eviction-run-interval-millis - you may
want to set this to control how often to check for problems
Docs: https://docs.spring.io/spring-ldap/docs/current/reference/#pool-configuration

grails 2.4.4 integration test not using the test datasource

Help!
Our plugin project integration tests should hit the database specified in the datasource.groovy, but for some reason they ignore it, and do it in memory.
Its a plugin which provides the core services (i.e. DB access) to several grails apps which are each a grails application.
Datasource.groovy looks like this:
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dialect = "org.hibernate.dialect.MySQL5InnoDBDialect"
}
environments {
development {
dataSource {
dbCreate = "create-drop"
url = "jdbc:mysql://127.0.0.1:3306/db"
username = "someuser"
password = "somepass"
}
}
test {
dataSource {
dbCreate = "update"
url = "jdbc:mysql://127.0.0.1:3306/db"
username = "someuser"
password = "somepass"
}
}
production {
dataSource {
}
}
}
The test (SiteIntegrationSpec.goovy)
import grails.test.mixin.TestFor
import grails.test.spock.IntegrationSpec
#TestFor(Site)
class SiteIntegrationSpec extends IntegrationSpec {
static transactional = false
def setup() {
}
def cleanup() {
}
void "test something"() {
when:
Site site
site = new Site(name: "asdf", description: "asdfsd").save(failOnError: true)
then:
site.id == 3
when:
Site site2 = Site.get(1L)
then:
site2.name == "root"
}
}
Data already existing in the site table:
ID name description
1 root root
2 test test
The first test should insert a record which will happen to have an ID of 3. It actually inserts with an ID of 1, i.e. its not seeing or hitting the test database, its using some mock or internal db which is not defined anywhere.
The second test fails as instead of retrieving "root" it retrieves "asdf"
What I have tried:
creating a separate DB for test. Didn't help.
specifying -Dgrails.env=test when running tests. Didn't help
running the tests with the DB down. This correctly fails with cant create pool type exception.
changing the test datasource password to an incorrect one - this correctly throws an exception.
grails -Dgrails.env=test test-app com.me.myproject.SiteIntegrationSpec --stacktrace --verbose
So grails is connecting to the test datasource, but then the integration tests are not using it!
Any ideas?
Edit: Site is a domain object:
class Site {
String name
String description
}
Plugin DataSource.groovy files aren't included in the plugin zip, and if you somehow manually or programmatically include them, they'll be ignored. The same goes for Config.groovy, UrlMappings.groovy, and BootStrap.groovy. In general when something is usable from a plugin, if the application has a file with the same name and location, it overrides the plugin's file, so that would keep this from working also.
You could define a dataSource bean in your plugin's doWithSpring that replaces the one Grails creates based on DataSource.groovy that uses values from a file that exists in the plugin zip, or that is located in the application if that makes sense. Note that there are really 3 DataSource beans and two of them are proxies of the "real" one, so you need to define yours as dataSourceUnproxied so the other two proxy yours and retain the behavior that they add.
Another thing that you will need to fix once you resolve this is your use of unit test annotations in an integration test. Never use Mock, TestFor, or any unit test mixin annotation or base class, since their purpose is to establish a fairly realistic environment that makes up for Spring, Hibernate, installed plugins, and lots of Grails functionality not being available, but in an integration test they are available, and the unit test stuff will stomp on the real instances.
Also - why are you using static transactional = false? This disables an important integration test feature where all of your test methods run in a transaction that is rolled back at the end of the tests pass or fail. This ensures that nothing you do in a test influences other tests - everything is independent. If you disable this, you need to undo all of the changes, and it's easy to miss some and introduce false negatives or worse - false positives - into your tests.

grails database commit's during a service's method

I am trying to have a grails service method which does database commits while running instead of commiting everything at the end.
serviceMethode(){
status = 0
doFirst
status = 1
doSecond
status = 2
doThird
status = 3
}
I am experiencing that 1, and 2 are never persistend to the database, even though I am doing DomainObject.save(flush:true). My guess is that this is due to transaction handling. How can i persist values while the service-method is running rather than at the end of the method.
Annotate your service method with #NotTransactional

Saving an object from Grails controller fails

There is a controller just updating an object:
def obj = Table.find(...)
obj.state = "END"
if(!obj.save(flush:true)) ...
// Session flushing does not help too.
// I am able to fetch that object again here but it is never persisted to DB
When deployed onto a Debian, the object is not actually persisted although there is no any errors reported (nor by manual checking nor by failOnError). But if deployed onto my local Windows machine the object is persisted correctly. The both machines are connected to the same DB.
Moving the save into a service or covering it withTransaction are workarounds. Yes, I know services are a better place to deal with DB. Instead I'm interesting to find what is the cause of such inconsistent and problematic behavior (and how to fix it in a single place). Or it is prohibited by design doing such things in controllers?
Grails 2.1.1, Tomcat
Excerptions from DataSource.groovy:
dataSource {
driverClassName = "oracle.jdbc.driver.OracleDriver"
dialect = org.hibernate.dialect.Oracle10gDialect
pooled = true
}
hibernate {
cache.use_second_level_cache = true
cache.use_query_cache = false
cache.region.factory_class = 'net.sf.ehcache.hibernate.EhCacheRegionFactory'
}
Please check if there is no validation errors.
obj.validate()
obj.errors.each {
if(obj.hasErrors)
println it
}

Grails Quartz Job Integration test - Not autowired Job

I'm writing the Integration test for a Quartz Job in a grails application.
I've the Job in grails-app/jobs folder, and if I start the application it works. The problem is that I want to get it in an integration test, but the autowire won't work. The test is like:
class MyJobTest{
MyJob myJob
def setUp(){
assert myJob != null
}
def testExecute(){
//test logic
}
}
but it fails because myJob is null...some help?
Quartz Jobs are not autowired like services are under the test environment. The documentation for the Quartz job also explicitly states that by default it will not execute on schedule under the test environment (you could change that if you want to but I wouldn't). I would just instantiate myJob = new MyJob() in your setUp and call the execute() method to test it. If you're trying to test the triggers you may want to find a way to look at what is inside the triggers {} maybe inspecting the metaClass?
EDIT IN RESPONSE TO COMMENT:
I've never gotten the services out of the application context so that might work. The way I would probably test it is as follows:
Assuming your class looks something like this:
class MyJob {
def myServiceA
def myServiceB
def execute() {
if(myJobLogicToDetermineWhatToDo) {
myServiceA.doStuff(parameter)
} else {
myServiceB.doStuff(parameter)
}
}
}
What you're really wanting to test here is the myJobLogicToDetermineWhatToDo. I would assume that you have (or can easily write) integration and/or unit tests against your services myServiceA and myServiceB to ensure that they are working correctly. I would then write unit tests to test the logic/wiring of your Job to the appropriate service.
#Test
void routeOne() {
def job = new MyJob()
def myServiceA = new Object()
def expectedParameter = "Name"
def wasCalled = false
myServiceA.metaClass.doStuff = {someParameter ->
assert expectedParameter == someParameter
wasCalled = true
}
job.myServiceA = myServiceA
//Setup data to cause myServiceA to be invoked
job.execute()
assert wasCalled
}
Then repeat this process for all of the routes you have through your Job. This way you can isolate your tests down to the smallest part possible and test the logic of the object that you're invoking not the services it is using. I would assume you're using a service because the logic in there is being used by another part of the system. If you're testing the service through this job and for some reason the job goes away then you have to re-write your tests to invoke the service directly. The way that I've proposed you have tests testing the service directly and tests that mock out those service calls. If the job goes away you would simply delete the tests associated with it and you won't loose any test coverage. Kinda long winded but that's how I would approach testing it.

Resources