I am experiencing the following very strange behavior with Grails GORM queries...
NOTE: I have 20 instances of MyDomain created. I.e 20 records
CASE 1
List<MyDomain> case1Results = MyDomain.where {
(isTypeA == true) && (relation1.relation2.relation3.organization == org)
}.list()
In this case, case1Results is an empty list.
CASE 2
List<MyDomain> case2Results = MyDomain.where {
(isTypeA == true)
}.list()
In this case, case2Results contains the 20 records I have stored.
This all seems to find up to this point. This shows that the 20 records are not associated with organization. However...
THE PROBLEM:
when I take case2Results and do the following...
println(case2Results.every {
it.relation1.relation2.relation3.organization == org
}) //true
I get true
OR:
case2Results = case2Results.findAll{
it.relation1.relation2.relation3.organization == org
}
Here, case2Results still contains 20 records as in CASE 2 above, not 0 as in CASE 1
I can't wrap my head around this behavior and I don't even know how to explain this if not with these examples. Does anyone know why this is happening and how to fix it?
PS: Each of my domain classes has a belongsTo and hasMany static maps, creating relationships between the domain classes.
PS: This behavior is being experienced when I run unit tests but works well in the actual feature being tested.
Related
My rails application is a residence application.
I have a model stay which is a third model in a has_many:through assocation between a tenant and a flat. A stay is defined by a checkin_date and a checkout_date. I would like to create a custom validations to prevent the creation of a new stay if there is already a tenant in a flat for a given period...but i never write a custom validation before so i'm super lost...
By logic, i know i need to compare the checkin_date and checkout_date of the "already existing record" and the "supposed new one". So I guess it would look like : (n+1= the new record vs. n= already existing record)
> def duplicate_stay
>
> if Stay.exists?(tenant_id: current_tenant.id, studio_id:
> current_studio.id) &&
> checkin_date(n+1) > checkin_date(n)
> checkout_date(n+1) > checkin_date(n)
> checkin_date(n+1) < checkout_date(n)
> checkout_date(n+1) < checkout_date(n)
> == false
> else
> == true (the model can be created)
end
Can someone help me? I keep looking but I know understand how to do it !
Try using the cover? method. You can use it like this:
today = Date.today
in_one_week = today + 7.days
search_date = today + 3.days
(today..in_one_week).cover? search_date
=> true
This is not tested, but if you added a scope to the Stay class:
scope :overlaps, ->(stay) {
where(
tenant_id: stay.tenant_id,
studio_id: stay.studio_id,
).
where(arel_table[:checkin_date].lt(stay.checkout_date)).
where(arel_table[:checkout_date].gt(stay.checkin_date))
}
... then for an instance of Stay #stay calling Stay.overlaps(#stay).exists? would return true if there was any instance that overlapped with #stay.
Let me know if that doesn't work. It's often helpful to look at the SQL that is executed as part of a validation if you're not sure why something isn't working the way you think it should.
Edit: one tricky thing here is that a stay overlaps with itself, so if #stay was saved successfully (ie. it has an id assigned) you might have a problem with that logic. You can provide different logic depending on whether the record is saved or not, though.
I think that this would do it:
scope :overlaps, ->(stay) {
logic = where(
tenant_id: stay.tenant_id,
studio_id: stay.studio_id,
).
where(arel_table[:checkin_date].lt(stay.checkout_date)).
where(arel_table[:checkout_date].gt(stay.checkin_date))
logic = logic.where.not(id: stay.id) unless stay.new_record?
logic
}
The fist part of my question is
suppose I have a poco class
public class shop{
public virtual string fruitName {get;set;}
public virtual double numberOfFruitsLeftToConsume {get;set;}
public virtual double numberOfFruitsLeftForStorage {get;set;}
public virtual List<Locations> shopLocations {get;set;}
}
I add new fruits in the db by creating a new object of shop and then add it via my context then save it.
Now to retrieve the data
will it be more efficient for me to first filter by fruit name get a List then in that collection should I run my query to sort by the number of fruits to consume , or should I just put it all into one query. Supposing that the site has more than 1000 hits/sec and a massive DB, which method will be efficient.
List<shop> sh = context.shopDB.Where(p => p.fruitName == "mango" &&
p.fruitName == "apple").ToList();
List<shop> sh = sh.Where(f => f.numberOfFruitsLeftToConsume >= 100 &&
f.numberOfFruitsLeftForStorage <= 100).ToList();
The example has no meaning , I just wanted to show the type of query I am using.
The second part of my question is, when I initialize the class shop I do not initialize the List within it. Later on when I try to add it it does not get saved, the shop is connected to the user.
ApplicationUser user = await usemanager.FindByEmailAsync("email");
if(user.shops.shopLocations == null){
user.shops.shopLocation = new List<Location>();
uset.shops.shopLocation.Add(someLocation);
await context.shopDB.SaveChangesAsync();
}
////already tried
//List<Location> loc = new List<Location>();
//loc.Add(someLocation);
//user.shops.shopLocation = loc;
//await context.shopDB.SaveChangesAsync();
I tried both the methods in a try catch block and no exception is thrown.
If you need any more details or if something is not clear to you please ask.
Thank you.
If I add Location and LocationId properties to shop, and then save, I can only view the LocationId , but the Location property still remains null.
To clear any question , If I save a location Individually it saves. So I don't think I'm providing wrong data.
Now to retrieve the data will it be more efficient for me to first filter by fruit name get a List then in that collection should I run my query to sort by the number of fruits to consume , or should I just put it all into one query. Supposing that the site has more than 1000 hits/sec and a massive DB, which method will be efficient.
You are the only one who can answer that question by measuring the query performance. Only as a general rule I can say that putting all into one query and let the database do the most of the job (eventually tuning it by creating appropriate indexes) is usually preferable.
What about the second part of the question (which basically is a different question), this block
if (user.shops.shopLocations == null)
{
user.shops.shopLocation = new List<Location>();
user.shops.shopLocation.Add(someLocation);
await context.shopDB.SaveChangesAsync();
}
looks suspicious. Your shopLocations member is declared as virtual, which means it's intended to use lazy loading, hence most probably will never be null. And even if it is null, you need to keep only the new part inside the if and do the rest outside, like this
if (user.shops.shopLocations == null)
user.shops.shopLocation = new List<Location>();
user.shops.shopLocation.Add(someLocation);
await context.shopDB.SaveChangesAsync();
1st Question
Because you are calling .ToList() at the end of your queries it will have to fetch all the rows from the db each time, so it will be much faster to do all your filtering in one LINQ .Where() call like this:
List<shop> sh = context.shopDB.Where(p => p.fruitName == "mango" && p.fruitName == "apple" && f.numberOfFruitsLeftToConsume >= 100 && f.numberOfFruitsLeftForStorage <= 100).ToList();
but if you don't call .ToList() at the end of first Linq query, spliting your query into two calls will be tottally fine and will yield the same performance as the previous approach like this:
var sh = context.shopDB.Where(p => p.fruitName == "mango" &&
p.fruitName == "apple");
List<shop> shList = sh.Where(f => f.numberOfFruitsLeftToConsume >= 100 &&
f.numberOfFruitsLeftForStorage <= 100).ToList();
2nd Question
when you initialize the Location for the shop, you must set the shopId property and then it should work, if not the problem might be with your database relationships.
I am importing a large amount of data from a csv file, (file size is over 100MB)
the code i'm using looks like this :
def errorLignes = []
def index = 1
csvFile.toCsvReader(['charset':'UTF-8']).eachLine { tokens ->
if (index % 100 == 0) cleanUpGorm()
index++
def order = Orders.findByReferenceAndOrganization(tokens[0],organization)
if (!order) {
order = new Orders()
}
if (tokens[1]){
def user = User.findByReferenceAndOrganization(tokens[1],organization)
if (user){
order.user = user
}else{
errorLignes.add(tokens)
}
}
if (tokens[2]){
def customer = Customer.findByCustomCodeAndOrganization(tokens[2],organization)
if (customer){
order.customer = customer
}else{
errorLignes.add(tokens)
}
}
if (tokens[3]){
order.orderType = Integer.parseInt(tokens[3])
}
// etc.....................
order.save()
}
and i'm using the cleanUpGorm method to clean session after each 100 entries
def cleanUpGorm() {
println "clean up gorm"
def session = sessionFactory.currentSession
session.flush()
session.clear()
propertyInstanceMap.get().clear()
}
I also turned 2nd level cache off
hibernate {
cache.use_second_level_cache = false
cache.use_query_cache = false
cache.provider_class = 'net.sf.ehcache.hibernate.EhCacheProvider'
}
the grails version of the project is 2.0.4 and as database i am using mysql
for every entry , i am doing 3 calls to a find
to check if the order already exists
to check if user is correct
to check if customer is correct
and finally i'm saving the order instance
the import process is too slow, i am wondering how can I speed up and optimise this code.
EDIT :
I found that the searchable plugin is also making it slower .
so , to get around this , I used the command :
searchableService.stopMirroring()
But it still not fast enough,I am finally changing the code to use groovy sql instead
This found this blog entry very useful:
http://naleid.com/blog/2009/10/01/batch-import-performance-with-grails-and-mysql/
You are already cleaning up GORM, but try cleaning every 100 entries:
def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP
propertyInstanceMap.get().clear()
Creating database indexes might help aswell and use default-storage-engine=innodb instead of MyISAM.
I'm also in the process of writing a number of services that will accomplish loads of very large datasets (multiple files of up to ~17million rows each). I initially tried the cleanUpGorm method you use, but found that, whilst it did improve things, the loading was still slow. Here's what I did to make it much faster:
Investigate what it is that is causing the app to actually be slow. I installed the Grails Melody plugin, then did a run-app then opened a browser at /monitoring. I could then see which routines took time to execute and what the worst-performing queries actually were.
Many of the Grails GORM methods map to a SQL ... where ... clause. You need to ensure that you have an index for each item used in a where clause for each query that you want to make faster, otherwise the method will become considerably slower the bigger your dataset is. This includes putting indexes on the id and version columns that are injected into each of your domain classes.
Ensure you have indexes set up for all of your hasMany and belongsTo relationships.
If the performance is still too slow, use Spring Batch. Even if you've never used it before, it should take you no time at all to set up a batch parse of a CSV file to parse into Grails domain objects. I suggest you use the grails-spring-batch plugin to do this and use the examples here to get a working implementation going quickly. It's extremely fast, very configurable and you don't have to mess around with cleaning up the session.
I had used batch insert while insert records, this is much faster than gorm cleanup method. Below example describes you how to implement it.
Date startTime = new Date()
Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
(1..50000).each {counter ->
Person person = new Person()
person.firstName = "abc"
person.middleName = "abc"
person.lastName = "abc"
person.address = "abc"
person.favouriteGame = "abc"
person.favouriteActor = "abc"
session.save(person)
if(counter.mod(100)==0) {
session.flush();
session.clear();
}
if(counter.mod(10000)==0) {
Date endTime =new Date()
println "Record inserted Counter =>"+counter+" Time =>"+TimeCategory.minus(endTime,startTime)
}
}
tx.commit();
session.close();
I have a mySQL database with a Website field that is of type VarChar. Some of these fields will be null on the database.
I have an MVC application that I sending the database information to.
I am trying to set up a filter on the Index page so I can filter by certain columns. I am using the Request.QueryString to do this.
switch (Request.QueryString["FilterOptionSelect"])
{ case "CountyName":
if (!string.IsNullOrEmpty(filteredText))
{
filteredText = filteredText.ToUpper();
var modelFiltered = from n in model
where n.CountyName.ToUpper().Contains(filteredText)
select n;
return View(modelFiltered);
}
break;
case "Website":
if (!string.IsNullOrEmpty(filteredText))
{
filteredText = filteredText.ToUpper();
var modelFiltered = from n in model
where n.CountyWebsite.ToUpper().Contains(filteredText)
select n;
return View(modelFiltered);
}
break;
}
The only problem I have is on the Website case. It gives me a Object reference not set to an instance of an object. on the WHERE CLAUSE of the Website case. When I debug, my model is not null (it has 130+ items inside...some with website info and some without website info).
I have already tried using the Lambda method (which had the same problem). I have also tried using where n.CountyWebsite.ToUpper().Contains(filteredText) && n.CountyWebsite != null which did not work either.
You were on the right track with your second attempt, but you need to switch the order of those statements.
where n.CountyWebsite != null && n.CountyWebsite.Contains(filteredText)
Just like all the other && operators, you don't want to evaulate any websites that are null, so do that operation first.
Also, .Contains in EF automatically is case-insensitive, so you don't need the ToUpper().
Try this, I think this will solve.
!String.IsNullOrEmpty(n.CountyWebsite) && n.CountyWebsite.Contains(filteredText)
Am trying to get the basics down before moving forward with neo4j. Love the querying aspect but now trying to delete using neo4jclient and stuck.
Simple setup
root-[:has_user]->user
and
user-[:friends_with]->friend`
For a user with the Id of 1 I would like to remove the specified from Id == 2. User 1 is no longer friends with User 2 :(
Anyway, using neo4jclient I first check to make sure that the users are friends in the first place with this:
if (client.Cypher.Start("root", client.RootNode)
.Match("root-[:HAS_USER]->user-[:FRIEND]->friend")
.Where((UserNode user, UserNode friend) => user.Id == 1 && friend.Id == id)
.Return<Node<UserNode>>("user")
.Results
.Count() == 1)
{
now I'm trying to delete:
client.Cypher.Start("root", client.RootNode)
.Match("root-[:HAS_USER]->user-[r]->friend")
.Where("user.Id = 1")
.And()
.Where("friend.Id = " + id)
.And()
.Where(string.Format("type(r) = 'FRIEND'"))
.Delete("r");
}
No errors, but the relationship is still there. Any ideas?
Update 11/12/2012
Got it working. I first updated by Neo4J instance with the stable 1.8. I think something with the latest neo4jclient and neo4j server were not working together. I first got the user's node based on the id, then from that node tested if the node had a relationship, then was able to remove it. Code below:
var currentUserNode = client.Cypher.Start("root", client.RootNode)
.Match("root-[:HAS_USER]->user")
.Where((UserNode user) => user.Id == 1)
.Return<Node<UserNode>>("user")
.Results.Single();
if (currentUserNode.StartCypher("user")
.Match("user-[r]->friend")
.Where("friend.Id = " + id).And()
.Where("type(r) = 'FRIEND'")
.Return<Node<UserNode>>("user")
.Results
.Count() == 1)
{
currentUserNode.StartCypher("user")
.Match("user-[r]->friend")
.Where("friend.Id = " + id).And()
.Where("type(r) = 'FRIEND'")
.Delete("r").ExecuteWithoutResults();
}
One way is to switch to use a CypherFluentQuery instead:
new CypherFluentQuery(client)
.Start("root", client.RootNode)
.Match("root-[:HAS_USER]->user-[r]->friend")
.Where("user.Val = 1").And()
.Where("friend.Val = " + 2).And()
.Where("type(r) = 'FRIEND'")
.Delete("r").ExecuteWithoutResults();
which will do what you want.
I believe this all stems from a bug: https://bitbucket.org/Readify/neo4jclient/issue/40/should-be-able-to-add-cypher-delete-clause
As to why client.Cypher.Start isn't working as it should, I'm not sure, the bug is fixed, and should be working from version 1.0.0.479 (current at time of writing is 1.0.0.496)
Use any Neo4jClient build after 1.0.0.500. See issue 45 if you're interested in why.
Don't forget to call ExecuteWithoutResults. The first code sample in your question is missing this. This is documented on https://bitbucket.org/Readify/neo4jclient/wiki/cypher