I have just started using Neo4jClient and Cypher and surprisingly I don't find any example on the net that's using the DateTime filed in the the cypher query where clause .
when I an trying to get some nodes filtered on a DateTime property, the query is not returning any results, here is an example of what I was trying:
Say I am looking for all make employees in HR deportment whose Date of birth is within a time range. the query I am trying to build is as shown below.
client.Cypher
.Start(new { company = companyNode.Reference})
.Match("(department)<-[:BELONGS_TO]-(employee)-[:BELONGS_TO]->(company)")
.Where<Department>(department=>department.Name=='Human Resource')
.AndWhere<Employee>(employee=> employee.DOB >= searchStart && employee.DOB<= searchEnd)
.ReturnDistinct((employee)=> new {name = employee.Name});
here Employee->DOB/searchStart/searchEnd are all DateTimeOffset fields and the data stored in the graph via the neo4jclient is represented as "1990-09-28T19:02:21.7576376+05:30"
when i am debugging the code I see that Neo4jClient is actually representing the query as something like this
AND ((employee.DOB >=10/3/1988 8:16:41 PM +03:00) AND (employee.DOB <=10/3/2003 8:16:41 PM +03:00))
when I get rid of the DOB where clause i do get results.
I would really appreciate if someone can point me to how the DateTimeOffset property can be used in the queries.
Regards, Kiran
Using the DateTimeOffset works fine for me:
private static IList<Node<DateOffsetNode>> Between(IGraphClient client, DateTimeOffset from, DateTimeOffset to)
{
ICypherFluentQuery<Node<DateOffsetNode>> query = new CypherFluentQuery(client)
.Start(new { n = All.Nodes })
.Where((DateOffsetNode n) => n.Date <= to && n.Date >= from)
.Return<Node<DateOffsetNode>>("n");
return query.Results.ToList();
}
Where DateOffsetNode is just:
public class DateOffsetNode { public DateTimeOffset Date { get;set; } }
But another way is to store the ticks value and compare with that instead:
.Where((DateObj o) => o.Ticks < DateTime.Now.Date.Ticks)
I typically define DateObj like:
public class DateObj {
public long Ticks { get;set; }
public int Year { get;set; }
public int Month { get;set;}
public int Day { get;set;}
public DateObj(){}
public DateObj(DateTime dt){
Ticks = dt.Date.Ticks;
Year = dt.Date.Year;
Month = dt.Date.Month;
Day = dt.Date.Day;
}
}
So I can also do things like:
.Where((DateObj o) => o.Year == 2013)
The equivalent for a time is to use something like the TotalMilliseconds property on the DateTime object as well:
.Where((TimeObj o) => o.TimeMs < DateTime.Now.TimeOfDay.TotalMilliseconds)
There's also whole chunk of the O'Reilly book about Neo4J (that they give an electronic copy of away free from the Neo4J Site here: http://www.neo4j.org/learn) about managing dates, which I believe echoes the information in the blog linked to above. I.e. they recommend putting dates into the database as nodes (i..e. year nodes with relationships to month nodes, with relationships to day nodes) and then linking all date sensitive nodes to the relevant day.
Would get a bit painful if you want to measure things in Milliseconds, though...
Related
I have the below example.
I was wondering what is the best and quickest way to add a list of nodes and edges in a single transaction? I use standard C# Neo4j .NET packages but open to the Neo4jClient as I've read that's faster. Anything that supports .NET and 4.5 to be honest.
I have an lists of about 60000 FooA objects that need to be added into Neo4j and it can take hours!
Firstly, FooB objects hardly change so I don't have to add them everyday. The performance issues is with adding new FooA objects twice a day.
Each FooA object has a list of FooB objects has two lists containing the relationships I need to add; RelA and RelB (see below).
public class FooA
{
public long Id {get;set;} //UniqueConstraint
public string Name {get;set;}
public long Age {get;set;}
public List<RelA> ListA {get;set;}
public List<RelB> ListB {get;set;}
}
public class FooB
{
public long Id {get;set;} //UniqueConstraint
public string Prop {get;set;}
}
public class RelA
{
public string Val1 {get;set;}
pulic NodeTypeA Node {get;set;
}
public class RelB
{
public FooB Start {get;set;}
public FooB End {get;set;}
public string ValExample {get;set;}
}
Currently, I check if Node 'A' exists by matching by Id. If it does then I completely skip and move onto the next item. If not, I create Node 'A' with its own properties. I then create the edges with their own unique properties.
That's quite a few transactions per item. Match node by Id -> add nodes -> add edges.
foreach(var ntA in FooAList)
{
//First transaction.
MATCH (FooA {Id: ntA.Id)})
if not exists
{
//2nd transaction
CREATE (n:FooA {Id: 1234, Name: "Example", Age: toInteger(24)})
//Multiple transactions.
foreach (var a in ListA)
{
MATCH (n:FooA {Id: ntA.Id}), (n2:FooB {Id: a.Id }) with n,n2 LIMIT 1
CREATE (n)-[:RelA {Prop: a.Val1}]-(n2)
}
foreach (var b in Listb)
{
MATCH (n:FooB {Id: b.Start.Id}), (n2:FooB {Id: b.End.Id }) with n,n2 LIMIT 1
CREATE (n)-[:RelA {Prop: b.ValExample}]-(n2)
}
}
How would one go about adding a list of FooA's using for example Neo4jClient and UNWIND or any other way apart from CSV import.
Hope that makes sense, and thanks!
The biggest problem is the nested lists, which mean you have to do your foreach loops, so you end up executing a minimum of 4 queries per FooA, which for 60,000 - well - that's a lot!
Quick Note RE: Indexing
First and foremost - you need an index on the Id property of your FooA and FooB nodes, this will speed up your queries dramatically.
I've played a bit with this, and have it storing 60,000 FooA entries, and creating 96,000 RelB instances in about 12-15 seconds on my aging computer.
The Solution
I've split it into 2 sections - FooA and RelB:
FooA
I've had to normalise the FooA class into something I can use in Neo4jClient - so let's introduce that:
public class CypherableFooA
{
public CypherableFooA(FooA fooA){
Id = fooA.Id;
Name = fooA.Name;
Age = fooA.Age;
}
public long Id { get; set; }
public string Name { get; set; }
public long Age { get; set; }
public string RelA_Val1 {get;set;}
public long RelA_FooBId {get;set;}
}
I've added the RelA_Val1 and RelA_FooBId properties to be able to access them in the UNWIND. I convert your FooA using a helper method:
public static IList<CypherableFooA> ConvertToCypherable(FooA fooA){
var output = new List<CypherableFooA>();
foreach (var element in fooA.ListA)
{
var cfa = new CypherableFooA(fooA);
cfa.RelA_FooBId = element.Node.Id;
cfa.RelA_Val1 = element.Val1;
output.Add(cfa);
}
return output;
}
This combined with:
var cypherable = fooAList.SelectMany(a => ConvertToCypherable(a)).ToList();
Flattens the FooA instances, so I end up with 1 CypherableFooA for each item in the ListA property of a FooA. e.g. if you had 2 items in ListA on every FooA and you have 5,000 FooA instances - you would end up with cypherable containing 10,000 items.
Now, with cypherable I call my AddFooAs method:
public static void AddFooAs(IGraphClient gc, IList<CypherableFooA> fooAs, int batchSize = 10000, int startPoint = 0)
{
var batch = fooAs.Skip(startPoint).Take(batchSize).ToList();
Console.WriteLine($"FOOA--> {startPoint} to {batchSize + startPoint} (of {fooAs.Count}) = {batch.Count}");
if (batch.Count == 0)
return;
gc.Cypher
.Unwind(batch, "faItem")
.Merge("(fa:FooA {Id: faItem.Id})")
.OnCreate().Set("fa = faItem")
.Merge("(fb:FooB {Id: faItem.RelA_FooBId})")
.Create("(fa)-[:RelA {Prop: faItem.RelA_Val1}]->(fb)")
.ExecuteWithoutResults();
AddFooAs(gc, fooAs, batchSize, startPoint + batch.Count);
}
This batches the query into batches of 10,000 (by default) - this takes about 5-6 seconds on mine - about the same as if I try all 60,000 in one go.
RelB
You store RelB in your example with FooA, but the query you're writing doesn't use the FooA at all, so what I've done is extract and flatten all the RelB instances in the ListB property:
var relBs = fooAList.SelectMany(a => a.ListB.Select(lb => lb));
Then I add them to Neo4j like so:
public static void AddRelBs(IGraphClient gc, IList<RelB> relbs, int batchSize = 10000, int startPoint = 0)
{
var batch = relbs.Select(r => new { StartId = r.Start.Id, EndId = r.End.Id, r.ValExample }).Skip(startPoint).Take(batchSize).ToList();
Console.WriteLine($"RELB--> {startPoint} to {batchSize + startPoint} (of {relbs.Count}) = {batch.Count}");
if(batch.Count == 0)
return;
var query = gc.Cypher
.Unwind(batch, "rbItem")
.Match("(fb1:FooB {Id: rbItem.StartId}),(fb2:FooB {Id: rbItem.EndId})")
.Create("(fb1)-[:RelA {Prop: rbItem.ValExample}]->(fb2)");
query.ExecuteWithoutResults();
AddRelBs(gc, relbs, batchSize, startPoint + batch.Count);
}
Again, batching defaulted to 10,000.
Obviously time will vary depending on the number of rels in ListB and ListA - My tests has one item in ListA and 2 in ListB.
I have the following entity:
#NodeEntity
public class Action {
...
#Index(unique = false)
private Date createDate;
...
}
I need to get the last Action that was created during some previous period of time.
In order to do this I have implemented the following repository method:
#Repository
public interface ActionRepository {
#Query("MATCH (a:Action)-[:CREATED_BY]->(u:User) WHERE a.entityType = {entityType} AND a.createDate <= {minCreateDate} AND u.id = {userId} RETURN a ORDER BY a.createDate DESC LIMIT 1")
Action findLastByEntityTypeForUser(#Param("entityType") String entityType, #Param("minCreateDate") Date minCreateDate, #Param("userId") Long userId);
}
I use the following code to test this method:
decisionDao.create("Decision2", "Decision2 description", null, false, null, user1);
Date minStartDate = DateUtils.addMilliseconds(new Date(), -1000 * 60);
Action user1LastAction = actionRepository.findLastByEntityTypeForUser(Decision.class.getSimpleName(), minStartDate, user1.getId());
assertNotNull(user1LastAction); // test fails here because of NPE
but without this part of the Cypher query AND a.createDate <= {minCreateDate} I can successfully find Action instance.
At Neo4j level my data looks like:
{
"updateDate":"2017-10-08T12:21:39.15
3Z",
"entityName":"General",
"en
tityType":"CriterionGroup",
"entityId":1,
"id":1,
"type":"CREATE",
"createDate":"2017-10-08T12:21:39.153Z"
}
What am I doing wrong and how to properly compare the dates with SDN/OGM and Cypher?
Also, is there any way to tell SDN/OGM to store java.util.Date object as long milliseconds and as String?
the minCreateDate parameter you use for your find-Method is of type Date and the createDate property is a String. So, this part a.createDate <= {minCreateDate} is basically comparing the String representation of minCreateDate and the String property createDate.
In my projects, I usually save the dates and timestamps as long both in the database and in my code.
Or even better, if the date attributes are crucial for my application, I'm using the "Time Tree Model" approach: https://graphaware.com/neo4j/2014/08/20/graphaware-neo4j-timetree.html
I'm trying to do a compiled query but I just want it to return an int
public Func<DataContext, DateTime, int>
GetNextTourNo = CompiledQuery.Compile((DataContext db, DateTime day) => ((from b in db.GetTable<BookingType>()
where b.RecordType == "H" && b.TourStartDateTime.Value.Date == day.Date
orderby b.TourID descending
select new { nextID = b.TourID +1 }).Single()));
You could just return nextID property from selected single anonymous object
select new { nextID = b.TourID +1 }).Single().nextID
Can you provide a bit more information on the anonymous type and the context of the compiled query?
Also if you are using the query directly in linq to Entity the date comparison will not work. Entity Functions need to be used for this. This could cause the invalid return.
var result =
(from bd in context.tblBasicDetails
from pd in context.tblPersonalDetails.Where(x => x.UserId == bd.UserId).DefaultIfEmpty()
from opd in context.tblOtherPersonalDetails.Where(x => x.UserId == bd.UserId).DefaultIfEmpty()
select new clsProfileDate()
{
DOB = pd.DOB
});
foreach (clsProfileDate prod in result)
{
prod.dtDOB = !string.IsNullOrEmpty(prod.DOB) ? Convert.ToDateTime(prod.DOB) : DateTime.Today;
int now = int.Parse(DateTime.Today.ToString("yyyyMMdd"));
int dob = int.Parse(prod.dtDOB.ToString("yyyyMMdd"));
string dif = (now - dob).ToString();
string age = "0";
if (dif.Length > 4)
age = dif.Substring(0, dif.Length - 4);
prod.Age = Convert.ToInt32(age);
}
GetFinalResult(result);
protected void GetFinalResult(IQueryable<clsProfileDate> result)
{
int from;
bool bfrom = Int32.TryParse(ddlAgeFrom.SelectedValue, out from);
int to;
bool bto = Int32.TryParse(ddlAgeTo.SelectedValue, out to);
result = result.AsQueryable().Where(p => p.Age >= from);
}
Here I am getting an exception:
The specified type member "Age" is not supported in LINQ to Entities.
Only initializers, entity members, and entity navigation properties
are supported.
Where Age is not in database it is property I created in clsProfileDate class to calculate Age from DOB. Any solution to this?
You cannot use properties that are not mapped to a database column in a Where expression. You must build the expression based on mapped properties, like:
var date = DateTime.Now.AddYears(-from);
result = result.Where(p => date >= p.DOB);
// you don't need `AsQueryable()` here because result is an `IQueryable` anyway
As a replacement for your not mapped Age property you can extract this expression into a static method like so:
public class clsProfileDate
{
// ...
public DateTime DOB { get; set; } // property mapped to DB table column
public static Expression<Func<clsProfileDate, bool>> IsOlderThan(int age)
{
var date = DateTime.Now.AddYears(-age);
return p => date >= p.DOB;
}
}
And then use it this way:
result = result.Where(clsProfileDate.IsOlderThan(from));
A lot of people are going to say this is a bad answer because it is not best practice but you can also convert it to a List before your where.
result = result.ToList().Where(p => date >= p.DOB);
Slauma's answer is better, but this would work as well. This cost more because ToList() will execute the Query against the database and move the results into memory.
You will also get this error message when you accidentally forget to define a setter for a property.
For example:
public class Building
{
public string Description { get; }
}
var query =
from building in context.Buildings
select new
{
Desc = building.Description
};
int count = query.ToList();
The call to ToList will give the same error message. This one is a very subtle error and very hard to detect.
I forgot to select the column (or set/map the property to a column value):
IQueryable<SampleTable> queryable = from t in dbcontext.SampleTable
where ...
select new DataModel { Name = t.Name };
Calling queryable.OrderBy("Id") will throw exception, even though DataModel has property Id defined.
The correct query is:
IQueryable<SampleTable> queryable = from t in dbcontext.SampleTable
where ...
select new DataModel { Name = t.Name, Id = t.Id };
In my case, I was getting this error message only in Production but not when run locally, even though my application's binaries were identical.
In my application, I'm using a custom DbModelStore so that the runtime-generated EDMX is saved to disk and loaded from disk on startup (instead of regenerating it from scratch) to reduce application startup time - and due to a bug in my code I wasn't invalidating the EDMX file on-disk - so Production was using an older version of the EDMX file from disk that referenced an older version of my application's types from before I renamed the type-name in the exception error message.
Deleting the cache file and restarting the application fixed it.
Advanced answer:
Search in edmx file EntitySetMapping and check if the field is mapped to a column in database:
<EntitySetMapping Name="MY_TABLE">
<EntityTypeMapping TypeName="MYMODEL.MY_TABLE">
<MappingFragment StoreEntitySet="MY_TABLE">
<ScalarProperty Name="MY_COLUMN" ColumnName="MY_COLUMN_NAME" />
</MappingFragment>
</EntityTypeMapping>
</EntitySetMapping>
I was having this problem because the edmx had changes I didn't want and through git I discarded too many changes...
Checking Count() before the WHERE clause solved my problem. It is cheaper than ToList()
if (authUserList != null && _list.Count() > 0)
_list = _list.Where(l => authUserList.Contains(l.CreateUserId));
how can you do multiple "group by's" in linq to sql?
Can you please show me in both linq query syntax and linq method syntax.
Thanks
Edit.
I am talking about multiple parameters say grouping by "sex" and "age".
Also I forgot to mention how would I say add up all the ages before I group them.
If i had this example how would I do this
Table Product
ProductId
ProductName
ProductQty
ProductPrice
Now imagine for whatever reason I had tons of rows each with the same ProductName, different ProductQty and ProductPrice.
How would I groupt hem up by Product Name and add together ProductQty and ProductPrice?
I know in this example it probably makes no sense why there would row after row with the same product name but in my database it makes sense(it is not products).
To group by multiple properties, you need to create a new object to group by:
var groupedResult = from person in db.People
group by new { person.Sex, person.Age } into personGroup
select new
{
personGroup.Key.Sex,
personGroup.Key.Age,
NumberInGroup = personGroup.Count()
}
Apologies, I didn't see your final edit. I may be misunderstanding, but if you sum the age, you can't group by it. You could group by sex, sum or average the age...but you couldn't group by sex and summed age at the same time in a single statement. It might be possible to use a nested LINQ query to get the summed or average age for any given sex...bit more complex though.
EDIT:
To solve your specific problem, it should be pretty simple and straightforward. You are grouping only by name, so the rest is elementary (example updated with service and concrete dto type):
class ProductInventoryInfo
{
public string Name { get; set; }
public decimal Total { get; set; }
}
class ProductService: IProductService
{
public IList<ProductInventoryInfo> GetProductInventory()
{
// ...
var groupedResult = from product in db.Products
group by product.ProductName into productGroup
select new ProductInventoryInfo
{
Name = productGroup.Key,
Total = productGroup.Sum(p => p.ProductCost * p.ProductQty)
}
return groupedResult.ToList();
}
}