I am trying to add these filters to an already reasonable complex Umbraco Examine query, and have seen that you can't mix the API with raw lucene query, so the whole thing might have to be done raw, which I was trying to avoid since its a query builder that has quite a few dimensions.
Is this sort of thing possible with the API? I saw the GroupedOr/And but I don't see how that cuts it since those are exclusive/inclusive sql "In" type queries.
AND ((_nodeTypAlias: 'Event' AND eventDate:(0xx TO 0xx)) OR (NOT _nodeTypAlias: 'Event'))
AND ((_nodeTypAlias: 'Article' AND postDate:(0xx TO 0xx)) OR (NOT _nodeTypAlias: 'Article'))
Achieved it with the most excellent Lucence query builder API
var q = new QueryBuilder()
.Must
.MatchSubQuery(and => and
.Should
.MatchSubQuery(qq => qq
.Must
.MatchTerm("__NodeTypAlias", "Event")
.MatchRange(
"comparableEventDate",
DateTime.Now.ToString("yyyyMMddHHmm00000"),
DateTime.Now.AddYears(100).ToString("yyyyMMddHHmm00000"))
.MatchSubQuery(qq => qq
.MustNot
.MatchTerm("__NodeTypAlias", "Event"))
)
.Query
.ToString();
Outputs:
+((+__NodeTypAlias:Event +eventDate:[42920 TO 79444]) (-__NodeTypAlias:Event))
and pass that into Umbraco Examine:
ISearchCriteria criteria = searcher.CreateSearchCriteria();
var filter = criteria.RawQuery(q);
var results = searcher.Search(filter, MaxResults);
The Should operator translates to "OR" whilst Must translates to AND (Lucene nomenclature).
For reference, you need also to write new comparable date fields to the index in gathering node data:
private void ExamineEvents_GatheringNodeData(object sender, IndexingNodeDataEventArgs e, UmbracoHelper umbraco)
{
if (e.IndexType != IndexTypes.Content) return;
try
{
var content = new Node(e.NodeId);
if (e.Fields.ContainsKey("postDate"))
e.Fields.Add("comparablePostDate", DateTime.Parse(e.Fields["postDate"]).ToString("yyyyMMddHHmm00000"));
if (e.Fields.ContainsKey("eventDate"))
e.Fields.Add("comparableEventDate", DateTime.Parse(e.Fields["eventDate"]).ToString("yyyyMMddHHmm00000"));
AddAuthor(e.Fields, content);
}
catch (Exception ex)
{
LogHelper.Error(this.GetType(), "Error in Umbers custom ExamineEvents_GatheringNodeData: ", ex);
//does nowt!
throw;
}
}
Related
I have a few methods that act on a query one that is for a simple where filter an the other with an expression to be used in the where filter and the third also has an expression as a parameter.
public static IQueryable<Employee> FilterExpression(this IQueryable<Employee> employees,
Expression<Func<Employee, Boolean>> expression)
{
return employees.Where(expression);
}
public static IQueryable<Employee> Active(this IQueryable<Employee> employees)
{
return employees.Where(e => e.IsActive);
}
The first two execute fine without any problems.
var active = db.Employees.Active().Take(5).ToList();
var activeExpression = db.Employees.Take(5).FilterExpression((e) => e.IsActive).ToList();
The next one is being called in another extension method that returns a delegate for a select on a tasks collection.
public static Expression<Func<SpwTask, TaskView>> Tasks(this TaskService service)
{
return (x) => new TaskView()
{
NotifyList = db.TaskContacts.JoinedEmployees
(t => t.TaskId == x.Id).Select(Projections.SimpleEmployees)
};
}
public static IQueryable<Employee> JoinedEmployees(this IQueryable<TaskContact> contacts,
Expression<Func<TaskContact, Boolean>> expression)
{
var id = ServicesRoot.Company.Id;
return from c in contacts.Where(expression)
join e in db.Employees on c.EmployeeId equals e.Id
where e.CompanyId == id
select e;
}
The calling code looks like this
// db is the DbContext and the Tasks is the extension method
...
return db.Tasks.Select(this.Tasks());
...
The error I get is this:
System.NotSupportedException: 'LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[SafetyPlus.Data.Models.Employee] JoinedEmployees(System.Linq.IQueryable1[SafetyPlus.Data.TaskContact], System.Linq.Expressions.Expression1[System.Func2[SafetyPlus.Data.TaskContact,System.Boolean]])' method, and this method cannot be translated into a store expression.'
Are there any ways to work around this problem? It would be really nice to reused this an other similar queries. It seems like this should work.
Inline version:
public static Expression<Func<SpwTask, TaskView>> Tasks(this TaskService service)
{
return (x) => new TaskView()
{
NotifyList =
(from c in service.db.TaskContacts.
where c.TaskId == x.Id
join e in db.Employees on c.EmployeeId equals e.Id
where e.CompanyId == id
select e).Select(Projections.SimpleEmployees)
};
}
When I run the code inline version of the query I get a single SQL produced which is optimized and what I want but is not reusable.
I had thought the Compile() solution given below was the answer that I needed, at first, but upon looking at the SQL Profiler I realize that each row of the outer task query is running a single query for each task instead of the single query for the whole dataset. This pretty much defeats the purpose of reusing the query.
You should pass it as a delegate without invoking it in the Select and compile before use.
Try this:
db.Tasks.Select(Tasks.Compile());
On my Swagger page, I am (mostly) able to order the operations as described on the Swashbuckle page.
Below the operations is a "Schemas" section showing the data structures used by the actions. These data structures appear in a arbitrary order. I would like to sort them.
The question Swagger sort Schema Defintions superficially looks like the same question, but in that question "sort" is used in the sense of "sorting the items into different bins", not "ordering a list" which is what I want.
I have made a document filter that "works", but when I look at the code I wrote, I die a little inside.
Is there a more correct way to do this?
Edit: To be specific, what I object to about this code is that it is "working" by sorting the entries in a Dictionary, which is just bad ( see this question ).
Turns out the answer was simply to use a SortedDictionary:
openApiDoc.Components.Schemas = new System.Collections.Generic.SortedDictionary<string, OpenApiSchema>(openApiDoc.Components.Schemas);
Actually, you were on the right track with the document filter!
In Startup.cs, ConfigureServices method:-
services.AddSwaggerGen(c =>
{
// ...
// For our document filtering needs.
c.DocumentFilter<DocumentFilter>();
});
And here is the document filter implementation:-
using System.Linq;
public class DocumentFilter : IDocumentFilter
{
public DocumentFilter()
{
}
// Implements IDocumentFilter.Apply().
public void Apply(OpenApiDocument swaggerDoc, DocumentFilterContext context)
{
if (swaggerDoc == null)
return;
// Re-order the schemas alphabetically.
swaggerDoc.Components.Schemas = swaggerDoc.Components.Schemas.OrderBy(kvp => kvp.Key, StringComparer.InvariantCulture)
.ToDictionary(kvp => kvp.Key, kvp => kvp.Value);
}
}
when I look at the code I wrote, I die a little inside - embrace the glory of LINQ, and you will be proud of your code!
This wouldn't change the final display order for me, but MikeBeaton's suggestion whilst reporting a Sort Schema Issue on GitHub worked like a charm..
app.UseSwagger(c =>
{
c.PreSerializeFilters.Add((swagger, _) =>
{
if (swagger.Components != null && swagger.Components.Schemas != null)
{
var replacement = new Dictionary<string, OpenApiSchema>();
foreach (var kv in swagger.Components.Schemas.OrderBy(p => p.Key))
{
replacement.Add(kv.Key, kv.Value);
}
swagger.Components.Schemas = replacement;
}
});
})
I have multiple subscriptions from Cloud PubSub to read based on certain prefix pattern using Apache Beam. I extend PTransform class and implement expand() method to read from multiple subscriptions and do Flatten transformation to the PCollectionList (multiple PCollection on from each subscription). I have a problem to pass subscription prefix as ValueProvider into the expand() method, since expand() is called on template creation time, not when launching the job. However, if I only use 1 subscription, I can pass ValueProvider into PubsubIO.readStrings().fromSubscription().
Here's some sample code.
public class MultiPubSubIO extends PTransform<PBegin, PCollection<PubsubMessage>> {
private ValueProvider<String> prefixPubsub;
public MultiPubSubIO(#Nullable String name, ValueProvider<String> prefixPubsub) {
super(name);
this.prefixPubsub = prefixPubsub;
}
#Override
public PCollection<PubsubMessage> expand(PBegin input) {
List<String> myList = null;
try {
// prefixPubsub.get() will return error
myList = PubsubHelper.getAllSubscription("projectID", prefixPubsub.get());
} catch (Exception e) {
LogHelper.error(String.format("Error getting list of subscription : %s",e.toString()));
}
List<PCollection<PubsubMessage>> collectionList = new ArrayList<PCollection<PubsubMessage>>();
if(myList != null && !myList.isEmpty()){
for(String subs : myList){
PCollection<PubsubMessage> pCollection = input
.apply("ReadPubSub", PubsubIO.readMessagesWithAttributes().fromSubscription(this.prefixPubsub));
collectionList.add(pCollection);
}
PCollection<PubsubMessage> pubsubMessagePCollection = PCollectionList.of(collectionList)
.apply("FlattenPcollections", Flatten.pCollections());
return pubsubMessagePCollection;
} else {
LogHelper.error(String.format("No subscription with prefix %s found", prefixPubsub));
return null;
}
}
public static MultiPubSubIO read(ValueProvider<String> prefixPubsub){
return new MultiPubSubIO(null, prefixPubsub);
}
}
So I'm thinking of how to use the same way PubsubIO.read().fromSubscription() to read from ValueProvider. Or am I missing something?
Searched links:
extract-value-from-valueprovider-in-apache-beam - Answer talked about using DoFn, while I need PTransform that receives PBegin.
Unfortunately this is not possible currently:
It is not possible for the value of a ValueProvider to affect transform expansion - at expansion time, it is unknown; by the time it is known, the pipeline shape is already fixed.
There is currently no transform like PubsubIO.read() that can accept a PCollection of topic names. Eventually there will be (it is enabled by Splittable DoFn), but it will take a while - nobody is working on this currently.
You can use MultipleReadFromPubSub from apache beam io module https://beam.apache.org/releases/pydoc/2.27.0/_modules/apache_beam/io/gcp/pubsub.html
topic_1 = PubSubSourceDescriptor('projects/myproject/topics/a_topic')
topic_2 = PubSubSourceDescriptor(
'projects/myproject2/topics/b_topic',
'my_label',
'my_timestamp_attribute')
subscription_1 = PubSubSourceDescriptor(
'projects/myproject/subscriptions/a_subscription')
results = pipeline | MultipleReadFromPubSub(
[topic_1, topic_2, subscription_1])
The model of my project is database first, and uses remote access to database on another server.
I need to use raw SQL query because my query is very complex and I feel more comfortable in SQl not LINQ.
This is how I do:
string query = "select * from Inquiry_TBL where ...";
using (educationEntities db = new educationEntities())
{
var list = db.Database.SqlQuery<Inquiry_TBL>(query);
ViewData["total"] = list.Count();
}
The problem is sometimes I get the query result within a second, sometimes it just keep loading for a long time and gives me an error that 'Calling 'Read' when the data reader is closed is not a valid operation.'
Why is that? Is there something wrong with my code, or because I'm using remote access to another server? Will switching to local server solve the problem?
The Entity Framework Code First API includes methods that enable you to pass SQL commands directly to the database. You have the following options:
• Use the DbSet.SqlQuery method for queries that return entity types. The returned objects must be of the type expected by the DbSet object, and they are automatically tracked by the database context unless you turn tracking off. (See the following section about the AsNoTracking method.)
• Use the Database.SqlQuery method for queries that return types that aren't entities. The returned data isn't tracked by the database context, even if you use this method to retrieve entity types.
• Use the Database.ExecuteSqlCommand for non-query commands.
Calling a Query that Returns Entities:
public async Task<ActionResult> Details(int? id)
{
if (id == null)
{
return new HttpStatusCodeResult(HttpStatusCode.BadRequest);
}
// Commenting out original code to show how to use a raw SQL query.
//Department department = await db.Departments.FindAsync(id);
// Create and execute raw SQL query.
string query = "SELECT * FROM Department WHERE DepartmentID = #p0";
Department department = await db.Departments.SqlQuery(query, id).SingleOrDefaultAsync();
if (department == null)
{
return HttpNotFound();
}
return View(department);
}
Calling a Query that Returns Other Types of Objects:
public ActionResult About()
{
//Commenting out LINQ to show how to do the same thing in SQL.
//IQueryable<EnrollmentDateGroup> = from student in db.Students
// group student by student.EnrollmentDate into dateGroup
// select new EnrollmentDateGroup()
// {
// EnrollmentDate = dateGroup.Key,
// StudentCount = dateGroup.Count()
// };
// SQL version of the above LINQ code.
string query = "SELECT EnrollmentDate, COUNT(*) AS StudentCount "
+ "FROM Person "
+ "WHERE Discriminator = 'Student' "
+ "GROUP BY EnrollmentDate";
IEnumerable<EnrollmentDateGroup> data = db.Database.SqlQuery<EnrollmentDateGroup>(query);
return View(data.ToList());
}
Calling an Update Query:
[HttpPost]
public ActionResult UpdateCourseCredits(int? credit)
{
if (credit != null)
{
ViewBag.RowsAffected = db.Database.ExecuteSqlCommand(
"UPDATE Course SET Credits = Credits * {0}", credit);
}
return View();
}
For more information have a look at Advanced Entity Framework 6 Scenarios for an MVC 5 Web Application (12 of 12).
I am using the repository pattern in a asp.net mvc application (v3) and EntityFramework (v4).
I would like to add to the repository paging functionality. I have the following sample code:
public PagedResult<T> GetPaged(int offset = 0, int limit = PagedResult<T>.NoLimit, Expression<Func<T, bool>> predicate = null)
{
var res = BaseQ.Where(_predicate);//.Skip(offset);
if (predicate != null)
res = res.Where(predicate);
res = res.Skip(offset);
if (limit != PagedResult<T>.NoLimit)
{
res = res.Take(limit);
}
return new PagedResult<T>(res, res.Count(), offset, limit);
}
However, this will not work because Entity framework throws an exception that I should call OrderBy before Skip.
I am wondering how can I implement this in a generic way so that the order by expression will be plugged in from the outside. I would then pass it as a parameter.
Perhaps there is also another way of working around that problem.
[EDIT] I found out that this can be done by passing in the name of the property and creating an expression out of that, but I would like to actually just use it as I am using it in the OrderBy. So just pass in (c=>c.ID) for example.
Thanks in advance for any ideas
I have decided to go with passing in a string and creating the expression from it