ElasticSearch MapperParsingExceptionn with geo_distance filter - grails

I'm using the grails elasticsearch plugin in my application, and I am running across an odd exception.
I have a method that builds a request using a geo_distance filter that way:
def activityList = ElasticSearchService.search(
[types:[DomainA, DomainB],
sort: orderBy, order: orderAs, size: (int)maxRes],
{
// bla bla bla some working closure building a working query
}
if(centerLat != -1){
filter{
geo_distance(
'distance': searchRadius,
'activityLocation': [lat: (Double)centerLat, lon: (Double)centerLon]
)
}
}
}
)
And whenever I try to use the method with the filter (that is, when I set for example searchRadius to '5km' and centerLat/ centerLon to correct coordinates, my ElasticSearch node goes crazy and keeps logging the following error until I shut it down:
| Error 2014-08-27 10:31:49,076 [elasticsearch[Agent Zero][bulk][T#2]] ERROR index.IndexRequestQueue - Failed bulk item: MapperParsingException[failed to parse]; nested: ElasticsearchParseException[field must be either 'lat', 'lon' or 'geohash'];
I tried looking around the web for the reason why this MappingParserException is thrown, and I ended up looking at the source code for the org.elasticsearch.common.geo.GeoUtils class. Apparenty, the exception is thrown because my lat and lon fields are not numbers (see lines 372 and 381).
Why is this happening? Did I declare my filter wrong?

I think you have a problem is with your index types mapping, in order to understand what is the problem you have to post some example of the data you are indexing and type mappings (you can get your current mappings from host:port/{index}/_mapping/{type}).
Try to define the mapping as explained here: http://noamt.github.io/elasticsearch-grails-plugin/guide/mapping.html look at 3.3.2 GeoPoint, make sure you define your data points correctly.

Assuming the mapping for activityLocation is "geo_point" then you shouldn't need to preface your coordindate with "lat" and "lon" and should just be able to send
'activityLocation': [(Double)centerLat,(Double)centerLon]
I think this "lat" "lon" preface might have been supported as a geo_point format in earlier versions, but as of 1.0.0 I don't think it still is.

It's been a while and this problem made me lose my nerves quite a bit, but I have finally solved it.
I was mapping the following classes that way:
class A{
static searchable={
activityLocation geoPoint:true, component:true
// some other mappings on some other fields
}
Location activityLocation
// some other fields
}
a
class Location{
static searchable = true
String locationName
Double lat
Double lon
}
There were two problems:
I didn't want to use the root false option when mapping my activityLocation objects, because I wanted to be able to search them directly, which caused my A objects to contain the following property (note the presence of the id field):
'activityLocation' : { 'id':'1', 'locationName' : 'foo', 'lat': '42.5464', 'lon': '65.5647' }
which isn't a geoPoint because it has more than just a lat and lon field.
I was mapping the locationName property as part of my activityLocation objects, which makes them not geoPoints.
I ended up solving the problem by changing the mapping to the following values for activityLocation:
ActivityLocation{
static searchable = {
only = ['lat', 'lon']
root false
}
String locationName
Double lat
Double lon
}
This solved my problem and my search went well from then on.
I have to add that I was pretty confused by the docs regarding this and I am a bit disappointed that I can't map a geoPoint with more attributes. I wish it were possible to have other fields in the mapping.
Furthermore, I wish the plugin gave me better access to the ES logs, it took me a while to figure out where to look for the error code.

I have managed to fix this with few lines of coded added to "DeepDomainClassMarshaller.groovy" and overriding this file.
Previous code:-
if (DomainClassArtefactHandler.isDomainClass(propertyClass)) {
String searchablePropertyName = getSearchablePropertyName()
if (propertyValue.class."$searchablePropertyName") {
// todo fixme - will throw exception when no searchable field.
marshallingContext.lastParentPropertyName = prop.name
marshallResult += [(prop.name): ([id: propertyValue.ident(), 'class': propertyClassName] + marshallingContext.delegateMarshalling(propertyValue, propertyMapping.maxDepth))]
} else {
marshallResult += [(prop.name): [id: propertyValue.ident(), 'class': propertyClassName]]
}
// Non-domain marshalling
}
New Code:-
if (DomainClassArtefactHandler.isDomainClass(propertyClass)) {
String searchablePropertyName = getSearchablePropertyName()
if (propertyValue.class."$searchablePropertyName") {
// todo fixme - will throw exception when no searchable field.
marshallingContext.lastParentPropertyName = prop.name
if(propertyMapping?.isGeoPoint())
marshallResult += [(prop.name): (marshallingContext.delegateMarshalling(propertyValue, propertyMapping.maxDepth))]
else
marshallResult += [(prop.name): ([id: propertyValue.ident(), 'class': propertyClassName] + marshallingContext.delegateMarshalling(propertyValue, propertyMapping.maxDepth))]
} else {
marshallResult += [(prop.name): [id: propertyValue.ident(), 'class': propertyClassName]]
}
// Non-domain marshalling
}
Let me know in case issue still persists.
Note:- I am still using elasticsearch:0.0.3.3.

Related

Hashset handling to avoid stuck in loop during iteration

I'm working on image mining project, and I used Hashset instead of array to avoid adding duplicate urls while gathering urls, I reached to the point of code to iterate the Hashset that contains the main urls and within the iteration I go and download the the page of the main URL and add them to the Hashet, and go on , and during iteration I should exclude every scanned url, and also exclude ( remove ) every url that end with jpg, this until the Hashet of url count reaches 0, the question is that I faced endless looping in this iteration , where I may get url ( lets call it X )
1- I scan the page of url X
2- get all urls of page X ( by applying filters )
3- Add urls to the Hashset using unioinwith
4- remove the scanned url X
the problem comes here when one of the URLs Y, when scanned bring X again
shall I use Dictionary and the key as "scanned" ?? I will try and post the result here, sorry it comes to my mind after I posted the question...
I managed to solve it for one url, but it seems it happens with other urls to generate loop, so how to handle the Hashset to avoid duplicate even after removing the links,,, I hope that my point is clear.
while (URL_Can.Count != 0)
{
tempURL = URL_Can.First();
if (tempURL.EndsWith("jpg"))
{
URL_CanToSave.Add(tempURL);
URL_Can.Remove(tempURL);
}
else
{
if (ExtractUrlsfromLink(client, tempURL, filterlink1).Contains(toAvoidLoopinLinks))
{
URL_Can.Remove(tempURL);
URL_Can.Remove(toAvoidLoopinLinks);
}
else
{
URL_Can.UnionWith(ExtractUrlsfromLink(client, tempURL, filterlink1));
URL_Can.UnionWith(ExtractUrlsfromLink(client, tempURL, filterlink2));
URL_Can.Remove(tempURL);
richTextBox2.PerformSafely(() => richTextBox2.AppendText(tempURL + "\n"));
}
}
toAvoidLoopinLinks = tempURL;
}
Thanks for All, I managed to solve this issue using Dictionary instead of Hashset, and use the Key to hold the URL , and the value to hold int , to be 1 if the urls is scanned , or 0 if the url still not processed, below is my code.
I used another Dictionary "URL_CANtoSave to hold the url that ends with jpg "my target"...and this loop of While..can loop until all the url of the website ran out based on the values you specify in the filter string variable that you parse the urls accordingly.
so to break the loop you can specify amount of images url to get in the URL_CantoSave.
return Task.Factory.StartNew(() =>
{
try
{
string tempURL;
int i = 0;
// I used to set the value of Dictionary Key, 1 or 0 ( 1 means scanned,
0 means not yet and to iterate until all the Dictionry Keys are scanned or you break in the middle based on how much images urls you collected in the other Dictionary
while (URL_Can.Values.Where(value => value.Equals(0)).Any())
{
// take 1 key and put it in temp variable
tempURL = URL_Can.ElementAt(i).Key;
// check if it ends with your target file extension. in this case image file
if (tempURL.EndsWith("jpg"))
{
URL_CanToSave.Add(tempURL,0);
URL_Can.Remove(tempURL);
}
// if not image go and download the page based on the url and keep analyzing
else
{
// if the url not scanned before then
if (URL_Can[tempURL] != 1)
{
// here it seems complex little bit, where Add2Dic is process to add to Dictionaries without adding the Key again ( solving main problem !! )
"ExtractURLfromLink" is another process that return dictionary with all links analyzed by downloading the document string of the url and analyzing it ,
you can add remove filter string based on you analysis
Add2Dic(ExtractUrlsfromLink(client, tempURL, filterlink1), URL_Can, false);
Add2Dic(ExtractUrlsfromLink(client, tempURL, filterlink2), URL_Can, false);
URL_Can[tempURL] = 1; // to set it as scanned link
richTextBox2.PerformSafely(() => richTextBox2.AppendText(tempURL + "\n"));
}
}
statusStrip1.PerformSafely(() => toolStripProgressBar1.PerformStep());
// here comes the other trick to keep this iteration keeps going until it scans all gathered links
i++; if (i >= URL_Can.Count) { i = 0; }
if (URL_CanToSave.Count >= 150) { break; }
}
richTextBox2.PerformSafely(() => richTextBox2.Clear());
textBox1.PerformSafely(() => textBox1.Text = URL_Can.Count.ToString());
return ProcessCompleted = true;
}
catch (Exception aih)
{
MessageBox.Show(aih.Message);
return ProcessCompleted = false;
throw;
}
{
richTextBox2.PerformSafely(()=>richTextBox2.AppendText(url+"\n"));
}
})

Unable to figure out how to use post method, for a suitescript written in Netsuite

I am trying to do use the post method for a simple suitescript program, i am very new to this.
In Netsuite i have written a suitescript as follows.
function restPost()
{
var i = nlapiLoadRecord('department', 115);
var memo = nlapisetfieldvalue('custrecord225', ' ');// this is a customfield, which i want to populate the memo field, using rest client in firefox
var recordId = nlapiSubmitRecord(i);
}
i have created a script record and uploaded this suitescript and even copied the external URL to paste it in restclient.
In Restclient(firefox plugin), pasted the external URL, i have given the method as post, header authorization given, content-type: application/json, and in body i put in {"memo":"mynamehere"};
In this the error i get is
message": "missing ) after argument list
I even tried it by writting other suitescript programs the errors i get is as follows:
Unexpected token in object literal (null$lib#3) Empty JSON string
Invalid data format. You should return TEXT.
I am kinda new to the programming world, so any help would be really good.
I think you are trying to create a RESTlet for POST method. Following is the sample code for POST method -
function createRecord(datain)
{
var err = new Object();
// Validate if mandatory record type is set in the request
if (!datain.recordtype)
{
err.status = "failed";
err.message= "missing recordtype";
return err;
}
var record = nlapiCreateRecord(datain.recordtype);
for (var fieldname in datain)
{
if (datain.hasOwnProperty(fieldname))
{
if (fieldname != 'recordtype' && fieldname != 'id')
{
var value = datain[fieldname];
if (value && typeof value != 'object') // ignore other type of parameters
{
record.setFieldValue(fieldname, value);
}
}
}
}
var recordId = nlapiSubmitRecord(record);
nlapiLogExecution('DEBUG','id='+recordId);
var nlobj = nlapiLoadRecord(datain.recordtype,recordId);
return nlobj;
}
So after deploying this RESTlet you can call this POST method by passing following sample JSON payload -
{"recordtype":"customer","entityid":"John Doe","companyname":"ABCTools Inc","subsidiary":"1","email":"jdoe#email.com"}
For Authorization you have to pass request headers as follows -
var headers = {
"Authorization": "NLAuth nlauth_account=" + cred.account + ", nlauth_email=" + cred.email +
", nlauth_signature= " + cred.password + ", nlauth_role=" + cred.role,
"Content-Type": "application/json"};
I can understand your requirement and the answer posted by Parsun & NetSuite-Expert is good. You can follow that code. That is a generic code that can accept any master record without child records. For Example Customer Without Contact or Addressbook.
I would like to suggest a small change in the code and i have given it in my solution.
Changes Below
var isExistRec = isExistingRecord(objDataIn);
var record = (isExistRec) ? nlapiLoadRecord(objDataIn.recordtype, objDataIn.internalid, {
recordmode: 'dynamic'
}) : nlapiCreateRecord(objDataIn.recordtype);
//Check for Record is Existing in Netsuite or Not using a custom function
function isExistingRecord(objDataIn) {
if (objDataIn.internalid != null && objDataIn.internalid != '' && objDataIn.internalid.trim().length > 0)
return true;
else
return false;
}
So whenever you pass JSON data to the REStlet, keep in mind you have
to pass the internalid, recordtype as mandatory values.
Thanks
Frederick
I believe you will want to return something from your function. An empty object should do fine, or something like {success : true}.
Welcome to Netsuite Suitescripting #Vin :)
I strongly recommend to go through SuiteScript API Overview & SuiteScript API - Alphabetized Index in NS help Center, which is the only and most obvious place to learn the basics of Suitescripting.
nlapiLoadRecord(type, id, initializeValues)
Loads an existing record from the system and returns an nlobjRecord object containing all the field data for that record. You can then extract the desired information from the loaded record using the methods available on the returned record object. This API is a core API. It is available in both client and server contexts.
function restPost(dataIn) {
var record = nlapiLoadRecord('department', 115); // returns nlobjRecord
record.setFieldValue('custrecord225', dataIn.memo); // set the value in custom field
var recordId = nlapiSubmitRecord(record);
return recordId;
}

accessing transition history via JIRA REST API

I found another person apparently having this issue but I thought I'd re-ask the question to see if I could make it more explicit.
I'm using the JIRA 6 REST web API and successfully pulling lots of data that matches our web cloud UI.
Now I'd like to see the transitions a given issue has been thru, preferably with info about who performed the transition.
I can see this transition history in our JIRA web UI but I haven't figured out how to access programmatically yet.
There's a promising sounding API:
http://example.com:8080/jira/rest/api/2/issue/{issueIdOrKey}/transitions [GET, POST]
And this is the API the previous asker seemed to have been using. From what I can tell it only returns the valid transitions you can ask for on the issue at a given point in time.
I would like a history of transitions, such as when the issue went to code review, QA, closed, etc.
I have done a expand=changelog but the change log does not correlate with the transitions that I can see.
Any tips would be appreciated. Thanks.
When you use expand=changelog, then all changes that have been done in issue are there. Exactly same info as in All tab in Activity section when viewing in web browser.
When I send:
http://jira.my.server.se/rest/api/2/issue/KEYF-42346?expand=changelog
Under changelogkey I find list of histories. Each historyhas list of items. Those items are changes performed on the certain field, with to and from values.
To find all status changes you need to do something like this:
for history in issue.changelog.histories:
for item in history.items:
if item.field == "status":
print item.toString # new value
print item.fromString # old value
Or use GET /rest/api/3/issue/{issueIdOrKey}/changelog like explained in the "get changelog" docs
You can try using the jql parameter for the REST API call.
So your call for,
JQL = project=XYZ and status was resolved
fields = key
will look like this,
http://example.com/rest/api/2/search?jql=project%3DXYZ%20and%20status%20was%20resolved&fields=key
where key will return only relevant information and not excessive for each issue.
public void changeStatus(IssueRestClient iRestClient,
List<Statuses> JiraStatuses, String key) {
String status = "To Do";
for (Statuses statuses : vOneToJiraStatuses) {
if (1 == statuses.compareTo(status)) {
try {
String _transition = statuses.getTransition();
Issue issue = iRestClient.getIssue(key).get();
Transition transition = getTransition(iRestClient, issue,
_transition);
if (!(isBlankOrNull(transition))) {
if (!(issue.getStatus().getName()
.equalsIgnoreCase(_transition)))
transition(transition, issue, null, iRestClient,
status);
}
} catch (Exception e) {
Constants.ERROR.info(Level.INFO, e);
}
break;
}
}
}
List is a pojo implementation where statuses and transitions defined in xml are injected through setter/constructor.
private void transition(Transition transition, Issue issue,
FieldInput fieldInput, IssueRestClient issueRestClient,
String status) throws Exception {
if (isBlankOrNull(fieldInput)) {
TransitionInput transitionInput = new TransitionInput(
transition.getId());
issueRestClient.transition(issue, transitionInput).claim();
Constants.REPORT.info("Status Updated for : " + issue.getKey());
} else {
TransitionInput transitionInput = new TransitionInput(
transition.getId());
issueRestClient.transition(issue, transitionInput).claim();
Constants.REPORT.info("Status Updated for : " + issue.getKey());
}
}
public Transition getTransition(IssueRestClient issueRestClient,
Issue issue, String _transition) {
Promise<Iterable<Transition>> ptransitions = issueRestClient
.getTransitions(issue);
Iterable<Transition> transitions = ptransitions.claim();
for (Transition transition : transitions) {
if (transition.getName().equalsIgnoreCase(_transition)) {
return transition;
}
}
return null;
}
In Short using Transition API of JIRA we can fetch all the transitions to set statuses

No such property: getFlatConfig when trying to access configuration

So have set up a couple of values in the groovy.config file which I want for my application.
Set them as follows:
environments {
development {
grails.logging.jul.usebridge = true
reslist = ['1400x1200','1200x1024','1024x800','800x600']
resdef = '1024x800'
mapregs = ['World', 'Europe', 'Asia', 'South America','Central America', 'Pacific','Africa']
mapdef = 'World'
Then I try to access them in a controller
if ( params.mapreq == null) {
mapreq = grailsApplication.config.grails.mapdef
} else {
mapreq = params.mapreq
}
It seems to work (kind a) I get something back, but looks like an object pointer in the format
groovy.util.ConfigObject#3764a904
Tried changing it to getFlatConfig
if ( params.mapreq == null) {
mapreq = grailsApplication.getFlatConfig.grails.mapdef
} else {
mapreq = params.mapreq
}
At which point I get a "No such property: getFlatConfig when trying to access configuration" instead
So any suggestions?
Also, would the same solution work for getting the lists (like the mapregs one)?
grailsApplication.config.grails.mapdef should be grailsApplication.config.mapdef since mapdef is at the top level of the config (within that environment block). Since there's nothing stored under grails.mapdef, the value will be a new ConfigObject. That's why config.a.b.c.d=1 works - each time you access a new level that doesn't exist, Groovy automatically creates a new ConfigObject to hold the value being set, but if you're getting and not setting, you end up with just the empty instance.
The 2nd one doesn't work because getFlatConfig should be getFlatConfig() or flatConfig. But you can't use the ConfigObject-style dots with the flat config because it's flattened. If mapdef was actually under grails you'd access it as grailsApplication.flatConfig.'grails.mapdef' or grailsApplication.flatConfig['grails.mapdef']. But like the other one it's not, so you'd use grailsApplication.flatConfig.mapdef.

neo4j 2.0 findNodesByLabelAndProperty not working

I'm currently trying the Neo4j 2.0.0 M3 and see some strange behaviour. In my unit tests, everything works as expected (using an newImpermanentDatabase) but in the real thing, I do not get results from the graphDatabaseService.findNodesByLabelAndProperty.
Here is the code in question:
ResourceIterator<Node> iterator = graphDB
.findNodesByLabelAndProperty(Labels.User, "EMAIL_ADDRESS", emailAddress)
.iterator();
try {
if (iterator.hasNext()) { // => returns false**
return iterator.next();
}
} finally {
iterator.close();
}
return null;
This returns no results. However, when running the following code, I see my node is there (The MATCH!!!!!!!!! is printed) and I also have an index setup via the schema (although that if I read the API, this seems not necessary but is important for performance):
ResourceIterator<Node> iterator1 = GlobalGraphOperations.at(graphDB).getAllNodesWithLabel(Labels.User).iterator();
while (iterator1.hasNext()) {
Node result = iterator1.next();
UserDao.printoutNode(emailAddress, result);
}
And UserDao.printoutNode
public static void printoutNode(String emailAddress, Node next) {
System.out.print(next);
ResourceIterator<Label> iterator1 = next.getLabels().iterator();
System.out.print("(");
while (iterator1.hasNext()) {
System.out.print(iterator1.next().name());
}
System.out.print("): ");
for(String key : next.getPropertyKeys()) {
System.out.print(key + ": " + next.getProperty(key).toString() + "; ");
if(emailAddress.equals( next.getProperty(key).toString())) {
System.out.print("MATCH!!!!!!!!!");
}
}
System.out.println();
}
I already debugged through the code and what I already found out is that I pass via the InternalAbstractGraphDatabase.map2Nodes to a DelegatingIndexProxy.getDelegate and end up in IndexReader.Empty class which returns the IteratorUtil.EMPTY_ITERATOR thus getting false for iterator.hasNext()
Any idea's what I am doing wrong?
Found it:
I only included neo4j-kernel:2.0.0-M03 in the classpath. The moment I added neo4j-cypher:2.0.0-M03 all was working well.
Hope this answer helps save some time for other users.
#Neo4j: would be nice if an exception would be thrown instead of just returning nothing.
#Ricardo: I wanted to but I was not allowed yet as my reputation wasn't good enough as a new SO user.

Resources