Elasticsearch create index errors out - post

I am very new to Elasticsearch.Using ES 5.1.1, I am trying to create the following simple index:
curl -XPOST "http://localhost:9200/user" -d'
{
"mappings": {
"post": {
"properties": {
"first_name": {
"type": "string"
},
"last_name": {
"type": "string"
},
"birth_date": {
"type": "date"
}
}
}
}
}'
But I am getting an HTTP 400 with the following error message:
No handler found for uri [/user] and method [POST]


Since ES 5, you must use PUT instead of POST when creating new indices:
curl -XPUT "http://localhost:9200/user" -d'
{
"mappings": {
"post": {
"properties": {
"first_name": {
"type": "string"
},
"last_name": {
"type": "string"
},
"birth_date": {
"type": "date"
}
}
}
}
}'

Related

Logstash output to ES gives me error code '400'

i am creating an ELK stack to fetch tweet and analyse them. When, i start my elk stack i got this error message from Logstash
Failed to install template {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://elasticsearch:9200/_index_template/twitter'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:412:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch.rb:578:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch.rb:344:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/outputs/elasticsearch.rb:300:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.9.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:154:in `block in after_successful_connection'"]}
I think i got an error inside my index template but even on internet i didn't found what is wrong with it.
I am using:
logstash:8.5.3
elastictsearch:8.5.3
kibana:8.5.3
this is my template:
{
"template": "twitter-*",
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index.mapping.total_fields.limit": 2000
},
"mappings": {
"_default_": {
"_all": {
"enabled": true
},
"properties": {
"#timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"created_at": {
"type": "date",
"format": "EEE MMM dd HH:mm:ss Z YYYY"
},
"text": {
"type": "text"
},
"user": {
"type": "object",
"properties": {
"description": {
"type": "text"
}
}
},
"coordinates": {
"type": "object",
"properties": {
"coordinates": {
"type": "geo_point"
}
}
},
"entities": {
"type": "object",
"properties": {
"hashtags": {
"type": "object",
"properties": {
"text": {
"type": "text",
"fielddata": true
}
}
}
}
},
"retweeted_status": {
"type": "object",
"properties": {
"text": {
"type": "text"
}
}
}
},
"dynamic_templates": [
{
"string_template": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
}
}
}
and this is my logstash config, i send my tweet via tcp because i have a python bot who fetch them for me.
input {
tcp {
port => 50000
}
}
filter {
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "elastic"
password => "${LOGSTASH_INTERNAL_PASSWORD}"
index => "twitter-%{+yyyy.MM.dd}"
document_type => "tweets"
template => "./templates/twitter.json"
template_name => "twitter"
template_overwrite => true
}
}
Thanks for any help <3
What did you try ?
I tried to modify some attributes of my template, but i still got this message.
What were you expecting?
I was expecting that logstash create my index_template, for incoming tweets.

Springfox error "Unable to find a model that matches key" for {namespace='java.lang', name='Class�ConstraintValidator�object,object��'}

I'm getting these error messages in logs:
[12:15 11:48:20.957] [ERROR][request id: ffdedde3-f147-4fc2-9f9f-ad0c76c1a868][springfox.documentation.swagger2.mappers.ReferenceModelSpecificationToPropertyConverter] - Unable to find a model that matches key ModelKey{qualifiedModelName=ModelName{namespace='java.lang', name='Class�ConstraintValidator�object,object��'}, viewDiscriminator=com.test.entity.view.View$External, validationGroupDiscriminators=[], isResponse=true}
[12:15 11:48:20.957] [ERROR][request id: ffdedde3-f147-4fc2-9f9f-ad0c76c1a868][springfox.documentation.swagger2.mappers.ReferenceModelSpecificationToPropertyConverter] - Unable to find a model that matches key ModelKey{qualifiedModelName=ModelName{namespace='java.lang', name='Class�object�'}, viewDiscriminator=com.test.entity.view.View$External, validationGroupDiscriminators=[], isResponse=true}
[12:15 11:48:20.958] [ERROR][request id: ffdedde3-f147-4fc2-9f9f-ad0c76c1a868][springfox.documentation.swagger2.mappers.ReferenceModelSpecificationToPropertyConverter] - Unable to find a model that matches key ModelKey{qualifiedModelName=ModelName{namespace='java.lang', name='Class�Payload�'}, viewDiscriminator=com.test.entity.view.View$External, validationGroupDiscriminators=[], isResponse=true}
[12:15 11:48:20.991] [ERROR][request id: ffdedde3-f147-4fc2-9f9f-ad0c76c1a868][springfox.documentation.swagger2.mappers.ReferenceModelSpecificationToPropertyConverter] - Unable to find a model that matches key ModelKey{qualifiedModelName=ModelName{namespace='java.lang', name='Class�ConstraintValidator�object,object��'}, viewDiscriminator=null, validationGroupDiscriminators=[], isResponse=true}
[12:15 11:48:20.991] [ERROR][request id: ffdedde3-f147-4fc2-9f9f-ad0c76c1a868][springfox.documentation.swagger2.mappers.ReferenceModelSpecificationToPropertyConverter] - Unable to find a model that matches key ModelKey{qualifiedModelName=ModelName{namespace='java.lang', name='Class�object�'}, viewDiscriminator=null, validationGroupDiscriminators=[], isResponse=true}
[12:15 11:48:20.992] [ERROR][request id: ffdedde3-f147-4fc2-9f9f-ad0c76c1a868][springfox.documentation.swagger2.mappers.ReferenceModelSpecificationToPropertyConverter] - Unable to find a model that matches key ModelKey{qualifiedModelName=ModelName{namespace='java.lang', name='Class�Payload�'}, viewDiscriminator=null, validationGroupDiscriminators=[], isResponse=true}
It seems as custom validation annotations are somehow involved and related to them fields are treated as models or smth.
Example:
#Target({ElementType.TYPE})
#Retention(RetentionPolicy.RUNTIME)
#Constraint(validatedBy = CardDocumentValidator.class)
#Documented
public #interface CardDocument {
String message();
Class<?>[] groups() default {};
Class<? extends Payload>[] payload() default {};
}
public class CardDocumentValidator implements ConstraintValidator<CardDocument, TestEntity> {...}
I'm using Spring Boot 2.5.5 with springfox-boot-starter 3.0.0.
However I'm still on OpenApi 2.0, so I have configured the property springfox:documentation:swagger:use-model-v3: false, which helped me to get rid of similar errors.
In api-docs json among definitions:
"ConstraintDescriptor«object»": {
"type": "object",
"properties": {
"annotation": {
"type": "object"
},
"attributes": {
"type": "object"
},
"composingConstraints": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/ConstraintDescriptor«object»"
}
},
"constraintValidatorClasses": {
"type": "array",
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«ConstraintValidator«object,object»»'}"
}
},
"groups": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«object»'}"
}
},
"messageTemplate": {
"type": "string"
},
"payload": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«Payload»'}"
}
},
"reportAsSingleViolation": {
"type": "boolean"
},
"validationAppliesTo": {
"type": "string",
"enum": [
"IMPLICIT",
"PARAMETERS",
"RETURN_VALUE"
]
},
"valueUnwrapping": {
"type": "string",
"enum": [
"DEFAULT",
"SKIP",
"UNWRAP"
]
}
},
"title": "ConstraintDescriptor«object»"
},
"ConstraintDescriptor«object»DefaultView": {
"type": "object",
"properties": {
"annotation": {
"type": "object"
},
"attributes": {
"type": "object"
},
"composingConstraints": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/ConstraintDescriptor«object»DefaultView"
}
},
"constraintValidatorClasses": {
"type": "array",
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«ConstraintValidator«object,object»»'}"
}
},
"groups": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«object»'}"
}
},
"messageTemplate": {
"type": "string"
},
"payload": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«Payload»'}"
}
},
"reportAsSingleViolation": {
"type": "boolean"
},
"validationAppliesTo": {
"type": "string",
"enum": [
"IMPLICIT",
"PARAMETERS",
"RETURN_VALUE"
]
},
"valueUnwrapping": {
"type": "string",
"enum": [
"DEFAULT",
"SKIP",
"UNWRAP"
]
}
},
"title": "ConstraintDescriptor«object»DefaultView"
},
"ConstraintDescriptor«object»ExternalView": {
"type": "object",
"properties": {
"annotation": {
"type": "object"
},
"attributes": {
"type": "object"
},
"composingConstraints": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/ConstraintDescriptor«object»ExternalView"
}
},
"constraintValidatorClasses": {
"type": "array",
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«ConstraintValidator«object,object»»'}"
}
},
"groups": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«object»'}"
}
},
"messageTemplate": {
"type": "string"
},
"payload": {
"type": "array",
"uniqueItems": true,
"items": {
"$ref": "#/definitions/Error-ModelName{namespace='java.lang', name='Class«Payload»'}"
}
},
"reportAsSingleViolation": {
"type": "boolean"
},
"validationAppliesTo": {
"type": "string",
"enum": [
"IMPLICIT",
"PARAMETERS",
"RETURN_VALUE"
]
},
"valueUnwrapping": {
"type": "string",
"enum": [
"DEFAULT",
"SKIP",
"UNWRAP"
]
}
},
"title": "ConstraintDescriptor«object»ExternalView"
},
"ConstraintValidator«object,object»": {
"type": "object",
"title": "ConstraintValidator«object,object»"
},
"ConstraintValidator«object,object»DefaultView": {
"type": "object",
"title": "ConstraintValidator«object,object»DefaultView"
},
"ConstraintValidator«object,object»ExternalView": {
"type": "object",
"title": "ConstraintValidator«object,object»ExternalView"
},
"ConstraintViolation": {
"type": "object",
"properties": {
"constraintDescriptor": {
"$ref": "#/definitions/ConstraintDescriptor«object»"
},
"executableParameters": {
"type": "array",
"items": {
"type": "object"
}
},
"executableReturnValue": {
"type": "object"
},
"invalidValue": {
"type": "object"
},
"leafBean": {
"type": "object"
},
"message": {
"type": "string"
},
"messageTemplate": {
"type": "string"
},
"propertyPath": {
"$ref": "#/definitions/Path"
},
"rootBean": {
"type": "object"
}
},
"title": "ConstraintViolation"
},
"ConstraintViolationDefaultView": {
"type": "object",
"properties": {
"constraintDescriptor": {
"$ref": "#/definitions/ConstraintDescriptor«object»DefaultView"
},
"executableParameters": {
"type": "array",
"items": {
"type": "object"
}
},
"executableReturnValue": {
"type": "object"
},
"invalidValue": {
"type": "object"
},
"leafBean": {
"type": "object"
},
"message": {
"type": "string"
},
"messageTemplate": {
"type": "string"
},
"propertyPath": {
"$ref": "#/definitions/PathDefaultView"
},
"rootBean": {
"type": "object"
}
},
"title": "ConstraintViolationDefaultView"
},
"ConstraintViolationExternalView": {
"type": "object",
"properties": {
"constraintDescriptor": {
"$ref": "#/definitions/ConstraintDescriptor«object»ExternalView"
},
"executableParameters": {
"type": "array",
"items": {
"type": "object"
}
},
"executableReturnValue": {
"type": "object"
},
"invalidValue": {
"type": "object"
},
"leafBean": {
"type": "object"
},
"message": {
"type": "string"
},
"messageTemplate": {
"type": "string"
},
"propertyPath": {
"$ref": "#/definitions/PathExternalView"
},
"rootBean": {
"type": "object"
}
},
"title": "ConstraintViolationExternalView"
},

Found the culprit :)
private Set validationErrors in one of the base entity classes.
So ConstraintViolation was added to Models and ConstraintDescriptor and its fields' classes which included:
private final List<Class<? extends ConstraintValidator<T, ?>>> constraintValidatorClasses;
private final Set<Class<?>> groups;
private final Set<Class<? extends Payload>> payloads;
Putting #JsonIgnore helped. Another way is to add #ApiModelProperty(hidden = true) which doesn't affect serialization by simply hiding the field from being documented.

Swagger 3.0 reDoc Discriminator JSON

I'm currently writing swagger 3.0 documentation and using reDoc to render as nice UI for it. I have a few scenarios in my documentation where based on a previous properties enum I would want to display different schema object properties. Sadly I cant seam to figure out how to wire this together properly in my documentation. So far I have the following test endpoint:
{
"post": {
"operationId" : "test",
"summary": "test",
"description": "test",
"tags": [ "test" ],
"consumes": "application/json",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"oneOf": [
{
"$ref": "./schemas/test1.json"
},
{
"$ref": "./schemas/test2.json"
}
],
"discriminator": {
"propertyName": "pet_type",
"mapping": {
"click": "./schemas/test1.json",
"open": "./schemas/test2.json"
}
}
}
}
}
},
"responses": {
"200": {
"description": "Success"
}
}
}
}
The test1.json looks like this:
{
"Cat": {
"type": "object",
"properties": {
"pet_type": {
"type": "string"
},
"hunts": {
"type": "boolean"
},
"age": {
"type": "integer"
}
},
"discriminator": {
"propertyName": "pet_type"
}
}
}
And the test2.json like this:
{
"Dog": {
"type": "object",
"properties": {
"pet_type": {
"type": "string"
},
"bark": {
"type": "boolean"
},
"breed": {
"type": "string",
"enum": [
"Dingo",
"Husky",
"Retriever",
"Shepherd"
]
}
},
"discriminator": {
"propertyName": "pet_type"
}
}
}
The desired out come would be to toggle between the two "test" jsons based on an enum (the drop down seen in the reDoc sample). What am I missing to get this result?
You can see an example of the discriminator result here under the feature section (the first gif)

After more digging I was able to figure out the issue... my structure for the most part.
On my index.json file I updated my components section to point at my components folder containing the schema as such:
"components": {
"$ref": "./components/test.json"
},
The test.json looks like the following:
{
"schemas": {
"Refinance": {
"description": "A representation of a cat",
"allOf": [
{
"$ref": "#/schemas/Pet"
},
{
"type": "object",
"properties": {
"huntingSkill": {
"type": "string",
"description": "The measured skill for hunting",
"default": "lazy",
"enum": [
"clueless",
"lazy",
"adventurous",
"aggressive"
]
}
},
"required": [
"huntingSkill"
]
}
]
},
"Purchase": {
"description": "A representation of a dog",
"allOf": [
{
"$ref": "#/schemas/Pet"
},
{
"type": "object",
"properties": {
"packSize": {
"type": "integer",
"format": "int32",
"description": "The size of the pack the dog is from",
"default": 1,
"minimum": 1
},
"foobar": {
"type": "string",
"description": "some ol bullshit"
}
},
"required": [
"packSize"
]
}
]
},
"Pet": {
"type": "object",
"discriminator": {
"propertyName": "petType"
},
"properties": {
"petType": {
"description": "Type of a pet",
"type": "string"
}
},
"xml": {
"name": "Pet"
}
}
}
}
And finally the schema for the endpoint gets referenced as follows:
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "../../index.json#/components/schemas/Pet"
}
}
}
},

ElasticSearch: Indexing with multiple mapping types

I am trying to fully comprehend indexing with multiple mapping types in ElasticSearch. In the docs it gives example code:
PUT my_index
{
"mappings": {
"user": {
"_all": { "enabled": false },
"properties": {
"title": { "type": "string" },
"name": { "type": "string" },
"age": { "type": "integer" }
}
},
"blogpost": {
"properties": {
"title": { "type": "string" },
"body": { "type": "string" },
"user_id": {
"type": "string",
"index": "not_analyzed"
},
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
With this mapping how would I then create and search on an object?
For create would it be:
POST my_index/user/blogspot
or
POST my_index/user,blogspot
For searching would it be:
GET my_index/user/blogspot
or
GET my_index/user,blogspot
or something else?
An example of a POST and GET with multiple mapping would really help me out. Thank you so much!

Retrieve only one object inside an array elasticsearch

I have an index named books which has reviews as an object which can handle arrays.
While retrieving data, in a particular case I only want the review having maximum rating.
"books" :{
"reviews": {
"properties": {
"rating": {
"type": "float"
},
"comments": {
"type": "string"
}
}
},
"author" : {
"type" : "string"
}
}
Many books can have many reviews each having some rating. For a particular use case I only want the result set to have the reviews having maximum rating. I need to build a search query for that kind of result.
POST books/_search
{
"size": 51,
"sort": [
{
"reviews.rating": {
"order": "asc",
"mode" : "min"
}
}
],
"fields": [
"reviews","author"]
}
By using script_fields one can build dynamic fields but not objects. Else I could have made a dynamic object reviews having one field as rating and another as comment.

script_fields can be used to build both dynamic fields and objects:
curl -XDELETE localhost:9200/test-idx
curl -XPUT localhost:9200/test-idx -d '{
"mappings": {
"books" :{
"reviews": {
"properties": {
"rating": {
"type": "float"
},
"comments": {
"type": "string"
}
}
},
"author" : {
"type" : "string"
}
}
}
}'
curl -XPOST "localhost:9200/test-idx/books?refresh=true" -d '{
"reviews": [{
"rating": 5.5,
"comments": "So-so"
}, {
"rating": 9.8,
"comments": "Awesome"
}, {
"rating": 1.2,
"comments": "Awful"
}],
"author": "Roversial, Cont"
}'
curl "localhost:9200/test-idx/books/_search?pretty" -d '{
"fields": ["author"],
"script_fields": {
"highest_review": {
"script": "max_rating = 0.0; max_review = null; for(review : _source[\"reviews\"]) { if (review.rating > max_rating) { max_review = review; max_rating = review.rating;}} max_review"
}
}
}'

Resources