Failed to deserialize exception response from stream - Grails ElasticSearch - grails

Question

I have a simple Grails app, which I am trying to set upo to use ElasticSearch.
I have a single-node ElasticSearch instance running on EC2, which is running happily enough. (For reference, I just followed the steps here: http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/), but using 0.90.7 and the cloud-aws plugin version 1.15.0)
I am using the Grails ElasticSearch GORM plugin (http://grails.org/plugin/elasticsearch-gorm) (Master branch) and i'm connecting to ES using the transport client mode (elasticSearch.client.mode = 'transport')
Here's where it gets really odd...
The first time I boot up my app, it will happily index my Domain data on ES, I can query, etc, no problems.
If I then restart my grails app, it won't launch at all. I get
Message: Error creating bean with name 'searchableClassMappingConfigurator': Invocation of init method failed; nested exception is org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
Line | Method
->> 262 | run in java.util.concurrent.FutureTask
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 724 | run in java.lang.Thread
Caused by TransportSerializationException: Failed to deserialize exception response from stream
->> 169 | handlerResponseError in org.elasticsearch.transport.netty.MessageChannelHandler
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 123 | messageReceived in ''
| 70 | handleUpstream in org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler
| 564 | sendUpstream in org.elasticsearch.common.netty.channel.DefaultChannelPipeline
| 791 | sendUpstream in org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext
| 296 | fireMessageReceived in org.elasticsearch.common.netty.channel.Channels
| 462 | unfoldAndFireMessageReceived in org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder
| 443 | callDecode in ''
| 310 | messageReceived in ''
| 70 | handleUpstream in org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler
| 564 | sendUpstream in org.elasticsearch.common.netty.channel.DefaultChannelPipeline
| 559 | sendUpstream in ''
| 268 | fireMessageReceived in org.elasticsearch.common.netty.channel.Channels
| 255 | fireMessageReceived in ''
| 88 | read . . in org.elasticsearch.common.netty.channel.socket.nio.NioWorker
| 108 | process in org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker
| 318 | run . . . in org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector
| 89 | run in org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker
| 178 | run . . . in org.elasticsearch.common.netty.channel.socket.nio.NioWorker
| 108 | run in org.elasticsearch.common.netty.util.ThreadRenamingRunnable
| 42 | run . . . in org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 724 | run in java.lang.Thread
Caused by StreamCorruptedException: unexpected end of block data
->> 1370 | readObject0 in java.io.ObjectInputStream
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1989 | defaultReadFields in ''
| 499 | defaultReadObject in ''
| 914 | readObject in java.lang.Throwable
| 1017 | invokeReadObject in java.io.ObjectStreamClass
| 1891 | readSerialData in java.io.ObjectInputStream
| 1796 | readOrdinaryObject in ''
| 1348 | readObject0 in ''
| 1989 | defaultReadFields in ''
| 499 | defaultReadObject in ''
| 914 | readObject in java.lang.Throwable
| 1017 | invokeReadObject in java.io.ObjectStreamClass
| 1891 | readSerialData in java.io.ObjectInputStream
| 1796 | readOrdinaryObject in ''
| 1348 | readObject0 in ''
| 370 | readObject in ''
| 167 | handlerResponseError in org.elasticsearch.transport.netty.MessageChannelHandler
| 123 | messageReceived in ''
| 70 | handleUpstream in org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler
| 564 | sendUpstream in org.elasticsearch.common.netty.channel.DefaultChannelPipeline
| 791 | sendUpstream in org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext
| 296 | fireMessageReceived in org.elasticsearch.common.netty.channel.Channels
| 462 | unfoldAndFireMessageReceived in org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder
| 443 | callDecode in ''
| 310 | messageReceived in ''
| 70 | handleUpstream in org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler
| 564 | sendUpstream in org.elasticsearch.common.netty.channel.DefaultChannelPipeline
| 559 | sendUpstream in ''
| 268 | fireMessageReceived in org.elasticsearch.common.netty.channel.Channels
| 255 | fireMessageReceived in ''
| 88 | read . . in org.elasticsearch.common.netty.channel.socket.nio.NioWorker
| 108 | process in org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker
| 318 | run . . . in org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector
| 89 | run in org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker
| 178 | run . . . in org.elasticsearch.common.netty.channel.socket.nio.NioWorker
| 108 | run in org.elasticsearch.common.netty.util.ThreadRenamingRunnable
| 42 | run . . . in org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 724 | run in java.lang.Thread
This happens until I change the elasticSearch host details - ie, I can't boot my ap at all, with the original host details, ever again.
Both the ES node and my Grails app are both using elasticSearch 0.90.7, my config for the ES plugin looks like so:#
elasticSearch.client.mode = 'transport'
elasticSearch.client.hosts = [[host:'<my EC2 DNS>', port:9300]]
elasticSearch.datastoreImpl = 'mongoDatastore'
elasticSearch.client.transport.sniff = true
The only domain object I am marking as 'searchable' is mapped with mongoDB, which looks like so:
class CompletedApplicationFormSearchEntry {
static searchable = true
Long formId
Long jobId
Long employerId
Long jobseekerId
Date applicationDate
static mapWith = "mongo"
static constraints = {
}
}
If I remove the searchable attribute from the domain class, then relaunch the app, it launches fine, so I assume that there's something going on in the bootstrapping process when the domain object is detected as being searchable, but of course, it only causes an issue when the app's been restarted.
There are a handful of threads kicking about where people are seeing similar issue,s where they have nodes running different ES versions, different JVM version,s etc. But in this case, I only have one node!
I am absolutely tearing my hair out over this - I just can't work out what on earth's going wrong. I've tried different plugin versions, elasticsearch versions, 32-bit EC2 instance, 64bit EC2 instance - no luck!

Answer 1

Not with grails es plugin, but I fixed similar issue by fixing the jvm versions in ide ( used by the es client) to be the same as es master running on different jvm version.
STEP 0 : Make sure api and actual es version are identical.
STEP 1 : check jvm versions used by the es nodes
In following case, I had two different jvm versions, as pointed by "jvm" json key of both es nodes.
$ curl -XGET "http://localhost:9200/_nodes?jvm=true&pretty=true"
{
"cluster_name" : "elasticsearch",
"nodes" : {
"A6PDUvlWSN-zN2GKRxrSHA" : {
"name" : "Madeline Joyce",
"transport_address" : "inet[/192.168.1.4:9301]",
"host" : "prayagupd",
"ip" : "127.0.1.1",
"version" : "1.3.2",
"build" : "dee175d",
"http_address" : "inet[/192.168.1.4:9201]",
"settings" : {
"path" : {
"logs" : "/usr/local/elasticsearch-1.3.2/logs",
"home" : "/usr/local/elasticsearch-1.3.2"
},
"cluster" : {
"name" : "elasticsearch"
},
"http" : {
"port" : "9201"
},
"transport" : {
"tcp" : {
"port" : "9301"
}
},
"foreground" : "yes",
"name" : "Madeline Joyce"
},
"os" : {
"refresh_interval_in_millis" : 1000,
"available_processors" : 4,
"cpu" : {
"vendor" : "Intel",
"model" : "Core(TM) i5 CPU M 480 # 2.67GHz",
"mhz" : 2667,
"total_cores" : 4,
"total_sockets" : 4,
"cores_per_socket" : 16,
"cache_size_in_bytes" : 3072
},
"mem" : {
"total_in_bytes" : 3803283456
},
"swap" : {
"total_in_bytes" : 5998899200
}
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 9036,
"max_file_descriptors" : 4096,
"mlockall" : false
},
"jvm" : {
"pid" : 9036,
"version" : "1.7.0_65",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "24.65-b04",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1421578674811,
"mem" : {
"heap_init_in_bytes" : 268435456,
"heap_max_in_bytes" : 1038876672,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 136314880,
"direct_max_in_bytes" : 1038876672
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
},
"thread_pool" : {
"generic" : {
"type" : "cached",
"keep_alive" : "30s",
"queue_size" : -1
},
"index" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "200"
},
"snapshot_data" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"bench" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"get" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"suggest" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1,
"queue_size" : -1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"search" : {
"type" : "fixed",
"min" : 12,
"max" : 12,
"queue_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
}
},
"network" : {
"refresh_interval_in_millis" : 5000,
"primary_interface" : {
"address" : "192.168.1.4",
"name" : "eth0",
"mac_address" : "20:6A:8A:2A:24:E6"
}
},
"transport" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9301]",
"publish_address" : "inet[/192.168.1.4:9301]"
},
"http" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9201]",
"publish_address" : "inet[/192.168.1.4:9201]",
"max_content_length_in_bytes" : 104857600
},
"plugins" : [ ]
},
"TWNkkYYZSWe8NnrOPU57mQ" : {
"name" : "Scarlet Spider",
"transport_address" : "inet[/192.168.1.4:9300]",
"host" : "prayagupd",
"ip" : "127.0.1.1",
"version" : "1.3.2",
"build" : "dee175d",
"http_address" : "inet[/192.168.1.4:9200]",
"attributes" : {
"client" : "true",
"data" : "false"
},
"settings" : {
"path" : {
"data" : "/var/lib/elasticsearch",
"work" : "/tmp/elasticsearch",
"conf" : "/etc/elasticsearch",
"logs" : "/var/log/elasticsearch"
},
"cluster" : {
"name" : "elasticsearch"
},
"node" : {
"client" : "true"
},
"name" : "Scarlet Spider"
},
"os" : {
"refresh_interval_in_millis" : 1000,
"available_processors" : 4
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 11028,
"max_file_descriptors" : 4096,
"mlockall" : false
},
"jvm" : {
"pid" : 11028,
"version" : "1.7.0_05",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "23.1-b03",
"vm_vendor" : "Oracle Corporation",
"start_time_in_millis" : 1421580829189,
"mem" : {
"heap_init_in_bytes" : 59426304,
"heap_max_in_bytes" : 846331904,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 136314880,
"direct_max_in_bytes" : 846331904
},
"gc_collectors" : [ "PS Scavenge", "PS MarkSweep" ],
"memory_pools" : [ "Code Cache", "PS Eden Space", "PS Survivor Space", "PS Old Gen", "PS Perm Gen" ]
},
"thread_pool" : {
"generic" : {
"type" : "cached",
"keep_alive" : "30s",
"queue_size" : -1
},
"index" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "200"
},
"snapshot_data" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"bench" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"get" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "1k"
},
"snapshot" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"merge" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"suggest" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "1k"
},
"bulk" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "50"
},
"optimize" : {
"type" : "fixed",
"min" : 1,
"max" : 1,
"queue_size" : -1
},
"warmer" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"flush" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
},
"search" : {
"type" : "fixed",
"min" : 12,
"max" : 12,
"queue_size" : "1k"
},
"percolate" : {
"type" : "fixed",
"min" : 4,
"max" : 4,
"queue_size" : "1k"
},
"management" : {
"type" : "scaling",
"min" : 1,
"max" : 5,
"keep_alive" : "5m",
"queue_size" : -1
},
"refresh" : {
"type" : "scaling",
"min" : 1,
"max" : 2,
"keep_alive" : "5m",
"queue_size" : -1
}
},
"network" : {
"refresh_interval_in_millis" : 5000
},
"transport" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0:9300]",
"publish_address" : "inet[/192.168.1.4:9300]"
},
"http" : {
"bound_address" : "inet[/0:0:0:0:0:0:0:0:9200]",
"publish_address" : "inet[/192.168.1.4:9200]",
"max_content_length_in_bytes" : 104857600
},
"plugins" : [ ]
}
}
}
STEP 2: update jvm version in ide (following shows for intellij ide)
Update to required jvm version, and add to project,
STEP 3 : Run both es nodes, the problem of TransportSerializationException should get fixed
$ curl -XGET "http://localhost:9200/_nodes?jvm=true&pretty=true"
{
"cluster_name": "elasticsearch",
"nodes": {
"GeRZFRiDSje8zLM_m90WRw": {
"name" : "Dougboy",
"jvm": {
"pid": 15223,
"version": "1.7.0_65",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "24.65-b04",
"vm_vendor": "Oracle Corporation",
"start_time_in_millis": 1421586819876,
"mem": {
"heap_init_in_bytes": 59426304,
"heap_max_in_bytes": 846200832,
"non_heap_init_in_bytes": 24576000,
"non_heap_max_in_bytes": 136314880,
"direct_max_in_bytes": 846200832
}
}
},
"A6PDUvlWSN-zN2GKRxrSHA": {
"name": "Madeline Joyce",
"jvm": {
"pid": 9036,
"version": "1.7.0_65",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "24.65-b04",
"vm_vendor": "Oracle Corporation",
"start_time_in_millis": 1421578674811,
"mem": {
"heap_init_in_bytes": 268435456,
"heap_max_in_bytes": 1038876672,
"non_heap_init_in_bytes": 24313856,
"non_heap_max_in_bytes": 136314880,
"direct_max_in_bytes": 1038876672
}
}
}
}
}
Reference
Java Client TransportSerializationException #3835, Oct 6, 2013

Answer 2

Looks like it was an issue with the plugin with Elasticsearch 0.90.7 throwing an exception which wasn't caught by the plugin.
Pull request here: https://github.com/mstein/elasticsearch-grails-plugin/pull/74 has a fix and includes ES 0.90.7