Error with Elasticsearch - ruby-on-rails

Question

I'm pretty new to Elasticsearch and i'm running into this error.
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable at /search
[503] {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}
This is the complete error log
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable - [503] {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}:
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/transport/base.rb:132:in `__raise_transport_error'
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/transport/base.rb:224:in `perform_request'
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/client.rb:104:in `perform_request'
elasticsearch-api (1.0.5) lib/elasticsearch/api/actions/search.rb:158:in `search'
elasticsearch-model (0.1.6) lib/elasticsearch/model/searching.rb:51:in `execute!'
elasticsearch-model (0.1.6) lib/elasticsearch/model/response.rb:31:in `response'
elasticsearch-model (0.1.6) lib/elasticsearch/model/response/records.rb:34:in `ids'
elasticsearch-model (0.1.6) lib/elasticsearch/model/adapters/active_record.rb:16:in `records'
elasticsearch-model (0.1.6) lib/elasticsearch/model/response/records.rb:64:in `respond_to?'
activesupport (4.1.4) lib/active_support/core_ext/object/blank.rb:17:in `blank?'
activesupport (4.1.4) lib/active_support/core_ext/object/blank.rb:24:in `present?'
() home/pavan/Roorah/app/controllers/visitors_controller.rb:34:in `search'
actionpack (4.1.4) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (4.1.4) lib/abstract_controller/base.rb:189:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (4.1.4) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (4.1.4) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:229:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:86:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:86:in `run_callbacks'
actionpack (4.1.4) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
activesupport (4.1.4) lib/active_support/notifications.rb:159:in `block in instrument'
activesupport (4.1.4) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.1.4) lib/active_support/notifications.rb:159:in `instrument'
actionpack (4.1.4) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
activerecord (4.1.4) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (4.1.4) lib/abstract_controller/base.rb:136:in `process'
actionview (4.1.4) lib/action_view/rendering.rb:30:in `process'
actionpack (4.1.4) lib/action_controller/metal.rb:196:in `dispatch'
actionpack (4.1.4) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
actionpack (4.1.4) lib/action_controller/metal.rb:232:in `block in action'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:82:in `call'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:50:in `call'
actionpack (4.1.4) lib/action_dispatch/journey/router.rb:71:in `block in call'
actionpack (4.1.4) lib/action_dispatch/journey/router.rb:59:in `each'
actionpack (4.1.4) lib/action_dispatch/journey/router.rb:59:in `call'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:678:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
omniauth (1.2.2) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.2) lib/omniauth/strategy.rb:164:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
omniauth (1.2.2) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.2) lib/omniauth/strategy.rb:164:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/error_collector.rb:50:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/agent_hooks.rb:26:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/browser_monitoring.rb:23:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/developer_mode.rb:48:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
warden (1.2.3) lib/warden/manager.rb:35:in `block in call'
warden (1.2.3) lib/warden/manager.rb:34:in `catch'
warden (1.2.3) lib/warden/manager.rb:34:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/etag.rb:23:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/conditionalget.rb:25:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/head.rb:11:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/params_parser.rb:27:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/flash.rb:254:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/session/abstract/id.rb:225:in `context'
rack (1.5.2) lib/rack/session/abstract/id.rb:220:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/cookies.rb:560:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activerecord (4.1.4) lib/active_record/query_cache.rb:36:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activerecord (4.1.4) lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activerecord (4.1.4) lib/active_record/migration.rb:380:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.1.4) lib/active_support/callbacks.rb:82:in `run_callbacks'
actionpack (4.1.4) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/reloader.rb:73:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
better_errors (2.0.0) lib/better_errors/middleware.rb:84:in `protected_app_call'
better_errors (2.0.0) lib/better_errors/middleware.rb:79:in `better_errors_call'
better_errors (2.0.0) lib/better_errors/middleware.rb:57:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
railties (4.1.4) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.1.4) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.1.4) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.1.4) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.1.4) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.1.4) lib/rails/rack/logger.rb:20:in `call'
quiet_assets (1.0.3) lib/quiet_assets.rb:23:in `call_with_quiet_assets'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/request_id.rb:21:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/runtime.rb:17:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activesupport (4.1.4) lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/static.rb:64:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/sendfile.rb:112:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
railties (4.1.4) lib/rails/engine.rb:514:in `call'
railties (4.1.4) lib/rails/application.rb:144:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/content_length.rb:14:in `call'
thin (1.6.3) lib/thin/connection.rb:86:in `block in pre_process'
thin (1.6.3) lib/thin/connection.rb:84:in `catch'
thin (1.6.3) lib/thin/connection.rb:84:in `pre_process'
thin (1.6.3) lib/thin/connection.rb:53:in `process'
thin (1.6.3) lib/thin/connection.rb:39:in `receive_data'
eventmachine (1.0.3) lib/eventmachine.rb:187:in `run_machine'
eventmachine (1.0.3) lib/eventmachine.rb:187:in `run'
thin (1.6.3) lib/thin/backends/base.rb:73:in `start'
thin (1.6.3) lib/thin/server.rb:162:in `start'
rack (1.5.2) lib/rack/handler/thin.rb:16:in `run'
rack (1.5.2) lib/rack/server.rb:264:in `start'
railties (4.1.4) lib/rails/commands/server.rb:69:in `start'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:81:in `block in server'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:76:in `tap'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:76:in `server'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:40:in `run_command!'
railties (4.1.4) lib/rails/commands.rb:17:in `<top (required)>'
() home/pavan/Roorah/bin/rails:8:in `require'
() home/pavan/Roorah/bin/rails:8:in `<top (required)>'
spring (1.1.3) lib/spring/client/rails.rb:27:in `load'
spring (1.1.3) lib/spring/client/rails.rb:27:in `call'
spring (1.1.3) lib/spring/client/command.rb:7:in `call'
spring (1.1.3) lib/spring/client.rb:26:in `run'
spring (1.1.3) bin/spring:48:in `<top (required)>'
spring (1.1.3) lib/spring/binstub.rb:11:in `load'
spring (1.1.3) lib/spring/binstub.rb:11:in `<top (required)>'
() home/pavan/Roorah/bin/spring:16:in `require'
() home/pavan/Roorah/bin/spring:16:in `<top (required)>'
() rails:3:in `load'
() rails:3:in `<main>'
This is the piece of code from visitors_controller.rb believed to be the reason behind the error.
#search_results = Property.where(property_active: true)
### new search
results = CityCoordinate.search(params[:q]).records
if params[:q].present? && results.present? #line 34
latitude = results.first.latitude
longitude = results.first.longitude
city = Property.new("latitude" => latitude, "longitude" => longitude)
#search_results = city.nearbys(200).where(property_active: true)
end
###
These are request parameters
{"utf8"=>"✓", "q"=>"Holtsville, NY, United States", "min"=>"", "max"=>"", "baths"=>"", "beds"=>"", "property_type"=>"", "pet"=>"", "garage"=>"", "controller"=>"visitors", "action"=>"search"}
Update:
Below is the log of things which are executed while search
Started GET "/search?utf8=%E2%9C%93&q=Holtsville%2C+NY%2C+United+States&min=&max=&baths=&beds=&property_type=&pet=&garage=" for 127.0.0.1 at 2015-03-16 09:53:30 +0530
Processing by VisitorsController#search as HTML
Parameters: {"utf8"=>"✓", "q"=>"Holtsville, NY, United States", "min"=>"", "max"=>"", "baths"=>"", "beds"=>"", "property_type"=>"", "pet"=>"", "garage"=>""}
CityCoordinate Load (283.1ms) SELECT "city_coordinates".* FROM "city_coordinates" ORDER BY "city_coordinates"."id" ASC LIMIT 1000
2015-03-16 09:54:30 +0530: POST http://localhost:9200/city_coordinates/city_coordinate/_bulk?type=city_coordinate [status:200, request:60.010s, query:60.005s]
2015-03-16 09:54:30 +0530: > {"index":{"_id":1}}
{"id":1,"latitude":40.81,"longitude":-73.04,"zipcode":544,"city":"Holtsville","state":"NY","created_at":"2015-03-10T06:21:50.061Z","updated_at":"2015-03-10T06:21:50.061Z"}
{"index":{"_id":2}}
{"id":2,"latitude":18.16,"longitude":-66.72,"zipcode":601,"city":"Adjuntas","state":"PR","created_at":"2015-03-10T06:21:50.120Z","updated_at":"2015-03-10T06:21:50.120Z"}
2015-03-16 09:54:30 +0530: < {"took":60005,"errors":true,"items":[{"index":{"_index":"city_coordinates","_type":"city_coordinate","_id":"1","status":503,"error":"UnavailableShardsException[[city_coordinates][2] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest#5f0340e0]"}},{"index":{"_index":"city_coordinates","_type":"city_coordinate","_id":"2","status":503,"error":"UnavailableShardsException[[city_coordinates][3] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest#2d813d7b]"}}]}
2015-03-16 09:54:30 +0530: GET http://localhost:9200/city_coordinates/city_coordinate/_search?q=Holtsville%2C+NY%2C+United+States [status:503, request:0.028s, query:N/A]
2015-03-16 09:54:30 +0530: < {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}
2015-03-16 09:54:30 +0530: [503] {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}
Completed 500 Internal Server Error in 60402ms
Log from elasticsearch
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ ./elasticsearch
[2015-03-16 09:49:18,455][INFO ][node ] [Ant-Man] version[1.4.4], pid[2980], build[c88f77f/2015-02-19T13:05:36Z]
[2015-03-16 09:49:18,499][INFO ][node ] [Ant-Man] initializing ...
[2015-03-16 09:49:18,518][INFO ][plugins ] [Ant-Man] loaded [], sites []
[2015-03-16 09:49:22,526][INFO ][node ] [Ant-Man] initialized
[2015-03-16 09:49:22,527][INFO ][node ] [Ant-Man] starting ...
[2015-03-16 09:49:22,711][INFO ][transport ] [Ant-Man] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.0.1.20:9300]}
[2015-03-16 09:49:22,740][INFO ][discovery ] [Ant-Man] elasticsearch/lZbmLwgxTdC9zwU0qcYrUw
[2015-03-16 09:49:26,516][INFO ][cluster.service ] [Ant-Man] new_master [Ant-Man][lZbmLwgxTdC9zwU0qcYrUw][Pavan-Inspiron-3542][inet[/10.0.1.20:9300]], reason: zen-disco-join (elected_as_master)
[2015-03-16 09:49:26,532][INFO ][http ] [Ant-Man] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.0.1.20:9200]}
[2015-03-16 09:49:26,532][INFO ][node ] [Ant-Man] started
[2015-03-16 09:49:27,317][INFO ][gateway ] [Ant-Man] recovered [2] indices into cluster_state
[2015-03-16 09:54:30,813][DEBUG][action.bulk ] [Ant-Man] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-03-16 09:54:30,815][DEBUG][action.bulk ] [Ant-Man] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-03-16 09:54:30,847][DEBUG][action.search.type ] [Ant-Man] All shards failed for phase: [query]
And i have these in the Gemfile
gem 'elasticsearch-model', '0.1.6'
gem 'elasticsearch-rails', '0.1.6'
gem 'bonsai-elasticsearch-rails', '0.0.4'
I can't figure what is happening and i'm stuck with this.In our rails app we use elasticsearch for the renters to search for the properties posted by the landlords with the city,state and country being the parameters of q. Currently there is a property with the given city Holtsville.
Please do ask if any information is required.
Update 1:
The output when i run curl -X GET 'http://localhost:9200/_cat/shards?v&pretty'
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ curl -X GET 'http://localhost:9200/_cat/shards?v&pretty'
index shard prirep state docs store ip node
city_coordinates 2 p UNASSIGNED
city_coordinates 2 r UNASSIGNED
city_coordinates 0 p UNASSIGNED
city_coordinates 0 r UNASSIGNED
city_coordinates 3 p UNASSIGNED
city_coordinates 3 r UNASSIGNED
city_coordinates 1 p UNASSIGNED
city_coordinates 1 r UNASSIGNED
city_coordinates 4 p UNASSIGNED
city_coordinates 4 r UNASSIGNED
properties 4 p UNASSIGNED
properties 4 r UNASSIGNED
properties 0 p STARTED 0 115b 127.0.1.1 Franz Kafka
properties 0 r UNASSIGNED
properties 3 p UNASSIGNED
properties 3 r UNASSIGNED
properties 1 p STARTED 0 115b 127.0.1.1 Franz Kafka
properties 1 r UNASSIGNED
properties 2 p STARTED 1 7.3kb 127.0.1.1 Franz Kafka
properties 2 r UNASSIGNED
Update 2:
I uncommented and added these things in elasticsearch.yml file
node.name: "Franz Kafka"
node.master: true
node.data: true
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 97
cluster.routing.allocation.disk.watermark.high: 99
bootstrap.mlockall: true
network.bind_host: localhost
script.disable_dynamic: true
network.publish_host: localhost
And i created an executable file called elasticsearch in /etc/init.d to add these environment variables
ES_MIN_MEM=256m
ES_MAX_MEM=1g
Update 3:
The output when i run curl -XGET http://localhost:9200/_cluster/health?pretty
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ curl -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 3,
"active_shards" : 3,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 17
}
Update 4:
According to this link,it seems like my machine didn't have the minimum 8GB of RAM(I have 4GB). Is this the reason for the error? And also the city_coordinates table(which the elasticsearch collects the data from) has 43k records which gives me enough reason to believe that the current memory(4GB) is not sufficient for elasticsearch.
Update 5:
The output when i run curl 'localhost:9200/_cat/nodes?v'
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ curl 'localhost:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
Pavan-Inspiron-3542 127.0.1.1 6 41 0.49 d * Franz Kafka

Answer 1

Your problem is logged here:
Primary shard is not active or isn't assigned is a known node.
This usually means that Elasticsearch is unable to create the primary shard for your index. Often, this is because your disk is too full:
cluster.routing.allocation.disk.watermark.low controls the low watermark for disk usage. It defaults to 85%, meaning ES will not allocate new shards to nodes once they have more than 85% disk used.
http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-allocation.html
You should probably set your Elasticsearch log level to DEBUG for more information.
Edit: Based on your shard debug output, it looks like you've created your index with 20 shards, but your nodes won't accomodate that many.
The index.routing.allocation.total_shards_per_node setting allows to control how many total shards (replicas and primaries) for an index will be allocated per node.
You should recreate your index with number_of_replicas and number_of_shards set to values such that number_of_shards * number_of_replicas is less than index.routing.allocation.total_shards_per_node, and number_of_replicas is no larger than the number of nodes in your cluster.
http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html