The production log on my Ruby on Rails app, has some of the following error types:
Mongo::Error::OperationFailure
[225:TransactionTooOld]: Retryable write with txnNumber 12 is prohibited on session b39c1b6b-54f5-4f5d-a010-7067c095bac7
The DB has a primary instance and two replicas. I would like to know how to prevent and fix the issue. Thank you in advance
Log
mongo (2.15.0) lib/mongo/operation/result.rb:348:in `raise_operation_failure'
mongo (2.15.0) lib/mongo/operation/result.rb:314:in `validate!'
mongo (2.15.0) lib/mongo/operation/shared/response_handling.rb:36:in `block (3 levels) in validate_result'
mongo (2.15.0) lib/mongo/operation/shared/response_handling.rb:107:in `add_server_diagnostics'
mongo (2.15.0) lib/mongo/operation/shared/response_handling.rb:35:in `block (2 levels) in validate_result'
mongo (2.15.0) lib/mongo/operation/shared/response_handling.rb:54:in `add_error_labels'
mongo (2.15.0) lib/mongo/operation/shared/response_handling.rb:34:in `block in validate_result'
mongo (2.15.0) lib/mongo/operation/shared/response_handling.rb:93:in `unpin_maybe'
mongo (2.15.0) lib/mongo/operation/shared/response_handling.rb:33:in `validate_result'
mongo (2.15.0) lib/mongo/operation/shared/write.rb:49:in `block in execute'
mongo (2.15.0) lib/mongo/server/connection_pool.rb:593:in `with_connection'
mongo (2.15.0) lib/mongo/server.rb:437:in `with_connection'
mongo (2.15.0) lib/mongo/operation/shared/write.rb:38:in `execute'
mongo (2.15.0) lib/mongo/collection.rb:607:in `block (2 levels) in insert_one'
mongo (2.15.0) lib/mongo/retryable.rb:230:in `write_with_retry'
mongo (2.15.0) lib/mongo/collection.rb:596:in `block in insert_one'
mongo (2.15.0) lib/mongo/client.rb:1057:in `with_session'
mongo (2.15.0) lib/mongo/collection.rb:585:in `insert_one'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:83:in `insert_as_root'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:29:in `block in insert'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:122:in `block (2 levels) in prepare_insert'
activesupport (6.1.4) lib/active_support/callbacks.rb:106:in `run_callbacks'
mongoid (7.3.2) lib/mongoid/interceptable.rb:135:in `run_callbacks'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:121:in `block in prepare_insert'
activesupport (6.1.4) lib/active_support/callbacks.rb:98:in `run_callbacks'
mongoid (7.3.2) lib/mongoid/interceptable.rb:135:in `run_callbacks'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:120:in `prepare_insert'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:25:in `insert'
mongoid (7.3.2) lib/mongoid/persistable/savable.rb:25:in `save'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:153:in `block in create'
mongoid (7.3.2) lib/mongoid/threaded/lifecycle.rb:163:in `_creating'
mongoid (7.3.2) lib/mongoid/persistable/creatable.rb:148:in `create'
app/models/email_message.rb:32:in `email_message'
app/controllers/application_controller.rb:17:in `render_error'
app/controllers/application_controller.rb:8:in `block in <class:ApplicationController>'
activesupport (6.1.4) lib/active_support/rescuable.rb:116:in `instance_exec'
activesupport (6.1.4) lib/active_support/rescuable.rb:116:in `block in handler_for_rescue'
activesupport (6.1.4) lib/active_support/rescuable.rb:92:in `rescue_with_handler'
activesupport (6.1.4) lib/active_support/rescuable.rb:165:in `rescue_with_handler'
actionpack (6.1.4) lib/action_controller/metal/rescue.rb:25:in `rescue in process_action'
actionpack (6.1.4) lib/action_controller/metal/rescue.rb:21:in `process_action'
actionpack (6.1.4) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
activesupport (6.1.4) lib/active_support/notifications.rb:203:in `block in instrument'
activesupport (6.1.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.1.4) lib/active_support/notifications.rb:203:in `instrument'
actionpack (6.1.4) lib/action_controller/metal/instrumentation.rb:33:in `process_action'
actionpack (6.1.4) lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
mongoid (7.3.2) lib/mongoid/railties/controller_runtime.rb:22:in `process_action'
actionpack (6.1.4) lib/abstract_controller/base.rb:165:in `process'
actionpack (6.1.4) lib/action_controller/metal.rb:190:in `dispatch'
actionpack (6.1.4) lib/action_controller/metal.rb:254:in `dispatch'
actionpack (6.1.4) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
actionpack (6.1.4) lib/action_dispatch/routing/route_set.rb:33:in `serve'
actionpack (6.1.4) lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
actionpack (6.1.4) lib/action_dispatch/routing/mapper.rb:49:in `serve'
actionpack (6.1.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (6.1.4) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (6.1.4) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (6.1.4) lib/action_dispatch/routing/route_set.rb:842:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack-pjax (1.1.0) lib/rack/pjax.rb:12:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
newrelic_rpm (7.2.0) lib/new_relic/rack/agent_hooks.rb:30:in `traced_call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
newrelic_rpm (7.2.0) lib/new_relic/rack/browser_monitoring.rb:33:in `traced_call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/cookies.rb:689:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
remotipart (1.4.4) lib/remotipart/middleware.rb:32:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/etag.rb:27:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/conditional_get.rb:40:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/head.rb:12:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/http/permissions_policy.rb:22:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/http/content_security_policy.rb:18:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/cookies.rb:689:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (6.1.4) lib/active_support/callbacks.rb:98:in `run_callbacks'
actionpack (6.1.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
logster (2.9.7) lib/logster/middleware/reporter.rb:43:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
railties (6.1.4) lib/rails/rack/logger.rb:37:in `call_app'
railties (6.1.4) lib/rails/rack/logger.rb:28:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/method_override.rb:24:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/runtime.rb:22:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
actionpack (6.1.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
newrelic_rpm (7.2.0) lib/new_relic/agent/instrumentation/middleware_tracing.rb:101:in `call'
act
database.yml
production:
clients:
default:
uri: "mongodb://user:password#[PRIMARY_IP_ADDRESS]:27017/[db_name is here]"
options:
auth_source: admin
read:
mode: :nearest
max_pool_size: 40
min_pool_size: 1
ssl: false
ssl_verify: false
It was resolved by adding the corresponding configuration for the server:
https://www.mongodb.com/docs/mongoid/7.1/tutorials/mongoid-configuration/#usage-with-forking-servers
I'm working o a Ruby on Rails application using PostgreSQL on Xubuntu (Xfce Desktop Environment). I downloaded postgresql using Linuxbrew
My database.yml looks like:
development:
adapter: postgresql
encoding: unicode
database: rado_dev_bouldin
host: localhost
port: 5432
pool: 5
username: bouldin.15
First I was getting the issue that a server was already running and to check the /tmp/pids/server.pid file, I did that and killed the process using kill -9 1420
I ran server s again get the following outut:
$ rails s
=> Booting Thin
=> Rails 4.1.12 application starting in development on http://0.0.0.0:3000
=> Run `rails server -h` for more startup options
=> Notice: server is listening on all interfaces (0.0.0.0). Consider using 127.0.0.1 (--binding option)
=> Ctrl-C to shutdown server
Thin web server (v1.6.4 codename Gob Bluth)
Maximum connections set to 1024
Listening on 0.0.0.0:3000, CTRL+C to stop
So it looks like it works, but once I actually open http://localhost:3000/ I get the following output
Started GET "/" for 127.0.0.1 at 2017-02-03 14:07:55 -0500
PG::ConnectionBad - could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
:
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `initialize'
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `new'
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:888:in `connect'
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:568:in `initialize'
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `new'
activerecord (4.1.12) lib/active_record/connection_adapters/postgresql_adapter.rb:41:in `postgresql_connection'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:435:in `new_connection'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:445:in `checkout_new_connection'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `acquire_connection'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
() home/bouldin.15/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:350:in `checkout'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:265:in `block in connection'
() home/bouldin.15/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:264:in `connection'
activerecord (4.1.12) lib/active_record/connection_adapters/abstract/connection_pool.rb:541:in `retrieve_connection'
activerecord (4.1.12) lib/active_record/connection_handling.rb:113:in `retrieve_connection'
activerecord (4.1.12) lib/active_record/connection_handling.rb:87:in `connection'
activerecord (4.1.12) lib/active_record/migration.rb:388:in `check_pending!'
activerecord (4.1.12) lib/active_record/migration.rb:377:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.1.12) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.1.12) lib/active_support/callbacks.rb:82:in `run_callbacks'
actionpack (4.1.12) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.1.12) lib/action_dispatch/middleware/reloader.rb:73:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.1.12) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
better_errors (2.1.1) lib/better_errors/middleware.rb:84:in `protected_app_call'
better_errors (2.1.1) lib/better_errors/middleware.rb:79:in `better_errors_call'
better_errors (2.1.1) lib/better_errors/middleware.rb:57:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.1.12) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.1.12) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
railties (4.1.12) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.1.12) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.1.12) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.1.12) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.1.12) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.1.12) lib/rails/rack/logger.rb:20:in `call'
quiet_assets (1.1.0) lib/quiet_assets.rb:27:in `call_with_quiet_assets'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.1.12) lib/action_dispatch/middleware/request_id.rb:21:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
rack (1.5.5) lib/rack/methodoverride.rb:21:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
rack (1.5.5) lib/rack/runtime.rb:17:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
activesupport (4.1.12) lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
rack (1.5.5) lib/rack/lock.rb:17:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.1.12) lib/action_dispatch/middleware/static.rb:84:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
rack (1.5.5) lib/rack/sendfile.rb:112:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
railties (4.1.12) lib/rails/engine.rb:514:in `call'
railties (4.1.12) lib/rails/application.rb:144:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
rack (1.5.5) lib/rack/content_length.rb:14:in `call'
thin (1.6.4) lib/thin/connection.rb:86:in `block in pre_process'
thin (1.6.4) lib/thin/connection.rb:84:in `catch'
thin (1.6.4) lib/thin/connection.rb:84:in `pre_process'
thin (1.6.4) lib/thin/connection.rb:53:in `process'
thin (1.6.4) lib/thin/connection.rb:39:in `receive_data'
eventmachine (1.2.0.1) lib/eventmachine.rb:194:in `run_machine'
eventmachine (1.2.0.1) lib/eventmachine.rb:194:in `run'
thin (1.6.4) lib/thin/backends/base.rb:73:in `start'
thin (1.6.4) lib/thin/server.rb:162:in `start'
rack (1.5.5) lib/rack/handler/thin.rb:16:in `run'
rack (1.5.5) lib/rack/server.rb:264:in `start'
railties (4.1.12) lib/rails/commands/server.rb:69:in `start'
railties (4.1.12) lib/rails/commands/commands_tasks.rb:81:in `block in server'
railties (4.1.12) lib/rails/commands/commands_tasks.rb:76:in `tap'
railties (4.1.12) lib/rails/commands/commands_tasks.rb:76:in `server'
railties (4.1.12) lib/rails/commands/commands_tasks.rb:40:in `run_command!'
railties (4.1.12) lib/rails/commands.rb:17:in `<top (required)>'
() home/bouldin.15/Psychology/RADO/bin/rails:8:in `require'
() home/bouldin.15/Psychology/RADO/bin/rails:8:in `<top (required)>'
spring (1.7.1) lib/spring/client/rails.rb:28:in `load'
spring (1.7.1) lib/spring/client/rails.rb:28:in `call'
spring (1.7.1) lib/spring/client/command.rb:7:in `call'
spring (1.7.1) lib/spring/client.rb:30:in `run'
spring (1.7.1) bin/spring:49:in `<top (required)>'
spring (1.7.1) lib/spring/binstub.rb:11:in `load'
spring (1.7.1) lib/spring/binstub.rb:11:in `<top (required)>'
() home/bouldin.15/Psychology/RADO/bin/spring:16:in `require'
() home/bouldin.15/Psychology/RADO/bin/spring:16:in `<top (required)>'
() rails:3:in `load'
() rails:3:in `<main>'
Started POST "/__better_errors/b6383117352c5f7e/variables" for 127.0.0.1 at 2017-02-03 14:07:55 -0500
Does anyone know how to resolve this?
Resolved...
I tried to start the PostgreSQL database by using the postgres -D /usr/local/pgsql/data command. I got an error that my postgres data didn't exist and used the command initdb PGDATA to create it. Then I used pg_ctl -D PGDATA -l logfile start to start the server. Then I had to recreate my database so I ran $bin\rake db:create db:migrate
Have a rails API app, hosted on cloud with phusion passenger and nginx. Requests to the API are getting timed out if the API is not accessed for some time.
Checking the logs, it seems the requests successfully hits rails and this is where it times out.
Here is the rails log
PG::UnableToSend (server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
activerecord (4.2.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:215:in `exec'
activerecord (4.2.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:215:in `dealloc'
activerecord (4.2.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:198:in `block in clear'
activerecord (4.2.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:197:in `each_value'
activerecord (4.2.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:197:in `clear'
activerecord (4.2.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:258:in `clear_cache!'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract_adapter.rb:291:in `reconnect!'
activerecord (4.2.5.1) lib/active_record/connection_adapters/postgresql_adapter.rb:275:in `reconnect!'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract_adapter.rb:328:in `verify!'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:456:in `block in checkout_and_verify'
activesupport (4.2.5.1) lib/active_support/callbacks.rb:88:in `__run_callbacks__'
activesupport (4.2.5.1) lib/active_support/callbacks.rb:778:in `_run_checkout_callbacks'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:455:in `checkout_and_verify'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:351:in `block in checkout'
/usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
activerecord (4.2.5.1) lib/active_record/connection_handling.rb:113:in `retrieve_connection'
activerecord (4.2.5.1) lib/active_record/connection_handling.rb:87:in `connection'
activerecord (4.2.5.1) lib/active_record/query_cache.rb:31:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
activerecord (4.2.5.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.2.5.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.2.5.1) lib/active_support/callbacks.rb:88:in `__run_callbacks__'
activesupport (4.2.5.1) lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
activesupport (4.2.5.1) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (4.2.5.1) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.2.5.1) lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
airbrake (5.2.3) lib/airbrake/rack/middleware.rb:22:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.2.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.2.5.1) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
railties (4.2.5.1) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.2.5.1) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.2.5.1) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.2.5.1) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.2.5.1) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.2.5.1) lib/rails/rack/logger.rb:20:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
actionpack (4.2.5.1) lib/action_dispatch/middleware/request_id.rb:21:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
rack (1.6.4) lib/rack/runtime.rb:18:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
activesupport (4.2.5.1) lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
railties (4.2.5.1) lib/rails/engine.rb:518:in `call'
railties (4.2.5.1) lib/rails/application.rb:165:in `call'
newrelic_rpm (3.15.1.316) lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in `call'
/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
database.yml configuration
database: xxx
username: xxx
password: xxx
host: xxx
port: 5432
pool: 10
Passenger configuration
server {
listen 80 default_server;
server_name {{server_name}};
passenger_enabled on;
rails_env {{env}};
root {{rails_app_root}}/{{rails_app_package_name}}-{{node_env}}/public;
passenger_min_instances 3;
}
passenger_max_request_queue_size 500;
passenger_pre_start http://{{server_name}}{{pre_start_url}};
passenger_spawn_method direct;
passenger_pool_idle_time 0;
passenger_max_preloader_idle_time 0;
Any pointers on why this could be happening?
This does not happen always, only if you access the service after it has been idle for sometime.
I've got a Rails app that has been live for over a year with no issues, running on a custom Ubuntu/Nginx/Puma setup.
Because of various issues with the hosting company, I moved the app over to EC2 this weekend - using ElasticBeanstalk on their Nginx/Puma/Ruby2.2 stack.
Everything is working fine - apart from file uploads.
The app itself uses the Paperclip gem for handling attachments - the actual files are only held momentarily in Paperclip, before being uploaded to a file-server controlled by my client (the very first version used S3 for storage, but now it uploads, attaches the "Job-Sheet" to the Job and then immediately uploads the "Job-Sheet" to the file-server).
As I say, this was all working fine on my old server. But now, on EC2, I'm getting a Encoding::UndefinedConversionError ("\xC3" from ASCII-8BIT to UTF-8) error.
The job sheet itself can be an Excel file or a CSV. It is defined as follows:
class Job < ActiveRecord::Base
...
has_attached_file :job_sheet
validates_attachment_file_name :job_sheet, matches: [/csv\Z/, /xls\Z/, /xlsx\Z/]
validates_attachment_presence :job_sheet
...
end
The controller looks like this:
class JobsController < ApplicationController
...
def create
#job = services['creates_job'].create_job job_params, session: current_session
redirect_to jobs_path
rescue ActiveRecord::RecordInvalid
...
end
...
end
The jobs_params does the StrongParams thing of permitting the parameters as required.
And the CreatesJob service object looks like:
def create_job params, session: nil
...
job = services['jobs'].create! params.merge(code: code)
...
end
So apart from the fact that the actual logic is in a separate "CreatesJob" class, there's nothing unusual going on - pretty much a standard Rails/Paperclip upload situation.
However, some uploads are failing, with the following in the production log:
I, [2016-03-22T10:12:58.701172 #23116] INFO -- : Started POST "/jobs" for 80.0.249.70 at 2016-03-22 10:12:58 +0000
F, [2016-03-22T10:12:58.706190 #23116] FATAL -- : Encoding::UndefinedConversionError ("\xC3" from ASCII-8BIT to UTF-8):
actionpack (4.2.0) lib/action_dispatch/http/upload.rb:31:in `encode'
actionpack (4.2.0) lib/action_dispatch/http/upload.rb:31:in `initialize'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:51:in `new'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:51:in `normalize_encode_params'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:57:in `block in normalize_encode_params'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:53:in `each'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:53:in `each_with_object'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:53:in `normalize_encode_params'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:57:in `block in normalize_encode_params'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:53:in `each'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:53:in `each_with_object'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:53:in `normalize_encode_params'
actionpack (4.2.0) lib/action_dispatch/http/request.rb:308:in `POST'
actionpack (4.2.0) lib/action_dispatch/http/parameters.rb:14:in `parameters'
actionpack (4.2.0) lib/action_dispatch/http/filter_parameters.rb:37:in `filtered_parameters'
actionpack (4.2.0) lib/action_controller/metal/instrumentation.rb:22:in `process_action'
actionpack (4.2.0) lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
activerecord (4.2.0) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (4.2.0) lib/abstract_controller/base.rb:137:in `process'
actionview (4.2.0) lib/action_view/rendering.rb:30:in `process'
actionpack (4.2.0) lib/action_controller/metal.rb:195:in `dispatch'
actionpack (4.2.0) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
actionpack (4.2.0) lib/action_controller/metal.rb:236:in `block in action'
actionpack (4.2.0) lib/action_dispatch/routing/route_set.rb:73:in `call'
actionpack (4.2.0) lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
actionpack (4.2.0) lib/action_dispatch/routing/route_set.rb:42:in `serve'
actionpack (4.2.0) lib/action_dispatch/journey/router.rb:43:in `block in serve'
actionpack (4.2.0) lib/action_dispatch/journey/router.rb:30:in `each'
actionpack (4.2.0) lib/action_dispatch/journey/router.rb:30:in `serve'
actionpack (4.2.0) lib/action_dispatch/routing/route_set.rb:802:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/rack/agent_hooks.rb:30:in `traced_call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/rack/browser_monitoring.rb:23:in `traced_call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
rack (1.6.1) lib/rack/etag.rb:24:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
rack (1.6.1) lib/rack/conditionalget.rb:38:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
rack (1.6.1) lib/rack/head.rb:13:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/params_parser.rb:27:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/flash.rb:260:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
rack (1.6.1) lib/rack/session/abstract/id.rb:225:in `context'
rack (1.6.1) lib/rack/session/abstract/id.rb:220:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/cookies.rb:560:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
activerecord (4.2.0) lib/active_record/query_cache.rb:36:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
activerecord (4.2.0) lib/active_record/connection_adapters/abstract/connection_pool.rb:647:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.2.0) lib/active_support/callbacks.rb:88:in `call'
activesupport (4.2.0) lib/active_support/callbacks.rb:88:in `_run_callbacks'
activesupport (4.2.0) lib/active_support/callbacks.rb:734:in `_run_call_callbacks'
activesupport (4.2.0) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (4.2.0) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
railties (4.2.0) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.2.0) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.2.0) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.2.0) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.2.0) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.2.0) lib/rails/rack/logger.rb:20:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/request_id.rb:21:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
rack (1.6.1) lib/rack/methodoverride.rb:22:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
rack (1.6.1) lib/rack/runtime.rb:18:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
activesupport (4.2.0) lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
actionpack (4.2.0) lib/action_dispatch/middleware/static.rb:113:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
rack (1.6.1) lib/rack/sendfile.rb:113:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
railties (4.2.0) lib/rails/engine.rb:518:in `call'
railties (4.2.0) lib/rails/application.rb:164:in `call'
newrelic_rpm (3.11.1.284) lib/new_relic/agent/instrumentation/middleware_tracing.rb:67:in `call'
puma (2.11.0) lib/puma/configuration.rb:82:in `call'
puma (2.11.0) lib/puma/server.rb:507:in `handle_request'
puma (2.11.0) lib/puma/server.rb:375:in `process_client'
puma (2.11.0) lib/puma/server.rb:262:in `block in run'
puma (2.11.0) lib/puma/thread_pool.rb:104:in `call'
puma (2.11.0) lib/puma/thread_pool.rb:104:in `block in spawn_thread'
The other answers related to this error all seem to be about forcing the encoding of the file. But the issue here seems to be 1) sometimes it's a binary (xls) and sometimes text (csv) so I can't go around mangling it and 2) the exception is in ActionDispatch; so looks like it's happening before I get to see the upload.
Any ideas what's going on and what I can do to fix it?
UPDATE: I can now reproduce the issue consistently. If I upload an XLSX file with an English filename, the upload works OK, if it's an XLSX file with Spanish accented in the filename, the upload fails as shown above.
The line in question is #original_filename.encode!(Encoding::UTF_8) in action_dispatch/http/upload.rb - however, the filename should be valid unicode, but it's obviously getting mangled in transit somewhere.
I'm pretty new to Elasticsearch and i'm running into this error.
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable at /search
[503] {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}
This is the complete error log
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable - [503] {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}:
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/transport/base.rb:132:in `__raise_transport_error'
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/transport/base.rb:224:in `perform_request'
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
elasticsearch-transport (1.0.5) lib/elasticsearch/transport/client.rb:104:in `perform_request'
elasticsearch-api (1.0.5) lib/elasticsearch/api/actions/search.rb:158:in `search'
elasticsearch-model (0.1.6) lib/elasticsearch/model/searching.rb:51:in `execute!'
elasticsearch-model (0.1.6) lib/elasticsearch/model/response.rb:31:in `response'
elasticsearch-model (0.1.6) lib/elasticsearch/model/response/records.rb:34:in `ids'
elasticsearch-model (0.1.6) lib/elasticsearch/model/adapters/active_record.rb:16:in `records'
elasticsearch-model (0.1.6) lib/elasticsearch/model/response/records.rb:64:in `respond_to?'
activesupport (4.1.4) lib/active_support/core_ext/object/blank.rb:17:in `blank?'
activesupport (4.1.4) lib/active_support/core_ext/object/blank.rb:24:in `present?'
() home/pavan/Roorah/app/controllers/visitors_controller.rb:34:in `search'
actionpack (4.1.4) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (4.1.4) lib/abstract_controller/base.rb:189:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (4.1.4) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (4.1.4) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:113:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:229:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:229:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:166:in `block in halting'
activesupport (4.1.4) lib/active_support/callbacks.rb:86:in `call'
activesupport (4.1.4) lib/active_support/callbacks.rb:86:in `run_callbacks'
actionpack (4.1.4) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
activesupport (4.1.4) lib/active_support/notifications.rb:159:in `block in instrument'
activesupport (4.1.4) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.1.4) lib/active_support/notifications.rb:159:in `instrument'
actionpack (4.1.4) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (4.1.4) lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
activerecord (4.1.4) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (4.1.4) lib/abstract_controller/base.rb:136:in `process'
actionview (4.1.4) lib/action_view/rendering.rb:30:in `process'
actionpack (4.1.4) lib/action_controller/metal.rb:196:in `dispatch'
actionpack (4.1.4) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
actionpack (4.1.4) lib/action_controller/metal.rb:232:in `block in action'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:82:in `call'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:50:in `call'
actionpack (4.1.4) lib/action_dispatch/journey/router.rb:71:in `block in call'
actionpack (4.1.4) lib/action_dispatch/journey/router.rb:59:in `each'
actionpack (4.1.4) lib/action_dispatch/journey/router.rb:59:in `call'
actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:678:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
omniauth (1.2.2) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.2) lib/omniauth/strategy.rb:164:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
omniauth (1.2.2) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.2) lib/omniauth/strategy.rb:164:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/error_collector.rb:50:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/agent_hooks.rb:26:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/browser_monitoring.rb:23:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/rack/developer_mode.rb:48:in `traced_call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:55:in `call'
warden (1.2.3) lib/warden/manager.rb:35:in `block in call'
warden (1.2.3) lib/warden/manager.rb:34:in `catch'
warden (1.2.3) lib/warden/manager.rb:34:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/etag.rb:23:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/conditionalget.rb:25:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/head.rb:11:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/params_parser.rb:27:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/flash.rb:254:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/session/abstract/id.rb:225:in `context'
rack (1.5.2) lib/rack/session/abstract/id.rb:220:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/cookies.rb:560:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activerecord (4.1.4) lib/active_record/query_cache.rb:36:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activerecord (4.1.4) lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activerecord (4.1.4) lib/active_record/migration.rb:380:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.1.4) lib/active_support/callbacks.rb:82:in `run_callbacks'
actionpack (4.1.4) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/reloader.rb:73:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
better_errors (2.0.0) lib/better_errors/middleware.rb:84:in `protected_app_call'
better_errors (2.0.0) lib/better_errors/middleware.rb:79:in `better_errors_call'
better_errors (2.0.0) lib/better_errors/middleware.rb:57:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
railties (4.1.4) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.1.4) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.1.4) lib/active_support/tagged_logging.rb:68:in `block in tagged'
activesupport (4.1.4) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (4.1.4) lib/active_support/tagged_logging.rb:68:in `tagged'
railties (4.1.4) lib/rails/rack/logger.rb:20:in `call'
quiet_assets (1.0.3) lib/quiet_assets.rb:23:in `call_with_quiet_assets'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/request_id.rb:21:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/runtime.rb:17:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
activesupport (4.1.4) lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
actionpack (4.1.4) lib/action_dispatch/middleware/static.rb:64:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/sendfile.rb:112:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
railties (4.1.4) lib/rails/engine.rb:514:in `call'
railties (4.1.4) lib/rails/application.rb:144:in `call'
newrelic_rpm (3.9.5.251) lib/new_relic/agent/instrumentation/middleware_tracing.rb:57:in `call'
rack (1.5.2) lib/rack/content_length.rb:14:in `call'
thin (1.6.3) lib/thin/connection.rb:86:in `block in pre_process'
thin (1.6.3) lib/thin/connection.rb:84:in `catch'
thin (1.6.3) lib/thin/connection.rb:84:in `pre_process'
thin (1.6.3) lib/thin/connection.rb:53:in `process'
thin (1.6.3) lib/thin/connection.rb:39:in `receive_data'
eventmachine (1.0.3) lib/eventmachine.rb:187:in `run_machine'
eventmachine (1.0.3) lib/eventmachine.rb:187:in `run'
thin (1.6.3) lib/thin/backends/base.rb:73:in `start'
thin (1.6.3) lib/thin/server.rb:162:in `start'
rack (1.5.2) lib/rack/handler/thin.rb:16:in `run'
rack (1.5.2) lib/rack/server.rb:264:in `start'
railties (4.1.4) lib/rails/commands/server.rb:69:in `start'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:81:in `block in server'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:76:in `tap'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:76:in `server'
railties (4.1.4) lib/rails/commands/commands_tasks.rb:40:in `run_command!'
railties (4.1.4) lib/rails/commands.rb:17:in `<top (required)>'
() home/pavan/Roorah/bin/rails:8:in `require'
() home/pavan/Roorah/bin/rails:8:in `<top (required)>'
spring (1.1.3) lib/spring/client/rails.rb:27:in `load'
spring (1.1.3) lib/spring/client/rails.rb:27:in `call'
spring (1.1.3) lib/spring/client/command.rb:7:in `call'
spring (1.1.3) lib/spring/client.rb:26:in `run'
spring (1.1.3) bin/spring:48:in `<top (required)>'
spring (1.1.3) lib/spring/binstub.rb:11:in `load'
spring (1.1.3) lib/spring/binstub.rb:11:in `<top (required)>'
() home/pavan/Roorah/bin/spring:16:in `require'
() home/pavan/Roorah/bin/spring:16:in `<top (required)>'
() rails:3:in `load'
() rails:3:in `<main>'
This is the piece of code from visitors_controller.rb believed to be the reason behind the error.
#search_results = Property.where(property_active: true)
### new search
results = CityCoordinate.search(params[:q]).records
if params[:q].present? && results.present? #line 34
latitude = results.first.latitude
longitude = results.first.longitude
city = Property.new("latitude" => latitude, "longitude" => longitude)
#search_results = city.nearbys(200).where(property_active: true)
end
###
These are request parameters
{"utf8"=>"✓", "q"=>"Holtsville, NY, United States", "min"=>"", "max"=>"", "baths"=>"", "beds"=>"", "property_type"=>"", "pet"=>"", "garage"=>"", "controller"=>"visitors", "action"=>"search"}
Update:
Below is the log of things which are executed while search
Started GET "/search?utf8=%E2%9C%93&q=Holtsville%2C+NY%2C+United+States&min=&max=&baths=&beds=&property_type=&pet=&garage=" for 127.0.0.1 at 2015-03-16 09:53:30 +0530
Processing by VisitorsController#search as HTML
Parameters: {"utf8"=>"✓", "q"=>"Holtsville, NY, United States", "min"=>"", "max"=>"", "baths"=>"", "beds"=>"", "property_type"=>"", "pet"=>"", "garage"=>""}
CityCoordinate Load (283.1ms) SELECT "city_coordinates".* FROM "city_coordinates" ORDER BY "city_coordinates"."id" ASC LIMIT 1000
2015-03-16 09:54:30 +0530: POST http://localhost:9200/city_coordinates/city_coordinate/_bulk?type=city_coordinate [status:200, request:60.010s, query:60.005s]
2015-03-16 09:54:30 +0530: > {"index":{"_id":1}}
{"id":1,"latitude":40.81,"longitude":-73.04,"zipcode":544,"city":"Holtsville","state":"NY","created_at":"2015-03-10T06:21:50.061Z","updated_at":"2015-03-10T06:21:50.061Z"}
{"index":{"_id":2}}
{"id":2,"latitude":18.16,"longitude":-66.72,"zipcode":601,"city":"Adjuntas","state":"PR","created_at":"2015-03-10T06:21:50.120Z","updated_at":"2015-03-10T06:21:50.120Z"}
2015-03-16 09:54:30 +0530: < {"took":60005,"errors":true,"items":[{"index":{"_index":"city_coordinates","_type":"city_coordinate","_id":"1","status":503,"error":"UnavailableShardsException[[city_coordinates][2] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest#5f0340e0]"}},{"index":{"_index":"city_coordinates","_type":"city_coordinate","_id":"2","status":503,"error":"UnavailableShardsException[[city_coordinates][3] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest#2d813d7b]"}}]}
2015-03-16 09:54:30 +0530: GET http://localhost:9200/city_coordinates/city_coordinate/_search?q=Holtsville%2C+NY%2C+United+States [status:503, request:0.028s, query:N/A]
2015-03-16 09:54:30 +0530: < {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}
2015-03-16 09:54:30 +0530: [503] {"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed]","status":503}
Completed 500 Internal Server Error in 60402ms
Log from elasticsearch
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ ./elasticsearch
[2015-03-16 09:49:18,455][INFO ][node ] [Ant-Man] version[1.4.4], pid[2980], build[c88f77f/2015-02-19T13:05:36Z]
[2015-03-16 09:49:18,499][INFO ][node ] [Ant-Man] initializing ...
[2015-03-16 09:49:18,518][INFO ][plugins ] [Ant-Man] loaded [], sites []
[2015-03-16 09:49:22,526][INFO ][node ] [Ant-Man] initialized
[2015-03-16 09:49:22,527][INFO ][node ] [Ant-Man] starting ...
[2015-03-16 09:49:22,711][INFO ][transport ] [Ant-Man] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.0.1.20:9300]}
[2015-03-16 09:49:22,740][INFO ][discovery ] [Ant-Man] elasticsearch/lZbmLwgxTdC9zwU0qcYrUw
[2015-03-16 09:49:26,516][INFO ][cluster.service ] [Ant-Man] new_master [Ant-Man][lZbmLwgxTdC9zwU0qcYrUw][Pavan-Inspiron-3542][inet[/10.0.1.20:9300]], reason: zen-disco-join (elected_as_master)
[2015-03-16 09:49:26,532][INFO ][http ] [Ant-Man] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.0.1.20:9200]}
[2015-03-16 09:49:26,532][INFO ][node ] [Ant-Man] started
[2015-03-16 09:49:27,317][INFO ][gateway ] [Ant-Man] recovered [2] indices into cluster_state
[2015-03-16 09:54:30,813][DEBUG][action.bulk ] [Ant-Man] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-03-16 09:54:30,815][DEBUG][action.bulk ] [Ant-Man] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-03-16 09:54:30,847][DEBUG][action.search.type ] [Ant-Man] All shards failed for phase: [query]
And i have these in the Gemfile
gem 'elasticsearch-model', '0.1.6'
gem 'elasticsearch-rails', '0.1.6'
gem 'bonsai-elasticsearch-rails', '0.0.4'
I can't figure what is happening and i'm stuck with this.In our rails app we use elasticsearch for the renters to search for the properties posted by the landlords with the city,state and country being the parameters of q. Currently there is a property with the given city Holtsville.
Please do ask if any information is required.
Update 1:
The output when i run curl -X GET 'http://localhost:9200/_cat/shards?v&pretty'
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ curl -X GET 'http://localhost:9200/_cat/shards?v&pretty'
index shard prirep state docs store ip node
city_coordinates 2 p UNASSIGNED
city_coordinates 2 r UNASSIGNED
city_coordinates 0 p UNASSIGNED
city_coordinates 0 r UNASSIGNED
city_coordinates 3 p UNASSIGNED
city_coordinates 3 r UNASSIGNED
city_coordinates 1 p UNASSIGNED
city_coordinates 1 r UNASSIGNED
city_coordinates 4 p UNASSIGNED
city_coordinates 4 r UNASSIGNED
properties 4 p UNASSIGNED
properties 4 r UNASSIGNED
properties 0 p STARTED 0 115b 127.0.1.1 Franz Kafka
properties 0 r UNASSIGNED
properties 3 p UNASSIGNED
properties 3 r UNASSIGNED
properties 1 p STARTED 0 115b 127.0.1.1 Franz Kafka
properties 1 r UNASSIGNED
properties 2 p STARTED 1 7.3kb 127.0.1.1 Franz Kafka
properties 2 r UNASSIGNED
Update 2:
I uncommented and added these things in elasticsearch.yml file
node.name: "Franz Kafka"
node.master: true
node.data: true
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 97
cluster.routing.allocation.disk.watermark.high: 99
bootstrap.mlockall: true
network.bind_host: localhost
script.disable_dynamic: true
network.publish_host: localhost
And i created an executable file called elasticsearch in /etc/init.d to add these environment variables
ES_MIN_MEM=256m
ES_MAX_MEM=1g
Update 3:
The output when i run curl -XGET http://localhost:9200/_cluster/health?pretty
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ curl -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 3,
"active_shards" : 3,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 17
}
Update 4:
According to this link,it seems like my machine didn't have the minimum 8GB of RAM(I have 4GB). Is this the reason for the error? And also the city_coordinates table(which the elasticsearch collects the data from) has 43k records which gives me enough reason to believe that the current memory(4GB) is not sufficient for elasticsearch.
Update 5:
The output when i run curl 'localhost:9200/_cat/nodes?v'
pavan#Pavan-Inspiron-3542:~/elasticsearch-1.4.4/bin$ curl 'localhost:9200/_cat/nodes?v'
host ip heap.percent ram.percent load node.role master name
Pavan-Inspiron-3542 127.0.1.1 6 41 0.49 d * Franz Kafka
Your problem is logged here:
Primary shard is not active or isn't assigned is a known node.
This usually means that Elasticsearch is unable to create the primary shard for your index. Often, this is because your disk is too full:
cluster.routing.allocation.disk.watermark.low controls the low watermark for disk usage. It defaults to 85%, meaning ES will not allocate new shards to nodes once they have more than 85% disk used.
http://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-allocation.html
You should probably set your Elasticsearch log level to DEBUG for more information.
Edit: Based on your shard debug output, it looks like you've created your index with 20 shards, but your nodes won't accomodate that many.
The index.routing.allocation.total_shards_per_node setting allows to control how many total shards (replicas and primaries) for an index will be allocated per node.
You should recreate your index with number_of_replicas and number_of_shards set to values such that number_of_shards * number_of_replicas is less than index.routing.allocation.total_shards_per_node, and number_of_replicas is no larger than the number of nodes in your cluster.
http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html