Sunspot Solr core initialization failure - ruby-on-rails

Question

When I run my tests, most of them fail with the following error.
Failure/Error: let(:user){ FactoryGirl.create(:user) }
RSolr::Error::Http:
RSolr::Error::Http - 500 Internal Server Error
Error: {msg=SolrCore 'test' is not available due to init failure: Error opening new searcher,trace=org.apache.solr.common.SolrException: SolrCore 'test' is not available due to init failure: Error opening new searcher
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:745)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:299)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
URI: http://localhost:8981/solr/test/update?wt=ruby
Request Headers: {"Content-Type"=>"text/xml"}
Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><add><doc><field name=\"id\">User 13497</field><field name=\"type\">User</field><field name=\"type\">ActiveRecord::Base</field><field name=\"class_name\">User</field><field name=\"phone_number_text\">[phone]</field><field name=\"name_text\">111.111.1200</field></doc></add>"
Development is working fine. I thought it may be a corrupt test index so did:
$ rm -rf solr/data/test/
$ rm -rf solr/test/
That didn't help. Looking at the development admin page: http://localhost:8982/solr/#/ I don't see any warnings or errors.
When I go to the test admin page:
http://localhost:8981/solr/#/
SolrCore Initialization Failures
development: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher
default: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher
test: org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher
In the logs for the test admin site I see:
null:org.apache.solr.common.SolrException: SolrCore 'test' is not available due to init failure: Error opening new searcher
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:745)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:299)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:873)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
... 8 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock#/Users/pramod/workspace/merlin/solr/test/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:89)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1528)
... 10 more
So my best guess is that something is wrong with how the write locks are being created/deleted. I checked solr/conf/solrconfig.xml and I had the following (other stackoverflow posts seem to say this was necessary):
<unlockOnStartup>true</unlockOnStartup>
I also rebooting my computer. However, it's still not working. Any suggestions on how to fix this? Thanks!
System info:
Mac OS X 10.11 El Capitan
Sunspot 2.2.0
ruby 2.2.2
rails 4.2.1
solr 4.10.2
I upgraded solr version from the default that comes with sunspot to 4.10.2 to match my production environment. This seems to be the action that screwed everything up.
------------PARTIAL WORKAROUND------------
In my solr/conf/solrconfig.xml I added the following:
<lockType>simple</lockType>
right above <unlockOnStartup>true</unlockOnStartup>
Restart the solr instance and it resolves the issue that time.
It seems that NativeFSLockFactory (the default that Solr uses) causes issues (http://wiki.apache.org/lucene-java/AvailableLockFactories)
However if I stop the Solr instance and then try to start it again, I get a different exception now:
Caused by: java.nio.file.NoSuchFileException: /solr/test/data/index/_5p.si
As a workaround I do
$ rm -rf solr/data/test/
$ rm -rf solr/test/
Whenever I close the solr instance or start it. That seems to solve the issue for now. I still don't understand what's going wrong.