ElasticSearch health check failed every time when spring boot start up - docker

I am working with Elastic Search 5.5.0 on Spring boot 1.5.8.RELEASE using the Java Transport Client library.
The Elastic Search was deployed with docker in a container. It works well. The queries from my java application works well too.
The problem is that the Elasticsearch Health check failed almost every time when Spring boot started from my local machine as the following exception says. I didn't call any health check explicitly in my application.
How can I remove the health check every time on start up or are there any way of passing the health check?
Thanks.
20180201 09:04:12.499 [restartedMain] INFO c.k.a.Application - Log info On
20180201 09:04:12.499 [restartedMain] WARN c.k.a.Application - Log warn On
20180201 09:04:12.499 [restartedMain] ERROR c.k.a.Application - Log error On
20180201 09:04:15.628 [RMI TCP Connection(9)-10.10.20.187] WARN o.s.b.a.h.ElasticsearchHealthIndicator - Health check failed
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:71)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:58)
at org.springframework.boot.actuate.health.ElasticsearchHealthIndicator.doHealthCheck(ElasticsearchHealthIndicator.java:52)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43)
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:85)
at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:35)
at org.springframework.boot.actuate.endpoint.jmx.DataEndpointMBean.getData(DataEndpointMBean.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1252)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:80)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1246)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1085)
at org.springframework.jmx.export.SpringModelMBean.invoke(SpringModelMBean.java:90)
at javax.management.modelmbean.RequiredModelMBean.getAttribute(RequiredModelMBean.java:1562)
at org.springframework.jmx.export.SpringModelMBean.getAttribute(SpringModelMBean.java:109)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:232)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:67)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:69)
... 47 common frames omitted

You can disable the Elasticsearch health check by adding the following line to your application.xml file
management.health.elasticsearch.enabled: false

When I went deep into the same issue happening in my sprint boot app too, I found that elasticsearch health check use org.elasticsearch.client.RestClient but I was using org.elasticsearch.client.RestHighLevelClient and had created Bean for the same.
In case you want to keep the recurring elasticsearch's healthcheck,
Create a Bean for RestClient using the right host and port.
Or add following entry in app config yml/xml to remove healthcheck itself:
management.health.elasticsearch.enabled: false

disable the health indicator auto configuration
#SpringBootApplication(exclude = {
ElasticSearchRestHealthIndicatorAutoConfiguration.class
})
public class YourApplication

Disable elastic search health check. Mention this property in your yml/xml file
management.health.elasticsearch.enabled=false

Related

Trouble Deploying Grails Application After Upgrading to Ver. 5.1.2 in Tomcat Server

Not sure why I'm getting this error while i deploy on tomcat server and i see its trying to find some yml file but i dont see any file on my code
I'm confused where to start trying various scenarios and still could not able to replicate the issue and solve it
ERROR --- [ NO_USER] [ main] org.apache.catalina.startup.HostConfig : Error deploying web application archive [E:\Tomcat\webapps\smp##build-533.war]
java.lang.IllegalStateException: Error starting child
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:729)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1023)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1910)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:824)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:474)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1617)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:318)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:478)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/smp##build-533]]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
... 37 common frames omitted
Caused by: org.springframework.boot.context.config.ConfigDataResourceNotFoundException: Config data resource 'file [E:\smp-config.yml]' via location 'E:\smp-config.yml' cannot be found
at org.springframework.boot.context.config.ConfigDataResourceNotFoundException.withLocation(ConfigDataResourceNotFoundException.java:97)
at org.springframework.boot.context.config.ConfigDataImporter.handle(ConfigDataImporter.java:145)
at org.springframework.boot.context.config.ConfigDataImporter.load(ConfigDataImporter.java:136)
at org.springframework.boot.context.config.ConfigDataImporter.resolveAndLoad(ConfigDataImporter.java:86)
at org.springframework.boot.context.config.ConfigDataEnvironmentContributors.withProcessedImports(ConfigDataEnvironmentContributors.java:121)
at org.springframework.boot.context.config.ConfigDataEnvironment.processInitial(ConfigDataEnvironment.java:240)
at org.springframework.boot.context.config.ConfigDataEnvironment.processAndApply(ConfigDataEnvironment.java:227)
at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:102)
at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:94)
at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:102)
at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:87)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131)
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85)
at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114)
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65)
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:338)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:296)
at grails.boot.GrailsApp.run(GrailsApp.groovy:99)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:175)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:155)
at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:97)
at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:174)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5211)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 38 common frames omitted
I'm having trouble deploying application after upgrading to grails 5 from 3 made all the necessary config changes. I dont see any issue while running application locally but i see the error as above while deploying not sure what am missing.
Please help me in this

Prometheus: Hadoop Monitoring using JMX Exporter

I am very new to Prometheus and monitoring and working on getting JMX-Exporter setup for my Windows-based Hadoop installation for small POC.
I tried putting in following line in "hadoop_installation\bin\hdfs.cmd\"
set HADOOP_OPTS=%HADOOP_OPTS% "-javaagent:C:\path\to\jmx_prometheus_javaagent-0.11.0.jar=1985:C:\path\to\conf\hadoop.yml"
following is my hadoop.yml
rules:
- pattern: Hadoop<service=ResourceManager, name=MetricsSystem, sub=Stats><>NumAllSources
name: sources
labels:
app_id: "hadoop_rm"
Now when I run start-all.cmd
It gives me errors,
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.NumberFormatException: For input string: "C"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:42)
... 6 more
FATAL ERROR in native method: processing of -javaagent failed
I am not sure whether I am on the right track for this?
Is there any other way except readymade FSImage exporter?

Jenkins throws NullPointerException after upgrading Splunk plugin

I got tired of the security prompt at the top of the Jenkins window telling me to upgrade my plugins, so I upgraded those plugins. When I rebooted, the familiar Jenkins landing page was replaced with the following stack trace:
java.lang.NullPointerException
at com.splunk.splunkjenkins.utils.SplunkLogService.enqueue(SplunkLogService.java:174)
at com.splunk.splunkjenkins.utils.SplunkLogService.send(SplunkLogService.java:170)
at com.splunk.splunkjenkins.utils.SplunkLogService.send(SplunkLogService.java:107)
at com.splunk.splunkjenkins.JdkSplunkLogHandler.publish(JdkSplunkLogHandler.java:43)
at java.util.logging.Logger.log(Logger.java:616)
at java.util.logging.Logger.doLog(Logger.java:641)
at java.util.logging.Logger.log(Logger.java:664)
at java.util.logging.Logger.info(Logger.java:1182)
at com.splunk.splunkjenkins.LoggingInitStep.setupSplunkJenkins(LoggingInitStep.java:22)
Caused: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:1089)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:47)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1113)
at jenkins.model.Jenkins.<init>(Jenkins.java:929)
at hudson.model.Hudson.<init>(Hudson.java:86)
at hudson.model.Hudson.<init>(Hudson.java:82)
at hudson.WebAppMain$3.run(WebAppMain.java:231)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:248)
Is there any way to recover from this aside from restoring a backup? I don't presently have shell access to the system, so please keep that in mind for your answers.
Jenkins version: 2.46.2
Splunk plugin version: 1.3.1
I was able to get SSH access to the system and did the following to solve my problem. Solution taken from the Removing and Disabling Plugins wiki page.
touch /var/lib/jenkins/plugins/splunk-devops.jpi.disabled
touch /var/lib/jenkins/plugins/splunk-devops-extend.jpi.disabled
Then, I rebooted. From the documentation, I assume that this is possible with any plugin.

NullPointerException in Embedded Jetty when using JMX

I get a NullPointerException every now and then in my Embedded Jetty application:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.jmx.ObjectMBean.getAttribute(ObjectMBean.java:349)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
...
Caused by: java.lang.NullPointerException
at org.eclipse.jetty.server.handler.ContextHandler.isShutdown(ContextHandler.java:682)
... 28 more
I have set up JMX like this:
Server server = new Server();
MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
MBeanContainer mbContainer = new MBeanContainer(mBeanServer);
server.addBean(mbContainer);
This happens randomly (not always) at server startup and only if I have my Prometheous JMX Exporter running on the same machine, most likely connecting to the JMX server at a phase when Jetty is not quite ready to handle the connection.
Any ideas how to fix this?
The JMX exporter should be gracefully handling the error, so you shouldn't have to worry on the JMX side.
Looking at the jetty code, that's just a synchronised accessor so I don't see how you can get an NPE out of it. Presumably the object is null, so this is likely worth filing a bug against jetty.

Cannot start jenkins after changed keystore password

I changed my keystore password today to a more secure one and I cannot restart jenkins. The log has the following error:
SEVERE: Container startup failed
java.io.IOException: Failed to start a listener: winstone.HttpsConnectorFactory
at winstone.Launcher.spawnListener(Launcher.java:207)
at winstone.Launcher.<init>(Launcher.java:149)
at winstone.Launcher.main(Launcher.java:352)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at Main._main(Main.java:246)
at Main.main(Main.java:91)
Caused by: winstone.WinstoneException: Error getting the SSL context object
at winstone.HttpsConnectorFactory.getSSLContext(HttpsConnectorFactory.java:241)
at winstone.HttpsConnectorFactory.createConnector(HttpsConnectorFactory.java:131)
at winstone.HttpsConnectorFactory.start(HttpsConnectorFactory.java:116)
at winstone.Launcher.spawnListener(Launcher.java:205)
... 8 more
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)
at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:146)
at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:96)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetKey(JavaKeyStore.java:70)
at java.security.KeyStore.getKey(KeyStore.java:1023)
at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:133)
at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70)
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256)
at winstone.HttpsConnectorFactory.getSSLContext(HttpsConnectorFactory.java:210)
I am assuming the password was stored somewhere and now they don't jive...
Turns out something got screwed up with keystore when I used a 3d party utility to combine a couple of certificates and change the password. I cleaned all that up and all is fine again.

Resources