Neo4j cluster not starting - neo4j

Question

I am building a Neo4j HA cluster and have followed the steps at https://neo4j.com/docs/operations-manual/current/tutorial/highly-available-cluster/
I have 3 machines running Neo4j in a single-instance Docker container.
Neo4j works fine as a standalone server on each machine, but when I add the HA config, it fails to start.
Each machine was created from the same prebuilt image so they are configured identically except for the ha.server_id which is unique for each machine (values are 1, 2, 3).
Firewall is off and I've verified connectivity between the machines.
The Neo4j HA config is:
ha.server_id=1
ha.initial_hosts=172.0.30.110:5001,172.0.31.39:5001,172.0.32.249:5001
dbms.mode=HA
To start Neo4j I'm running the following on each machine:
/usr/bin/docker run \
--publish=7474:7474 --publish=7687:7687 --publish=5001:5001 \
--volume=/var/lib/neo4j/data:/data \
--volume=/var/lib/neo4j/logs:/logs \
--volume=/var/lib/neo4j/conf:/conf \
--name=neo4j \
neo4j:3.0-enterprise
When I run the startup command on machines 2 and 3, and sometimes on machine 1, I get an error following 'Attempting to join cluster'. Occasionally (but not all the time) when I run the command on machine 1, it realizes it is the first machine and goes ahead to initialize itself as master.
Below are the full failure and success messages. I can't figure out why only one machine (incidentally the first) is doing what it's supposed to do sometimes and not all the time. I also need to figure out how to get the other two machines to register themselves with the cluster.
Why is one node sometimes launching Neo4j with clustering and sometimes not?
Why are the 2nd and 3rd nodes not registering with the cluster?
Where should I look next?
Error (seen on all 3 machines):
> 2017-05-30 09:18:35.502+0000 INFO Starting... 2017-05-30
> 09:18:35.833+0000 INFO Write transactions to database disabled
> 2017-05-30 09:18:36.116+0000 INFO Bolt enabled on 0.0.0.0:7687.
> 2017-05-30 09:18:36.131+0000 INFO Initiating metrics... 2017-05-30
> 09:18:36.937+0000 INFO Attempting to join cluster of
> [172.0.30.110:5001, 172.0.31.39:5001, 172.0.32.249:5001] 2017-05-30
> 09:19:06.996+0000 ERROR Failed to start Neo4j: Starting Neo4j failed:
> Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception. Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception. org.neo4j.server.ServerStartupException: Starting
> Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
> at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
> at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:215)
> at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:90)
> at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:67)
> at org.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:32)
> Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
> at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
> at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
> ... 3 more Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory,
> /var/lib/neo4j/data/databases/fraud.db
> at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
> at org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory.newFacade(HighlyAvailableFacadeFactory.java:42)
> at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphDatabase.java:41)
> at org.neo4j.server.enterprise.EnterpriseNeoServer.lambda$static$0(EnterpriseNeoServer.java:80)
> at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
> ... 5 more Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.cluster.client.ClusterJoin#5d2e6054' was successfully
> initialized, but failed to start. Please see attached cause exception.
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
> at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
> at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
> at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
> ... 10 more Caused by: java.util.concurrent.TimeoutException: Conversation-response mapping:
> {3/13#=ResponseFuture{conversationId='3/13#',
> initiatedByMessageType=join, response=null}}
> at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:314)
> at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:143)
> at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:82)
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
> ... 14 more ubuntu#ip-172-0-32-249:~$ sudo /usr/bin/docker rm neo4j neo4j ubuntu#ip-172-0-32-249:~$ sudo /usr/bin/docker run
> --publish=7474:7474 --publish=7687:7687 --publish=5001:5001 --volume=/var/lib/neo4j/data:/data --volume=/var/lib/neo4j/logs:/logs --volume=/var/lib/neo4j/conf:/conf --name=neo4j neo4j:3.0-enterprise Starting Neo4j. 2017-05-30 09:20:17.877+0000 INFO No SSL certificate
> found, generating a self-signed certificate.. 2017-05-30
> 09:20:18.387+0000 INFO Starting... 2017-05-30 09:20:18.676+0000 INFO
> Write transactions to database disabled 2017-05-30 09:20:19.036+0000
> INFO Bolt enabled on 0.0.0.0:7687. 2017-05-30 09:20:19.046+0000 INFO
> Initiating metrics... 2017-05-30 09:20:19.779+0000 INFO Attempting to
> join cluster of [172.0.30.110:5001, 172.0.31.39:5001,
> 172.0.32.249:5001] 2017-05-30 09:20:49.851+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception. Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception. org.neo4j.server.ServerStartupException: Starting
> Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
> at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
> at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:215)
> at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:90)
> at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:67)
> at org.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:32)
> Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase#1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
> at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
> at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
> ... 3 more Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory,
> /var/lib/neo4j/data/databases/fraud.db
> at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
> at org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory.newFacade(HighlyAvailableFacadeFactory.java:42)
> at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphDatabase.java:41)
> at org.neo4j.server.enterprise.EnterpriseNeoServer.lambda$static$0(EnterpriseNeoServer.java:80)
> at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
> ... 5 more Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.cluster.client.ClusterJoin#6c1c89e2' was successfully
> initialized, but failed to start. Please see attached cause exception.
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
> at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
> at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
> at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
> ... 10 more Caused by: java.util.concurrent.TimeoutException: Conversation-response mapping:
> {3/13#=ResponseFuture{conversationId='3/13#',
> initiatedByMessageType=join, response=null}}
> at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:314)
> at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:143)
> at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:82)
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
> ... 14 more
Success (seen on one machine only):
2017-05-30 09:20:22.963+0000 INFO Starting...
2017-05-30 09:20:23.319+0000 INFO Write transactions to database disabled
2017-05-30 09:20:23.619+0000 INFO Bolt enabled on 0.0.0.0:7687.
2017-05-30 09:20:23.628+0000 INFO Initiating metrics...
2017-05-30 09:20:24.352+0000 INFO Attempting to join cluster of [172.0.30.110:5001, 172.0.31.39:5001, 172.0.32.249:5001]
2017-05-30 09:20:31.382+0000 INFO Could not join cluster of [172.0.30.110:5001,172.0.31.39:5001, 172.0.32.249:5001]
2017-05-30 09:20:31.382+0000 INFO Creating new cluster with name [neo4j.ha]...
2017-05-30 09:20:31.396+0000 INFO Instance 1 (this server) entered the cluster
2017-05-30 09:20:31.401+0000 INFO Instance 1 (this server) was elected as coordinator
2017-05-30 09:20:31.413+0000 INFO I am 1, moving to master
2017-05-30 09:20:31.465+0000 INFO Instance 1 (this server) was elected as coordinator
2017-05-30 09:20:31.492+0000 INFO I am 1, successfully moved to master
2017-05-30 09:20:31.510+0000 INFO Instance 1 (this server) is available as master at ha://172.17.0.2:6001?serverId=1 with StoreId{creationTime=1496122128110,randomId=-2708986986476371425, storeVersion=15531981201765894, upgradeTime=1496122128110, upgradeId=1}
2017-05-30 09:20:31.612+0000 INFO Instance 1 (this server) is available as backup at backup://127.0.0.1:6362 with StoreId{creationTime=1496122128110, randomId=-2708986986476371425, storeVersion=15531981201765894, upgradeTime=1496122128110, upgradeId=1}
2017-05-30 09:20:31.709+0000 INFO Database available for write transactions
2017-05-30 09:20:32.929+0000 INFO Started.
2017-05-30 09:20:33.087+0000 INFO Mounted REST API at: /db/manage
2017-05-30 09:20:33.828+0000 INFO Remote interface available at http://0.0.0.0:7474/