I have a question regarding Flink. I am running an application in a local cluster, with 1 TaskManager and 4 Taskslots.
After some time of running the application, I got an Timeout error:
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id feea6a6702a0cf960ae2847b5bd25665 timed out.
I have seen some posts with this topic but any answer to it. Could you help me to see the root cause, or a posible troubleshooting?
I am using flink version 1.5.3
It seems that the docker container of taskmanagers and JobManager are stopped when this happens.
Let me add the error trace from the JobManager container logs:
2019-06-09 13:31:06,300 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Socket Window NgsiEvent (ef3a860de48d54544d973754c6170d8b) switched from state FAILING to FAILED.
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id 63dbab620797b84da023b33578478238 timed out.
at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1609)
at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:339)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:154)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-06-09 13:31:06,308 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Could not restart the job Socket Window NgsiEvent (ef3a860de48d54544d973754c6170d8b) because the restart strategy prevented it.
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id 63dbab620797b84da023b33578478238 timed out.
at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1609)
at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:339)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:154)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-06-09 13:31:06,317 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job ef3a860de48d54544d973754c6170d8b.
2019-06-09 13:31:06,322 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down
2019-06-09 13:31:06,331 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f]
2019-06-09 13:31:06,351 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job ef3a860de48d54544d973754c6170d8b reached globally terminal state FAILED.
2019-06-09 13:31:06,434 INFO org.apache.flink.runtime.jobmaster.JobMaster - Stopping the JobMaster for job Socket Window NgsiEvent(ef3a860de48d54544d973754c6170d8b).
2019-06-09 13:31:06,447 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Suspending SlotPool.
2019-06-09 13:31:06,448 INFO org.apache.flink.runtime.jobmaster.JobMaster - Close ResourceManager connection 883e842633b0fd9a2e53ab45778581fe: JobManager is shutting down..
2019-06-09 13:31:06,449 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcActor - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-06-09 13:31:06,457 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Disconnect job manager 00000000000000000000000000000000#akka.tcp://flink#jobmanager:6123/user/jobmanager_2 for job ef3a860de48d54544d973754c6170d8b from the resource manager.
2019-06-09 13:31:06,459 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Stopping SlotPool.
2019-06-09 13:31:06,460 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManagerRunner already shutdown.
2019-06-09 13:31:16,304 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f: Name or service not known]
2019-06-09 13:31:26,320 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f: Name or service not known]
2019-06-09 13:31:36,286 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f]
Thanks in advance!
Related
When using neo4j-admin memrec there is a recommendation to activate the following in the neo4j.conf file:
dbms.jvm.additional=-XX:+ExitOnOutOfMemoryError
However if I add the line in, I am unable to restart the neo4j service at all. If I check the debug log the service does not even begin to startup. Is there an error in that particular suggested config line that prevents the service from starting before it has even checked anything else?
:~$ sudo service neo4j restart
:~$ tail -f /var/log/neo4j/debug.log
2022-01-24 14:49:26.937+0000 INFO [o.n.d.d.DefaultDatabaseManager] Stopped 'DatabaseId{21dafb04[neo4j]}' successfully.
2022-01-24 14:49:26.937+0000 INFO [o.n.d.d.DefaultDatabaseManager] Stopping 'DatabaseId{00000000[system]}'.
2022-01-24 14:49:26.938+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] [system/00000000] Requirement `Database unavailable` makes database system unavailable.
2022-01-24 14:49:26.938+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] [system/00000000] DatabaseId{00000000[system]} is unavailable.
2022-01-24 14:49:26.939+0000 INFO [o.n.k.d.Database] [system/00000000] Waiting for closing transactions.
2022-01-24 14:49:26.940+0000 INFO [o.n.k.d.Database] [system/00000000] All transactions are closed.
2022-01-24 14:49:26.940+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [system/00000000] Checkpoint triggered by "Database shutdown" # txId: 71 checkpoint started. ..
2022-01-24 14:49:26.956+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [system/00000000] Checkpoint triggered by "Database shutdown" # txId: 71 checkpoint completed in 15ms
2022-01-24 14:49:26.956+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] [system/00000000] No log version pruned. The strategy used was '1 days'.
2022-01-24 14:49:26.974+0000 INFO [o.n.d.d.DefaultDatabaseManager] Stopped 'DatabaseId{00000000[system]}' successfully.
I install kylin by https://github.com/cas-packone/ambari-kylin-service/
2020-01-13 01:52:16,673 INFO [main] utils.Compatibility:41 : Running in ZooKeeper 3.4.x compatibility mode
2020-01-13 01:52:16,710 INFO [main] imps.CuratorFrameworkImpl:284 : Starting
2020-01-13 01:52:16,715 INFO [main] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=node1:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState#6c000e0c
2020-01-13 01:52:16,717 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:1013 : Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-01-13 01:52:16,719 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:856 : Socket connection established, initiating session, client: /127.0.0.1:49768, server: node1/127.0.0.1:2181
2020-01-13 01:52:16,721 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:1273 : Session establishment complete on server node1/127.0.0.1:2181, sessionid = 0x16f798087cc03af, negotiated timeout = 60000
2020-01-13 01:52:16,722 INFO [main] imps.CuratorFrameworkImpl:326 : Default schema
2020-01-13 01:52:16,725 DEBUG [main] util.ZookeeperDistributedLock:142 : 6521#node1 trying to lock /kylin/kylin_metadata/create_htable/kylin_metadata/lock
2020-01-13 01:52:16,730 INFO [main-EventThread] state.ConnectionStateManager:237 : State change: CONNECTED
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata#hbase
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:83)
... 3 more
Caused by: java.lang.NoSuchMethodError: org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModePathAndBytesable;
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:145)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:166)
at org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:305)
at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:110)
at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:91)
... 8 more
2020-01-13 01:52:16,772 INFO [Curator-Framework-0] imps.CuratorFrameworkImpl:924 : backgroundOperationsLoop exiting
2020-01-13 01:52:16,773 INFO [Thread-1] zookeeper.ReadOnlyZKClient:344 : Close zookeeper connection 0x7b94089b to node1:2181
2020-01-13 01:52:16,775 INFO [ReadOnlyZKClient-node1:2181#0x7b94089b] zookeeper.ZooKeeper:692 : Session: 0x16f798087cc03ae closed
2020-01-13 01:52:16,775 INFO [ReadOnlyZKClient-node1:2181#0x7b94089b-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down
2020-01-13 01:52:16,777 INFO [Thread-4] zookeeper.ZooKeeper:692 : Session: 0x16f798087cc03af closed
2020-01-13 01:52:16,777 INFO [main-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down
ERROR: Unknown error. Please check full log.
I get this error when start Alwayson sql, tried many ways but the results still same. any ideas why?
Im using 1 cluster, 1 analytics+search center, 2 ubuntu 16.04 nodes.
INFO [ALWAYSON-SQL] 2019-02-14 11:36:01,348 ALWAYSON-SQL AlwaysOnSqlRunner.scala:304 - Shutting down AlwaysOn SQL.
INFO [ALWAYSON-SQL] 2019-02-14 11:36:01,617 ALWAYSON-SQL AlwaysOnSqlRunner.scala:328 - Set status to stopped
INFO [ALWAYSON-SQL] 2019-02-14 11:36:01,620 ALWAYSON-SQL AlwaysOnSqlRunner.scala:382 - Reserve port for AlwaysOn SQL
INFO [ALWAYSON-SQL] 2019-02-14 11:36:04,621 ALWAYSON-SQL AlwaysOnSqlRunner.scala:375 - Release reserved port
INFO [ALWAYSON-SQL] 2019-02-14 11:36:04,622 ALWAYSON-SQL AlwaysOnSqlRunner.scala:805 - Set InCluster token to DseFs client
INFO [ForkJoinPool-1-worker-1] 2019-02-14 11:36:04,650 AlwaysOnSqlRunner.scala:740 - dsefs server heartbeat response: pong
INFO [ForkJoinPool-1-worker-3] 2019-02-14 11:36:04,757 AlwaysOnSqlRunner.scala:704 - Create DseFs directory /var/log/spark/alwayson_sql
INFO [ForkJoinPool-1-worker-3] 2019-02-14 11:36:04,758 AlwaysOnSqlRunner.scala:805 - Set InCluster token to DseFs client
ERROR [ForkJoinPool-1-worker-3] 2019-02-14 11:36:04,788 AlwaysOnSqlRunner.scala:722 - Failed to check dsefs directory alwayson_sql
com.datastax.bdp.fs.model.AccessDeniedException: Insufficient permissions to path /
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:258)
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:232)
at spray.json.JsValue.convertTo(JsValue.scala:31)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:48)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:44)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
INFO [ALWAYSON-SQL] 2019-02-14 11:36:04,788 ALWAYSON-SQL AlwaysOnSqlRunner.scala:247 - ALWAYSON-SQL caused an exception in state RUNNING : com.datastax.bdp.fs.model.AccessDeniedException: Insufficient permissions to path /
com.datastax.bdp.fs.model.AccessDeniedException: Insufficient permissions to path /
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:258)
at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:232)
at spray.json.JsValue.convertTo(JsValue.scala:31)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:48)
at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$331$1.apply(RestResponse.scala:44)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run(Thread.java:748)
I have seen this problem too! It was a permissions problem in dsefs! To fix, login with the root Cassandra user, and change permissions of the your alwayson log directory to the alwayson user.
I'm trying to run confluence image on a docker container. This behavior is Intermittent - most of the time I can't run it, but at sometimes I can.
After downloading the image I run these commands in order put confluence online:
sudo chown -R daemon /data/confluence
sudo chown -R daemon /var/atlassian/application-data/confluence
docker run -v /data/your-confluence-home:/var/atlassian/application-data/confluence --name="confluence" -d -p 8090:8090 -p 8091:8091 atlassian/confluence-server
But then, I get this error on docker logs confluence
If you encounter issues starting up Confluence, please see the Installation guide at http://confluence.atlassian.com/display/DOC/Confluence+Installation+Guide
Server startup logs are located in /opt/atlassian/confluence/logs/catalina.out
---------------------------------------------------------------------------
Using Java: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
2017-05-23 15:33:41,519 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] A Context element for ${confluence.context.path}/synchrony-proxy is found in /opt/atlassian/confluence/conf/server.xml. No further action is required
---------------------------------------------------------------------------
23-May-2017 15:33:42.013 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
23-May-2017 15:33:42.026 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
23-May-2017 15:33:42.028 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 381 ms
23-May-2017 15:33:42.033 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Tomcat-Standalone
23-May-2017 15:33:42.033 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.41
SLF4J: A number (1) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
23-May-2017 15:33:43.517 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Spring WebApplicationInitializers detected on classpath: [com.atlassian.synchrony.proxy.SynchronyDispatcherServletInitializer#335dedfb]
23-May-2017 15:33:43.814 INFO [localhost-startStop-2] org.apache.catalina.core.ApplicationContext.log Initializing Spring FrameworkServlet 'dispatcher'
23-May-2017 15:33:43.814 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization started
23-May-2017 15:33:43.819 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.prepareRefresh Refreshing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue May 23 15:33:43 UTC 2017]; root of context hierarchy
23-May-2017 15:33:43.864 INFO [localhost-startStop-2] org.springframework.web.context.support.AnnotationConfigWebApplicationContext.loadBeanDefinitions Registering annotated classes: [class com.atlassian.synchrony.proxy.websocket.WebSocketConfig,class com.atlassian.synchrony.proxy.web.SynchronyWebMvcConfig]
2017-05-23 15:33:44,007 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.2.0 [build 7111 based on commit hash 583f3f4010922d699723a091097524472954c767] - synchrony version 0.3.1-release-confluence_6.1-e20fb944
23-May-2017 15:33:44.371 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler.initialize Initializing ExecutorService 'defaultSockJsTaskScheduler'
23-May-2017 15:33:44.670 INFO [localhost-startStop-2] org.springframework.web.socket.server.support.WebSocketHandlerMapping.registerHandler Mapped URL path [/sockjs/v1/**] onto handler of type [class org.springframework.web.socket.sockjs.support.SockJsHttpRequestHandler]
23-May-2017 15:33:44.854 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/healthcheck]}" onto public com.atlassian.synchrony.proxy.web.HealthCheckResult com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.getSynchronyProxyInfo()
23-May-2017 15:33:44.855 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.register Mapped "{[/reload],methods=[PUT]}" onto public org.springframework.http.ResponseEntity com.atlassian.synchrony.proxy.web.SynchronyProxyRestController.reloadConfiguration(com.atlassian.synchrony.proxy.web.SynchronyProxyConfigPayload)
23-May-2017 15:33:44.873 INFO [localhost-startStop-2] org.springframework.web.servlet.handler.SimpleUrlHandlerMapping.registerHandler Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.DefaultServletHttpRequestHandler]
23-May-2017 15:33:44.939 INFO [localhost-startStop-2] org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.initControllerAdviceCache Looking for #ControllerAdvice: WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue May 23 15:33:43 UTC 2017]; root of context hierarchy
23-May-2017 15:33:45.056 INFO [localhost-startStop-2] org.springframework.context.support.DefaultLifecycleProcessor.start Starting beans in phase 2147483647
23-May-2017 15:33:45.094 INFO [localhost-startStop-2] org.springframework.web.servlet.DispatcherServlet.initServletBean FrameworkServlet 'dispatcher': initialization completed in 1279 ms
2017-05-23 15:33:46,688 WARN [localhost-startStop-1] [com.atlassian.config.ApplicationConfig] setApplicationHome Application home does not exist. Creating directory: /var/atlassian/application-data/confluence/shared-home
2017-05-23 15:33:46,783 ERROR [localhost-startStop-1] [atlassian.config.bootstrap.DefaultAtlassianBootstrapManager] init Home is not configured properly:
com.atlassian.config.ConfigurationException: Couldn't save confluence.cfg.xml to /var/atlassian/application-data/confluence directory.
at com.atlassian.config.xml.AbstractDom4jXmlConfigurationPersister.saveDocument(AbstractDom4jXmlConfigurationPersister.java:86)
at com.atlassian.config.xml.DefaultDom4jXmlConfigurationPersister.save(DefaultDom4jXmlConfigurationPersister.java:24)
at com.atlassian.config.ApplicationConfig.save(ApplicationConfig.java:330)
at com.atlassian.confluence.setup.DefaultBootstrapManager.populateSynchronyConfiguration(DefaultBootstrapManager.java:298)
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:509)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:71)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:36)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:127)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:60)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at com.atlassian.config.xml.AbstractDom4jXmlConfigurationPersister.saveDocumentAtomically(AbstractDom4jXmlConfigurationPersister.java:92)
at com.atlassian.config.xml.AbstractDom4jXmlConfigurationPersister.saveDocument(AbstractDom4jXmlConfigurationPersister.java:82)
... 17 more
What am I doing wrong?
Have you tried via named volume (if using a docker version >= 1.9)?
$ sudo docker volume create --name confluenceVolume
$ sudo docker run -v confluenceVolume:/var/atlassian/application-data/confluence --name="confluence" -d -p 8090:8090 -p 8091:8091 atlassian/confluence-server
Add ENTRYPOINT script and use it. Example - https://github.com/AfsmNGhr/dockemacs/blob/master/Dockerfile#L34. See also - http://www.johnzaccone.io/entrypoint-vs-cmd-back-to-basics/.
I'm trying access our TFS server when VPN'd from home but it only gives me the following message:
Connecting to TFS...
git-tf: Permission denied: connect
I'm using a fully qualified URL and everything else works just fine over the VPN. Are there any authentication config settings I could try? I can't find mention of anything online anywhere.
And of course when I'm in the office it works just fine.
EDIT - Stack trace from logs
2012-10-12 10:07:31,231 INFO [main] (com.microsoft.tfs.core.config.httpclient.DefaultHTTPClientFactory) HttpClient configured for http://svrtfs2010.ms.innovativesol.com:8080/tfs/DefaultCollection, authenticating as innovative/jrusso
2012-10-12 10:07:31,535 INFO [main] (com.microsoft.tfs.core.httpclient.HttpMethodDirector) I/O exception (java.net.SocketException) caught when processing request: Permission denied: connect
2012-10-12 10:07:31,535 INFO [main] (com.microsoft.tfs.core.httpclient.HttpMethodDirector) Retrying request
2012-10-12 10:07:31,535 INFO [main] (com.microsoft.tfs.core.httpclient.HttpMethodDirector) I/O exception (java.net.SocketException) caught when processing request: Permission denied: connect
2012-10-12 10:07:31,535 INFO [main] (com.microsoft.tfs.core.httpclient.HttpMethodDirector) Retrying request
2012-10-12 10:07:31,535 INFO [main] (com.microsoft.tfs.core.httpclient.HttpMethodDirector) I/O exception (java.net.SocketException) caught when processing request: Permission denied: connect
2012-10-12 10:07:31,535 INFO [main] (com.microsoft.tfs.core.httpclient.HttpMethodDirector) Retrying request
2012-10-12 10:07:31,544 WARN [main] (git-tf) Error executing command: pull --deep --rebase
com.microsoft.tfs.core.exceptions.TECoreException: Permission denied: connect
at com.microsoft.tfs.core.exceptions.mappers.TECoreExceptionMapper.map(TECoreExceptionMapper.java:99)
at com.microsoft.tfs.core.exceptions.mappers.LocationExceptionMapper.map(LocationExceptionMapper.java:32)
at com.microsoft.tfs.core.clients.framework.location.internal.LocationWebServiceProxy.connect(LocationWebServiceProxy.java:76)
at com.microsoft.tfs.core.clients.framework.location.LocationService.connect(LocationService.java:754)
at com.microsoft.tfs.core.clients.framework.location.LocationService.ensureConnected(LocationService.java:884)
at com.microsoft.tfs.core.clients.framework.location.LocationService.ensureAuthenticated(LocationService.java:923)
at com.microsoft.tfs.core.TFSConnection.ensureAuthenticated(TFSConnection.java:739)
at com.microsoft.gittf.client.clc.commands.framework.Command.getConnection(Command.java:354)
at com.microsoft.gittf.client.clc.commands.framework.Command.getConnection(Command.java:326)
at com.microsoft.gittf.client.clc.commands.framework.Command.getVersionControlClient(Command.java:474)
at com.microsoft.gittf.client.clc.commands.framework.Command.getVersionControlService(Command.java:492)
at com.microsoft.gittf.client.clc.commands.PullCommand.run(PullCommand.java:140)
at com.microsoft.gittf.client.clc.Main.main(Main.java:319)
Caused by: com.microsoft.tfs.core.ws.runtime.exceptions.TransportException: Permission denied: connect
at com.microsoft.tfs.core.ws.runtime.client.SOAPService.executeSOAPRequestInternal(SOAPService.java:744)
at com.microsoft.tfs.core.ws.runtime.client.SOAPService.executeSOAPRequest(SOAPService.java:473)
at ms.ws._LocationWebServiceSoap12Service.connect(_LocationWebServiceSoap12Service.java:384)
at com.microsoft.tfs.core.clients.framework.location.internal.LocationWebServiceProxy.connect(LocationWebServiceProxy.java:70)
... 10 more
Caused by: java.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.microsoft.tfs.core.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:147)
at com.microsoft.tfs.core.httpclient.HttpConnection.open(HttpConnection.java:862)
at com.microsoft.tfs.core.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1690)
at com.microsoft.tfs.core.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:495)
at com.microsoft.tfs.core.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:197)
at com.microsoft.tfs.core.httpclient.HttpClient.executeMethod(HttpClient.java:464)
at com.microsoft.tfs.core.httpclient.HttpClient.executeMethod(HttpClient.java:376)
at com.microsoft.tfs.core.ws.runtime.client.SOAPService.executeSOAPRequestInternal(SOAPService.java:588)
... 13 more
This is related to Java bug 703177. Java 7 brings IPv6 support, and Cisco's VPN client reportedly rejects IPv6 traffic. You can force Java to use IPv4 as its transit with the java.net.preferIPv4Stack system property.
You will need to edit the git-tf launcher scripts that start the JVM in order to do this. You can add the -Djava.net.preferIPv4Stack=true argument to the launch line in both git-tf and git-tf.cmd.
The last lines of git-tf should be:
exec java -Xmx512m -classpath "$GITTF_CLASSPATH" \
"-Dcom.microsoft.tfs.jni.native.base-directory=$BASE_DIRECTORY/native" \
-Djava.net.preferIPv4Stack=true \
com.microsoft.gittf.client.clc.Main "$#"
The 47th line of git-tf.cmd should be:
java -ea -Xmx512M -cp %GITTF_CLASSPATH% "-Dcom.microsoft.tfs.jni.native.base-directory=%BASE_DIRECTORY%native" -Djava.net.preferIPv4Stack=true com.microsoft.gittf.client.clc.Main %*