SCDF server failing to start with the k8s version 1.20 - spring-cloud-dataflow

We recently upgraded to k8s version 1.20.9 and not sure if that is the root cause but SCDF server pod fails to come up with the error below.
I usually deploy scdf server using kubectl based deployment.
Anyone has any idea ? Attached error below.
2022-01-05 05:08:56.207 INFO 1 --- [ main]
o.a.coyote.http11.Http11NioProtocol : Starting ProtocolHandler
["http-nio-80"] 2022-01-05 05:08:56.300 WARN 1 --- [ main]
ConfigServletWebServerApplicationContext : Exception encountered
during context initialization - cancelling refresh attempt:
org.springframework.context.ApplicationContextException: Failed to
start bean 'webServerStartStop'; nested exception is
org.springframework.boot.web.server.WebServerException: Unable to
start embedded Tomcat server 2022-01-05 05:08:56.798 INFO 1 --- [
main] j.LocalContainerEntityManagerFactoryBean : Closing JPA
EntityManagerFactory for persistence unit 'default' 2022-01-05
05:08:56.893 INFO 1 --- [ main]
com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown
initiated... 2022-01-05 05:08:57.194 INFO 1 --- [ main]
com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown
completed. 2022-01-05 05:08:57.197 INFO 1 --- [ main]
o.a.coyote.http11.Http11NioProtocol : Pausing ProtocolHandler
["http-nio-80"] 2022-01-05 05:08:57.197 INFO 1 --- [ main]
o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2022-01-05 05:08:57.292 INFO 1 --- [ main]
o.a.coyote.http11.Http11NioProtocol : Stopping ProtocolHandler
["http-nio-80"] 2022-01-05 05:08:57.293 INFO 1 --- [ main]
o.a.coyote.http11.Http11NioProtocol : Destroying ProtocolHandler
["http-nio-80"] 2022-01-05 05:08:57.793 ERROR 1 --- [ main]
o.s.boot.SpringApplication : Application run failed
org.springframework.context.ApplicationContextException: Failed to
start bean 'webServerStartStop'; nested exception is
org.springframework.boot.web.server.WebServerException: Unable to
start embedded Tomcat server Caused by:
org.springframework.boot.web.server.WebServerException: Unable to
start embedded Tomcat server Caused by:
java.lang.IllegalArgumentException:
standardService.connector.startFailed Caused by:
org.apache.catalina.LifecycleException: Protocol handler start failed
Caused by: java.net.SocketException: Permission denied

What stands out in the trace is SocketException: permission denied It is likely due to some security configuration change in the upgrade affecting the TCP layer. I would start with your security configuration. Keep us posted.

Related

Jenkins agent jobs killed when running Spring Boot Tests with DB

I am running a self-hosted OKD 4 cluster with minimum production requirements (3 control planes and two compute nodes). This setup includes a Jenkins installation - installed via Helm (https://www.jenkins.io/doc/book/installing/kubernetes/) So far everything worked fine: builds start automatically when changes are pushed to Github and when they are successful are deployed to the same cluster where Jenkins runs in.
But currently I am facing the problem that when a build job executes a Spring Boot test which fires up a persistence context. The build agent (a jdk-11 image, see additionalAgent configuration below) gets killed as soon as Spring starts up the persistence context. Downloading dependencies and compilation works fine, btw.
additionalAgents:
jdk-11:
podName: jdk-11
customJenkinsLabels: jdk-11
image: jenkins/jnlp-agent-jdk11
tag: latest
...
When the tests are disabled the job runs fine. But as soon as the persistence gets initialised the agent gets killed.
Those are the configurations I have tried for the test:
Starting with an in-memory h2 database and flyway provisioning.
Without flyway provisioning.
Even without the database connection string set.
The time where the job gets killed is almost the same:
For 1. it is
2021-10-20 22:44:06.637 INFO 299 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2021-10-20 22:44:07.032 INFO 299 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 310 ms. Found 2 JPA repository interfaces.
2021-10-20 22:44:08.240 INFO 299 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=1c9e8306-7514-338e-8a9f-3cfba5c1169b
2021-10-20 22:44:10.527 INFO 299 --- [ main] o.f.c.internal.license.VersionPrinter : Flyway Community Edition 7.7.3 by Redgate
2021-10-20 22:44:10.532 INFO 299 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-10-20 22:44:11.744 INFO 299 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-10-20 22:44:12.041 INFO 299 --- [ main] o.f.c.i.database.base.DatabaseType : Database: jdbc:h2:mem:testdb (H2 1.4)
Killed
For 2.
2021-10-21 19:50:51.604 INFO 306 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2021-10-21 19:50:52.005 INFO 306 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 391 ms. Found 2 JPA repository interfaces.
2021-10-21 19:50:53.510 INFO 306 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=0fd77ef3-b5a2-35cb-b157-6d27c0cfe9a5
2021-10-21 19:50:56.405 INFO 306 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
2021-10-21 19:50:56.708 INFO 306 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.32.Final
2021-10-21 19:50:57.503 INFO 306 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
Killed
And for 3.
2021-10-21 22:02:48.810 INFO 309 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2021-10-21 22:02:49.198 INFO 309 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 380 ms. Found 2 JPA repository interfaces.
2021-10-21 22:02:50.509 INFO 309 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=0fd77ef3-b5a2-35cb-b157-6d27c0cfe9a5
2021-10-21 22:02:53.523 INFO 309 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
2021-10-21 22:02:53.898 INFO 309 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.4.32.Final
Killed
The log of the Jenkins pod just states
Terminated Kubernetes instance for agent jenkins/jdk-11-bjtz5
Disconnected computer jdk-11-bjtz5
2021-10-21 22:02:57.342+0000 [id=465] INFO o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent jenkins/jdk-11-bjtz5
2021-10-21 22:02:57.342+0000 [id=465] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jdk-11-bjtz5
2021-10-21 22:02:57.356+0000 [id=436] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#56] for jdk-11-bjtz5 terminated: java.nio.channels.ClosedChannelException
In all cases there are no exceptions, stacktraces or suspicious events. And these steps are reproducable - when I run the build with the same configuration again the agents gets killed at exactly the same step in the test.
The setup:
Jenkins Version 2.303.2
Jenkins uses a MySQL database running in the same cluster
all Jenkins plugins are up-to-date
OKD currently running at version 4.8.0-0.okd-2021-10-10-030117
currently there are no resource quotas set and the system still has plenty of free resources
I am presuming that a little bit of configuration is missing to make this work. But I just cannot find what it could be. So I am asking: have had anyone the same issue here? Or any guesses what the missing part could be?
When there is some information missing please point it out and I will add it.
After a little step back I took a look into the actual pod which runs the build. And found out that the memory limit of the agents was the problem.
So increasing the limit solved the problem!
I have done that by modifying the local jenkins-values.yaml and updated the limits section of the agent: block in it.
A little confusing for me was the fact that no log entry stated the exceed of memory usage.
Next thought is that I will set a memory limit for the test step via Java options to kill the maven process before the pod exceeds the limit. Guess it would be more transparent in the build.
And as a sidenote: the limit has been set to 512Mi previously and has been exceeded by ~10 MiB -.-
Luckily for me I have found it at this point and the other build-jobs where running fine was just due to the lack of resource-usage (haven't figured that just starting Hibernate would exceed the 512 MiB mark)

SCDF dataflow server 2.4.2 composed task

I use SCDF with skipper server 2.3.2 and dataflow server 2.4.2. deployed with Docker-compose
I build a task with composed-task-runner 2.1.3 release
I try with my task but even with samples, composed-task-runner doesn't launch tasks
Example :
composed-task-runner && mytask && myTask
2020-04-14 14:40:41.633 INFO 231 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'taskExecutor'
2020-04-14 14:40:41.752 INFO 231 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: POSTGRES
2020-04-14 14:40:41.885 INFO 231 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2020-04-14 14:40:42.083 DEBUG 231 --- [ main] o.s.c.t.a.c.DataFlowConfiguration : Not configuring basic security for accessing the Data Flow Server
2020-04-14 14:40:46.922 DEBUG 231 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for postgresql database
2020-04-14 14:40:47.461 DEBUG 231 --- [ main] BatchConfiguration$ReferenceTargetSource : Initializing lazy target object
2020-04-14 14:40:47.552 DEBUG 231 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Starting: TaskExecution{executionId=102, parentExecutionId=null, exitCode=null, taskName='testComposedTask2', startTime=Tue Apr 14 14:40:47 GMT 2020, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[xx=7777, --spring.cloud.data.flow.platformname=default, --spring.cloud.task.executionid=102, --spring.cloud.data.flow.taskappname=composed-task-runner]}
2020-04-14 14:40:47.675 INFO 231 --- [ main] .t.a.c.ComposedtaskrunnerTaskApplication : Started ComposedtaskrunnerTaskApplication in 33.768 seconds (JVM running for 38.059)
2020-04-14 14:40:47.743 DEBUG 231 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution with executionId=102 with the following {exitCode=0, endTime=Tue Apr 14 14:40:47 GMT 2020, exitMessage='null', errorMessage='null'}
2020-04-14 14:40:47.831 INFO 231 --- [ Thread-7] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'taskExecutor'
2020-04-14 14:40:47.836 INFO 231 --- [ Thread-7] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2020-04-14 14:40:47.852 INFO 231 --- [ Thread-7] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2020-04-14 14:40:47.909 INFO 231 --- [ Thread-7] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
Could you help me? I am with a Postgres DB
New remark :
I build a new composed task as said in the documentation :
testComposedTaskA with definition myTask && timestamp
dataflow:>task list
╔═══════════════════════════╤═══════════════════╤═══════════╤═══════════╗
║ Task Name │ Task Definition │description│Task Status║
╠═══════════════════════════╪═══════════════════╪═══════════╪═══════════╣
║testComposedTaskA-myTask │myTask │ │UNKNOWN ║
║testComposedTaskA-timestamp│timestamp │ │UNKNOWN ║
║testComposedTaskA │myTask && timestamp│ │UNKNOWN ║
╚═══════════════════════════╧═══════════════════╧═══════════╧═══════════╝
dataflow:>task launch testComposedTaskA
Launched task 'testComposedTaskA' with execution id 135
dataflow:>task execution list
╔═════════════════╤═══╤════════════════════════════╤════════════════════════════╤═════════╗
║ Task Name │ID │ Start Time │ End Time │Exit Code║
╠═════════════════╪═══╪════════════════════════════╪════════════════════════════╪═════════╣
║testComposedTaskA│135│Fri Apr 17 15:04:42 GMT 2020│Fri Apr 17 15:04:42 GMT 2020│0 ║
╚═════════════════╧═══╧════════════════════════════╧════════════════════════════╧═════════╝
Sub-tasks are never executed . In the samples, we can see the sub task as executed
What is wrong?
As request, the dataflow server start log
Hi the dataflow server start log :
dataflow-server | 2020-04-20 07:43:24.047 INFO 1 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
dataflow-server | 2020-04-20 07:43:24.061 INFO 1 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
dataflow-server | 2020-04-20 07:43:27.735 DEBUG 1 --- [ main] o.s.c.t.c.SimpleTaskAutoConfiguration : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
dataflow-server | 2020-04-20 07:43:27.745 DEBUG 1 --- [ main] o.s.c.t.c.DefaultTaskConfigurer : EntityManager was found, using JpaTransactionManager
dataflow-server | 2020-04-20 07:43:28.282 INFO 1 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: POSTGRES
skipper | 2020-04-20 07:43:28.294 INFO 1 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
dataflow-server | 2020-04-20 07:43:28.505 INFO 1 --- [ main] o.s.c.d.s.b.SimpleJobServiceFactoryBean : No database type set, using meta data indicating: POSTGRES
dataflow-server | 2020-04-20 07:43:28.975 WARN 1 --- [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
dataflow-server | 2020-04-20 07:43:29.392 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
dataflow-server | 2020-04-20 07:43:32.150 INFO 1 --- [ main] .s.c.DataFlowControllerAutoConfiguration : Skipper URI [http://skipper-server:7577/api]
dataflow-server | 2020-04-20 07:43:32.614 INFO 1 --- [ main] o.a.coyote.http11.Http11NioProtocol : Starting ProtocolHandler ["http-nio-9393"]
dataflow-server | 2020-04-20 07:43:32.764 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 9393 (http) with context path ''
dataflow-server | 2020-04-20 07:43:32.780 INFO 1 --- [ main] o.s.c.d.s.s.DataFlowServerApplication : Started DataFlowServerApplication in 68.947 seconds (JVM running for 78.165)
dataflow-server | 2020-04-20 07:43:33.396 INFO 1 --- [ main] .s.c.d.s.s.LauncherInitializationService : Added 'Local' platform account 'default' into Task Launcher repository.
The log at the start of the task :
2020-04-20 08:01:58.886 INFO 115 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data repositories in DEFAULT mode.
2020-04-20 08:01:59.008 INFO 115 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 68ms. Found 0 repository interfaces.
2020-04-20 08:02:00.545 INFO 115 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=56219e3d-5a2d-3fb3-8901-c3dad5be90af
2020-04-20 08:02:00.943 INFO 115 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$cd22d842] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-04-20 08:02:01.049 INFO 115 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$e93cdb3f] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-04-20 08:02:01.062 INFO 115 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration' of type [org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration$$EnhancerBySpringCGLIB$$529283ce] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-04-20 08:02:01.094 INFO 115 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration' of type [org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration$$EnhancerBySpringCGLIB$$9ae88dd1] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-04-20 08:02:02.529 INFO 115 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2020-04-20 08:02:03.239 INFO 115 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2020-04-20 08:02:04.489 INFO 115 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [
name: default
...]
2020-04-20 08:02:07.075 INFO 115 --- [ main] org.hibernate.Version : HHH000412: Hibernate Core {5.3.13.Final}
2020-04-20 08:02:07.111 INFO 115 --- [ main] org.hibernate.cfg.Environment : HHH000206: hibernate.properties not found
2020-04-20 08:02:08.896 INFO 115 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
2020-04-20 08:02:10.736 INFO 115 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL95Dialect
2020-04-20 08:02:10.825 INFO 115 --- [ main] o.h.e.j.e.i.LobCreatorBuilderImpl : HHH000422: Disabling contextual LOB creation as connection was null
2020-04-20 08:02:10.843 INFO 115 --- [ main] org.hibernate.type.BasicTypeRegistry : HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType#3eb91815
2020-04-20 08:02:11.457 INFO 115 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2020-04-20 08:02:11.675 DEBUG 115 --- [ main] o.s.c.t.c.SimpleTaskAutoConfiguration : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
2020-04-20 08:02:11.679 DEBUG 115 --- [ main] o.s.c.t.c.DefaultTaskConfigurer : EntityManager was found, using JpaTransactionManager
2020-04-20 08:02:11.902 INFO 115 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'taskExecutor'
2020-04-20 08:02:12.005 INFO 115 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: POSTGRES
2020-04-20 08:02:12.107 INFO 115 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2020-04-20 08:02:12.203 DEBUG 115 --- [ main] o.s.c.t.a.c.DataFlowConfiguration : Not configuring basic security for accessing the Data Flow Server
2020-04-20 08:02:15.942 DEBUG 115 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for postgresql database
2020-04-20 08:02:16.551 DEBUG 115 --- [ main] BatchConfiguration$ReferenceTargetSource : Initializing lazy target object
2020-04-20 08:02:16.617 DEBUG 115 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Starting: TaskExecution{executionId=136, parentExecutionId=null, exitCode=null, taskName='testComposedTaskA', startTime=Mon Apr 20 08:02:16 GMT 2020, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--spring.cloud.data.flow.platformname=default, --spring.cloud.task.executionid=136, --spring.cloud.data.flow.taskappname=composed-task-runner]}
2020-04-20 08:02:16.686 INFO 115 --- [ main] .t.a.c.ComposedtaskrunnerTaskApplication : Started ComposedtaskrunnerTaskApplication in 31.176 seconds (JVM running for 34.541)
2020-04-20 08:02:16.717 DEBUG 115 --- [ main] o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution with executionId=136 with the following {exitCode=0, endTime=Mon Apr 20 08:02:16 GMT 2020, exitMessage='null', errorMessage='null'}
2020-04-20 08:02:16.778 INFO 115 --- [ Thread-7] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'taskExecutor'
2020-04-20 08:02:16.784 INFO 115 --- [ Thread-7] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2020-04-20 08:02:16.803 INFO 115 --- [ Thread-7] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2020-04-20 08:02:16.854 INFO 115 --- [ Thread-7] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
the docker compose used :
dataflow-server:
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgres:5432/dataflow
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=rootpw
- SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver
- spring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults=false
- spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL95Dialect
entrypoint: "./wait-for-it.sh -t 240 postgres:5432 -- java -jar /maven/spring-cloud-dataflow-server.jar"
skipper-server:
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgres:5432/dataflow
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=rootpw
- SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver
entrypoint: "./wait-for-it.sh -t 240 postgres:5432 -- java -Djava.security.egd=file:/dev/./urandom -jar /spring-cloud-skipper-server.jar"
I hope, it will be an help.
Regards
Frederic
From the log: composed-task-runner && mytask && myTask, the syntax for creating the Composed Task definition looks incorrect.
The app name composed-task-runner should not be part of the task DSL.
You can check here for the detailed documentation on how to create and manage composed tasks.

Kylin start fail java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata#hbase

I install kylin by https://github.com/cas-packone/ambari-kylin-service/
2020-01-13 01:52:16,673 INFO [main] utils.Compatibility:41 : Running in ZooKeeper 3.4.x compatibility mode
2020-01-13 01:52:16,710 INFO [main] imps.CuratorFrameworkImpl:284 : Starting
2020-01-13 01:52:16,715 INFO [main] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=node1:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState#6c000e0c
2020-01-13 01:52:16,717 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:1013 : Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-01-13 01:52:16,719 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:856 : Socket connection established, initiating session, client: /127.0.0.1:49768, server: node1/127.0.0.1:2181
2020-01-13 01:52:16,721 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:1273 : Session establishment complete on server node1/127.0.0.1:2181, sessionid = 0x16f798087cc03af, negotiated timeout = 60000
2020-01-13 01:52:16,722 INFO [main] imps.CuratorFrameworkImpl:326 : Default schema
2020-01-13 01:52:16,725 DEBUG [main] util.ZookeeperDistributedLock:142 : 6521#node1 trying to lock /kylin/kylin_metadata/create_htable/kylin_metadata/lock
2020-01-13 01:52:16,730 INFO [main-EventThread] state.ConnectionStateManager:237 : State change: CONNECTED
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata#hbase
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:83)
... 3 more
Caused by: java.lang.NoSuchMethodError: org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModePathAndBytesable;
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:145)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:166)
at org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:305)
at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:110)
at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:91)
... 8 more
2020-01-13 01:52:16,772 INFO [Curator-Framework-0] imps.CuratorFrameworkImpl:924 : backgroundOperationsLoop exiting
2020-01-13 01:52:16,773 INFO [Thread-1] zookeeper.ReadOnlyZKClient:344 : Close zookeeper connection 0x7b94089b to node1:2181
2020-01-13 01:52:16,775 INFO [ReadOnlyZKClient-node1:2181#0x7b94089b] zookeeper.ZooKeeper:692 : Session: 0x16f798087cc03ae closed
2020-01-13 01:52:16,775 INFO [ReadOnlyZKClient-node1:2181#0x7b94089b-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down
2020-01-13 01:52:16,777 INFO [Thread-4] zookeeper.ZooKeeper:692 : Session: 0x16f798087cc03af closed
2020-01-13 01:52:16,777 INFO [main-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down
ERROR: Unknown error. Please check full log.

jBPM : Not able to start business application in dev mode

I am running jBPM (v7.18) in docker on localhost using the following docker-compose configuration:
version: '2'
services:
postgres:
image: postgres:10.4
volumes:
- ./volumes/psql/:/var/lib/postgresql/data/
environment:
- POSTGRES_USER=jbpm
- POSTGRES_PASSWORD=jbpm
ports:
- 5432:5432
jbpm:
image: jboss/jbpm-server-full
environment:
JBPM_DB_DRIVER: postgres
JBPM_DB_HOST: postgres
ports:
- 8080:8080
- 8001:8001
volumes:
- "/Users/guest/prac/jbpm/quickfox:/opt/jboss/quickfox"
depends_on:
- postgres
I generated the business application from https://start.jbpm.org/
I am starting the service of the business application in dev mode as follows.
./launch-dev.sh clean install
As per the documentation ,
KIE server configuration needs to be as follows:
kieserver.serverId=business-application-service
kieserver.serverName=business-application-service
kieserver.location=http://localhost:8090/rest/server
kieserver.controllers=http://localhost:8080/jbpm-console/rest/controller
(which are the default settings in application-dev.properties)
But when I start the service it is not able to connect to business-central. I get the following logs
2019-05-01 11:56:50.789 INFO 47000 --- [ main] o.k.s.s.j.u.f.r.BootstrapFormRenderer : Boostrap Form renderer templates loaded successfully.
2019-05-01 11:56:50.795 INFO 47000 --- [ main] o.k.s.s.j.u.f.r.PatternflyFormRenderer : patternfly Form renderer templates loaded successfully.
2019-05-01 11:56:50.799 INFO 47000 --- [ main] o.k.s.s.j.u.f.r.PatternflyFormRenderer : workbench Form renderer templates loaded successfully.
2019-05-01 11:56:50.801 INFO 47000 --- [ main] o.k.server.services.impl.KieServerImpl : jBPM-UI KIE Server extension has been successfully registered as server extension
2019-05-01 11:56:50.802 INFO 47000 --- [ main] o.k.server.services.impl.KieServerImpl : DMN KIE Server extension has been successfully registered as server extension
2019-05-01 11:56:50.806 INFO 47000 --- [ main] o.k.s.s.impl.policy.PolicyManager : Registered KeepLatestContainerOnlyPolicy{interval=0 ms} policy under name KeepLatestOnly
2019-05-01 11:56:50.807 INFO 47000 --- [ main] o.k.s.s.impl.policy.PolicyManager : Policy manager started successfully, activated policies are []
2019-05-01 11:56:50.817 WARN 47000 --- [ main] o.kie.server.common.KeyStoreHelperUtil : Unable to load key store. Using password from configuration
2019-05-01 11:56:50.933 WARN 47000 --- [ main] o.k.s.s.i.c.DefaultRestControllerImpl : Exception encountered while syncing with controller at http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev error Error while sending PUT request to http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev response code 405
2019-05-01 11:56:50.933 WARN 47000 --- [ main] o.k.s.s.i.ControllerBasedStartupStrategy : Unable to connect to any controllers, delaying container installation until connection can be established
2019-05-01 11:56:50.934 WARN 47000 --- [ntrollerConnect] o.kie.server.common.KeyStoreHelperUtil : Unable to load key store. Using password from configuration
2019-05-01 11:56:50.950 WARN 47000 --- [ntrollerConnect] o.k.s.s.i.c.DefaultRestControllerImpl : Exception encountered while syncing with controller at http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev error Error while sending PUT request to http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev response code 405
2019-05-01 11:56:51.009 INFO 47000 --- [ main] o.k.s.s.a.KieServerAutoConfiguration : KieServer (id business-application-service-dev) started successfully
2019-05-01 11:56:51.339 INFO 47000 --- [ main] org.apache.cxf.endpoint.ServerImpl : Setting the server's publish address to be /
2019-05-01 11:56:51.652 INFO 47000 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8090 (http) with context path ''
2019-05-01 11:56:51.658 INFO 47000 --- [ main] com.quickfox.service.Application : Started Application in 13.158 seconds (JVM running for 13.969)
2019-05-01 11:57:00.954 WARN 47000 --- [ntrollerConnect] o.kie.server.common.KeyStoreHelperUtil : Unable to load key store. Using password from configuration
2019-05-01 11:57:00.961 WARN 47000 --- [ntrollerConnect] o.k.s.s.i.c.DefaultRestControllerImpl : Exception encountered while syncing with controller at http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev error Error while sending PUT request to http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev response code 405
2019-05-01 11:57:10.963 WARN 47000 --- [ntrollerConnect] o.kie.server.common.KeyStoreHelperUtil : Unable to load key store. Using password from configuration
2019-05-01 11:57:10.972 WARN 47000 --- [ntrollerConnect] o.k.s.s.i.c.DefaultRestControllerImpl : Exception encountered while syncing with controller at http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev error Error while sending PUT request to http://localhost:8080/jbpm-console/rest/controller/server/business-application-service-dev response code 405
But If I use the following configuration it works.
kieserver.serverId=business-application-service-dev
kieserver.serverName=business-application-service Dev
kieserver.location=http://localhost:8080/kie-server/services/rest/server
kieserver.controllers=http://localhost:8080/business-central/rest/controller
Can someone tell me what is the reason for this behavior? Please correct me if I am missing anything.
The 2nd set of URLs that you used are correct URLs. It seems like documentation needs to be corrected. "jbpm-console" end point were used in older versions.

Spring Cloud Task App - composed-task-runner doesnt shutdown

I have created a composed task on SCDF platform(kubernetes) in my local environment. But I wonder why the composed task doesnt stop after all the individual tasks stop.
spring-cloud-dataflow-server-kubernetes: 1.7.2.RELEASE
Composed task runner version: docker:springcloudtask/composedtaskrunner-task:2.1.0.M1
Composed task definition:
task create --name dbsales-pipeline-st --definition "ftp-source-task-st: ftp-source-task --host=ftp-host --username=ftp-user --port=21 --remote-directory='/trial/' --delete-remote-file=true --password=password --source-exchange-name=xml-transform-exchange-source --source-exchange-routing-key=st --name=ftp-source-task-st && xml-transform-source-st: xml-transform-task --sink-exchange-name=xml-transform-exchange-sink --source-queue-name=xml-transform-source-st --sink-exchange-routing-key=st --name=xml-transform-source-st && <ftp-sink-task-aaa: ftp-sink-task --host=ftp-host --username=ftp-user --port=21 --remote-directory='/trial-output/aaa/' --password=password --source-queue-name=xml-transform-sink-st-aaa --name=ftp-sink-task-aaa || ftp-sink-task-bbb: ftp-sink-task --host=ftp-host --username=ftp-user --port=21 --remote-directory='/trial-output/bbb/' --password=password --source-queue-name=xml-transform-sink-st-bbb --name=ftp-sink-task-bbb >"
Kubernetes Log
NAME READY STATUS RESTARTS AGE
dbsales-pipeline-st-ez28melww9 1/1 Running 0 14m
dbsales-pipeline-st-ftp-sink-task-aaa-lwenvxo7ry 0/1 Completed 0 12m
dbsales-pipeline-st-ftp-sink-task-bbb-vmwpnd250d 0/1 Completed 0 12m
dbsales-pipeline-st-ftp-source-task-st-jep9zyzpzv 0/1 Completed 0 14m
dbsales-pipeline-st-xml-transform-source-st-rodzk0voz7 0/1 Completed 0 13m
Log from the composed task execution:
2019-02-11 09:01:34.598 INFO 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : http://localhost:8888
2019-02-11 09:01:34.828 INFO 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Connect Timeout Exception on Url - http://localhost:8888. Will be trying the next url if available
2019-02-11 09:01:34.829 WARN 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/composedtaskrunner-task/default": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
2019-02-11 09:01:34.832 INFO 1 --- [ main] .t.a.c.ComposedtaskrunnerTaskApplication : No active profile set, falling back to default profiles: default
2019-02-11 09:01:36.589 INFO 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=8b0b06e3-848d-3c1a-9a81-67298b0a58c4
2019-02-11 09:01:36.848 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$c1b98ce] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-02-11 09:01:36.943 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$28359bcb] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-02-11 09:01:36.948 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration' of type [org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration$$EnhancerBySpringCGLIB$$918b445a] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-02-11 09:01:37.397 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2019-02-11 09:01:37.570 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2019-02-11 09:01:37.656 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'taskExecutor'
2019-02-11 09:01:39.641 INFO 1 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: MYSQL
2019-02-11 09:01:39.701 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2019-02-11 09:01:40.381 INFO 1 --- [ main] .t.a.c.ComposedtaskrunnerTaskApplication : Started ComposedtaskrunnerTaskApplication in 8.093 seconds (JVM running for 9.199)
2019-02-11 09:01:40.383 INFO 1 --- [ main] o.s.b.a.b.JobLauncherCommandLineRunner : Running default command line with: [--spring.datasource.username=root, --spring.datasource.url=jdbc:mysql://10.102.85.203:3306/mysql, --spring.datasource.driverClassName=org.mariadb.jdbc.Driver, --dataflow-server-uri=http://10.99.69.126:80, --spring.cloud.task.name=dbsales-pipeline-st, --graph=dbsales-pipeline-st-ftp-source-task-st && dbsales-pipeline-st-xml-transform-source-st && <dbsales-pipeline-st-ftp-sink-task-aaa || dbsales-pipeline-st-ftp-sink-task-bbb>, --spring.datasource.password=yourpassword, --increment-instance-enabled=true, --split-thread-wait-for-tasks-to-complete-on-shutdown=true, --dataflow-server-username=user, --dataflow-server-password=password, id=1, --spring.cloud.task.executionid=735]
2019-02-11 09:01:40.603 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=dbsales-pipeline-st]] launched with the following parameters: [{-spring.cloud.task.executionid=735, -spring.datasource.username=root, -spring.cloud.task.name=dbsales-pipeline-st, -graph=dbsales-pipeline-st-ftp-source-task-st && dbsales-pipeline-st-xml-transform-source-st && <dbsales-pipeline-st-ftp-sink-task-aaa || dbsales-pipeline-st-ftp-sink-task-bbb>, -dataflow-server-password=password, incrementInstanceEnabled=true, -spring.datasource.driverClassName=org.mariadb.jdbc.Driver, -increment-instance-enabled=true, -spring.datasource.url=jdbc:mysql://10.102.85.203:3306/mysql, -dataflow-server-username=user, -split-thread-wait-for-tasks-to-complete-on-shutdown=true, run.id=4, -dataflow-server-uri=http://10.99.69.126:80, id=1, -spring.datasource.password=yourpassword}]
2019-02-11 09:01:40.617 WARN 1 --- [ main] o.s.c.t.b.l.TaskBatchExecutionListener : This job was executed outside the scope of a task but still used the task listener.
2019-02-11 09:01:40.647 INFO 1 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [dbsales-pipeline-st-ftp-source-task-st_0]
2019-02-11 09:03:11.499 INFO 1 --- [ main] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution dbsales-pipeline-st-ftp-source-task-st_0
2019-02-11 09:03:11.502 INFO 1 --- [ main] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution dbsales-pipeline-st-ftp-source-task-st_0 with taskExecution 736
2019-02-11 09:03:11.526 INFO 1 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [dbsales-pipeline-st-xml-transform-source-st_0]
2019-02-11 09:03:22.288 INFO 1 --- [ main] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution dbsales-pipeline-st-xml-transform-source-st_0
2019-02-11 09:03:22.295 INFO 1 --- [ main] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution dbsales-pipeline-st-xml-transform-source-st_0 with taskExecution 737
2019-02-11 09:03:22.431 INFO 1 --- [ taskExecutor-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [dbsales-pipeline-st-ftp-sink-task-aaa_0]
2019-02-11 09:03:22.441 INFO 1 --- [ taskExecutor-2] o.s.batch.core.job.SimpleStepHandler : Executing step: [dbsales-pipeline-st-ftp-sink-task-bbb_0]
2019-02-11 09:04:53.824 INFO 1 --- [ taskExecutor-2] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution dbsales-pipeline-st-ftp-sink-task-bbb_0
2019-02-11 09:04:53.828 INFO 1 --- [ taskExecutor-2] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution dbsales-pipeline-st-ftp-sink-task-bbb_0 with taskExecution 738
2019-02-11 09:04:53.836 INFO 1 --- [ taskExecutor-1] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing for stepExecution dbsales-pipeline-st-ftp-sink-task-aaa_0
2019-02-11 09:04:53.840 INFO 1 --- [ taskExecutor-1] .t.a.c.ComposedTaskStepExecutionListener : AfterStep processing complete for stepExecution dbsales-pipeline-st-ftp-sink-task-aaa_0 with taskExecution 739
2019-02-11 09:04:53.864 INFO 1 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=dbsales-pipeline-st]] completed with the following parameters: [{-spring.cloud.task.executionid=735, -spring.datasource.username=root, -spring.cloud.task.name=dbsales-pipeline-st, -graph=dbsales-pipeline-st-ftp-source-task-st && dbsales-pipeline-st-xml-transform-source-st && <dbsales-pipeline-st-ftp-sink-task-aaa || dbsales-pipeline-st-ftp-sink-task-bbb>, -dataflow-server-password=password, incrementInstanceEnabled=true, -spring.datasource.driverClassName=org.mariadb.jdbc.Driver, -increment-instance-enabled=true, -spring.datasource.url=jdbc:mysql://10.102.85.203:3306/mysql, -dataflow-server-username=user, -split-thread-wait-for-tasks-to-complete-on-shutdown=true, run.id=4, -dataflow-server-uri=http://10.99.69.126:80, id=1, -spring.datasource.password=yourpassword}] and the following status: [COMPLETED]
I did try with a simple composed task and it does stop.
task create foo --definition "t1: timestamp && t2: timestamp"
Is there any parameters that can force the composed task to stop after the completion?
The usual signs for this to occur is when the spring.cloud.task.closecontextEnabled is not set to "true" for the composed-task-runner.
You can read more about the property in Spring Cloud Task docs.
Recently also, this was discussed at spring-cloud-task-app-starters/composed-task-runner#60 and spring-cloud/spring-cloud-dataflow#2667.

Resources