Violation of Foreign Key constraint in "task_metadata_fk" when launching a Task in SCDF Dashboard - spring-cloud-dataflow

Question

I am trying to launch a task using the SCDF Dashboard.
I am using CockroachDB as the underlying persistence layer with Hibernate ORM version 5.4.22.Final which is supported by the 2.7.0 release of SCDF as well as 2.8.0-SNAPSHOT.
I am able to create the application using a file URI to a jar and I am able to create a task. When I try to launch the task with program arguments I get this exception in the console output of the SCDF server:
org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [INSERT INTO task_execution_metadata (id, task_execution_id, task_execution_manifest) VALUES (?, ?, ?)]; ERROR: insert on table "task_execution_metadata" violates foreign key constraint "task_metadata_fk"
Detail: Key (task_execution_id)=(2) is not present in table "task_execution".; nested exception is org.postgresql.util.PSQLException: ERROR: insert on table "task_execution_metadata" violates foreign key constraint "task_metadata_fk"
Detail: Key (task_execution_id)=(2) is not present in table "task_execution".
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:251)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1443)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:633)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:862)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:883)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:321)
at org.springframework.cloud.dataflow.server.repository.JdbcDataflowTaskExecutionMetadataDao.save(JdbcDataflowTaskExecutionMetadataDao.java:111)
at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService.executeTask(DefaultTaskExecutionService.java:340)
at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService$$FastClassBySpringCGLIB$$422cda43.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at org.springframework.cloud.dataflow.server.service.impl.DefaultTaskExecutionService$$EnhancerBySpringCGLIB$$dbc35f76.executeTask(<generated>)
at org.springframework.cloud.dataflow.server.controller.TaskExecutionController.launch(TaskExecutionController.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:103)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: ERROR: insert on table "task_execution_metadata" violates foreign key constraint "task_metadata_fk"
Detail: Key (task_execution_id)=(2) is not present in table "task_execution".
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:473)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:393)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:130)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$0(JdbcTemplate.java:867)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:617)
... 74 common frames omitted
Looking into the "task_execution" table I do see the tasks and their execution ids. So it looks to me like there might be some timing issue here, where the insert into the "task_execution_metadata" is being executed before the insert into the "task_execution" table.
This occurs on both 2.7.0 and 2.8.0-SNAPSHOT. Any help as to why this is happening would be appreciated.
EDIT:
After debugging a bit more I have come to conclusion that this is either a bug in the way the postgresql driver sends the query or how CockroachDB interprets the query.
In the request log on my CockroachDB node I can see the queries coming in the correct order:
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"SELECT taskdeploy0_.id AS id1_4_, taskdeploy0_.object_version AS object_v2_4_, taskdeploy0_.created_on AS created_3_4_, taskdeploy0_.platform_name AS platform4_4_, taskdeploy0_.task_definition_name AS task_def5_4_, taskdeploy0_.task_deployment_id AS task_dep6_4_ FROM task_deployment AS taskdeploy0_ WHERE taskdeploy0_.task_definition_name = $1 ORDER BY taskdeploy0_.created_on ASC LIMIT $2"› ‹{$1:"'run-batch-client'", $2:"1"}› 1.053 0 ‹""› 0
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"SELECT taskdefini0_.definition_name AS definiti1_3_0_, taskdefini0_.description AS descript2_3_0_, taskdefini0_.definition AS definiti3_3_0_ FROM task_definitions AS taskdefini0_ WHERE taskdefini0_.definition_name = $1"› ‹{$1:"'run-batch-client'"}› 0.904 1 ‹""› 0
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"SELECT appregistr0_.id AS id1_0_, appregistr0_.object_version AS object_v2_0_, appregistr0_.default_version AS default_3_0_, appregistr0_.metadata_uri AS metadata4_0_, appregistr0_.name AS name5_0_, appregistr0_.type AS type6_0_, appregistr0_.uri AS uri7_0_, appregistr0_.version AS version8_0_ FROM app_registration AS appregistr0_ WHERE ((appregistr0_.name = $1) AND (appregistr0_.type = $2)) AND (appregistr0_.default_version = true)"› ‹{$1:"'batch-client'", $2:"4"}› 0.953 1 ‹""› 0
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"SELECT nextval('TASK_SEQ')"› ‹{}› 32.725 1 ‹""› 0
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"INSERT INTO task_execution(task_execution_id, exit_code, start_time, task_name, last_updated, external_execution_id, parent_execution_id) VALUES ($1, $2, $3, $4, $5, $6, $7)"› ‹{$1:"14", $2:"NULL", $3:"NULL", $4:"'run-batch-client'", $5:"'2020-12-07 16:55:55.733'", $6:"NULL", $7:"NULL"}› 1.612 1 ‹""› 0
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"SELECT task_execution_manifest FROM task_execution_metadata AS m INNER JOIN task_execution AS e ON m.task_execution_id = e.task_execution_id WHERE e.task_name = $1 ORDER BY e.task_execution_id DESC LIMIT 1 OFFSET 0"› ‹{$1:"'run-batch-client'"}› 2.138 1 ‹""› 0
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"SELECT nextval('task_execution_metadata_seq')"› ‹{}› 3.576 1 ‹""› 0
‹exec› ‹"PostgreSQL JDBC Driver"› ‹{}› ‹"INSERT INTO task_execution_metadata(id, task_execution_id, task_execution_manifest) VALUES ($1, $2, $3)"› ‹{$1:"13", $2:"14", $3:"'{\"taskDeploymentRequest\":{\"definition\":{\"name\":\"run-batch-client\",\"properties\":{\"management.metrics.tags.service\":\"task-application\",\"spring.datasource.username\":\"SA\",\"spring.datasource.url\":\"jdbc:postgresql://localhost:26257/batchdb?ssl=false\",\"spring.datasource.driverClassName\":\"org.postgresql.Driver\", <SNIP>\"--spring.cloud.task.executionid=14\"]},\"platformName\":\"default\"}'"}› 1.007 1 ‹"insert on table \"task_execution_metadata\" violates foreign key constraint \"task_metadata_fk\""› 0
From the above log one can see the postgresql driver sends the parameterized query INSERT INTO task_execution_metadata(id, task_execution_id, task_execution_manifest) VALUES ($1, $2, $3)"› ‹{$1:"13", $2:"14", $3:"<BLAHBLAH>"}> after the task has been entered into the task table. I think CockroachDB is interpreting the parameterized task id as a string instead of a long and then throwing a foreign key constraint violation.
So, this is either a case of the driver sending invalid queries, or the database interpreting the queries incorrectly. I think it's the latter, and I will open a thread on CockroachDB's forums

Answer 1

The problem seems to be with how Database Transactions are managed within Spring Cloud Task and Spring Cloud Task Dataflow. The issue is described in detail here: https://github.com/spring-cloud/spring-cloud-dataflow/issues/4286