Spring Cloud Data Flow : Sample application to have multiple inputs and outputs.
Can anyone have a sample application to demonstrate spring cloud data flow with multiple inputs and outputs support
These samples apps uses multiple inputs and outputs.
The code from the acceptance tests provides an illustration of usage.
In dataflow you start by adding only one app to dsl with one of the required properties, then on deployment, you remove any extra properties and provide all the properties to connect the various streams.
The properties refer to each app by its registered name.
Related
In Spring Cloud Dataflow (Stream) what's the right generic way to implement the conditional flow based on a processor output
For example, in below case to have the execution flow through different path based on the product price spit by a transformer.
Example Process Diagram
You can use the router-sink to dynamically decide where to send the data based on the upstream event-data.
There is a simple SpEL as well as comprehensive support via a Groovy script as options, which can help with decision making and conditions. See README.
If you foresee more complex conditions and process workflows, alternatively, you could build a custom processor that can do the dynamic routing to various downstream named-channel destinations. This approach can also help with unit/IT testing the business-logic standalone.
I am new to opendaylight and I seek help regarding following:
How can I make two different applications communicate with each other?
Can I have something like a communication bus for my selected set of applications that can transfer data to each other? Or Do I need to have a single application with submodules (with different features ) to achieve the same task, i.e feature communication in this case.
The main feature that sets OpenDaylight (ODL) apart from other SDN controllers is the Model-Driven Service Abstraction Layer (MD-SAL), which provides a framework to share structured/modeled data and send notifications between ODL applications, among other things.
If you want to transfer data between ODL applications, you first need to model it using YANG, and include the YANG model in one of the applications.
To take advantage of the features offered by MD-SAL, please take a look at the official documentation. Once you understand the architecture, you should look at the source code of existing applications to see examples of how to take advantage of the power of MD-SAL.
Can we get some sample application describing the behaviour of flow mentioned in its documentation.
https://docs.wolkenkit.io/2.0.0/reference/creating-stateful-flows/overview/
A stateful flow is a state machine whose transitions are caused by events. Whenever a stateful flow transitions, it is able to run a reaction, such as sending commands or running tasks. Using their state they have knowledge of their past. This way you can use them to create complex workflows that include conditions and loops, e.g. to notify a user once an invoice has been rejected for the third time in a row.
I think a good place to start is the wolkenkit-boards sample application (see a list of all available sample applications here). It basically is an application for collaboratively organizing notes, similar to tools such as Trello.
It contains a tip of the day workflow, which actually uses a stateful flow under the hoods.
I hope this helps 😊
how to set different id in distributed system for different computer?
Twitter's snowflake arithmetic can generate id in distributed system. Every different computer has a 'workerId'.
so How to set a different workerId when I start tomcat and tell applation.
Arthur, for the context of this question, are you talking about this: https://www.callicoder.com/distributed-unique-id-sequence-number-generator/
In this article they talk about Twitter snowflake, not Snowflake, the Cloud Data Warehouse.
What you can do with Snowflake if you are considering moving your sharded database for your application to a high performing data base check out that solution that was linked above.
Hope that helps!
I'd like to test my pipeline.
My pipeline extract data from BigQuery, then store data to GCS and S3.
Although there are some information about pipeline test here,
https://cloud.google.com/dataflow/pipelines/testing-your-pipeline,
it does not include about data model of extracting data from BigQuery.
I found following example for it, but it lacks of comment, so little bit difficult to understand.
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/examples/src/test/java/com/google/cloud/dataflow/examples/cookbook/BigQueryTornadoesTest.java
Are there any good documents for test my pipeline?
In order to properly integration test your entire pipeline, please create a small amount of sample data stored in BigQuery. Also, please create a sample bucket/folder in S3 and GCS to store your output. Then run your pipeline as you normally would, using PipelineOptions to specify the test BQ table. You can use the DirectPipelineRunner if you want to run locally. It will probably be easiest to create a script which will first run the pipeline, then down the data from S3 and GCS and verify you see what you expect.
If you want to just test your pipeline's transforms on some offline data, then please follow the WordCount example.
In Google Cloud is easy to create end-to-end test using real resources like Pub/Sub topics and BigQuery tables.
By using Junit5 extension model https://junit.org/junit5/docs/current/user-guide/#extensions you can hide the complexity of creation and deleting of the required resources.
You can find a demo/seed here https://github.com/gabihodoroaga/dataflow-e2e-demo and a blog post here https://hodo.dev/posts/post-31-gcp-dataflow-e2e-tests/