i am looking at the documentation for spring Cloud Data flow.
https://dataflow.spring.io/docs/recipes/rabbitmq/rabbit-source-sink/
This example that uses RabbitMQ as source and sink is using Spring Cloud Streams framework - which is fine. But it doesn't show how these 3 apps - source, sink and processor can be deployed to Spring Cloud Data Flow (SCDF), it simply just runs three jars locally and they talk to each other via RabbitMQ Queues.
I am not sure how this shows the use of SCDF in this case. There's no involvement of SCDF in this case. A proper example that shows how to deploy this jars as apps inside the SCDF needs to be provided. am i missing anything in this case?. i am hoping somebody else has tried them and can share their feedback about my concern.
The documentation here covers the SCDF part of how to manage those source, processor and sink applications.
Related
Is there a recommended way to customize the user interface of Spring Cloud Data Flow?
I am trying to find a way to change the text of the dashboard, colors and logos.
I found the GitHub repo for the UI on https://cloud.spring.io/spring-cloud-dataflow-ui/
But I cannot understand how to integrate that with my backend because the APIs and docs are not provided. Neither style guides nor backend logic.
The Spring Cloud Data Flow UI is tightly coupled to the Spring Cloud Data Flow server (as one of the REST clients of the SCDF server) and hence it is not designed to be customizable or extensible. All the documentation we have for the SCDF UI is pertinent to the SCDF use cases only.
I am trying to setup Spring Cloud Data Flow (SCDF) to run in Local mode and how few questions which may help me decide if its a suitable platform for my requirements.
Even though the recommendation is to use Cloud Foundry, Kubernetes etc as task execution environment my preference to run things on production is local mode mainly because I don't have a lot of workload and cant deal with all the additional complexity. Now in local mode will I be able to run all types of SCDF apps, namely Streams, Jobs and tasks with no limitations? Some parts of the document mentions that only Jobs can be run in local mode.
Security - I am looking to put controls in place around deployment of apps and operational access to the tool (dashboard) and do see the support for LDAP with roles as an option but the whole concept of using Cloudfoundry UAA, another product to drive the user managements seems like an overkill. Is there no way to configure the tool with an existing LDAP server? Found the following in one of the LDAP issues in Github but its not clear whether it uses UAA in its docker image. Worst case I wont mind if the dashboard can be run in a view/read only mode.
https://github.com/spring-cloud/spring-cloud-dataflow/issues/2871
If Spring Cloud Data Flow were a purely monolithic application, integrating all aspects of security directly into the app is definitely easier to to wrap one's mind around. This is how Spring Cloud Data Flow originally started out from a security perspective and thus, versions of Spring Cloud Dataflow <2.0.0 supported what we labelled traditional security.
However, even before 2.0.x Spring Cloud Data Flow:
Had to integrate with external platforms such as Cloud Foundry
Became more and more microservices oriented (e.g. by using Skipper)
As a result 2 parallel security architectures had emerged, one using traditional security and the other one driven by OAuth2/OpenID Connect.
This started to become increasingly harder to maintain and for 2.0.x we decided to exclusively focus on OAuth2/OpenID Connect. However, we still had to support a rich set of enterprise features such as Roles, LDAP integration etc. As such, we find that the open-source, production-ready CloudFoundry User Account and Authentication (UAA) Server is an excelling choice. Its LDAP support and features actually exceeds the features offered by Spring Cloud Dataflow <2.0.0.
So yes, in order to setup security for Spring Cloud Data Flow locally, you need to run the UAA. And the UAA would also provide the LDAP support. Technically, Spring Cloud Data Flow has no awareness of the LDAP setup at all.
I hope this provides some background regarding how the Spring Cloud Data Flow security architecture emerged. Please have a look at the reference documentation and the aforementioned SCDF Security with UAA + LDAP example. Don't hesitate to reach out in case of further questions!
Disclaimer: I am a committer on the project.
Starting from v2.0, we delegate to UAA for authentication and authorization. There are a variety of write-ups on this matter; a more comprehensive one to look at is the end-to-end sample on how all this could be put together locally. You do not need CF or K8s, all this can run locally also. We rely on UAA as the gateway to standardize on end-to-end SSO across all the client tools, including shell, dashboard, RESTful APIs, CTR, etc.,
Sample: SCDF Security with UAA + LDAP. For further reading, please refer to the security section in the ref. guide.
Lastly, we do not recommend Local for a production install, but I understand that resiliency and/or restartability of apps under failure condition is not a requirement for some workloads.
I need a Spring Batch Admin-like application to embed in my own SB-powered Spring Boot application.
The Spring website says it's deprecated and been moved to the Spring Attic. They recommend making use of Spring Cloud Dataflow Console.
I investigated this, and it appears that there is a lot of additional functionality I don't need -- all I want to do is inspect and retry batch job executions.
Is there a means of getting only this functionality, short of carving out the Jobs controllers out of the REST API implementation, and building my own admin screens?
Yes, it is possible; however, you'd still have to use SCDF to gain access to the REST-APIs.
Once when you have SCDF running, you'd get access to the Task/Batch-job specific REST endpoints and that you can use in your custom dashboard tooling.
I want to design an application using AWS as IAAS, Docker as PAAS and Spring Boot and Spring cloud as application technology.
For this, I googled and read a lot of blogs and watch videos but could not find any answer for that.
I developed one application using Spring Boot and Spring cloud technology, and the application architecture looks like below image.
This design looks good and working fine as per expectation.
Now the new task is, I need to use the cloud (AWS) as Infrastructure and Docker.
For that, I designed one more architecture, and it looks like below image.
The component as follows:
ELB - (Elastic Load Balancer) -> Target Group (Part of Auto Scaling) -> EC2 instance (will be created more on demand)
Now if I want to integrate my previous design then I think there is not need of Zuul server here because this load balancing is done by ELB, the second I do not need Service Discovery component as well because it will be done by Target AutoScale group.
I am a little bit confused here with Spring Cloud and AWS infrastructure.
Could someone help me to make really simple how I can integrate these components to work together?
Thanks
Why Spring Cloud with AWS ?
Let's take example when you need Spring Cloud even if your architecture is deployed on AWS infra :
Imagine your Product service need to communicate with your Order Service, in this case you will see Spring Cloud utility.
You don't see the necessity of Spring Cloud because you don't have an internal communication (between your services) and this is the role of Registry service.
Why Gateway service (Zuul in your architecture) ?
Again, your current architecture don't use (need) the powerful of Gateway pattern.
Let's assume your system need to aggregate multiple results from different services to response to client request. You can do this in Gateway (Zuul in your case).
Another advantage to use Gateway service is you can use it as a unified front door to your system, which allows a browser, mobile app or other user interface to consume services from multiple hosts without managing cross-origin resource sharing (CORS) and authentication for each one.
Important :
It's fine to not use Spring Cloud, is not a rule or THE right way to implement microservices architecture. If you don't need it don't use it.
I use Spring Cloud approach for building few microservices which supposed to interact with each other. For messaging between microservices I intended to use RabbitMQ and Spring AMQP, but after I looked at Spring Cloud Stream I feel lost. In my mind Spring Cloud Stream is next level of abstraction (probably too strong, but you should get overall impression) with many very useful features. So I wonder why would someone use Spring AMQP for new development? Could you please provide any Spring AMQP benefits over Spring Cloud Stream for pretty basic case when one microservice sends message to another microservice and receives reply?
Thanks.
Spring Cloud Stream provides an opinionated configuration model that connects to the external system (Binder, Consumer Group etc.,). This is mainly intended for Streaming applications where the applications are connected via pipeline. The applications that don't fit this opinionated model can be configured directly from Spring Integration (+ Spring AMQP).
For instance, Spring Cloud Stream doesn't provide direct support for request/reply scenarios for example. You can read this SO question and the github issue here