I am newbie in SCDF. I have few micro services running behind Spring Cloud platform. Each services got multiple nodes. Can we use those existing services into SCDF platform as either SOURCE, PROCESSOR or SINK? If so, how would I get them into dashboard as they are already deployed as services!
SCDF doesn't probe on a given K8s cluster/namespace to automatically build the streaming data pipelines.
Today, it is imperative that the streaming/task "definitions" are created and deployed in SCDF first, and only then it is possible to monitor, scale, and manage the applications.
In case it wasn't apparent already, SCDF can only orchestrate the deployment and the management for event-streaming and batch/task Spring Boot applications. Not all kinds of application workloads are possible.
Related
spring cloud or kubernates for service govern？
i want to use spring cloud to build microservice,but if the many microservices is online,it will have many online problem,such as Gray release ,monitor,version rollback and so on?what technolygy should use to manage microservices?
Spring Cloud and Kubernetes are the best environments for developing and running microservices, but they are both very different in nature and address different concerns.Spring cloud covers logging, monitoring, service discovery but not scaling and high availability which are very important for Microservice architecture.
Spring cloud has a rich set of java libraries for runtime concerns like client-side service discovery, load balancing, configuration update and metrics. Whereas Kubernetes is not only for Java platform and it doesn't need any specific libraries for service discovery, load balancing, metrics and scheduled jobs. For some areas like logging both use third-party tools like ELK.
Spring Cloud
Rich set of libraries help the developers to integrates different services easily.
Limited to only java
Scaling and high availability is not achieved unless taken care by an orchestrator.
Kubernetes
Is an open-source container management platform which helps to create different environments like dev, test, demo..
Allows to provision resource constraints, scaling and high availability.
It all depends on the use case.Hope this helps.
References :
https://developers.redhat.com/blog/2016/12/09/spring-cloud-for-microservices-compared-to-kubernetes/
http://blog.christianposta.com/microservices/netflix-oss-or-kubernetes-how-about-both/
I am planning to start my project but a bit confuse between choosing Amazon ECS and Kubernetes perhaps I am really a beginner with Micro-services architecture.
I would really appreciate if someone can show some path for deploying my docker container on a fast easier to handle platform.
Thanks
Here a list of differences from the top of my head:
AWS ECS / Kubernetes:
Proprietary AWS implementation / Open source solution
Runs on AWS / Supported by most cloud providers and on premise
Task Definitions / PODs have different features
Runs on your EC2 machines or allows for serverless with Fargate (in beta) / Runs on any cluster of (physical/virtual/cloud) machines running the kubernetes controller.
Support for AWS VPCs / Support for multiple networking models
I would also argue that kubernetes has a slightly steeper learning curve but ultimately provides more freedom and is probably a safer bet for the future given the wide adoption.
Features supported in both systems:
Horizontal application scalability
Cluster Scalability
Load Balancing
Rolling upgrades
Logging (with additional logging systems)
Container Health Checks
APIs
Amazon has bowed to customer pressure and currently has a managed kubernetes support in beta (EKS).
*edit: EKS is released now - but with an upcharge for the cluster controller nodes, as compared to google GKE for example.
Here is one article about the topic.
What's the best practice to set-up SCDF server for Failover? I am talking about SCDF server itself. Not the stream and Tasks that you deploy in SCDF. I am planning to use Kubernetes as the runtime.
Depending on the runtime platform of your choice, you can scale multiple SCDF server instances and with a load-balancer in front, you'd be able to route the traffic among server instances. The idea is that you'd have a backup (server) instance to serve the traffic under failure scenarios.
That said, in Kubernetes, the replication controller keeps track of the pod that's running the SCDF server and upon failure, it automatically creates a new pod to re-establish the server operation, anyway.
Similar capability is also available for PCF, Mesos, and Yarn implementations of SCDF.
We're looking at using Spring Cloud Task / Spring Cloud Data Flow for our batch processing needs as we're modernising from a legacy system. We don't want or need the whole microservices offering ... we want to be able to deploy jobs/tasks, kick off batch processes, have them log to a log file, and share a database connection pool and message queue. We don't need the whole PaaS that's provided by Spring Cloud Foundry, and we don't want to pay for that, but we do want the Data Flow / Task framework to be commercially supported. Is such an option available?
Spring Cloud Data Flow (SCDF) builds upon spring-cloud-deployer abstraction to deploy stream/task workloads to a variety of runtimes including Cloud Foundry, Kubernetes, Mesos and Yarn - see this visual.
You'd need a runtime for SCDF to orchestrate these workloads in production setting. If there's no scope for cloud infrastructure, the YARN based deployment could be a viable option for standalone bare-metal installation. Please review the reference guide and Apache Ambari provisioning tools for more details. There's a separate commercial support option available for this type of installation.
Recently I came across Mantl ( microservices infrastructure management project by Cisco). Its an opensource one and they have pushed it on github. I didn't understood their basic working. Does anyone have any idea about that ?
From my understanding, Mantl is a collection of tools/applications that ties together to create a cohesive docker-based application platform. Mantl is ideally deployed on virtualized/cloud environments (AWS, OpenStack, GCE), but I have just recently able to deploy it on bare-metal.
The main component in Mantl is Mesos, which manages dockers, handles scheduling and task isolation. Marathon is a mesos framework that manages long running tasks, such as web services, this is where most application reside. The combination of mesos-marathon handles application high-availability, resiliency and load-balancing. Tying everything together is consul, which handles service discovery. I use consul to do lookups for each application to communication to each other. Mantl also includes the ELK stack for logging, but I haven't had any success in monitoring any of my applications, yet. There is also Chronos, where scheduled tasks are handles ala cron. Traefik acts as a reverse-proxy, where application/service endpoints are mapped to URLs for external services to communicate.
Basically, your microservices should be self-contained in docker images, initiate communications via consul lookup and logs into standard io. Then you deploy your app, using the Marathon API, and monitor it in Marathon UI. When deploying your dockerized-app, marathon will register you docker image names in consul, along with its' exposed port. Scheduled tasks should be deployed in Chronos, where you will be able to monitor running tasks and pending scheduled tasks.