Currently I can see that Spring Cloud Data Flow has these servers: Local, YARN, Cloud Foundry, Mesos, and Kubernetes; is there any plan for Swarm support?
Caleb: Spring Cloud Data Flow's deployer implementation is based on Spring Cloud Deployer's service provider interface (SPI), so there's currently SPI implementation for Local, CF, Yarn, Kubernetes, and Mesos. These implementations are managed in separate repos, too.
This decoupling provides flexibility and it is easy to extend to add new deployers. Though we haven't attempted to add Docker Swarm deployer yet, we would love to review contributions from the community.
Related
I am new to devops. I want to install jenkins. So out of all options available to install jenkins provided in official documentation which one should I use. I am zeroed on docker or kubernetes. So parameters I am looking for decision are below.
portability - can be installed on any major os or cloud provider.
minimal changes to move to production.
Kubernetes is a container orchestrator that may use Docker as its container runtime. So, they are quite different things—essentially, different levels of abstraction.
You could theoretically run an application at both of these abstraction levels. Here's a comparison:
Docker
You can run an application as a Docker container on any machine that has Docker installed (i.e. any OS or cloud provider instance that supports Docker). However, you would need to implement any operations-related features that are relevant for production, such as health checks, replication, load balancing, etc. yourself.
Kubernetes
Running an application on Kubernetes requires a Kubernetes cluster. You can run a Kubernetes cluster either on-premises, in the cloud, or use a managed Kubernetes service (such as Amazon EKS, Google GKE, or Azure AKS). The big advantage of Kubernetes is that it provides all the production-relevant features mentioned above (health checks, replication, load balancing, etc.) as part of the platform. So, you don't need to implement them yourself but just use the primitives that Kubernetes provides to you.
Regarding your two requirements, Kubernetes provides both of them, while using Docker alone does not provide easy production-readiness (requirement 2). So, if you're opting for production stability, setting up a Kubernetes cluster is certainly worth the effort.
I am newbie in SCDF. I have few micro services running behind Spring Cloud platform. Each services got multiple nodes. Can we use those existing services into SCDF platform as either SOURCE, PROCESSOR or SINK? If so, how would I get them into dashboard as they are already deployed as services!
SCDF doesn't probe on a given K8s cluster/namespace to automatically build the streaming data pipelines.
Today, it is imperative that the streaming/task "definitions" are created and deployed in SCDF first, and only then it is possible to monitor, scale, and manage the applications.
In case it wasn't apparent already, SCDF can only orchestrate the deployment and the management for event-streaming and batch/task Spring Boot applications. Not all kinds of application workloads are possible.
On DockerCon 2019 Docker announced the Docker Kubernetes Service (DKS).
Quoted from here:
Docker Enterprise 3.0’s Docker Kubernetes Service (DKS) integrates
Kubernetes container orchestration from the developer desktop to the
production server.
...It also provides an automated way to install and configure
Kubernetes applications across hybrid and multi-cloud deployments.
Other capabilities include security, access control, and lifecycle
management
And from here:
The Docker platform includes a secure and fully-conformant Kubernetes
environment for developers and operators of all skill levels,
providing out-of-the-box integrations for common enterprise
requirements while still enabling complete flexibility for expert
users.
After some searching and research I haven't succeed to fully understand the different solutions and features that DKS has to offer. So, my question is:
What DKS has to offer regarding topics like: Security, Networking, Access Management, etc'?
I'll start with what I discovered so far as an entry point for the discussion, hopefully that others will share there own understanding and experience and maybe provide some references and examples.
This is very basic - but I'll share what I found so far - starting with the product page as my entry point for research.
Security
Secure Kubernetes cluster with TLS authentication and encryption.
Integrated security for the application lifecycle with Docker Content Trust.
Integration with validated and certified 3rd party tools (monitoring, logging, storage,
networking, etc') .
Access control
Restricting visibility for different user groups and operate multi-tenant environments - I found only this: restrict services to worker nodes.
Advanced Access Controls Docker Enterprise includes integrated RBAC that works with corporate LDAP, Active Directory, PKI certificates and/or SAML 2.0 identity provider solutions - I found only this: Configure native Kubernetes role-based access control.
Networking
Found only this which is related to installation of cni plugins.
I think DKS offers much more regarding to integrating with 3rd party networking solutions - Quoted from the product page:
Out-of-the-box Networking Docker Enterprise includes Project Calico by
Tigera as the “batteries included” Kubernetes CNI plug-in for a highly
scalable, networking and routing solution. Get access to overlay
(IPIP), no overlay, and hybrid data-plane networking models in
addition to native Kubernetes ingress controllers for load balancing.
I have a docker-compose.yml file which have environment variable and certificates. I like to deploy these in cloud foundry dev version.
I want to deploy microgateway on cloud foundry link for microgateway is below-
https://github.com/CAAPIM/Microgateway
In cloud native world, you instantiate the services to your foundation beforehand. You can use prebuilt services (auto-scaler) available from the market place.
If the service you want is not available, you can install a tile (e.g redis, mysql, rabbitmq), which will add services to the market place. Lot of vendors provide tiles that can be installed on PCF (check on newtork.pivotal.io for the full list).
If you have services that are outside of cloud foundry (e.g. Oracle, Mongo, or MS Sql Server), and you wish to inject them into your cloud foundry foundation, you can create do that by creating User Provide Services (cups).
Once you have a service, you have to create a service instance. Think of it as provisioning a service for you. After you have provisioned i.e. created a service instance, then you can bind it to one or more apps.
A service instance is scoped to an org and a space. All apps within a org - space, can be bound to that service instance.
You deploy your app individually, by itself, to cloud foundry (jar, war, zip). You then bind any needed services to your app (e.g db, scaling, caching etc).
Use a manifest file to do all these steps in one deployment.
PCF 2.0 is introducing PKS - Pivotal Container Service. It is implementation of Kubo within PCF. It is still not GA.
Kubo, Kubernetes, and PKS allow you to deployed your containerized applications.
I have played with MiniKube and little bit of Kubo. Still getting my hands wet on PKS.
Hope this helps!
We're looking at using Spring Cloud Task / Spring Cloud Data Flow for our batch processing needs as we're modernising from a legacy system. We don't want or need the whole microservices offering ... we want to be able to deploy jobs/tasks, kick off batch processes, have them log to a log file, and share a database connection pool and message queue. We don't need the whole PaaS that's provided by Spring Cloud Foundry, and we don't want to pay for that, but we do want the Data Flow / Task framework to be commercially supported. Is such an option available?
Spring Cloud Data Flow (SCDF) builds upon spring-cloud-deployer abstraction to deploy stream/task workloads to a variety of runtimes including Cloud Foundry, Kubernetes, Mesos and Yarn - see this visual.
You'd need a runtime for SCDF to orchestrate these workloads in production setting. If there's no scope for cloud infrastructure, the YARN based deployment could be a viable option for standalone bare-metal installation. Please review the reference guide and Apache Ambari provisioning tools for more details. There's a separate commercial support option available for this type of installation.