We are currently running "composer-2.0.7-airflow-2.2.3" on GKE v1.21.x.
Looking to upgrade to GKE v1.22.x there are warnings of deprecated API's no longer available in v1.22 being used.
Can you advise if Cloud Composer will fix / or has fixed the use of deprecated API's in any later versions ?
This message seems to be occurring due to the deprecated APIs in Kubernetes where Beta APIs have since graduated from Beta (v1beta1) to GA (v1) as given in this documentation.
According to this official documentation, it should not affect Autopilot Clusters in GKE as whenever an autopilot Cluster is created,auto-upgrade is enabled on the cluster and the node pools by default which results in less distruption in the services.Since Autopilot clusters are enabled with auto-upgrade, they always run in the same version of GKE as the cluster itself.
Related
I've read the Cloud Composer overview (https://cloud.google.com/composer/) and documentation (https://cloud.google.com/composer/docs/).
It doesn't seem to mention failover.
I'm guessing it does, since it runs on Kubernetes cluster. Does it?
By failover I mean if the airflow webserver or scheduler stops for some reason, does it get started automatically again?
Yes, since Cloud Composer is built on Google Kubernetes Engine, it benefits from all the fault tolerance of any other service running on Kubernetes Engine. Pod and machine failures are automatically healed.
I am planning to start my project but a bit confuse between choosing Amazon ECS and Kubernetes perhaps I am really a beginner with Micro-services architecture.
I would really appreciate if someone can show some path for deploying my docker container on a fast easier to handle platform.
Thanks
Here a list of differences from the top of my head:
AWS ECS / Kubernetes:
Proprietary AWS implementation / Open source solution
Runs on AWS / Supported by most cloud providers and on premise
Task Definitions / PODs have different features
Runs on your EC2 machines or allows for serverless with Fargate (in beta) / Runs on any cluster of (physical/virtual/cloud) machines running the kubernetes controller.
Support for AWS VPCs / Support for multiple networking models
I would also argue that kubernetes has a slightly steeper learning curve but ultimately provides more freedom and is probably a safer bet for the future given the wide adoption.
Features supported in both systems:
Horizontal application scalability
Cluster Scalability
Load Balancing
Rolling upgrades
Logging (with additional logging systems)
Container Health Checks
APIs
Amazon has bowed to customer pressure and currently has a managed kubernetes support in beta (EKS).
*edit: EKS is released now - but with an upcharge for the cluster controller nodes, as compared to google GKE for example.
Here is one article about the topic.
istio An open platform to connect, manage, and secure micro-services looks very interesting, but supports only Kubernetes. I couldn't find a roadmap or mention of future support for other container management platforms, specifically Docker Swarm
The project's github site does state the following explicitly:
Istio currently only supports the Kubernetes platform, although we
plan support for additional platforms such as Cloud Foundry, and Mesos
in the near future.
I don't know about the plans for Docker Swarm however I believe it probably would figure in the plans.
The roadmap at https://istio.io/docs/reference/release-roadmap.html shows that VM support is planned for 0.2
You can see that work is happening in the Cloud Foundry world when you see issues such as this.
The docker team indicated recently they are very interested in looking at istio and docker swarm integration so stay tuned this may happen in the next few quarters before you know it :)
Does any one of know which version of calico works with Docker in Swarm Mode?
Per https://docs.docker.com/engine/extend/plugins_network/ :
Docker 1.12 adds support for cluster management and orchestration called swarm mode. Docker Engine running in swarm mode currently only supports the built-in overlay driver for networking. Therefore existing networking plugins will not work in swarm mode.
I have heard this may be fixed for the upcoming 1.13.1 - so this could change very soon.
In order to work in swarm-mode, the calico plugin must be implemented as a managed plugin.
Calico project should also be swarm-mode compatible and support it.
Some more discussion here https://github.com/projectcalico/calicoctl/issues/1133 on expecting version Calico v2.3.0, Calico v2.2.0, but there is no official confirmation on that.
Though expectation was with Docker 1.13, no official confirmation or documentation on this.
Hope that help
I am interested in installing OpenStack to a couple of physical we have lying around, and then, somehow, deploying CloudFoundry on top of of it, as the PaaS.
I am also interested in playing around with Docker and CoreOS, and see that an integration between OpenStack and CoreOS already exists.
My question: if I have OpenStack/Nova spinning up VMs running CoreOS, and hence be Docker/container-based, will this be compatible with CloudFoundry, or is CloudFoundry somehow incompatible with Docker containers?
Cloud Foundry is installed using a specialised tool called Bosh. It has support for Openstack and I think would require deployment using Ubuntu VMs (open to correction on this point). Cloud Foundry has not integrated Docker yet, that is coming in the next version, google "Cloud Foundry" and "Diego".
maybe I'm not fully understanding here, but I was under the impression
that containers can't just stand on their own. They would require
living inside a VM. So my thinking/hope was that I could use
CloudFoundry to spin up VM instances, and inside those instances,
deploy containers. Thoughts?
Containers are completely standalone, they are a form of lightweight virtualization. Cloud Foundry is a platform for deploying your application. It runs on virtual machines (or physical servers) and instances of your application are compiled and run on the CF hosts within containers. Currently the container tech used by CF is something called Warden. Diego is a new CF component coming in 2015 that will offer Docker support.
then what is the difference between CF Diego and Kubernetes, which
also seems to be about deploying/distributing your container across
pools of nodes? Do they serve different, similar or identical
purposes? In other words, would there be a use case for having both CF
Diego and Kubernetes managing your app deployments, if so, what?
Kubernetes is a Google sponsored project for orchestrating containers across multiple hosts. Cloud Foundry goes further because it also contains features for building and versioning applications that are deployed. It's worth noting that Redhat have a competing PAAS solution called Openshift. The next version (already available in github) has integrated Kubernetes and added in all the missing application build support, making it comparable to what Cloud Foundry offers. Both CF Diego and Openshift V3 are due for delivery sometime in 2015.
Update
I see from your other questions, you're familiar with Camel. You'd be interested in the fabric8 framework which has recently integrated Openshift V3. (Fabric is the upstream project for the JBoss Fuse product)