I am new to devops. I want to install jenkins. So out of all options available to install jenkins provided in official documentation which one should I use. I am zeroed on docker or kubernetes. So parameters I am looking for decision are below.
portability - can be installed on any major os or cloud provider.
minimal changes to move to production.
Kubernetes is a container orchestrator that may use Docker as its container runtime. So, they are quite different things—essentially, different levels of abstraction.
You could theoretically run an application at both of these abstraction levels. Here's a comparison:
Docker
You can run an application as a Docker container on any machine that has Docker installed (i.e. any OS or cloud provider instance that supports Docker). However, you would need to implement any operations-related features that are relevant for production, such as health checks, replication, load balancing, etc. yourself.
Kubernetes
Running an application on Kubernetes requires a Kubernetes cluster. You can run a Kubernetes cluster either on-premises, in the cloud, or use a managed Kubernetes service (such as Amazon EKS, Google GKE, or Azure AKS). The big advantage of Kubernetes is that it provides all the production-relevant features mentioned above (health checks, replication, load balancing, etc.) as part of the platform. So, you don't need to implement them yourself but just use the primitives that Kubernetes provides to you.
Regarding your two requirements, Kubernetes provides both of them, while using Docker alone does not provide easy production-readiness (requirement 2). So, if you're opting for production stability, setting up a Kubernetes cluster is certainly worth the effort.
I'm using Docker I have implemented a system to deploy environments (on a single server) based on Git branches using Traefik (*.dev.domain.com) and Docker Compose templates.
I like Kubernetes and I've never switched to it since I'm limited to one single server for my infrastructure. I've only used it using local installations (Docker for Windows).
So, my question is: does it make sense to run a Kubernetes "cluster" (master and nodes) on a single server to orchestrate and route containers (in place of Traefik/Rancher/Docker Compose)?
This use is for development and staging only for the moment, so high availability is not a prerequisite.
Thanks.
If it is not a production environment, it doesn't matter how many nodes you are using. So yes, it should be just fine in this case. But make sure all the k8s features you will need in production are available in test/dev, to keep things similar and portable.
AFAIU,
I do not see a requirement for kubernetes unless we are doing below at least for single host using native docker run or docker-compose or docker engine swarm mode -
Make sure there are enough(>=2) replicas of your app in a single server and you are balancing the load across those apps docker containers.
If you want to go bit advanced, we should be able to scale up & down dynamically (docker swarm mode supports this out of the box else use jwilder nginx proxy).
Your deployment should not cause a downtime. Make sure a single container is always healthy at any instant of time while deploying.
Container should auto heal(restart automatically) in case your HTTP or TCP health check fails.
Doing all of the above will certainly put you in a better place but single host is still a single source of failure which you got to deal with at regular intervals.
Preferred : if possible try to start with docker engine swarm mode or kubernetes single master or minikube. This will automatically take care of all the above scenarios out of the box and will also allow you to further scale up anytime by adding more nodes without changing much in your YML files for docker swarm or kubernetes.
Ref -
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
https://docs.docker.com/engine/swarm/
I would use single host k8s only if I managed clusters with the same project that I would like to deploy to the said host. This enables you to reuse manifests and all the automation you've created for your clusters.
Have I had single host environments only, I would probably stick to docker-compose.
If you're looking to try it out your easiest options are probably minikube (easy to run single-node cluster locally but without some features) or using one of the free trial accounts for a managed Kubernetes service from one of the big cloud providers (fully-featured and multi-node but limited use before you have to pay).
I have a web application consisting of a few services - web, DB and a job queue/worker. I host everything on a single Google VM and my deployment process is very simple and naive:
I manually install all services like the database on the VM
a bash script scheduled by crontab polls a remote git repository for changes every N minutes
if there were changes, it would simply restart all services using supervisord (job queue, web, etc)
Now, I am starting a new web project where I enjoy using docker-compose for local development. However, I seem to suck in analysis paralysis deciding between available options for production deployment - I looked at Kubernetes, Swarm, docker-compose, container registries and etc.
I am looking for a recipe that will keep me productive with a single machine deployment. Ideally, I should be able to scale it to multiple machines when the time comes, but simplicity and staying frugal (one machine) is more important for now. I want to consider 2 options - when the VM already exists and when a new bare VM can be allocated specifically for this application.
I wonder if docker-compose is a reasonable choice for a simple web application. Do people use it in production and if so, how does the entire process look like from bare VM to rolling out an updated application? Do people use Kubernetes or Swarm for a simple single-machine deployment or is it an overkill?
I wonder if docker-compose is a reasonable choice for a simple web application.
It can be, sure, if the development time is best spent focused on the web application and less on the non-web stuff such as the job queue and database. The other asterisk is whether the development environment works ok with hot-reloads or port-forwarding and that kind of jazz. I say it's a reasonable choice because 99% of the work of creating an application suitable for use in a clustered environment is the work of containerizing the application. So if the app already works under docker-compose, then it is with high likelihood that you can take the docker image that is constructed on behalf of docker-compose and roll it out to the cluster.
Do people use it in production
I hope not; I am sure there are people who use docker-compose to run in production, just like there are people that use Windows batch files to deploy, but don't be that person.
Do people use Kubernetes or Swarm for a simple single-machine deployment or is it an overkill?
Similarly, don't be a person that deploys the entire application on a single virtual machine or be mentally prepared for one failure to wipe out everything that you value. That's part of what clustering technologies are designed to protect against: one mistake taking down the entirety of the application, web, queuing, and persistence all in one fell swoop.
Now whether deploying kubernetes for your situation is "overkill" or not depends on whether you get benefit from the other things that kubernetes brings aside from mere scaling. We get benefit from developer empowerment, log aggregation, CPU and resource limits, the ability to take down one Node without introducing any drama, secrets management, configuration management, using a small number of Nodes for a large number of hosted applications (unlike creating a single virtual machine per deployed application because the deployments have no discipline over the placement of config file or ports or whatever). I can keep going, because kubernetes is truly magical; but, as many people will point out, it is not zero human cost to successfully run a cluster.
Many companies I have worked with are shifting their entire production environment towards Kubernetes. That makes sense because all cloud providers are currently pushing Kubernetes and we can be quite positive about Kubernetes being the future of cloud-based deployment. If your application is meant to run in any private or public cloud, I would personally choose Kubernetes as operating platform for it. If you plan to add additional services, you will be easily able to connect them and scale your infrastructure with a growing number of requests to your application. However, if you already know that you do not expect to scale your application, it may be over-powered to use a Kubernetes cluster to run it although Google Cloud etc. make it fairly easy to setup such a cluster with a few clicks.
Regarding an automated development workflow for Kubernetes, you can take a look at my answer to this question: How to best utilize Kubernetes/minikube DNS for local development
Good day
We have a development environment that consists of 6 virtual machines. Currently we are using Vagrant and Ansible with VirtualBox.
As you can imagine, hosting this environment is a maintenance nightmare particularly as versions of software/OS change. Not too mention resource load for developer machines.
We have started migrating some virtual machines to docker. But this itself poses problems around orchestration, correct configurations, communication etc. This led me to Kubernetes.
Would someone be so kind as to provide some reasoning as to whether Kubernetes would or wouldn't be the right tool for the job? That is managing and orchestrating 'development' docker containers.
Thanks
This is quite complex topic and many things have to be considered if it's worth to use k8s as local dev environment. Especially I used it when I wanted to have my local developer environment very close to production one which was running on Kubernetes. This helped to avoid many configuration bugs.
In my opinion Kubernetes(k8s) will provide you all you need for a development environment.
It gives you much flexibility and does much configuration itself. Few examples:
An easy way to deploy new version into local kubernetes stack
You prepare k8s replication controller files for each of your application module (keep in mind that they need to be stateless modules)
In replication controller you specify the docker image and that's it.
Using this approach you can push new docker images to local docker_registry and then using kubectl control the lifecycle of your application.
Easy way to scale your application modules
For example:
kubectl scale rc your_application_service --replicas=3
This way k8s will check how many pods you have running for your service and if it recognises that the number is smaller then the replicas value it will create new to satisfy the replicas number.
It's endless topic and many other things come to my mind, but I would suggest you to try it out.
There is a https://github.com/kubernetes/kubernetes/blob/master/docs/devel/developer-guides/vagrant.md project for running the k8s cluster in vagrant.
Of course you have to remember that if you have many services all of them have to be pushed to local repository and run by k8s. This will require some time but if you automate local deploy with some custom scripts you won't regret.
As wsl mentioned before, it is a quite complex topic. But i'm doing this as well at the moment. So let me summaries some things for you:
With Kubernetes (k8s) you're going to orchestrate your SaaS Application. In best case, it is a Cloud-native Application. The properties/requirements for a Cloud-native Application are formulated by the Cloud Native Computing Foundation (CNCF), which basically were formed around k8s, after Google donates it to the Linux Foundation.
So the properties/requirements for a Cloud-native Application are: Container packaged, Dynamically managed and Micro-services oriented (cncf.io/about/charter). You will benefit mostly from k8s, if your applications are micro-service based and every service has a separate container.
With micro-service based applications, every service can be developed independently. The developer only needs to follow the 12Factor Method (12factor.net) for example (use env var instead of hard coded IP addresses, etc).
In the next step the developer build the container for a service and pushes it the a container registry. For a local develop environment, you may need to run a container registry inside the cluster as well, so the developer can push and test his code locally.
Then you're able to define your k8s replication-controllers, services, PetSets, etc. with Ports, Port-mapping, env vars, Container Images... and create and run it inside the cluster.
The k8s-documentation recommend Minikube for running k8s locally (kubernetes.io/docs/getting-started-guides/minikube/). With Minikube you got features like DNS, NodePorts, ConfigMaps and Secrets
Dashboards.
But I choose the multi node CoreOS Kubernetes with Vagrant Cluster for my Development Environment as Puja Abbassi mentioned in the Blog "Finding The Right Local Kubernetes Development Environment" (https://deis.com/blog/2016/local-kubernetes-development-environment/), it is closer to the my production environment (12Factor: 10 - Dev/prod parity).
With the Vagrant Environment you got features like:
Networking with flannel
Service Discovery with etcd
DNS names for a set of containers with SkyDNS
internal load balancing
If you want to know, how everything works look inside this Github repo github.com/coreos/coreos-kubernetes/tree/master/multi-node (vagrant and generic folder).
So you have to ask yourself, if you or your developers really need to run a complete "cloud environment" locally. In many cases a developer can develop a service (based on micro-services and containers) independently.
But sometimes it is necessary to have multiple or all services run on your local machine as a dev-environment.
I am trying to wrap my head around CoreOS and I perused their official docs, some random articles, and even watched this excellent presentation by their CTO.
My understanding of CoreOS is that its a stripped down, bare bones Linux distribution that requires anything running on it to be an OCF-compliant container, not just a Docker container.
My understanding of fleet is that its systemd at the cluster level
My understanding of flannel is that its a network layer that is used by both etcd and fleet to route network requests to containers living in the cluster
So first off, if my above assertions are incorrect or misled in any way, please begin by correcting me! Assuming that I'm more or less on track, I have a few concerns here:
What concrete benefit(s) does CoreOS offer Docker-contained apps that is not present with other Linux distros, such as Ubuntu or Debian? In other words, what objective benefits do I gain by going Docker/CoreOS vs. Docker/Ubuntu?
Fleet just seems like a scheduling engine, like Mesos or Kubernetes. Is it a direct competitor to these projects, or do they handle scheduling at different "layers" (different types of responsibilities)? If so, what are these distinctions?
There are a lot of moving parts here. The answer already posted is very good. I think there are going to be opinions in any answer you get. I thought I'd go through your punch list in my attempt at 100 bounty points :-)
I've been using CoreOS/Flannel/Kubernetes/Fleet everyday now for about 6 months. When you posted the url to the introduction I decided to watch it. Wow, great presentation. I think Brandon Philips is a very good teacher. I like the way he built upon each technology as he introduced it. I would recommend that tutorial to anyone.
CoreOS is a linux based operating system. It is very stripped down, nothing extra running. For me, it does these things:
Auto updates. Does this well. Dual partitions, updates non-active, swaps active, falls back (I think, I have never experienced a fallback). The have tackled the 'how to update your operating system after you deploy' issue and made it relatively painless.
systemd init system. This one took me a bit longer to like (being a /etc/init.d guy) but, after a while it grows on you. There is a pretty steep learning curve. Once you get what is going on you will like how systemd keeps the machine running specific things, dependencies, restarts (if you want), listening on sockets (like super initd) and spawning processes, d-bus (although I don't know much about this part yet). systemd lets you specify 'units' and units can have dependencies, pre and post processes, etc.
basic services. I've copied the brief description line from each of the services that are running on my CoreOS system.
systemd - It provides a system and service manager that runs as PID 1 and starts the rest of the system
docker - Docker is an open source project to pack, ship and run any application as a lightweight container
etcd - etcd is a distributed, consistent key-value store for shared configuration and service discovery
sshd - sshd (OpenSSH Daemon) is the daemon program for ssh(1). Together these programs replace rlogin and rsh, and provide secure encrypted communications between two untrusted hosts over an insecure network.
locksmithd - locksmith is a reboot manager for the CoreOS update engine which uses etcd to ensure that only a subset of a cluster of machines are rebooting at any given time. locksmithd runs as a daemon on CoreOS machines and is responsible for controlling the reboot behaviour after updates.
journald - systemd-journald is a system service that collects and stores logging data.
timesyncd - systemd-timesyncd is a system service that may be used to synchronize the local system clock with a remote Network Time Protocol server
update_engine
udevd - systemd-udevd listens to kernel uevents. For every event, systemd-udevd executes matching instructions specified in udev rules. See udev(7).
logind - systemd-logind is a system service that manages user logins.
resolved - systemd-resolved is a system service that manages network name resolution. It implements a caching DNS stub resolver and an LLMNR resolver and responder.
hostnamed - This is a tiny daemon that can be used to control the host name and related machine meta data from user programs.
networkd - systemd-networkd is a system service that manages networks. It detects and configures network devices as they appear, as well as creating virtual network devices.
CoreOS doesn't necessarily require that everything that you want to run must be a container. It will run anything that a unix box will run. yum and apt-get are conspicuously missing, but wget is included. So, you can 'install' programs, libraries, even apt-get via wget and be on your way to polluting the CoreOS base. That wouldn't be good, though. You really do want to keep it pristine. To that end, they include a 'toolbox' which lets you run a container like sandbox to do your work that goes away when you log out of it.
My favorite part of CoreOS is the cloud-config. On first boot you can provide user_data called a cloud-config. It is a yaml file which tells the base CoreOS what to do when it boots the first time. This is where you install things like fleet, flannel, kubernetes, etc. It is a real easy way to get a repeatable install of a combination of your choosing on a VM. In a typical cloud-config I will write configuration files, copy files from other machines to install on the new machine, and create unit files that control the other processes we want CoreOS' systemd to manage (like flannel, fleet, etc). And it is completely repeatable.
Here is another interesting thing about CoreOS. You can modify the dependency and configuration of existing units. For example, CoreOS starts docker. But, I want to modify the startup sequence of docker, so I can add a drop-in configuration that augments the existing system docker configuration. I use this to drop-in the dependency for flannel before docker starts, so I can configure docker to use a flannel provided network. This isn't necessarily CoreOS, but, it does make it all fit together.
I think you can use cloud-config with Ubuntu as well as CoreOS, and you can do the same things. So, I think the benefit you get from CoreOS over Ubuntu would be that you get a new release often, the operating system is auto-updated, and you don't have anything 'extra' running (it's lean, and a reduced attack vector is fallout). CoreOS is tuned for docker (it is already running) and ubuntu doesn't have it already running. Although, you can create a cloud-config file that will make ubuntu run docker... In summary, I think you have CoreOS understood.
Another thing that you can get with CoreOS is support, directly from the company, either paid or unpaid. I have had many questions answered by the people at CoreOS via this forum and CoreOS Dev/CoreOS User Google groups.
Your fleet description is also pretty good. Fleet manages a cluster. A cluster is one or more CoreOS machines. So, if you are going to use fleet you must use CoreOS, I guess this would be another of those benefits of CoreOS over Ubuntu.
Much like how a Unit File for systemd controls running a process on a host, a Unit File for fleetd controls running a process on a cluster. There is a bit of syntactic sugar, but a Unit file for fleet is about the same as a unit file for systemd. They fit very well together. Fleet's unit files are saved in etcd's database, so once ingested the unit is persistent, even if the machine(s) that host the unit service go down, the unit description exists in etcd.
Fleet also has commands for listing my machines in my cluster, listing a unit file, showing the units that are running, etc. Basically you can submit units to run on the cluster (or all machines, or on a specific kind of machine (like with ssd drives), or on the same machine as something else is running (affinity), etc, etc).
Fleet keeps it running. If the machine goes away its units are going to be run on some other machine in the cluster.
In the tutorial you reference Brandon uses Fleet to launch Kubernetes. It is very simple to do. By making the Fleet unit files place Kubernetes on all machines in the fleet cluster, as machines are added and subtracted from the fleet cluster Kubernetes automatically uses that machine and schedules the Kubernetes to run on them. I have run my Kubernetes cluster like this as well. However, I don't do that much anymore. I am sure there is a benefit that I don't see, but, I feel like it is not necessary in my environment. Since I already boot my machines with a cloud-config file, it is trivial to put the Kubernetes node services directly in there. In fact, with cloud-config, if I wanted to use Fleet to boot the Kubernetes stuff, I would have to write the Fleet unit files, start Fleet, submit the unit files I wrote to Fleet, when I could just write a unit file to start the Kubernetes node. But I digress...
Fleet is a scheduling mechanism, just like Kubernetes. However, Fleet can start any executable just like systemd via a unit file, where Kubernetes is geared towards containers. Kubernetes allows definition of:
replication controllers
services
pods
containers
(other stuff as well).
So, the assertion that Fleet is just a different 'layer' of scheduling is a good one. You might add that Fleet schedules different things. In my work I don't use the Fleet layer, I just jump directly to the Kubernetes because I am working only with containers.
Finally, the assertion about flannel is incorrect. Flannel uses etcd for its database. Flannel creates a private network for each host that it routes between them. The flannel network is handed to docker, and docker is told to use that network to assign ip addresses from. So, docker processes that use flannel can communicate with each other over ip. All of the port mapping stuff can be skipped since each container gets its own ip address. These docker processes can communicate infra and intra machine on the flannel network. I could be wrong, but I don't think there is any connection between Fleet and flannel. Also, I don't think etcd or Fleet use flannel to route their data. Etcd and Fleet route whether or not flannel is being used. Docker containers route their traffic over flannel.
-g
Yes, your understanding is pretty much correct.
Coreos is designed as a more secure operating system that autoupdates itself by default and runs the bare minimum in services to lessen any attack vector. http://www.activestate.com/blog/2013/08/alex-polvi-explains-coreos
Everything needs to either run in a container, be statically compiled (golang binaries as an example), or be a shell script. There is no python or ruby installed.
Containers/systemd units started by Fleet can be rescheduled on another node should its server fail (assuming you container is ephemeral) and should keep the requested number of instances running over the cluster, whilst obeying deployment constraints. https://coreos.com/using-coreos/clustering/
Mesos is more of a framework for a scheduler, you still need something else (chronos/marathon) to provide jobs to execute, but it is very flexible in that regard and handles utilising server resources better.
I don't have much experience with Flannel, but the new networking plugins coming in a future version of Docker may give you more options for container networking. http://blog.docker.com/2015/06/networking-receives-an-upgrade/
what objective benefits do I gain by going Docker/CoreOS vs.
Docker/Ubuntu?
Technical benefits
The features that attract me to CoreOS are:
It's a cluster, not a single-machine, OS
It's built with failure in mind
It's self-updating
CoreOS is a cluster, while Ubuntu is a single machine. With CoreOS when the machine a container is on disappears, the cluster starts the container somewhere else. When that Ubuntu server fails, its containers go down with it. CoreOS allows the machine to be disposable, which is a good thing.
With that said, keep in mind CoreOS does not handle data persistence; data stored in a container does not exist! ;) In my case I dynamically attach EBS volumes where needed.
Design benefits
To me more importantly, the technical benefits above bring along design benefits. Going into a system designing knowing, "this process will randomly disappear," is great for building resiliency. From the beginning, services are stateless and because you literally have no idea what system a dependent service is on, they must also be discoverable. CoreOS's etcd, a distributed configuration store, can be used to discover where a service is located. Finally, because processes may not be on the same machine, network-accessible services -- a must for horizontally-scalable systems -- are the only way to go.
All in all, I find CoreOS a great for building Twelve-Factor Apps and you get Chaos Monkey for free.
Fleet just seems like a scheduling engine, like Mesos or Kubernetes.
Is it a direct competitor to these projects, or do they handle
scheduling at different "layers" (different types of
responsibilities)? If so, what are these distinctions?
Yes, Fleet schedules a container and determines where in a cluster it runs. If that machine disappears, Fleet also takes responsibility for re-launching it on a working machine.
I haven't taken a deep dive into Kubernetes, but there does appear to be overlap. The way I understand it thus far is that Fleet handles running a single container (a "unit"), while Kubernetes is complementary and orchestrates multiple units comprising a system. For example, Fleet ensures Postgres stays running; Kubernetes ensures your application, e.g. comprised of Postgres, Redis, and Django, are all humming away.