I want to create a tool that couples together a lot (~10, perhaps more) of other CLI tools to automate some stuff. This tool needs to be able to just be dropped-in on any VPS and work, hence the Docker containers. Work in this case means running central program (made by me) that orchestrates all the other tools and aggregates their results in a single database to browse/export later. The tools' containers need to have network access.
In my limited knowledge of Docker I've concluded that multi-stage build to fit all the tools in a single container is a bad design here, and very cumbersome. I've thought of networking the tools' containers to the central one and doing some sort of TCP piping, but that seems less than ideal too. What should the approach here be like? Are there some ready-made solutions to this issue?
Thanks
How about docker-compose?
You can use this tool to deploy all your dockerized tools inside docker network and then communicate with them via your orchestrator. Additionally you can pack this composed dockers into another docker and create docker-in-docker environment and expose only your orchestrator as a gate to your all-in-one tool.
Cheers,
I am exploring use of containers in a new application and have looked at a fair amount of content and created a sandbox environment to explore docker and containers. My struggle is more an understanding what components needs to be containerized individually vs bundling multiple components into my own container. And what points to consider when architecting this?
Example:
I am building a python back end service to be executed via webservice call.
The service would interact with both Mongo DB, and RabbitMQ.
My questions are:
Should I run individual OS container (EG Ubuntu), Python Container, MongoDB Container, Rabbit MQ container etc? Combined they all form part of my application and by decoupling everything I have the ability to scale individually?
How would I be able to bundle/link these for deployment without losing the benefits of decoupling/decomposing into individual containers
Is an OS and python container actually required as this will all be running on an OS with python anyways?
Would love to see how people have approached this problem?
Docker's philosophy: using microservices in containers. The term "Microservice Architecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services.
Some advantages of microservices architecture are:
Easier upgrade management
Eliminates long-term commitment to a single technology stack
Improved fault isolation
Makes it easier for a new developer to understand the functionality of a service
Improved Security
...
Should I run individual OS container (EG Ubuntu), Python Container,
MongoDB Container, Rabbit MQ container etc? Combined they all form
part of my application and by decoupling everything I have the ability
to scale individually?
You dont need an individual OS ontainer. Each container will use Docker host's kernel, and will contain only binaries required, python binaries for example.
So you will have, a python container for you python service, MongoDB container and RabbitMQ container.
How would I be able to bundle/link these for deployment without losing
the benefits of decoupling/decomposing into individual containers?
For deployments, You will use dockerfiles + docker-compose file. Dockerfiles include instructions to create a docker image. If you are just using official library images, you don't need dockerfiles.
docker-compose will help you orchestrate the container builds (from docker files), start ups, Creating required networks, Mounting required volumes and etc.
I read on Rancher Official Page
Rancher is an open source software platform that enables organizations
to run containers in production. With Rancher, organizations no longer
have to build a container services platform from scratch using a
distinct set of open source technologies. Rancher supplies the entire
software stack needed to manage containers in production.
Base on this description, I think Rancher is a container orchestration like docker-compose. But as I read on same page:
Many users choose to run containerized applications using a container
orchestration and scheduling framework. Rancher includes a
distribution of all popular container orchestration and scheduling
frameworks today, including Docker Swarm, Kubernetes, and Mesos.
This paragraph makes me think Rancher is not a container orchestration but something that controls those thing. Please tell me what is the difference between Rancher and other container orchestration.
[Rancher Labs employee]
Basically we are unopinionated about what orchestration system you want to use. Rancher contains our own container orchestration system called Cattle, which has a full UI, API, and supports YAML syntax that matches docker-compose plus additional things not available in compose.
Obviously we like our own but are aware that other choices exist in the ecosystem and many people want to use them. And managing their installations is not a trivial task... so Rancher contains (Cattle) templates for Kubernetes, Mesos, and Swarm.
When you create an environment you choose an orchestration system, and if you pick e.g. K8s we use Cattle to orchestrate the installation and configuration of K8s, plus integrating other Rancher services like access controls, load balancing, etc. You can then use the standard tools like kubectl, their API, or we have a fairly complete custom UI for K8s built-in.
I apologize if my question is too basic, but I just started to learn docker and there are some concepts that are not clear to me.
I think of docker as a fully functional virtual machine, and therefore I thought of transforming a server with a set of services, into a container. Then I started reading about "dockerizing" applications (for instance postgresql), and I learned that a Docker file can only have one default instruction, when executing a container. I read that it is possible to coordinate multiple executing instructions using a supervisor, but I started wondering if that is really the best approach, or if it would be better to lean towards a microservices architecture?
To state better my point, I would like to describe my use case. I want to create an environment that provides tomcat (with some services deployed through servlets) and a postgreSQL database. Ideally, I would like the services (and the database), to run in the same host (on different ports).
Is it a best practice to create one container for Tomcat and one for the Database, or is it better to ship them in the same container?
And if I create two different containers, which framework should I use to orchestrate them? Is it Docker compose appropriated for this task?
1)
I think of docker as a fully functional virtual machine, and therefore I thought of transforming a server with a set of services, into a container.
NO NO and NO! It is not virtual machine
Run only one process per container
In almost all cases, you should only run a single process in a single container. Decoupling applications into multiple containers makes it much easier to scale horizontally and reuse containers. If that service depends on another service, make use of container linking.
Read Best practices for writing Dockerfiles https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/
2)
And if I create two different containers, which framework should I use to orchestrate them? Is it Docker compose appropriated for this task?
You need to start with docker-compose. When you will be more familiar with it you make you own well-founded decision.
I am trying to wrap my head around CoreOS and I perused their official docs, some random articles, and even watched this excellent presentation by their CTO.
My understanding of CoreOS is that its a stripped down, bare bones Linux distribution that requires anything running on it to be an OCF-compliant container, not just a Docker container.
My understanding of fleet is that its systemd at the cluster level
My understanding of flannel is that its a network layer that is used by both etcd and fleet to route network requests to containers living in the cluster
So first off, if my above assertions are incorrect or misled in any way, please begin by correcting me! Assuming that I'm more or less on track, I have a few concerns here:
What concrete benefit(s) does CoreOS offer Docker-contained apps that is not present with other Linux distros, such as Ubuntu or Debian? In other words, what objective benefits do I gain by going Docker/CoreOS vs. Docker/Ubuntu?
Fleet just seems like a scheduling engine, like Mesos or Kubernetes. Is it a direct competitor to these projects, or do they handle scheduling at different "layers" (different types of responsibilities)? If so, what are these distinctions?
There are a lot of moving parts here. The answer already posted is very good. I think there are going to be opinions in any answer you get. I thought I'd go through your punch list in my attempt at 100 bounty points :-)
I've been using CoreOS/Flannel/Kubernetes/Fleet everyday now for about 6 months. When you posted the url to the introduction I decided to watch it. Wow, great presentation. I think Brandon Philips is a very good teacher. I like the way he built upon each technology as he introduced it. I would recommend that tutorial to anyone.
CoreOS is a linux based operating system. It is very stripped down, nothing extra running. For me, it does these things:
Auto updates. Does this well. Dual partitions, updates non-active, swaps active, falls back (I think, I have never experienced a fallback). The have tackled the 'how to update your operating system after you deploy' issue and made it relatively painless.
systemd init system. This one took me a bit longer to like (being a /etc/init.d guy) but, after a while it grows on you. There is a pretty steep learning curve. Once you get what is going on you will like how systemd keeps the machine running specific things, dependencies, restarts (if you want), listening on sockets (like super initd) and spawning processes, d-bus (although I don't know much about this part yet). systemd lets you specify 'units' and units can have dependencies, pre and post processes, etc.
basic services. I've copied the brief description line from each of the services that are running on my CoreOS system.
systemd - It provides a system and service manager that runs as PID 1 and starts the rest of the system
docker - Docker is an open source project to pack, ship and run any application as a lightweight container
etcd - etcd is a distributed, consistent key-value store for shared configuration and service discovery
sshd - sshd (OpenSSH Daemon) is the daemon program for ssh(1). Together these programs replace rlogin and rsh, and provide secure encrypted communications between two untrusted hosts over an insecure network.
locksmithd - locksmith is a reboot manager for the CoreOS update engine which uses etcd to ensure that only a subset of a cluster of machines are rebooting at any given time. locksmithd runs as a daemon on CoreOS machines and is responsible for controlling the reboot behaviour after updates.
journald - systemd-journald is a system service that collects and stores logging data.
timesyncd - systemd-timesyncd is a system service that may be used to synchronize the local system clock with a remote Network Time Protocol server
update_engine
udevd - systemd-udevd listens to kernel uevents. For every event, systemd-udevd executes matching instructions specified in udev rules. See udev(7).
logind - systemd-logind is a system service that manages user logins.
resolved - systemd-resolved is a system service that manages network name resolution. It implements a caching DNS stub resolver and an LLMNR resolver and responder.
hostnamed - This is a tiny daemon that can be used to control the host name and related machine meta data from user programs.
networkd - systemd-networkd is a system service that manages networks. It detects and configures network devices as they appear, as well as creating virtual network devices.
CoreOS doesn't necessarily require that everything that you want to run must be a container. It will run anything that a unix box will run. yum and apt-get are conspicuously missing, but wget is included. So, you can 'install' programs, libraries, even apt-get via wget and be on your way to polluting the CoreOS base. That wouldn't be good, though. You really do want to keep it pristine. To that end, they include a 'toolbox' which lets you run a container like sandbox to do your work that goes away when you log out of it.
My favorite part of CoreOS is the cloud-config. On first boot you can provide user_data called a cloud-config. It is a yaml file which tells the base CoreOS what to do when it boots the first time. This is where you install things like fleet, flannel, kubernetes, etc. It is a real easy way to get a repeatable install of a combination of your choosing on a VM. In a typical cloud-config I will write configuration files, copy files from other machines to install on the new machine, and create unit files that control the other processes we want CoreOS' systemd to manage (like flannel, fleet, etc). And it is completely repeatable.
Here is another interesting thing about CoreOS. You can modify the dependency and configuration of existing units. For example, CoreOS starts docker. But, I want to modify the startup sequence of docker, so I can add a drop-in configuration that augments the existing system docker configuration. I use this to drop-in the dependency for flannel before docker starts, so I can configure docker to use a flannel provided network. This isn't necessarily CoreOS, but, it does make it all fit together.
I think you can use cloud-config with Ubuntu as well as CoreOS, and you can do the same things. So, I think the benefit you get from CoreOS over Ubuntu would be that you get a new release often, the operating system is auto-updated, and you don't have anything 'extra' running (it's lean, and a reduced attack vector is fallout). CoreOS is tuned for docker (it is already running) and ubuntu doesn't have it already running. Although, you can create a cloud-config file that will make ubuntu run docker... In summary, I think you have CoreOS understood.
Another thing that you can get with CoreOS is support, directly from the company, either paid or unpaid. I have had many questions answered by the people at CoreOS via this forum and CoreOS Dev/CoreOS User Google groups.
Your fleet description is also pretty good. Fleet manages a cluster. A cluster is one or more CoreOS machines. So, if you are going to use fleet you must use CoreOS, I guess this would be another of those benefits of CoreOS over Ubuntu.
Much like how a Unit File for systemd controls running a process on a host, a Unit File for fleetd controls running a process on a cluster. There is a bit of syntactic sugar, but a Unit file for fleet is about the same as a unit file for systemd. They fit very well together. Fleet's unit files are saved in etcd's database, so once ingested the unit is persistent, even if the machine(s) that host the unit service go down, the unit description exists in etcd.
Fleet also has commands for listing my machines in my cluster, listing a unit file, showing the units that are running, etc. Basically you can submit units to run on the cluster (or all machines, or on a specific kind of machine (like with ssd drives), or on the same machine as something else is running (affinity), etc, etc).
Fleet keeps it running. If the machine goes away its units are going to be run on some other machine in the cluster.
In the tutorial you reference Brandon uses Fleet to launch Kubernetes. It is very simple to do. By making the Fleet unit files place Kubernetes on all machines in the fleet cluster, as machines are added and subtracted from the fleet cluster Kubernetes automatically uses that machine and schedules the Kubernetes to run on them. I have run my Kubernetes cluster like this as well. However, I don't do that much anymore. I am sure there is a benefit that I don't see, but, I feel like it is not necessary in my environment. Since I already boot my machines with a cloud-config file, it is trivial to put the Kubernetes node services directly in there. In fact, with cloud-config, if I wanted to use Fleet to boot the Kubernetes stuff, I would have to write the Fleet unit files, start Fleet, submit the unit files I wrote to Fleet, when I could just write a unit file to start the Kubernetes node. But I digress...
Fleet is a scheduling mechanism, just like Kubernetes. However, Fleet can start any executable just like systemd via a unit file, where Kubernetes is geared towards containers. Kubernetes allows definition of:
replication controllers
services
pods
containers
(other stuff as well).
So, the assertion that Fleet is just a different 'layer' of scheduling is a good one. You might add that Fleet schedules different things. In my work I don't use the Fleet layer, I just jump directly to the Kubernetes because I am working only with containers.
Finally, the assertion about flannel is incorrect. Flannel uses etcd for its database. Flannel creates a private network for each host that it routes between them. The flannel network is handed to docker, and docker is told to use that network to assign ip addresses from. So, docker processes that use flannel can communicate with each other over ip. All of the port mapping stuff can be skipped since each container gets its own ip address. These docker processes can communicate infra and intra machine on the flannel network. I could be wrong, but I don't think there is any connection between Fleet and flannel. Also, I don't think etcd or Fleet use flannel to route their data. Etcd and Fleet route whether or not flannel is being used. Docker containers route their traffic over flannel.
-g
Yes, your understanding is pretty much correct.
Coreos is designed as a more secure operating system that autoupdates itself by default and runs the bare minimum in services to lessen any attack vector. http://www.activestate.com/blog/2013/08/alex-polvi-explains-coreos
Everything needs to either run in a container, be statically compiled (golang binaries as an example), or be a shell script. There is no python or ruby installed.
Containers/systemd units started by Fleet can be rescheduled on another node should its server fail (assuming you container is ephemeral) and should keep the requested number of instances running over the cluster, whilst obeying deployment constraints. https://coreos.com/using-coreos/clustering/
Mesos is more of a framework for a scheduler, you still need something else (chronos/marathon) to provide jobs to execute, but it is very flexible in that regard and handles utilising server resources better.
I don't have much experience with Flannel, but the new networking plugins coming in a future version of Docker may give you more options for container networking. http://blog.docker.com/2015/06/networking-receives-an-upgrade/
what objective benefits do I gain by going Docker/CoreOS vs.
Docker/Ubuntu?
Technical benefits
The features that attract me to CoreOS are:
It's a cluster, not a single-machine, OS
It's built with failure in mind
It's self-updating
CoreOS is a cluster, while Ubuntu is a single machine. With CoreOS when the machine a container is on disappears, the cluster starts the container somewhere else. When that Ubuntu server fails, its containers go down with it. CoreOS allows the machine to be disposable, which is a good thing.
With that said, keep in mind CoreOS does not handle data persistence; data stored in a container does not exist! ;) In my case I dynamically attach EBS volumes where needed.
Design benefits
To me more importantly, the technical benefits above bring along design benefits. Going into a system designing knowing, "this process will randomly disappear," is great for building resiliency. From the beginning, services are stateless and because you literally have no idea what system a dependent service is on, they must also be discoverable. CoreOS's etcd, a distributed configuration store, can be used to discover where a service is located. Finally, because processes may not be on the same machine, network-accessible services -- a must for horizontally-scalable systems -- are the only way to go.
All in all, I find CoreOS a great for building Twelve-Factor Apps and you get Chaos Monkey for free.
Fleet just seems like a scheduling engine, like Mesos or Kubernetes.
Is it a direct competitor to these projects, or do they handle
scheduling at different "layers" (different types of
responsibilities)? If so, what are these distinctions?
Yes, Fleet schedules a container and determines where in a cluster it runs. If that machine disappears, Fleet also takes responsibility for re-launching it on a working machine.
I haven't taken a deep dive into Kubernetes, but there does appear to be overlap. The way I understand it thus far is that Fleet handles running a single container (a "unit"), while Kubernetes is complementary and orchestrates multiple units comprising a system. For example, Fleet ensures Postgres stays running; Kubernetes ensures your application, e.g. comprised of Postgres, Redis, and Django, are all humming away.