I am wondering how do we make machines that host docker to be easily replaceable. I would like something like a Dockerfile that contains instructions on how to set-up the machine that will host docker. Is there a way to do that?
The naive solution would be to create an official "docker host" binary image to install on new machines, but I would like to have something that is reproducible and transparent like the dockerfile?
It seems like tools like Vagrant, Puppet, or Chef may be useful but they appear to be for virtual machine procurement and they seem to all require set-up of some sort of "master node" server. I am not going to be spinning up and tearing down regularly so a master server is a waste of a server, I just want something that is reproducible in the event i need to set-up or replace a new machine.
this is basically what docker-machine does for you https://docs.docker.com/machine/overview/
and other "orchestration" systems will make this automated and easier, as well
There are lots of solutions to this with no real one size fits all answer.
Chef and Puppet are the popular configuration management tools that typically use a centralized server. Ansible is another option that typically runs without a server and just connects with ssh to configure the host. All three of these works very similarly, so if your concern is simply managing the CM server, Ansible may be the best option for you.
For VM's Vagrant is the typical solution and it can be combined with other tools like Ansible to provision the VM after creating it.
In the cloud space, there's tools like Terraform or vendor specific tools like CloudFormation.
Docker is working on a project called Infrakit to deploy infrastructure the way compose deploys containers. It includes hooks for several of the above tools, including Terraform and Vagrant. For your own requirements, this may be overkill.
Lastly, for designing VM images, Docker recently open sourced their Moby project which creates the VM image containing a minimal container OS, the same one used under the covers in Docker for Windows, Docker for Mac, and possibly some of the cloud hosing providers.
We automate Docker installation on hosts using Ansible + Jenkins. Given the propper SSH access, provisioning new Docker hosts is a matter of triggering a Jenkins job.
Related
I want to create a tool that couples together a lot (~10, perhaps more) of other CLI tools to automate some stuff. This tool needs to be able to just be dropped-in on any VPS and work, hence the Docker containers. Work in this case means running central program (made by me) that orchestrates all the other tools and aggregates their results in a single database to browse/export later. The tools' containers need to have network access.
In my limited knowledge of Docker I've concluded that multi-stage build to fit all the tools in a single container is a bad design here, and very cumbersome. I've thought of networking the tools' containers to the central one and doing some sort of TCP piping, but that seems less than ideal too. What should the approach here be like? Are there some ready-made solutions to this issue?
Thanks
How about docker-compose?
You can use this tool to deploy all your dockerized tools inside docker network and then communicate with them via your orchestrator. Additionally you can pack this composed dockers into another docker and create docker-in-docker environment and expose only your orchestrator as a gate to your all-in-one tool.
Cheers,
This question already has answers here:
Vagrant, Docker, Puppet, Chef
(3 answers)
Closed 5 years ago.
What is the common part and the differences between docker and chef?
As I understand, docker can include chef, but actually I don't have an exact example how do the related to each other...
Chef is a configuration management software. It allows you to automate things like provision, manage, and configure infrastructure. You basically have a central place to manage your infrastructure as code. Think of it as a tool that can:
Interact with multiple infrastructure provisioners (VMWare, AWS, ...)
Manage and Install software on machines
Deploy and start applications
...
Docker on the other hand is quite different. Simply put Docker is a technology to package your application with all of its dependencies in a sandbox which allows you to run the container anywhere since it is self-contained.
As I understand, docker can include chef, but actually I don't have an exact example how do the related to each other...
Nowadays it is very popular to package and distribute applications in docker container. Chef is also a program which can be packaged in a docker container. In fact the docker images for Chef already exist on Dockerhub. You can get
Chef by running the docker container. This will make it easy to get Chef up and running without having to directly install it on your machine. This is what is meant by "docker can include chef".
On the other hand, a Docker container is also a runnable software from the persective of the OS. You can tell Chef to also run a docker container on a machine that chef is managing.
In conclusion, there is no overlap between Docker and Chef. Both tools are used side by side, each for a different purpose. Chef is for managing infrastructure and machines, whereas docker is for packaging, shipping and deploying applications in containers.
How should applications be scripted/automatically deployed when in LXD containers?
For example is best way to deploy applications in LXD containers to use a bash script (which deploys an application)? How to execute this bash script inside the container by executing a command on the host?
Are there any tools/methods of doing this in a similar way to Docker recipes?
In my case, I use Ansible to:
build the LXD containers (web, database, redis for example).
connect to the containers and deploy the services and code needed.
you can build your own images for example with the services and/or code already deployed and build specific containers from this images.
I was doing this from before LXD had Ansible support (Ansible 2.2) i prefer to use ssh instead of lxd connection, when i connect to the containers to deploy services/code. they comes with a profile where i had setup my ssh public key (to have direct ssh connection by keys ... no passwords)
Take a look at my open source project on bitbucket devops_lxd_containers It includes:
Scripts to build lxd image templates including Apache, tomcat, haproxy.
Scripts to demonstrate custom application image builds such as Apache hosting and key/value content and haproxy configured as a router.
Code to launch the containers and map ports so they are accessible to the larger network
Code to configure haproxy as layer 7 proxy to route http requests between boxes and containers based on uri prefix routing. Based on where it previously deployed and mapped ports.
At the higher level it accepts a data drive spec and will deploy an entire environment compose of many containers spread across many hosts and hook them all up to act as a cohesive whole via a layer 7 proxy.
Extensive documentation showing how I accomplished each major step using code snippets before automating.
Code to support zero-outage upgrades using the layer7 ability to gracefully bleed off old connections while accepting new connections at the new layer.
The entire system is built on the premise that image building is best done in layers. We build a updated Ubuntu image. From it we build a hardened Ubuntu image. From it we build a basic Apache image. From it we build an application specific image like our apacheKV sample. The goal is to never rebuild any more than once and to re-use the common functionality such as the basicJDK as the source for all JDK dependent images so we can avoid having duplicate code in any location. I have strived to keep Image or template creation completely separate from deployment and port mapping. The exception is that I could not complete creation of the layer 7 routing image until we knew everything about how other images would be mapped.
I've been using Hashicorp Packer with the ansible provisioner using ansible_connection = lxd
Some notes here for constructing a template
When iterating through local files on your host system you may need to be using ansible_connection = local (e.g for stat & friends)
Using local_action in ansible with the lxd connection is still
action inside the container when using stat (but not with include_vars & lookup function for files)
Using lots of debug messages in Ansible is helpful to know which local environment ansible is actually operating in.
I'm surprised no one here mentioned Canonicals own tool for managing LXD.
https://juju.is
it is super simple, well supported, and the only caveat is it requires you turn off ipv6 at the LXD/LXC side of things (in the network bridge)
snap install juju --classic
juju bootstrap localhost
from there you can learn about juju models, deploy machines or prebaked images like ubuntuOS
juju deploy ubuntu
I have little experience using chef for managing the DevOps. I was using AWS opsworks, there you can create a stack with layers and can configure it to create a new server instance for each layer or use existing instance. Now I'm trying to make my hands dirty with Docker. From my reading about Docker so far you have to spin off a new server instance, ssh to it and then run a Docker image on it. Is there a way I can create new server instances and install images on them using scripts?
If you're just concerned with playing around in a development environment, look at docker-machine, and docker-compose. These are tools for orchestrating docker images and containers from your primary machine on a local VM running docker.
If you're interested in orchestrating containers and images with multiple servers on a production level, you will still need a configuration management tool like chef. However you will be doing less application environment setup in chef, just the basic docker configuration and container management.
If you aren't too attached to chef. I'd look at ansible. It has some configuration management modules specifically for docker that are great for starting out.
We're thinking about using mesos and mesosphere to host our docker containers. Reading the docs it says that a prerequisite is that:
Docker version 1.0.0 or later needs to be installed on each slave
node.
We don't want to manually SSH into each new machine and install the correct version of the Docker daemon. Instead we're thinking about using something like Ansible to install Docker (and perhaps other services that may be required on each slave).
Is this a good way to solve it or does Mesosphere/DCOS or any of Mesos ecosystem components have other ways of dealing with this?
I've seen the quick intro where someone from Mesosphere just use dcos resize to change the cluster size on the Google Cloud Platform. Is there a way to hook in to this process and install additional services on the (google) container when it has booted? Or is this something we should avoid and instead just use a "pre-baked image"?
In your own datacenter using your favorite configuration tool such as ansible, salt, ... is probably a good choice.
On the cloud it might be easier to use virtual machine images providing docker, so for example dcos on aws uses coreOS which comes with docker out of the box. Shouldn't be too difficult with Ubuntu either...