I have some tasks defined as Docker containers, each one will consume one full CPU whilst it runs.
Id like to run them as cost efficiently as possible so that VMs are only active when the tasks are running.
Whats the best way to do this on Google Cloud Platform?
It seems Kuberenetes and other cluster managers assume you will be running some service, and have spare capacity in your cluster to schedule the containers.
Is there any program or system that I can use to define my tasks, and then start/stop VMs on a schedule to run those tasks?
I think the best way is to just use the GCP/Docker APIs and script it myself?
You're right, all the major cloud container services provide you a cluster for running containers - GCP Container Engine, EC2 Container Service, and Azure Container Service.
In all those cases, the compute is charged by the VMs in the cluster, so you'll pay while the VMs are running. If you have an occasional workload you'll need to script creating or starting the VMs before running your containers, and stopping or deleting them when you're done.
An exception is Joyent's cloud, which let's you run Docker containers and charges per container - that could fit your scenario.
Disclaimer - I don't work for Google, Amazon, Microsoft or Joyent. Or Samsung.
Related
Docker Swarm is comparatively easier to maintain compared to Kubernetes so I was thinking of migrating from a Kubernetes cluster to a Docker Swarm cluster.
In my Kubernetes cluster, I used Jobs extensively to spin up workers ad-hoc and is very critical to the main functions of the whole infrastructure. Best thing about Kubernetes Jobs is that they will shut down after a certain time limit after Job completion and recycle the resources.
I want to ask if there's a Kubernetes Jobs equivalent in Docker Swarm? So far I couldn't find official documentation on Docker website supporting this feature.
so I'm currently developing a distributed system with microservices and I'm running out of memory to run all of them in my laptop. I wonder how other people manage this situation. Is there a way to configure docker-compose to run some of them in a remote machine but keep behavior as if they were on my laptop (for inter-service communication)?
I know that I might achieve it with k8s but it seems to be so much burden for day-to-day development flow.
Any ideas?
well if it's about day to day development, then you can figure out only the most essential services to run on your system. even if with the most essential microservices you are getting out of memory then you know the answer that you can't run them all on your local machine, unless you increase your resources or you can try Docker swarm (similar to k8s but a bit easier).
docker swarm - you will need to create a cluster among multiple machines and you will be able to offload some of the services on the other machines within the cluster.
Also, you will need to do minor changes in your docker-compose file to run on the docker swarm.
Sorry for this question, but I just started with Docker and Docker Compose and I really didn't need any of this until I read that I need to use Docker Swarn or Kuebernetes to have more stability in production. I started reading about Docker Swarn and they mentioned nodes and clusters.
I was really happy not knowing about this as I understood docker-compose:
Is that I could manage my services/containers from a single file
and only have to run several commands to launch, build, delete, etc.
all my services based on the docker-compose configuration.
But now the nodes and cluster have come out and I've really gone a bit crazy, and that's why if you can help me understand this next step in the life of containers. I've been googling and it's not very clear to me.
I hope you can help me and explain it to me in a way that I can understand.
Thank you!
A node is just a physical or virtual machine.
In Kubernetes/Docker Swarm context each node must have the relevant binaries installed (Docker Engine, kubelet etc..)
A cluster is a grouping of one or more nodes.
If you have just been testing on your local machine you have a single node.
If you were to add a second machine and link both machines together using docker swarm/kubernetes then you would have created a 2 node cluster
You can then use docker swarm/kubernetes to run your services/containers on any or all nodes in your cluster. This allows your services to be more resilient and fault tolerant.
By default Docker Compose runs a set of containers on a single system. If you need to run more containers than fit on one system, or you're just afraid of that system crashing, you need more than one system to do it. The cluster is the group of all of the systems (physical computers, virtual machines, cloud instances) that are working together to run the containers. Each of those individual systems is a node.
The other important part of the cluster container setups is that you can generally run multiple replicas of a give container, and you don't care where in the cluster they run. Say you have five nodes, and a Web server container, and you'd like to run three copies of it for redundancy. Instead of having to pick a node, ssh to it, and manually docker run there, you just tell the cluster manager "run me three of these", and it chooses a node and launches the container for you. You can also scale the containers up and down at runtime, or potentially set the cluster to do the scaling on its own based on load.
If your workload is okay running a single copy of containers on a single server, you don't need a cluster setup. (You might have some downtime during updates or if the single server dies.) Swarm has the advantages of being bundled with Docker and being able to use Docker-native tools (docker-compose can deploy to a Swarm cluster). Kubernetes is much more complex, but at this point most public cloud providers will sell you a preconfigured Kubernetes cluster, and it has better stories around security, storage management, and autoscaling. There are also a couple other less-prominent alternatives like Nomad and Mesos out there.
I thought a major benefit of Docker was the ability to deploy a single unit of work (a container) that is cheap, lightweight, and boots fast, instead of having to deploy a more expensive and heavy VM that boots slowly. But everywhere I look (eg AWS, Docker Cloud, IBM, Azure, Google Cloud, kubernetes), deploying single containers is not an option. Instead, a single customer must deploy entire VMs that will run instances of the docker engine which will then host clusters of containers.
Is there any CaaS that allows you to deploy only as few containers as you need? I thought many cloud provider companies would offer this service, coordinating the logistics of which containers submitted by which customers to group together and distribute among the companies' docker engines. I see this service is unnecessary for those customers that will be deploying enough containers that a full docker engine instance is necessary. But what about those customers that want the cheap option of only deploying a single container?
If this service is not available, I see Docker containers as no cheaper nor lighter in weight than full VMs. In both cases, you pay for a heavy VM. The only remaining benefit would be isolation of processes and the ability to quickly change them.
Again, is there any cloud service available to deploy only a single container?
As far as I see here, the problem is the point of view of your approach, not Docker.
Any machine that runs a GNU-Linux distro can run the docker daemon and therefore, run your docker containers.
There are solutions like Elastic Beanstalk that allow you to deploy docker containers with a high level of abstraction, making your "ops" part a little bit easier.
Nevertheless, I wonder, how do you actually try to deploy your application? what do you mean with:
"Instead, a single customer must deploy entire VMs that will run
instances of the docker engine which will then host clusters of
containers."
?
For example, kubernetes is a framework that allows you to deploy containers in other machines, so yes, you have to have a Framework for that or, instead, use a Framework as a service as Elastic Beankstalk is.
I hope my answer helps!
I'm running several micro-services in docker containers. Each service consists of 2 or more containers that I start/stop using docker-compose.
As the number of containers grows I'd like to monitor them, but I haven't found a method or (self-hosted) tool to monitor linked containers as a logical unit (service) instead of every single container, yet.
Do you have any recommendations?
That should be included in a project like Kubernetes, which groups containers in a pod.
You can then run Heapster, a cluster-wide aggregator of monitoring and event data. The Heapster pod discovers all nodes in the cluster and queries usage information from the nodes’ Kubelets, the on-machine Kubernetes agent.
You can try Ruxit that provides service-oriented monitoring for Docker containers. It automatically detects the services in the containers and show them as separate entities on the dashboard even if a service is provided by multiple containers across hosts.