So I have been using docker with docker-compose for quite some time in a development environment, I love how easy it is.
Until now, I also used docker-compose to serve my apps on my own server as I could afford short down times like docker-compose restart.
But in my current project, we need rolling updates.
We still have one node, and it shall remain as we don't plan on having scalability issue for quite some time.
I read I need to use docker swarm, fine, but when I look for some tutorials on how to set it up, along with using my docker-compose.yml files, I can't find any developer-oriented (instead of devops) resources that would simply tell me the steps to achieve this, even though I don't understand everything it is ok, as I will along the way.
Are there any tutorials to learn how to set this up out there? If not, shouldn't we build it here?
I am definitely sure we are quite numerous to have the issue, as docker is now a must have for devs, but we (devs) still don't want to dive too deep into the devops world.
Cheers, hope it gets attention instead of criticism.
After giving multiple tries to docker swarm, I did struggle a lot with concurrency and orchestration issues, hence I decided to stick with docker-compose which I'm much more comfortable with.
Here's how I achieved rolling updates: https://github.com/docker/compose/issues/1786#issuecomment-579794865
It works actually pretty nice though observers told me it was a similar strategy to what swarm does by default.
So I guess most of my issues went away by removing replication of nodes.
When I get time, I'll give swarm another try. For now, compose does a great job for me.
Related
My main goal is not to have a reproducible environment but rather an independent one.
I can achieve this by using docker and namely docker-compose where I can describe services I need and start/stop them with ease. That way I can have two versions of the database to be used in two different projects without polluting my global-space with them. That all sounds nice and shiny but I am on macOs and all of that is particularly slow. To the extend when it is even unusable.
As a slight alternative to docker/containers people often propose nix. I like the idea of it. You have reproducible and isolated environments without any virtualizition/containerization on top of it. Cool! I could have said if there was any info on services and how to use them with nix. The only thing I found is that there is such thing as shellHook which allows to do anything like starting a db when you enter nix shell. But you can't automatically stop it if you leave it or if you simply close the terminal.
Is there something in the nix world which helps manage services with the ease it helps to manage libraries/languages/frameworks?
Sounds like you need a process manager.
You might be able to use nix-processmgmt to write service configurations that you can run with supervisord on macOS. Launchd is also supported, but project configuration shouldn't be mixed with system configuration, if avoidable.
I haven't tried this yet, because I can do my backend work on NixOS with arion, which uses docker compose as a backend. I'll be interested to know what you think of nix-processmgmt.
I'm working on a scientific computing project. For this work, I need many Python modules as well as C++ packages. The C++ packages require specific versions of other software, so setting up the environment should be done carefully, and after the setup the dependencies should not be updated. So, I thought it should be good to make a Docker container and work inside it, in order to make the work reproducible in the future. However, I didn't understand why people in the internet recommend to use different Docker containers for different processes. For me it seems more natural that I setup the environment, which is a pain, and then use it for the entire project. Can you please explain what I have to be worried about in this case?
It's important that you differentiate between a Docker image and a Docker container.
People recommend using one process per container because this results in a more flexible, scalable environment: if you need to scale out your frontend web servers, or upgrade your database, you can do that without bringing down your entire stack. Running a single process per container also allows Docker to manage those processes in a sane fashion, e.g. by restarting things that have unexpectedly failed. If you're running multiple processes in a container, you end up having to hide this information from Docker by running some sort of process manager, which complicates your containers and can make it difficult to orchestrate a complex application.
On the other hand, it's quite common for someone to use a single image as the basis for a variety of containers all running different services. This is particularly true if you're build a project where a single source tree results in several commands; in that case, it makes sense to have bundle that all into a single image, and then choose which command to run when you start the container.
The only time this becomes a problem is when someone decides to do something like bundle, say, MySQL and Apache into a single image: which is a problem because there are already well maintained official images for those projects, and by building your own you've taking on the burden of properly configuring those services and maintaining the images going forward.
To summarize:
One process/service per container tends to make your life easier
Bundling things together in a single image can be okay
I have a repo with a piece of software, and a docker for users who have installation problems. I need to re-build the docker every time I publish a new version, and also want to use automated testing after it. DockerHub has such functionality, but builds are too long and are killed by timeout. Also I can't use tests there, as some tests use ~8 Gb RAM.
Are there any other services for these tasks? I'm fine with paying for it, but don't want to spend time for long configuration and maintenance (e.g. for having my own build server).
TravisCI.
It's fairly easy to start, hosted CI service which is free as long as you're keeping the repo public.
It's well known, common and you will find thousands of helpful questions and answers under the [travisci] tag
I'm adding a link to their documentation with example on how to build Dockerfile.
https://docs.travis-ci.com/user/docker/#building-a-docker-image-from-a-dockerfile
Also, I've tried to find memory and time limitations but couldn't find any in quick search.
I have an API server which writes some data to the DB and should eventually generate other containers - according to the different parameters it gets.
How should I do that? both in development and in production.
You need to work on a dockerfile generator. Like here.
Thats a lot of work for your cause, but worth doing it. Friendly advise, have a control on the number of containers you create by reusing them for similiar functions.
It seems that the only tutorials out there talking about using Amazon's SimpleDB in a rails site are using AWSDBProxy... Personally, I find this counter-intuitive to scaling out, considering the server layout of a typical Rails site below (using AWSDBProxy):
Plugin here: http://agilewebdevelopment.com/plugins/aws_sdb_proxy
Image here: http://www.freeimagehosting.net/uploads/91be4e0617.png
As you can see, even if we add more mongrels, we have two problems.
We have a single point of failure far less stable than our load balancer
We have to force all our information through this one WEBrick server
The solution is, of course, to add more AWSDBProxies... but why not then just use the following code in say, a class, skipping the proxy all together?
service = AwsSdb::Service.new(Logger.new(nil),
CONFIG['aws_access_key_id'],
CONFIG['aws_secret_access_key'])
service.query(domain, query)
So what I'm getting at, is if you are using AWSDBProxy, what are you justifications for it? And if you are indeed using it, what is your performance like? If you have hard numbers, this would be even more appreciated!
I'm not using it, nor have I ever heard of it, but this is what I would think are reasonable reasons.
You're running your main app server on EC2, so the chance of Internet FAIL doesn't really affect you more than once.
You run one proxy on each of your app servers. So it's connection going down is no worse than it's connection(s) to the database going down.
Because it can be done. This is as good a reason as any in an open source project. Sometimes it takes building a thing before you know whether said thing is a good/bad idea.
You don't have the traffic levels to need a load balancer. Then your diagram squashes down to a line, if not a single machine.