My main goal is not to have a reproducible environment but rather an independent one.
I can achieve this by using docker and namely docker-compose where I can describe services I need and start/stop them with ease. That way I can have two versions of the database to be used in two different projects without polluting my global-space with them. That all sounds nice and shiny but I am on macOs and all of that is particularly slow. To the extend when it is even unusable.
As a slight alternative to docker/containers people often propose nix. I like the idea of it. You have reproducible and isolated environments without any virtualizition/containerization on top of it. Cool! I could have said if there was any info on services and how to use them with nix. The only thing I found is that there is such thing as shellHook which allows to do anything like starting a db when you enter nix shell. But you can't automatically stop it if you leave it or if you simply close the terminal.
Is there something in the nix world which helps manage services with the ease it helps to manage libraries/languages/frameworks?
Sounds like you need a process manager.
You might be able to use nix-processmgmt to write service configurations that you can run with supervisord on macOS. Launchd is also supported, but project configuration shouldn't be mixed with system configuration, if avoidable.
I haven't tried this yet, because I can do my backend work on NixOS with arion, which uses docker compose as a backend. I'll be interested to know what you think of nix-processmgmt.
Related
I'm working on a scientific computing project. For this work, I need many Python modules as well as C++ packages. The C++ packages require specific versions of other software, so setting up the environment should be done carefully, and after the setup the dependencies should not be updated. So, I thought it should be good to make a Docker container and work inside it, in order to make the work reproducible in the future. However, I didn't understand why people in the internet recommend to use different Docker containers for different processes. For me it seems more natural that I setup the environment, which is a pain, and then use it for the entire project. Can you please explain what I have to be worried about in this case?
It's important that you differentiate between a Docker image and a Docker container.
People recommend using one process per container because this results in a more flexible, scalable environment: if you need to scale out your frontend web servers, or upgrade your database, you can do that without bringing down your entire stack. Running a single process per container also allows Docker to manage those processes in a sane fashion, e.g. by restarting things that have unexpectedly failed. If you're running multiple processes in a container, you end up having to hide this information from Docker by running some sort of process manager, which complicates your containers and can make it difficult to orchestrate a complex application.
On the other hand, it's quite common for someone to use a single image as the basis for a variety of containers all running different services. This is particularly true if you're build a project where a single source tree results in several commands; in that case, it makes sense to have bundle that all into a single image, and then choose which command to run when you start the container.
The only time this becomes a problem is when someone decides to do something like bundle, say, MySQL and Apache into a single image: which is a problem because there are already well maintained official images for those projects, and by building your own you've taking on the burden of properly configuring those services and maintaining the images going forward.
To summarize:
One process/service per container tends to make your life easier
Bundling things together in a single image can be okay
So I have been using docker with docker-compose for quite some time in a development environment, I love how easy it is.
Until now, I also used docker-compose to serve my apps on my own server as I could afford short down times like docker-compose restart.
But in my current project, we need rolling updates.
We still have one node, and it shall remain as we don't plan on having scalability issue for quite some time.
I read I need to use docker swarm, fine, but when I look for some tutorials on how to set it up, along with using my docker-compose.yml files, I can't find any developer-oriented (instead of devops) resources that would simply tell me the steps to achieve this, even though I don't understand everything it is ok, as I will along the way.
Are there any tutorials to learn how to set this up out there? If not, shouldn't we build it here?
I am definitely sure we are quite numerous to have the issue, as docker is now a must have for devs, but we (devs) still don't want to dive too deep into the devops world.
Cheers, hope it gets attention instead of criticism.
After giving multiple tries to docker swarm, I did struggle a lot with concurrency and orchestration issues, hence I decided to stick with docker-compose which I'm much more comfortable with.
Here's how I achieved rolling updates: https://github.com/docker/compose/issues/1786#issuecomment-579794865
It works actually pretty nice though observers told me it was a similar strategy to what swarm does by default.
So I guess most of my issues went away by removing replication of nodes.
When I get time, I'll give swarm another try. For now, compose does a great job for me.
I hope that this question will not be marked as primarily opinion-based, but that there is an objective answer to it.
I have read Introducing dumb-init, an init system for Docker containers, which extensively describes why and how to use dumb-init. To be honest, for someone not too experienced with how the Linux process structure works, this sounds pretty dramatic - and it feels as if you are doing things entirely wrong if you don't use dumb-init.
This is why I'm thinking about using it within my very own Docker images… what keeps me from doing this is the fact that I have not yet found an official Docker image that uses it.
Take mongo as an example: They call mongod directly.
Take postgres as an example: They call postgres directly.
Take node as an example: They call node directly.
…
If dumb-init is so important - why is apparently nobody using it? What am I missing here?
Something like dumb-init or tini can be used if you have a process that spawns new processes and you don't have good signal handlers implemented to catch child signals and stop your child if your process should be stopped etc.
If your process doesn't spawn new processes (e.g. Node.js), then this may not be necessary.
I guess that MongoDB, PostgreSQL, ... which may run child processes have good signal handlers implemented. Otherwise there would have been zombie processes and someone would have filed an issue to fix this.
Only problem may be the official language images, like node, ruby, golang. They don't have dumb-init/tini in it as you normally don't need them. But it's up to the developer which may implement bad child execution code to either fix the signal handlers or use helper as PID 1.
When running our Erlang application in our system tests, I sometimes want to turn on and capture a debug trace.
The Erlang node is started using a relx start script (called as _rel/bin/foo foreground), so I don't have any control over the startup options. The system test runner (written in Python) is capturing stdout from the node.
How do I connect to an Erlang node, using -remsh, turn on dbg-tracing, and have that output written to stdout on the original node? And how do I do this all in a Python-friendly way (though I'm happy to write an escript if that'll make it easier).
To complicate this further, the relx generated release doesn't include the runtime_tools library, so dbg: isn't actually available, so I'll also add this question.
There are quite few way you could do that. All depends on what you are familiar with, and what your use case is.
I would start from doing everything by hand. That way you have greatest control on that's going one, and how effects look like (if you are turning too much debugging or not enough). That's I'm most familiar with, and in the end you almost always will have to connect to remote shell and do something by hand (from my experience)
One feature of dbg that not too many people talk about i ability of saving/loading trace pasterns from files. I find those easiest way to store and share debugging information in between sessions; but lack of readability might be too big trade-off.
You don't have to use dbg if you don't want to interfere with your live system too much. You could use erlang:trace which is given by default, but you must be cautious about state you leave your VM in (dbg should turn off all tracing upon exit; with erlang:trace that's your responsibility)
If you debug session is part of python script, writng escript and calling it from python would be my way to go. You just have to remember that escripts are run in new VM, and -remsh will not allow you to just run your code on other VM. You will have to use rpc module for that.
Since you are using application is released you might look into logging. One might assume that there should already be some logging in place, quite possible lager which is somewhat standard in Erlang, and which have possibility to change logging level during runtime.
Personally I would try some mix of first and last option, and just experiment.
I've read the many answers online on how to use SRVANY.exe to create a Windows service out of anything. My service is a batch file that sets up the environment (i need to set env vars and map drives) and then spawns my c++ app. But when i do a NET STOP, the srvany.exe process goes away, and my c++ app stays alive. Is there any way to have it killed when it receives the stop command? I'd need to be able to bounce it in case of any config file changes.
The reason i picked cmd shell is the easy drive mapping. In theory i can wrap it with either perl or python, whichever is easier to get this behavior, but then i'd need to shell out anyway to map the drives. Does this make sense?
AlwaysUp is a commercial alternative to SrvAny which covers shortcomings like this one in addition to adding more useful features.
NSSM is a open source alternative with slightly fewer features than AlwaysUp but still it can kill the underlying process when you stop the service.
no, srvany was not designed to stop your applications. The main purpose was to be able to start applications as a service that were not designed to run as a service.
As a clumsy workaround you can run a scheduled task that will monitor if srvany runs and if not it will terminate your application.