I have a Gitlab runner that runs all kind of jobs using Docker executors (host is Ubuntu 20, guests are various Linux images). The runner runs containers as unprivileged.
I am stumped on an apparently simple requirement - I need to deploy some artifacts on a Windows machine that exposes the target path as an authenticated share (\\myserver\myapp). Nothing more than replacing files on the target with the ones on the source - a simple rsync would be fine.
Gitlab Runner does not allow specifying mounts in the CI config (see https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28121), so I tried using mount.cifs, but I discovered that by default Docker does not allow mounting anything inside the container unless running privileged, which I would like to avoid.
I also tried the suggestion to use --cap-add as described at Mount SMB/CIFS share within a Docker container but they do not seem to be enough for my host, there are probably other required capabilities and I have no idea how to identify them. Also, this looks just slightly less ugly than running privileged.
Now, I do not strictly need to mount the remote folder - if there were an SMB-aware rsync command for example I would be more than happy to use that. Unfortunately I cannot install anything on the Windows machine (no SSH, no SCP, no FTP).
Do you have any idea on how achieve this?
Unfortunately I cannot install anything on the Windows machine (no SSH, no SCP, no FTP).
You could simply copy an executable (that you can build anywhere else) in Go, listening to a port, ready to receive a file.
This this implementation for instance: file-receive.go. It listens on port 8080 (can be changed) and copies the file content to a local folder.
No installation or setup required: just copy the exe to the target machine and run it.
From your GitLab runner, you can 'curl' send a file to the remote PC machine, port 8080.
Related
I use docker-compose to spin up a few containers as part of an application I'm developing. One of the containers needs to start a docker swarm service on the host machine. On Docker for Windows and Docker for Mac, I can connect to the host docker daemon using the REST Api by using the "host.docker.internal" DNS name and this works great. However, if I run the same compose file on linux, "host.docker.internal" does not work (yet, seems it may be coming in the next version of docker). To make matters worse, on Linux I can use network mode of "host" to work around the issue but that isn't supported on Windows or Mac.
How can I either:
Create a docker-compose file or structure a containerized application to be slightly different based on the host platform (windows|mac|linux) without having to create multiple docker-compose.yml files or different application code?
Access the host docker daemon in a consistent way regardless of the host OS?
If it matters, the container that is accessing the docker daemon of the host is using the docker python sdk and making api calls to docker over tcp without TLS (this is used for development only).
Update w/ Solution Detail
For a little more background, there's a web application (aspnet core/C#) that allows users to upload a zip file. The zip file contains, among other things, an exported docker image file. There's also an nginx container in front of all of this to allow for ssl termination and load balancing. The web application pulls out the docker image, then using the docker daemon's http api, loads the image, re-tags the image, then pushes it to a private docker repository (which is running somewhere on the developer's network, external to docker). After that, it posts a message to a message queue where a separate python application uses the python docker library to deploy the docker image to a docker swarm.
For development purposes, the applications all run as containers and thus need to interact with docker running on the host machine as a stand alone swarm node. SoftwareEngineer's answer lead me down the right path. I mapped the docker socket from the host into the web application container at first but ran into a limitation of .net core that won't be resolved until .net 5 which is that there's no clean way of doing http over a unix socket.
I worked around that issue by eventually realizing that nginx can reverse proxy http traffic to a unix socket. I setup all containers (including the dynamically loaded swarm service from the zips) to be part of an overlay network to give them all access to each other and allowing me to hit an http endpoint to control the host machine's docker/swarm daemon over http.
The last hurdle I ran into was that nginx couldn't write to the mapped in /var/run/docker.sock file so I modified nginx.conf to allow it to run as root within the container.
As far as I can tell, the docker socket is available at the path /var/run/docker.sock on all systems. I have personally verified this with a recent Linux distro (Ubuntu), Windows 10 Pro running Docker for Windows (2.2.0) with both WSL2 (Ubuntu and Alpine) and the windows cmd (cli) and powershell. From memory, it works with OSX too, and I used to do the same thing in WSL1.
Mapping this into a container is achieved on any terminal with the -v, --volume, or --mount flags. So,
docker container run -v /var/run/docker.sock:/var/run/docker.sock
Mounts the socket into an identical path within the container. This means that you can access the socket using the standard docker client (docker) from within the container with no extra configuration. Using this path inside a Linux container is recommended because the standard location and is likely to be less confusing to anyone maintaining your code in the future (including yourself).
I am trying to create the docker file (Image file) for the web application I am creating. Basically, the web application is written in Node.js and Vue.js. In order to create a docker container for the application, I have got the documentation from vue.js to create a docker file. The steps given are working file. I just wanted to clear my understanding in this part.
link:- https://cli.vuejs.org/guide/deployment.html#docker-nginx
If the necessary package Node/Python is installed in the OS (Not in the container), would the container be able to pick up the npm scripts and execute python scripts also? If yes, is it really dependent on the OS software packages as well?
Please help me with the understanding.
Yes, you need to install Node or Python or whatever software you need in your application in your container. The reason is that the container should be able to run on any host machine that has Docker installed, regardless of how the host machine is set up or what it software it has installed.
It might be a bit tedious at first to make sure that your Dockerfile installs all the software that is needed, but it becomes very useful when you want to run your container on another machine. Then all you have to do is type docker run and it should work!
Like David said above, Docker containers are isolated from your host machine and it should be treated as a completely different machine/host. The way containers can communicate with other containers or sometimes the host is through network ports.
One "exception" to the isolation between the container and the host is that the container can sometimes write to files in the host in order to persist data even after the container has been stopped. You can use volumes or mounts to allow containers to write to files on the host.
I would suggest the Docker Overview for more information about Docker.
I have a project on which we have set up a development environment using docker-compose. We are using volumes to sync files from host to docker containers. The performance for sync is kinda bad on Mac.
I recently saw some extension for VS code which allows you to edit files inside the docker container. Here is the link to that extension.
Can I do something similar with vim?
Thanks a lot in advance!
I tried to ssh into docker container but I wasn't successful. I will have to use docker exec to ssh into it.
You can use vim to edit the files remotely provided that you have the SSH access to that container. In order to get that, you'll have to generate and place SSH keys on your machine and the public keys inside the container. But, you need to have SSH server running inside the container and the SSH port exposed.
If editing the file from the host machine using Vim is the absolute requirement, this is the way to go.
But, if you only want to make the debugging easier, consider using Bind mounts. You bind the target file from your host machiner and edit it locally. The container accessing your file will immediately see the changes reflected inside it.
Before I burn hours trying it out I wanted to ask the community is this even possible?
Scenario:
Running Goland on host (may be any OS)
Running Go dev env in Alpine based container
Code on host volume mapped to container
Can I attach the Goland debugger (Delve) to a Go process in the container? I'm assuming I can run delve in the container headless and run the client on the host, punching whatever port is required? Will I have binary compatibility issues if the host is not linux?
I'd rather not duplicate the entire post in this answer, but have a look at this resource on how to use containers to run applications you write https://blog.jetbrains.com/go/2018/04/30/debugging-containerized-go-applications/
To answer this specifically, as long as you have Go, the application sources, and all dependencies installed on the host machine, you can develop in GoLand and then, using a mapped volume, you can also run it from the container.
However, this workflow sounds more like the workflow you'd normally have using VMs not containers, which is why in the above article all the running/debugging is done using the actual containers, rather than using bash inside a container to run those commands.
As mentioned in the title, I thinking about a dockerized jenkins. I have a running container that run all tests but now I want to run some deployment job.
The files (.py, .conf, .sh) will be copied into folders which are mounted by other container (app container). As I seen some recommend do not use docker as well.
Now I'm wondering if I should continue to use jenkins in a container (so i must find a way to run the deployment script) or prefer to install it on the server ?
If you are running dockerized Jenkins for production, It is good practice to have its volume mounted on Docker host.
I personally do not prefer dockerized Jenkins for production due to non static IP for Jenkins, and reliability issues with docker networking. For non-production use, i dockerize Jenkins.
We're experimenting with containerizing Jenkins in production - the flexibility of being able to easily set up or move instances offsets the learning pain, and that pain is :
1 - Some build jobs are themselves containerized, requiring that you run docker-in-docker. This is possible by passing the host docker.sock into the Jenkins' container. (more reading : https://getintodevops.com/blog/the-simple-way-to-run-docker-in-docker-for-ci). It requires that the host and Jenkins container are running identical versions of Docker, but I can live with that.
2 - SSH keys are a bigger issue. ssh agent forwarding in Docker is notorious for its unreliability, and we've always copied keys into containers (ignoring security questions for the context of this question). In an on-the-host Jenkins instance we put our ssh keys in Jenkins' home folder and everything works seamlessly. But, dockerized Jenkins has its home folder inside a Docker volume, which is owned by the host system, so keys are too open. We got around this by copying the keys to a folder outside Jenkins' home, chown/chmod'ing those keys to the Jenkins container user, then adding the key path to the container's /etc/ssh/ssh_config.