I've got a repository that has a series of documents (multimarkdown files, PDFs, GSN arguments, etc.) that need to use our internal (currently) proprietary tool to assemble those documents into a HTML-like document. The internal tool is quite complicated to use and isn't (yet) deployable.
What I tried doing was compiling the internal tool on the Ubuntu VM that I knew would be used for this job and then not tell GitLab (we're using self-hosted GitLab) to use any docker image when it tried to assemble the documents. Alas, when the CI job was run, I saw:
Pulling docker image alpine:latest ...
And then, of course, none of the stuff I installed on the VM itself was available.
Is there a way to have GitLab run the CI job without any Docker image?
If not (or if this alternative is just plain "better"), what is a good resource for reading how to install this complicated internal tool into a Docker image?
NB: The current methodology for "installing" the complicated internal tool, in addition to a lot of installing packages via apt-get, etc., (which I already have examples of how to do in Docker), is to clone the repository, and then run npm install and rake install in the cloned directory.
This is controlled by your GitLab-runner configuration. When the runner uses the docker executor it will always use a docker image for the build. If you want to run a GitLab job without using docker, you will need to configure a GitLab runner with the "shell" executor on your VM.
However, using image: ubuntu:focal or similar is likely enough. You usually don't have to be concerned about the fact that an executor happens to run your job inside of a container. This is also beneficial, as it means your build environment is reproducible and that process will be defined in your job.
myjob:
image: ubuntu:focal
script:
- apt update && apt install -y nodejs ruby # or whatever else
# - npm install
# - gem install
# - rake install
# etc...
-
Or better yet, if you can produce a docker image with your core dependencies installed you can just use image: my-special-image in your GitLab job to use that image as your build environment.
Related
I wonder if it is possible to make Docker automatically mount volumes during build or run container phase. With podman it is easy, using /usr/share/containers/mounts.conf, but I need to use Docker CE.
If it is not, may I somehow use host RHEL subscription during Docker build phase? I need to use RHEL UBI image and I have to use companys Satellite
A container image build in docker is designed to be self contained and portable. It shouldn't matter whether you run the build on your host or a CI server in the cloud. To do that, they rely on the build context and args to the build command, rather than other settings on the host, where possible.
buildah seems to have taken a different approach with their tooling, allowing you to use components from the host in your build, giving you more flexibility, but also fragility.
That's a long way of saying the "feature" doesn't exist in docker, and if it gets created, I doubt it would look like what you're describing. Instead, with buildkit, they allow you to inject secrets from the build command line, which are mounted into the steps where they are required. An example of this is available in the buildkit docs:
# syntax = docker/dockerfile:1.3
FROM python:3
RUN pip install awscli
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ...
And to build that Dockerfile, you would pass the secret as a CLI arg:
$ docker build --secret id=aws,src=$HOME/.aws/credentials .
Fair warning: I'm new to all of this, so there there might be some mistakes in my thinking process.
I want to system test an application we are developing, and we ship this application via Docker, so that's what I want to test.
For GitLab CI, this means creating a Docker image which has Docker in Docker and Cypress, since that is what I'd like to use.
So just from checking the Docker docs I can see that Docker can be installed on a multitude of Linux distros, but not on Alpine. The official image however is Alpine.
The Cypress docs however show that Cypress can not be installed to Alpine. Only the package managers "apt-get" and "yum" are supported, which is Ubuntu and Fedora, respectively.
So as far as I can tell, it's not possible to have both of these at once? Which would be absolutely baffling (but so is the package manager chaos I just learned about).
What I tried:
used the Docker image as a base and tried to install Cypress (does not work because there is no installation manual and the packages you need to install via apt-get don't exist for apk)
used the Cypress image as a base and tried to install Docker (does not work because the Cypress images don't work)
used another image and tried to install both (does not work because installing Docker inside the Docker container does not work, that's why they have the image provided)
used DinD with another distro (cruizba/ubuntu-dind, fails with " dockerd is not running after max time")
So... what am I missing? Is there any way to get to the point where I can use both Cypress and DinD in the same image?
There is an image named blackholegalaxy/cypress-dind which combines DinD and Cypress.
Sadly it's really old and there is no way to update Docker to the newest version easily.
What I want to build, without Docker, would look like this:
AWS EC2 Debian OS where I can install:
Python 3
Anaconda
Run python scripts that will create new files as output
SSH into it so that I can explore the new files created
I am starting to use Docker an my first approach was to build everything into a Dockerfile but I don't know how to add multiple FROM so that I can install the official docker image of Debian, Python3 and Anaconda.
In addition, with this approach is it possible to get into the container and explore the files that have been created?
I have a Gitlab server running on a Docker container: gitlab docker
On Gitlab there is a project with a simple Makefile that runs pdflatex to build pfd file.
On the Docker container I installed texlive and make, I also installed docker runner, command:
curl -sSL https://get.docker.com/ | sh
the .gitlab-ci.yml looks like follow:
.build:
script: &build_script
- make
build:
stage: test
tags:
- Documentation Build
script: *build
The job is stuck running and a message is shown:
This build is stuck, because the project doesn't have any runners online assigned to it
any idea?
The top comment on your link is spot on:
"Gitlab is good, but this container is absolutely bonkers."
Secondly looking at gitlab's own advice you should not be using this container on windows, ever.
If you want to use Gitlab-CI from a Gitlab Server, you should actually be installing a proper Gitlab server instance on a proper Supported Linux VM, with Omnibus, and should not attempt to use this container for a purpose it is manifestly unfit for: real production way to run Gitlab.
Gitlab-omnibus contains:
a persistent (not stateless!) data tier powered by postgres.
a chat server that's entire point in existing is to be a persistent log of your team chat.
not one, but a series of server processes that work together to give you gitlab server functionality and web admin/management frontend, in a design that does not seem ideal to me to be run in production inside docker.
an integrated CI build manager that is itself a Docker container manager. Your docker instance is going to contain a cache of other docker instances.
That this container was built by Gitlab itself is no indication you should actually use it for anything other than as a test/toy or for what Gitlab themselves actually use it for, which is probably to let people spin up Gitlab nightly builds, probably via kubernetes.
I think you're slightly confused here. Judging by this comment:
On the Docker container I installed texlive and make, I also installed
docker runner, command:
curl -sSL https://get.docker.com/ | sh
It seems you've installed docker inside docker and not actually installed any runners? This won't work if that's the case. The steps to get this running are:
Deploy a new gitlab runner. The quickest way to do this will be to deploy another docker container with the gitlab runner docker image. You can't run a runner inside the docker container you've deployed gitlab in. You'll need to make sure you select an executor (I suggest using the shell executor to get you started) and then you need to register the runner. There is more information about how to do this here. What isn't detailed here is that if you're using docker for gitlab and docker for gitlab-runner, you'll need to link the containers or set up a docker network so they can communicate with each other
Once you've deployed and registered the runner with gitlab, you will see it appear in http(s)://your-gitlab-server/admin/runners - from here you'll need to assign it to a project. You can also make it as "Shared" runner which will execute jobs from all projects.
Finally, add the .gitlab-ci.yml as you already have, and the build will work as expected.
Maybe you've set the wrong tags like me. Make sure the tag name with your available runner.
tags
- Documentation Build # tags is used to select specific Runners from the list of all Runners that are allowed to run this project.
see: https://docs.gitlab.com/ee/ci/yaml/#tags
I want to create development environment for all our programmers so I want to install jdk, jdeveloper, maven and svn in one docker container.
How can I do that?
First of all, you need to go to the docker site and learn how to create a Dockerfile. The file will run and create a container with whatever you want to install in it. For example:
FROM debian
RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -yqq\
openjdk-8-jre\
maven\
svn\
....
RUN ... (to run commands inside the container when it's created)
EXPOSE 80 8080... (whatever ports you want to expose)
This is a very simple example, you have to read the docs and see what is available. I would recommend looking at the Dockerfile for from the github repo that has all the library docker files and get an idea of what's happening so you know how to create your container.
On a side note, I would not install svn in the same container, it would be better to have it in a separate container so that you never have more than one service per container since each container runs in a separate process. You can link containers, but that would require reading the docs to see how that is done.
Why using jdeveloper? For eclipse you can find several examples in the docker hub.
Agree with #Hatem-Jaber seperate svn from the dev environment.