Running a Gitlab project that uses a Docker image I created.
Problem: Gitlab job execution log shows that image is not being pulled.
Here is the .gitlab-ci.yml file, with the company stuff removed:
default:
image:
name: guythedocker/jmeter-mssql-windows:latest
entrypoint: [""]
api test:
stage: test
script:
- get-variable
- $env:path -split ";"
- echo $WORKDIR
- Get-ChildItem -Path / -File
- entrypoint.ps1 --version
- |
/entrypoint.ps1 -n -t ./JMeter/xxx.jmx -l ./xxx.log -e -o ./testresults/xxx-Jthreads=$xx-Jrampup=$xxx -JtestCases=$xxx -Jhost=xxx.com -f
retry: 2
only:
- schedules
artifacts:
paths:
- testresults
tags:
- win2019
Here is the Dockerfile, which is essentially copied from QAInsights' Dockerfile:
# Dockerfile for Apache JMeter for Windows
# Indicates that the windowsservercore along with OpenJDK will be used as the base image.
# Based on work by NaveenKumar Namachivayam
FROM openjdk:8-windowsservercore
ARG JMETER_VERSION="5.4.3"
ENV JMETER_HOME /apache-jmeter-$JMETER_VERSION/apache-jmeter-$JMETER_VERSION/
# Metadata indicating an image maintainer.
LABEL maintainer="Guy L."
# Downloads JMeter from one of the mirrors, if you prefer to change, you can change the URL
RUN Invoke-WebRequest -URI https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-$env:JMETER_VERSION.zip \
-UseBasicParsing -Outfile /apache-jmeter-$env:JMETER_VERSION.zip
# Extract the downloaded zip file
RUN Expand-Archive /apache-jmeter-$env:JMETER_VERSION.zip -DestinationPath /apache-jmeter-$env:JMETER_VERSION
# For JDBC
RUN Invoke-WebRequest -URI https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/9.4.1.jre8/mssql-jdbc-9.4.1.jre8.jar -Outfile mssql-jdbc-9.4.1.jre8.jar
RUN Invoke-WebRequest -URI https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc_auth/9.4.1.x86/mssql-jdbc_auth-9.4.1.x86.dll -Outfile mssql-jdbc_auth-9.4.1.x86.dll
COPY ./mssql-jdbc-9.4.1.jre8.jar ${JMETER_HOME}/lib/
COPY ./mssql-jdbc_auth-9.4.1.x86.dll ${JMETER_HOME}/lib/
# Copies the entrypoint.ps1
COPY /entrypoint.ps1 /entrypoint.ps1
COPY /jmeter-plugins-install.ps1 /jmeter-plugins-install.ps1
RUN ["powershell.exe","/jmeter-plugins-install.ps1"]
# Sets the Working directory
WORKDIR ${JMETER_HOME}/bin
# Sets a command or process that will run each time a container is run from the new image. For detailed instruction, go to entrypoint.ps1 file.
ENTRYPOINT ["powershell.exe", "/entrypoint.ps1"]
The image was successfully published.
So why is my Gitlab project not pulling this image?
It is a Windows runner (since I'm using the -tag, and as I can see in the Job history).
Your runner is configured to use shell executor (as you can see line 3 of your printscreen) but to run Docker image, you have to use docker or docker-windows executor (depending of if container you want to run is Linux or Windows based).
Related
For my projects I sometimes need to unittest my code against a ms sql server.
At the moment, I created a monstrous docker image, containing all the tools I need, including the sql server.
Now this image got to about 15 GB in size, which is really not cool.
So I'm trying to use the GitLab CICD Service part (as described here https://docs.gitlab.com/ee/ci/services/index.html).
But when I'm trying to add my (custom) image as a service I keep getting the health check issue:
*** WARNING: Service runner-hrtjgacu-project-1489-concurrent-0-2ae3f3cd2099f19a-gitlab.mydomain.lcaol__windowsdockerimages__mssql-0 probably didn't start properly.
Health check error:
service "runner-hrtjgacu-project-1489-concurrent-0-2ae3f3cd2099f19a-gitlab.mydomain.local__windowsdockerimages__mssql-0-wait-for-service" health check: exit code 1
Health check container logs:
2023-01-19T09:58:09.913696500Z FATAL: No HOST or PORT found
Service container logs:
*********
Now I found something online about needing to expose the ports of the service in the Dockerfile, so I tried that, but that did not help.
This is what my MSSQL DockerFile currently looks like:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
# Download Links:
ENV exe "https://go.microsoft.com/fwlink/?linkid=840945"
ENV box "https://go.microsoft.com/fwlink/?linkid=840944"
ENV sa_password="_" \
attach_dbs="[]" \
ACCEPT_EULA="Y" \
sa_password_path="C:\ProgramData\Docker\secrets\sa-password"
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
# make install files accessible
COPY start.ps1 /
WORKDIR /
RUN Invoke-WebRequest -Uri $env:box -OutFile SQL.box ; \
Invoke-WebRequest -Uri $env:exe -OutFile SQL.exe ; \
Start-Process -Wait -FilePath .\SQL.exe -ArgumentList /qs, /x:setup ; \
.\setup\setup.exe /q /ACTION=Install /INSTANCENAME=MSSQLSERVER /FEATURES=SQLEngine /UPDATEENABLED=0 /SQLSVCACCOUNT='NT AUTHORITY\System' /SQLSYSADMINACCOUNTS='BUILTIN\ADMINISTRATORS' /TCPENABLED=1 /NPENABLED=0 /IACCEPTSQLSERVERLICENSETERMS ; \
Remove-Item -Recurse -Force SQL.exe, SQL.box, setup
RUN stop-service MSSQLSERVER ; \
set-itemproperty -path 'HKLM:\software\microsoft\microsoft sql server\mssql14.MSSQLSERVER\mssqlserver\supersocketnetlib\tcp\ipall' -name tcpdynamicports -value '' ; \
set-itemproperty -path 'HKLM:\software\microsoft\microsoft sql server\mssql14.MSSQLSERVER\mssqlserver\supersocketnetlib\tcp\ipall' -name tcpport -value 1433 ; \
set-itemproperty -path 'HKLM:\software\microsoft\microsoft sql server\mssql14.MSSQLSERVER\mssqlserver\' -name LoginMode -value 2 ;
HEALTHCHECK CMD [ "sqlcmd", "-Q", "select 1" ]
EXPOSE 1433/tcp
EXPOSE 4022/tcp
EXPOSE 135/tcp
EXPOSE 1434/tcp
EXPOSE 1434/udp
CMD .\start -sa_password $env:sa_password -ACCEPT_EULA $env:ACCEPT_EULA -attach_dbs \"$env:attach_dbs\" -Verbose
And this is is my .gitlab-ci.yml file:
default:
image:
name: gitlab.mydomain.local:4567/windowsdockerimages/basicnetframeworkimage:latest
tags:
- windows
- docker
# The stages during this build
stages:
- build
build:
stage: build
script:
- echo "Hello world"
- ping mssql
- dotnet run
services:
- name: gitlab.mydomain.local:4567/windowsdockerimages/mssql
alias: mssql
Anyone who can help me get closer to the solution?
*Note, the ping mssql also fails, the container in which we run cannot find the DNS
entry
I would use an existing official image for MSSQL server (https://hub.docker.com/_/microsoft-mssql-server) and run unit tests in one container and the database in another one.
Problem
Inside a Dockerfile, I want to do something like this:
ENV CONTAINER_INFO_GCC=\"$(gcc --version | head -n 1)\"
In a perfect world, at build time, this would produce this environment variable:
CONTAINER_INFO_GCC="gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609"
Motivation
I want to see what is installed in my docker container with the Azure DevOps system capabilities windows.
For example, if I want to know which version of gcc is installed in my build environment, I can just take a look here:
Question
I have tried doing something like this, without the desired effect.
RUN echo CONTAINER_INFO_GCC=\"$(gcc --version | head -n 1)\" >> /etc/environment
Is there a way to use the Dockerfile ENV command dynamically?
As each dockerfile command is run, it will generate a intermediate container.
So the RUN , COPY, CMD ... command couldn't pass the environment to next container.
You need to use ENV set the Environment, but the ENV will not execute the command.
In Azure Devops, you could use Self-hosted Agent(Running in Docker) to create a variable.
Here is the steps:
Step1: Create a Self-Hosted Agent running in Docker.
Step2: In Build Pipeline, you could run the gcc --version | head -n 1
Here is a Blog about create Self-Hosted Agent running in Docker.
Update:
You could try to add the container resource to Azure Pipeline, then you could run the script on the container.
Here is a doc about this feature.
Here is the Yaml example:
resources:
containers:
- container: python
image: python:3.8
trigger:
- none
pool:
vmimage: ubuntu-16.04
steps:
- script: |
gcc --version | head -n 1
echo "##vso[task.setvariable variable=test]$(gcc --version | head -n 1)"
displayName: 'Run a multi-line script'
target:
container: python
commands: restricted
- script: |
echo "$(test)"
displayName: 'Run a multi-line script'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$token = "PAT"
$url="https://dev.azure.com/{Organization Name}/_apis/distributedtask/pools/{Pool Id}/agents/{AgentID}/usercapabilities?api-version=5.0"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$JSON = #'
{
"Gcc-version":"$(test)"
}
'#
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method PUT -Body $JSON -ContentType application/json
Result:
The Rest API is used to update the Agent Capabilities.
Note: We can only manually change the user defined Capabilities.
On the other hand, you still could create a Self-hosted agent running in docker.
Then you could directly run the same script on the agent and get the tool version.
I'm trying to do some basic containers in Windows. I've been using Docker on Linux for years, but this issue is new for me.
Running the command
docker build -f windowsTest3.df -t dockertest . results in a good, tagged build.
...
---> 04064df75127
Step 13/13 : ENTRYPOINT C:/BuildTools/Common7/Tools/VsDevCmd.bat
---> Using cache
---> 9e098cff37a2
Successfully built 9e098cff37a2
Successfully tagged dockertest:latest
However, attempting to run an interactive shell inside the container gives an error. The system cannot find the path specified.
Edit: Can't believe I forgot to list the command...
To start the container interactively, I'm running docker run -it dockertest, but I've also tried docker run -it dockertest cmd and variations of that.
Running docker images shows that the tagged image exists, so I can't figure out what's causing the error.
docker images
C:\Users\devuser.DESKTOP-UV8CO47\Desktop\tmp>docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
dockertest latest 9e098cff37a2 41 minutes ago 12.3GB
Here are my path locations:
C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Python37-32\Scripts\;C:\Program Files (x86)\Python37-32\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Program Files (x86)\IAR Systems\Embedded Workbench 8.2\common\bin;C:\Program Files\Amazon\AWSCLI\bin\;C:\Program Files (x86)\GnuWin32\bin;C:\Program Files\CMake\bin;C:\Program Files\dotnet\;C:\Windows\Microsoft.NET\Framework\v4.0.30319;C:\ProgramData\chocolatey\bin;C:\PRQA\PRQA-Framework-2.4.0\common\bin;C:\Users\DevUser\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\IAR Systems\Embedded Workbench 8.2\arm\bin;C:\Program Files\Git\bin;C:\Program Files\7-Zip;C:\Program Files\nssm-2.24\win64
Here is a slightly abridged version of the dockerfile
FROM mcr.microsoft.com/windows:10.0.17763.316-amd64
# Restore the default Windows shell for correct batch processing.
SHELL ["cmd", "/S", "/C"]
# Download the Build Tools bootstrapper.
ADD https://aka.ms/vs/16/release/vs_buildtools.exe C:/tmp/vs_buildtools.exe
# Install Build Tools excluding workloads and components with known issues.
RUN C:/tmp/vs_buildtools.exe --quiet --wait --norestart --nocache \
--installPath C:\BuildTools \
--all \
--remove Microsoft.VisualStudio.Component.Windows10SDK.10240 \
--remove Microsoft.VisualStudio.Component.Windows10SDK.10586 \
--remove Microsoft.VisualStudio.Component.Windows10SDK.14393 \
--remove Microsoft.VisualStudio.Component.Windows81SDK \
|| IF "%ERRORLEVEL%"=="3010" EXIT 0
ENV chocolateyUseWindowsCompression=false
RUN powershell set-executionpolicy remotesigned
RUN powershell -Command Invoke-Expression ((New-Object Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
RUN powershell -Command Install-PackageProvider -Name chocolatey -Force
RUN powershell -command "choco install -y git"
ENTRYPOINT C:/BuildTools/Common7/Tools/VsDevCmd.bat
Please check if VsDevCmd.bat is available inside the container when it's starting, at C:/BuildTools/Common7/Tools/ path
Also as per this Doc reference
On Windows, file paths specified in the CMD instruction must use
forward slashes or have escaped backslashes \.
CMD c:\Apache24\bin\httpd.exe -w
Maybe try your ENTRYPOINT like this.
ENTRYPOINT C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat
You can also use CMD
CMD C:\\BuildTools\\Common7\\Tools\\VsDevCmd.bat
Can also try this as well but recommended is above one
ENTRYPOINT C:\BuildTools\Common7\Tools\VsDevCmd.bat
I have the following script thats run in my jenkins job
set +x
SERVICE_ACCOUNT=`cat "$GCLOUD_AUTH_FILE"`
docker login -u _json_key -p "${SERVICE_ACCOUNT}" https://gcr.io
set -x
docker pull gcr.io/$MYPROJECT/automation:master
docker run --rm --attach STDOUT -v "$(pwd)":/workspace -v "$GCLOUD_AUTH_FILE":/gcloud-auth/service_account_key.json -v /var/run/docker.sock:/var/run/docker.sock -e "BRANCH=master" -e "PROJECT=myproject" gcr.io/myproject/automation:master "/building/buildImages.sh" "myapp"
if [ $? -ne 0 ]; then
exit 1
fi
I am now trying to do this in cloudbuild.yaml such that I can run my script using my own automation image (which has a bunch of dependencies docker/jdk/pip etc installed) , and mount my git folders in my workspace directory
I tried putting my cloudbuild.yaml at the top level in my directory in my git repo and set it up as this
steps:
- name: 'gcr.io/myproject/automation:master'
volumes:
- name: 'current-working-dir'
path: /mydirectory
args: ['bash', '-c','/building/buildImages.sh', 'myapp']
timeout: 4000s
But this gives me errors saying the
invalid build: Volume "current-working-dir" is only used by one step
Just FYI, my script buildImages.sh, copies folders and dockerfiles, runs pip install/ npm/ and gradle commands and then docker build commands (kind of all in one solution).
Whats the way to translate my script to cloudbuild.yaml
try this in your cloudbuild.yaml:
steps:
- name: 'gcr.io/<your-project>/<image>'
args: ['sh','<your-script>.sh']
using this I was able to pull the image from Google Cloud Registry that has my script, then run the script using 'sh'. It didn't matter where the script is. I'm using alpine in my Dockerfile as base image.
I'm attempting to get RabbitMQ up and running inside a Windows container but without a whole lot of luck. I've copied into the container the installation directories for RabbitMQ and Erlang but when I attempt to run erl.exe I'm told that beam.smp.dll is not able to be loaded.
PS C:\Program Files\erl8.2\bin> .\erl.exe
Unable to load emulator DLL
(C:\Program Files\erl8.2\erts-8.2\bin\beam.smp.dll)
Running the same command on the same installation directory on the host machine works just fine. I've checked that the file exists and that the checksums match. My bet is that there is some subtile difference in how the container loads the file and how the host loads the file. I'm just not sure where to even start looking.
Here is my Dockerfile which works. I can connect to RabbitMQ and the logs show it is running correctly, and I can login to the management UI using the guest/guest login from my host using IP/hostname of container.
# start with this container as the base
FROM microsoft/windowsservercore
# erlang installer download url
ENV erlang_download_url "http://erlang.org/download/otp_win64_19.3.exe"
# erlang will install to this location and rabbitmq will use this environment variable to locate it
ENV ERLANG_HOME c:\\erlang
# rabbitmq version used in download url and to rename folder extracted from zip file
ENV rabbitmq_version "3.6.9"
# rabbitmq zip package download url
ENV rabbit_download_url "https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.9/rabbitmq-server-windows-$rabbitmq_version.zip"
# setup powershell options for RUN commands
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
# download and install erlang using silent install option, and remove installer when done
RUN Invoke-WebRequest -Uri $env:erlang_download_url -OutFile erlang_install.exe ; \
Start-Process -Wait -FilePath .\erlang_install.exe -ArgumentList /S, /D=$env:ERLANG_HOME ; \
Remove-Item -Force erlang_install.exe
# download and extract rabbitmq, and remove zip file when done
RUN Invoke-WebRequest -Uri $env:rabbit_download_url -OutFile rabbitmq.zip ; \
Expand-Archive -Path .\rabbitmq.zip -DestinationPath "c:\\" ; \
Remove-Item -Force rabbitmq.zip
# remove version from rabbitmq folder name
RUN Rename-Item c:\rabbitmq_server-$env:rabbitmq_version c:\rabbitmq
# enable managment plugin
RUN c:\rabbitmq\sbin\rabbitmq-plugins.bat enable rabbitmq_management --offline
# tell rabbitmq where to find our custom config file
ENV RABBITMQ_CONFIG_FILE "c:\rabbitmq"
RUN ["cmd", "/c", "echo [{rabbit, [{loopback_users, []}]}].> c:\\rabbitmq.config"]
# run server when container starts - container will shutdown when this process ends
CMD "c:\rabbitmq\sbin\rabbitmq-server.bat"