Check Syntax errors in Dockerfile [duplicate] - docker

If a Dockerfile is written with mistakes for example:
CMD ["service", "--config", "/etc/service.conf] (missing quote)
Is there a way to lint it to detect such mistake before building?

Try:
Either the Haskell Dockerfile Linter ("hadolint"), also available online. hadolint parses the Dockerfile into an AST and performs checking and validation based on best practice Docker images rules. It also uses Shellcheck to lint the Bash code on RUN commands.
Or dockerlinter (node.js-based).
I've performed a simple test against of a simple Docker file with RUN, ADD, ENV and CMD. dockerlinter was smart about grouping the same violation of rules together but it was not able to inspect as thorough as hadolinter possibly due to the lack of Shellcheck to statically analyze the Bash code.
Although dockerlinter falls short in the scope it can lint, it does seem to be much easier to install. npm install -g dockerlinter will do, while compiling hadolinter requires a Haskell compiler and build environment that takes forever to compile.
$ hadolint ./api/Dockerfile
L9 SC2046 Quote this to prevent word splitting.
L11 SC2046 Quote this to prevent word splitting.
L8 DL3020 Use COPY instead of ADD for files and folders
L10 DL3020 Use COPY instead of ADD for files and folders
L13 DL3020 Use COPY instead of ADD for files and folders
L18 DL3020 Use COPY instead of ADD for files and folders
L21 DL3020 Use COPY instead of ADD for files and folders
L6 DL3008 Pin versions in apt get install. Instead of `apt-get install <package>` use `apt-get install <package>=<version>`
L6 DL3009 Delete the apt-get lists after installing something
L6 DL3015 Avoid additional packages by specifying `--no-install-recommends`
$ dockerlint ./api/Dockerfile
WARN: ADD instruction used instead of COPY on line 8, 10, 13, 18, 21
ERROR: ./api/Dockerfile failed.
Update in 2018. Since hadolint has the official Docker repository now, you can get the executable quickly:
id=$(docker create hadolint/hadolint:latest)
docker cp "$id":/bin/hadolint .
docker rm "$id"
or you can use this command
docker container run --rm -i hadolint/hadolint hadolint - < Dockerfile
This is a statically compiled executable (according to ldd hadolint), so it should run regardless of installed libraries. A reference on how the executable is built: https://github.com/hadolint/hadolint/blob/master/docker/Dockerfile.

If you have a RedHat subscription, you can access the "Linter for Dockerfile" application directly at https://access.redhat.com/labs/linterfordockerfile/; information about the application is located at https://access.redhat.com/labsinfo/linterfordockerfile
This Node.js application is also available on GitHub https://github.com/redhataccess/dockerfile_lint if you prefer to run it locally.

I use very successfully in my CI pipeline npm's dockerfile_lint. You can add or extend rules. Using the package.json you can create different configs for the different jobs. There are both
Docker CLI
docker run -it --rm --privileged -v `pwd`:/root/ \
projectatomic/dockerfile-lint \
dockerfile_lint [-f Dockerfile]
docker run -it --rm --privileged -v `pwd`:/root/ \
-v /var/run/docker.sock:/var/run/docker.sock \
projectatomic/dockerfile-lint \
dockerfile_lint image <imageid>
and Atomic CLI available
atomic run projectatomic/dockerfile-lint
atomic run projectatomic/dockerfile-lint image <imageid>
Also you can lint your images for tagging.

I created dockerfile-validator as an extension for VS Code, which uses the dockerfile-lint mentioned in a previous answer. By default it uses dockerfile-lint default rules, but in VS code User Settings (dockerfile-validator.rulefile.path) you can specify a path to a custom rule file with your own coding standards.

Recently, I cam across dockerfilelint which is NodeJS based.
dockerfilelint Dockerfile
Supports following rules and rudimentary CMD checks
required_params
uppercase_commands
from_first
invalid_line
sudo_usage
apt-get_missing_param
apt-get_recommends
apt-get-upgrade
apt-get-dist-upgrade
apt-get-update_require_install
apkadd-missing_nocache_or_updaterm
apkadd-missing-virtual
invalid_port
invalid_command
expose_host_port
label_invalid
missing_tag
latest_tag
extra_args
missing_args
add_src_invalid
add_dest_invalid
invalid_workdir
invalid_format
apt-get_missing_rm
deprecated_in_1.13
Hadolint seems like a better option but this may suffice for simple needs. Also, Github's super-linter uses this.

I'm not too familiar with go but it looks like you can simply call the Parse method as is done in the test suite here. If that does not return an err then your lint passes. I'm assuming that's trivial to expose to a script or something to call during development.

Related

Using JMeter plugins with justb4/jmeter Docker image results in error

Goal
I am using Docker to run JMeter in Azure Devops. I am trying to use Blazemeter's Parallel Controller, which is not native to JMeter. So, according to the justb4/jmeter image documentation, I used the following command to get the image going and run the JMeter test:
docker run --name jmetertest -i -v /home/vsts/work/1/s/plugins:/plugins -v $ROOTPATH:/test -w /test justb4/jmeter ${#:2}
Error
However, it produces the following error while trying to accommodate for the plugin (I know the plugin makes the difference due to testing without the plugin):
cp: can't create '/test/lib/ext': No such file or directory
As far as I understand, this is an error produced when one of the parent directories of the directory you are trying to make does not exist. Is there something I am doing wrong, or is there actually something wrong with the image?
References
For reference, I will include links to the image documentation and the repository.
Image: https://hub.docker.com/r/justb4/jmeter
Repository: https://github.com/justb4/docker-jmeter
Looking into the Dockerfile:
ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
Looking into entrypoint.sh
if [ -d /plugins ]
then
for plugin in /plugins/*.jar; do
cp $plugin $(pwd)/lib/ext
done;
fi
It basically copies the plugins from /plugins folder (if it is present) to /lib/ext folder relative to current working directory
I don't know why did you add this stanza -w /test to your command line but it explicitly "tells" the container that local working directory is /test, not /opt/apache-jmeter-xxxx, that's why the script is failing to copy the files.
In general I don't think that the approach is very valid because:
In Azure DevOps you won't have your "local" folder (unless you want to add plugins binaries under the version control system)
Some JMeter Plugins have other .jars as the dependencies so when you're installing the plugin you should:
put the plugin itself under /lib/ext folder of your JMeter installation
put the plugin dependencies under /lib folder of your JMeter installation
So I would recommend amending the Dockerfile, download JMeter Plugins Manager and installed the plugin(s) you need from the command line
Something like:
RUN wget https://jmeter-plugins.org/get/ -O /opt/apache-jmeter-${JMETER_VERSION}/lib/ext/jmeter-plugins-manager.jar
RUN wget https://repo1.maven.org/maven2/kg/apc/cmdrunner/2.2/cmdrunner-2.2.jar -P /opt/apache-jmeter-${JMETER_VERSION}/lib/
RUN java -cp /opt/apache-jmeter-${JMETER_VERSION}/lib/ext/jmeter-plugins-manager.jar org.jmeterplugins.repository.PluginManagerCMDInstaller
RUN /opt/apache-jmeter-${JMETER_VERSION}/bin/./PluginsManagerCMD.sh install bzm-parallel

Docker Alpine Latext texmf-dist styles not found

I'm attempting to add some necessary latext style files to the pandoc/latex docker container (which itself is built over Alpine) and the underlying latex to pdf conversion does is unable to find some of the style files I've added. Here's my Dockerfile, which adds the packages I need.
FROM pandoc/latex:latest
RUN apk --no-cache add texlive-xetex texmf-dist-pictures texmf-dist-latexextra poppler-utils && texhash
I can see that the style file I need is installed.
$ docker run --rm -it --entrypoint /bin/sh -v $PWD:/data mypandoc:latest
/data # find / -name tikzpagenodes.sty
/usr/share/texmf-dist/tex/latex/tikzpagenodes/tikzpagenodes.sty
But, when I try to do the actual build, I get the following error.
Error producing PDF.
! LaTeX Error: File `tikzpagenodes.sty' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)
Enter file name:
! Emergency stop.
<read *>
l.103 \usepackage
I had hoped that the call to texhash in the container build would fix this, but it did not. Is there some trick to getting the texmf-dist directories into the search path? The problem appears to be with all of the styles I've added from texmf-dist, not just this one. Thanks.
Use the tlmgr from TeXLive to install more packages. The pandoc/latex Docker container does not use the TeX installation from Alpine in order to give users more flexibility and to reduce image size – the Alpine packages are far less fine-grained.
Usually, one will add the following to the Dockerfile
# Install additional LaTeX packages
RUN tlmgr update --self && tlmgr install \
pgf # <list of packages goes here>
Alternatively, you could use the pandoc/core image and build your own LaTeX container using the Alpine system packages.

Separating Docker files and application source files to optimize production environment

I have a bunch of (Ruby) scripts stored on a server. Up until now, my team has used them by opening an accessor app that launches a list of the script names, and they select the script they want to run in that instance on the files in their working folder. The scripts are run directly from the server, so updates made to the script files are automatically reflected when a user runs the script.
The scripts require a fair amount of specific dependencies, so I'm trying to move to a Docker-based workflow to eliminate the problems we encounter with incongruent computer environments. I've been able to successfully build an image with our script library and run an instance of it on my computer.
However, all of the documentation and tutorials include the application source files when building an image, so that all the files are copied over by the Dockerfile. From my understanding, this means that any time the code in the application files needs to be updated, all the users will need to rebuild the image before trying to run anything. I would very rarely ever need to make changes to the environment settings/dependencies, but the app code is changed relatively frequently, so it seems like having every user rebuild an image every single time a line of app code is changed would actually slow down everyone's workflow considerably.
My question is this: Is it not possible to have Docker simply create the environment that a user must have to run the applications, but have the applications themselves still run directly off the server where they were originally stored? And does a new container need to be created every single time a user wants to run any one of the scripts? (The users are not tech-savvy.)
Generally you'd do this by using a Docker image instead of the checked-out tree of scripts. You can use a Docker registry to store a built copy of the image somewhere on the network; Docker Hub works for this, most large public-cloud providers have some version of this (AWS ECR, Google GCR, Azure ACR, ...), or you can run your own. The workflow for using this would generally look like
# Get any updates to the "latest" version of the image
# (can be run infrequently)
docker pull ourorg/scripts
# Actually run the script, injecting config files and credentials
docker run --rm \
-v $PWD/config:/config \
-v $HOME/.ssh:/config/.ssh \
ourorg/scripts \
some_script.rb
# Nothing in this example actually requires a local copy of the scripts
I'm envisioning a directory that has kind of a mix of scripts and support files and not a lot of organization to it. Still, you could write a simple Dockerfile that looks like
FROM ruby:2.7
WORKDIR /opt/scripts
# As of Bundler 2.1, there is no compatibility between Bundler
# versions; this must match exactly what is in Gemfile.lock
RUN gem install bundler -v 2.1.4
# Copy the scripts in and do basic installation
COPY Gemfile Gemfile.lock .
RUN bundle install
COPY . .
ENV PATH /opt/scripts:$PATH
# Prefix all commands with...
ENTRYPOINT ["bundle", "exec"]
# The default command to run is...
CMD ["ls"]
On the back end you'd need a continous integration service (Jenkins is popular if a little unwieldy; there are a large selection of cloud-hosted ones) that can rebuild the Docker image whenever there's a commit to the source repository. You can generally rig this up so that it happens automatically whenever anybody pushes anything.
This process makes more sense of most people are just using the set of scripts and few of them are developing them. It's also a little bit difficult to discover what the scripts are (you might be able to docker run --rm ourorg/scripts ls though).
Is it not possible to have Docker simply create the environment that a user must have to run the applications, but have the applications themselves still run directly off the server where they were originally stored?
This always strikes me as an ineffective use of Docker. You have all of the fiddly steps of your current workflow that require everyone to run a git pull or equivalent routinely, but you also have to inject the host source tree into the container. If there are OS incompatibilities in, for example, native gems in the vendor tree, you have to work around that.
# You still need to do this periodically
git pull
# And you also need to
sudo docker run \
--rm \
-v $PWD:/app \
-v $HOME/config:/config \
-v $HOME/.ssh:/config/.ssh \
-w /app \
ruby:2.7 \
bundle exec ./some_script.rb
Some of these details (especially the config file and credentials) you'd have to deal with even if you did build an image; some others of the details you could improve by building an image. Inside the image you need to correct the ownership and permissions on the ssh keys and replace the $PWD/vendor tree with something the container can run, without modifying the mounted host directories.
Is it not possible to have Docker simply create the environment that a user must have to run the applications, but have the applications themselves still run directly off the server where they were originally stored?
You can build an image with all the environment already installed then mount the directory with the scripts so the container can read the scripts from the host. Something like
docker run -it --rm -v /opt/myscripts:/myscripts myimage somescript.rb
Then your image Dockerfile would end with:
WORKDIR /myscripts
ENTRYPOINT ["/usr/bin/ruby"]
And does a new container need to be created every single time a user wants to run any one of the scripts?
Of course, a container is just an isolated process managed by docker, you could make a wrapper so the users wouldn't need to type the full docker run command.

Creating a dockerfile to compile source code

I am trying to follow the 2 steps mentioned below:
1) Downloaded source code of
https://sourceforge.net/projects/hunspell/files/Hyphen/2.8/hyphen-2.8.8.tar.gz/download
2) Compiled it and you will get binary named example:
hyphen-2.8.8$ ./example ~/dev/smc/hyphenation/hi_IN/hyph_hi_IN.dic
~/hi_sample.text
I have downloaded and uncompressed the tar file. My question is how to create a dockerfile to automate this?
There are only 3 commands involved:
./configure
make all-recursive
make install
I can select the official python image as a base container. But how do I write the commands in a docker file?
You can do that with a RUN command:
FROM python:<version number here>
RUN ./configure && make-recursive && make install
CMD ['<some command here>']
what you use for <some command here> depends on what the image is meant to do. Remember that docker containers only run as long as that command is executing, so if you put the configure/make/install steps in a script and use that as your entry point, it's going to build your program, and then the container will halt.
Also you need to get the downloaded files into the container. That can be done using a COPY or an ADD directive (before the RUN of course). If you have the tar.gz file saved locally, then ADD will both copy the file into the container and expand it into a directory automatically. COPY will not expand it, so if you do that, you'll need to add a tar -zxvf or similar to the RUN.
If you want to download the file directly into the container, that could be done with ADD <source URL>, but in that case it won't expand it, so you'll have to do that in the RUN. COPY doesn't allow sourcing from a URL. This post explains COPY vs ADD in more detail.
You can have the three commands in a shell script and then use the following docker commands
COPY ./<path to your script>/<script-name>.sh /
ENTRYPOINT ["/<script-name>.sh"]
CMD ["run"]
For reference, you can create your docker file as they have created for one of the projects I worked on Apache Artemis Active Mq:
https://github.com/apache/activemq-artemis/blob/master/artemis-docker/Dockerfile-ubuntu

envsubst command getting stuck in a container

I have a requirement that before an application runs, some part of it needs to read the environmental variable. For this I have the following docker file
FROM nodesource/jessie:0.12.7
# install gettext for envsubst
RUN apt-get update
RUN apt-get install -y gettext-base
# cache package.json and node_modules to speed up builds
ADD package.json package.json
RUN npm install
# Add source files
ADD src src
# Substiture value for backend endpoint env var
RUN envsubst < src/js/envapp.js > src/js/app.js
ADD node_modules node_modules
EXPOSE 8000
CMD ["npm","start"]
The above envsubst line reads (should read) an env variable $MYENV and substitutes it. But when I open the file app.js, its empty.
I checked if the environmental variable exists in the container and it does. Any reason its value is not read and substituted?
I also tried the same command in teh container and it works. It only does not work when I run the image
This is likely because $MYENV is not available for envsubst when you run the image.
Each RUN command runs on its own shell.
From the Docker documentations:
RUN (the command is run in a shell - /bin/sh -c - shell form)
You need to source your profile as well, for example if the $MYENV environment variable is available in the .bashrc file, you can modify your Dockerfile like this:
RUN source ~/.bashrc && envsubst < src/js/envapp.js > src/js/app.js
I encountered the same issues, and after much research and fishing through the internet. I managed to find a few work arounds to this issue. Below I'll list them and identifiable risks at the time of this "Answer post"
Solutions:
1.) apt-get install -y gettext its a standard GNU package language library, one of these libraries that it includes is envsubst` and I can confirm that it works for docker UBUNTU:latest and it should work for every flavored version.
2.) npm install envsub dependent on the "use case" - this approach would be better supported by node based projects.
3.) enstub cli project lib in my opinion it seems a bit overkill to downloading a custom cli from a random stranger but, it's also another option.
Risk:
apt-get install -y gettext:
1.) gettext - this approach would NOT be ideal for VM's as with any package library, it requires maintenance and updates as time passes. However, this isn't necessary for docker because once an a container is initialized and up and running we can create a bashscript to add the package, substitute env vars and then remove the package.
2.) It's a bad idea for VM's because it can be used to execute arbitrary code
npm install ensub
1.) envsub - updating packages and this approach wouldn't be ideal if your dealing with a different stack and not using nodejs.
NOTE:
There's also a PHP version for those developing a PHP application and it seems to work PHP's cli if you need a custom environment.
Resources:
GetText package library info: https://www.gnu.org/software/gettext/
GetText Risk - https://ubuntu.com/security/notices/USN-3815-2
PHP-GetText - apt-get install -y php-gettext
Custom ensubst cli: https://github.com/a8m/envsubst
I suggest that since your are using Node, you use the npm envsub module.
This module is well tested and is developed with docker in mind.
It avoids the need for relying on other dependencies when you already have the full Node arsenal at your fingertips.
envsub is described as
envsub is envsubst for NodeJS
NodeJS global CLI module providing file-level environment variable substitution via Handlebars
I am the author of the package. I think you will enjoy it.
I had some issues with envsubst in Docker.
For some reasons envsubst doesn't work when I try to copy the output in the same file. For example, this is not working:
RUN envsubst < file.conf > file.conf
But when I when I tried to use a temp file the issue disappeared:
RUN envsubst < file.conf > file.conf.temp && cp -f file.conf.temp file.conf

Resources