I have a multi module SBT project that I'm using to build a Docker image out of. There is one module that depends on all the other modules and I'm actually trying to build the Docker image for this module. Here is a snippet from my build.sbt
lazy val impute = (project in file(MODULE_NAME_IMPUTE)).dependsOn(core% "compile->compile;test->test", config)
.settings(
commonSettings,
enablingCoverageSettings,
name := MODULE_NAME_IMPUTE,
description := "Impute the training data"
)
.enablePlugins(JavaAppPackaging, DockerPlugin)
lazy val split = (project in file(MODULE_NAME_SPLIT)).dependsOn(core% "compile->compile;test->test", config)
.settings(
commonSettings,
enablingCoverageSettings,
dockerSettings("split"),
name := MODULE_NAME_SPLIT,
description := "Split the dataset into train and test"
)
.enablePlugins(JavaAppPackaging, DockerPlugin)
lazy val run = (project in file(MODULE_NAME_RUN)).dependsOn(core % "compile->compile;test->test", config, cleanse, encode, feature, impute, split)
.settings(
commonSettings,
dockerSettings("run"),
enablingCoverageSettings,
name := MODULE_NAME_RUN,
description := "To run the whole setup as a pipeline locally"
)
.enablePlugins(JavaAppPackaging, DockerPlugin)
As you can see, the run module depends on all the other modules and I'm actually trying to build the Docker image for the run module for which I used the following command:
sbt run/docker:publishLocal
This works fine and the Docker image is also built, but when I inspect my Docker image, especially on the ENTRYPOINT, I get to see the following:
"Entrypoint": [
"/opt/docker/bin/run"
],
But instead, I would have expected to see something like this:
"Entrypoint": [
"java",
"-cp",
"com.mypackage.housingml.run.Main"
],
Is there anything else that I'm missing? Here is my dockerSettings() function from my build.sbt:
def dockerSettings(name: String) = {
Seq(
// Always use latest tag
dockerUpdateLatest := true,
maintainer := s"$projectMaintainer",
// https://hub.docker.com/r/adoptopenjdk/openjdk13
// Remember to use AshScriptPlugin if you are using an alpine based image
dockerBaseImage := "adoptopenjdk/openjdk13:alpine-slim",
// If you want to publish to a remote docker repository, uncomment the following:
//dockerRepository := Some("remote-docker-hostname"),
Docker / packageName := s"joesan/$projectName-$name",
// If we're running in a docker container, then export logging volume.
Docker / defaultLinuxLogsLocation := "/opt/docker/logs",
dockerExposedVolumes := Seq((Docker / defaultLinuxLogsLocation).value),
dockerEnvVars := Map(
"LOG_DIR" -> (Docker / defaultLinuxLogsLocation).value,
)
)
}
The sbt-native-packager creates a executable bash script to run your code and Docker ENTRYPOINT is by default set to that executable script as you can see.
This is customisable, see the docs for reference.
Also, the bash script is customisable - docs.
Related
I want to run KubernetesPodOperator in Airflow that reads some file and send the content to XCOM.
Definition looks like:
read_file = DefaultKubernetesPodOperator(
image = 'alpine:3.16',
cmds = ['bash', '-cx'],
arguments = ['cat file.json >> /airflow/xcom/return.json'],
name = 'some-name',
task_id = 'some_name',
do_xcom_push = True,
image_pull_policy = 'IfNotPresent',
)
but I am getting: INFO - stderr from command: cat: can't open '/***/xcom/return.json': No such file or directory
When I use ubuntu:22.04 it works, but I want it make faster by using smaller (Alpine) image. Why it is not working with alpine and how to overcome that?
I'd like to slim down a debian 10 Docker image using Bazel and then flatten the result into a single layer image.
Here's the code I have:
load("#io_bazel_rules_docker//container:container.bzl", "container_image", "container_flatten", "container_bundle", "container_import")
load("#io_bazel_rules_docker//docker/util:run.bzl", "container_run_and_commit")
container_run_and_commit(
name = "debian10_slimmed_layers",
commands = ["rm -rf /usr/share/man/*"],
image = "#debian10//image",
)
# Create an image just so we can flatten it.
container_image(
name = "debian10_slimmed_image",
base = ":debian10_slimmed_layers",
)
# Flatten the layers to a single layer.
container_flatten(
name = "debian10_flatten",
image = ":debian10_slimmed_image",
)
Where I'm stuck is I can't figure out how to use the output of debian10_flatten to produce a runnable Docker image.
I tried:
container_image(
name = "debian10",
base = ":debian10_flatten",
)
That fails with:
2021/06/27 13:16:25 Unable to load docker image from bazel-out/k8-fastbuild/bin/debian10_flatten.tar:
file manifest.json not found in tar
container_flatten gives you the filesystem tarball. You need to add the tarball as tars in debian10, instead of deps:
container_image(
name = "debian10",
tars = [":debian10_flatten.tar"],
)
deps is for another container_image rule (or equivalent). If you had a docker save-style tarball, container_load would be the way to get the container_image equivalent.
I figured this out looking at the implementation in container/flatten.bzl. The docs could definitely use some improvements if somebody wants to open a PR (they're generated from the python-style docstring in that .bzl file).
I am using the Go exec package to execute a docker pull debian command:
import (
"bufio"
"os/exec"
"strings"
)
func main() {
cmd := exec.Command("docker", "pull", "debian")
stdout, _ := cmd.StdoutPipe()
cmd.Start()
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
return nil
}
But it never shows me the progress bar. It only shows an update when it is fully complete. For larger images over a GB it is hard to see if there is progess being made. This is what it shows:
e9afc4f90ab0: Pulling fs layer
e9afc4f90ab0: Verifying Checksum
e9afc4f90ab0: Download complete
e9afc4f90ab0: Pull complete
Is it possible to get output similar to what I see when I run docker pull debian in the terminal or something that I can use to show progress?:
e9afc4f90ab0: Downloading [==========> ] 10.73MB/50.39MB
As David mentionned, you would rather use the official docker engine SDK to interact with docker.
Initialize the docker client
cli, _ := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
Pull the image
reader, _ := cli.ImagePull(context.Background(), "hello-world", types.ImagePullOptions{})
Parse the json stream
id, isTerm := term.GetFdInfo(os.Stdout)
_ = jsonmessage.DisplayJSONMessagesStream(reader, os.Stdout, id, isTerm, nil)
You will get the same output as the docker cli provide when you do a docker pull hello-world
I am trying to test docker and go project. Here is my dockerfile
FROM golang
ARG app_env
ENV APP_ENV $app_env
COPY ./ /go/src/github.com/user/myProject/app
WORKDIR /go/src/github.com/user/myProject/app
RUN go get ./
RUN go build
CMD if [ ${APP_ENV} = production ]; \
then \
app; \
else \
go get github.com/pilu/fresh && \
fresh; \
fi
EXPOSE 8080
It runs fine. Then i added a package "testpack" to my go program.
package main
import(
"fmt"
"time"
"testpack"
)
var now = time.Now()
var election = time.Date(2016, time.November, 8, 0, 0, 0, 0, time.UTC)
func main() {
//get duration between election date and now
tillElection := election.Sub(now)
//get duration in nanoseconds
toNanoseconds := tillElection.Nanoseconds()
//calculate hours from toNanoseconds
hours := toNanoseconds/3600000000000
remainder := toNanoseconds%3600000000000
//derive minutes from remainder of hours
minutes := remainder/60000000000
remainder = remainder%60000000000
//derive seconds from remainder of minutes
seconds := remainder/1000000000
//calculate days and get hours left from remainder
days := hours/24
hoursLeft := hours%24
fmt.Printf("\nHow long until the 2016 U.S. Presidential election?\n\n%v Days %v Hours %v Minutes %v Seconds\n\n", days, hoursLeft, minutes, seconds)
}
Now i ran=> docker build ./
I am getting an error
package testpack: unrecognized import path "testpack" (import path does not begin with hostname)
I tried this Error 'import path does not begin with hostname' when building docker with local package but couldn't resolve
Any help is appreciated.
It is obviously trying to load it from the Internet because it isn't finding "testpack" in your GOPATH.
You didn't show us your GOPATH setting or where you copied "testpack" to, so other than saying "It's missing" that's all I can tell you.
Read https://golang.org/cmd/go/#hdr-Relative_import_paths
Try either
import "./testpack"
Set GOPATH to "/go" in your Dockerfile
import "github.com/user/myProject/app/testpack"
It sounds an awful lot like you're having a problem building your app inside the docker build process. This is likely a dependency issue (you have a dependency installed on your local $GOPATH that is not installed inside the image's go environment). You could install the dependency before the build command in the Dockerfile, but I would pretty seriously consider building the app outside of the Dockerfile, and copying the executable into the image on build.
One of the biggest advantages of Golang is the statically compiled executables. Once it is compiled, you should be able to run it in any equivalent architecture. By default, go will try to compile a static executable but if you would like to enforce it (and you are not doing anything fancy with CGO) you can build with the CGO_ENABLED env var set to 0 like so: CGO_ENABLED=0 go build -o <output.name> <file.to.build>
At this point, your Dockerfile becomes much simpler (and SMALLER, check the image size of the resulting images), something like:
FROM scratch
#copy executable into container
ADD <output.name> <output.name>
#set entrypoint
ENTRYPOINT [./<output.name>]
#set exposed port
EXPOSE 8080
This should solve your dependency issue, and make your runtime container much smaller (probably < 20MB), which will decrease build times and increase deployment speed.
I am trying to build a Docker image on a Play 2.2 project. I am using Docker version 1.2.0 on Ubuntu Linux.
My Docker specific settings in Build.scala looks like this:
dockerBaseImage in Docker := "dockerfile/java:7"
maintainer in Docker := "My name"
dockerExposedPorts in Docker := Seq(9000, 9443)
dockerExposedVolumes in Docker := Seq("/opt/docker/logs")
Generated Dockerfile:
FROM dockerfile/java:latest
MAINTAINER
ADD files /
WORKDIR /opt/docker
RUN ["chown", "-R", "daemon", "."]
USER daemon
ENTRYPOINT ["bin/device-guides"]
CMD []
Output looks like the dockerBaseImage is being ignored, and the default
(dockerfile/java:latest) is not handled correctly:
[project] $ docker:publishLocal
[info] Wrote /..../project.pom
[info] Step 0 : FROM dockerfile/java:latest
[info] ---> bf7307ff060a
[info] Step 1 : MAINTAINER
[error] 2014/10/07 11:30:12 Invalid Dockerfile format
[trace] Stack trace suppressed: run last docker:publishLocal for the full output.
[error] (docker:publishLocal) Nonzero exit value: 1
[error] Total time: 2 s, completed Oct 7, 2014 11:30:12 AM
[project] $ run last docker:publishLocal
java.lang.RuntimeException: Invalid port argument: last
at scala.sys.package$.error(package.scala:27)
at play.PlayRun$class.play$PlayRun$$parsePort(PlayRun.scala:52)
at play.PlayRun$$anonfun$play$PlayRun$$filterArgs$2.apply(PlayRun.scala:69)
at play.PlayRun$$anonfun$play$PlayRun$$filterArgs$2.apply(PlayRun.scala:69)
at scala.Option.map(Option.scala:145)
at play.PlayRun$class.play$PlayRun$$filterArgs(PlayRun.scala:69)
at play.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$1.apply(PlayRun.scala:97)
at play.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$1.apply(PlayRun.scala:91)
at scala.Function7$$anonfun$tupled$1.apply(Function7.scala:35)
at scala.Function7$$anonfun$tupled$1.apply(Function7.scala:34)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Invalid port argument: last
[error] Total time: 0 s, completed Oct 7, 2014 11:30:16 AM
What needs to be done to make this work?
I am able to build the image using Docker from the command line:
docker build --force-rm -t device-guides:1.0-SNAPSHOT .
Packaging/publishing settings are per-project settings, rather than per-build settings.
You were using a Build.scala style build, with a format like this:
object ApplicationBuild extends Build {
val main = play.Project(appName, appVersion, libraryDependencies).settings(
...
)
}
The settings should be applied to this main project. This means that you call the settings() method on the project, passing in the appropriate settings to set up the packaging as you wish.
In this case:
object ApplicationBuild extends Build {
val main = play.Project(appName, appVersion, libraryDependencies).settings(
dockerBaseImage in Docker := "dockerfile/java:7",
maintainer in Docker := "My name",
dockerExposedPorts in Docker := Seq(9000, 9443),
dockerExposedVolumes in Docker := Seq("/opt/docker/logs")
)
}
To reuse similar settings across multiple projects, you can either create a val of type Seq[sbt.Setting], or extend sbt.Project to provide the common settings. See http://jsuereth.com/scala/2013/06/11/effective-sbt.html for some examples of how to do this (e.g. Rule #4).
This placement of settings is not necessarily clear if one is used to using build.sbt-type builds instead, because in that file, a line that evaluates to an SBT setting (or sequence of settings) is automatically appended to the root project's settings.
It's a wrong command you executed. I didn't saw it the first time.
run last docker:publishLocal
remove the run last
docker:publishLocal
Now you get your docker image build as expected