I would like to configure a CI such as TravisCI to build my application from Docker. My application has two part: Javascript and Python.
I thought to use docker-compose to do this:
version: '3'
services:
node:
image: node:12.8.0-buster
volumes:
- .:/srv
python:
image: python:3.7.4-buster
volumes:
- .:/src
I would like to have a Makefile such as:
all: foo bar
foo:
docker-compose exec node /bin/bash -c ' \
cd /workdir; \
npm install; \
npm run build'
bar:
docker-compose exec python /bin/bash -c ' \
cd /workdir; \
pip install sphinx; \
make html'
Is this correct to use docker compose like this? And what should I change to make it work?
docker compose not only support container run, but also image build, see this.
So, for your scenario, you should add your package build in Dockerfile and exeucte it with docker-compose up -d --build which will first build out a docker image then start the service base on the new docker image.
A simple fake code is as next, note next is just to explain the main idea, not a fully workable example, you need to add your stuff base on your real situation.
Dockerfile.node:
FROM node:12.8.0-buster
# Add related to build
ADD . /srv
# Add all package install
RUN cd /workdir && npm install && npm run build
# Others
......
Dockerfile.python:
FROM python:3.7.4-buster
# Add related to build
ADD . /srv
# Add all package install
RUN cd /workdir && pip install sphinx && make html
# Others
......
docker-compose.yaml:
version: '3'
services:
node:
build:
context: .
dockerfile: Dockerfile.node
volumes:
- .:/srv
python:
build:
context: .
dockerfile: Dockerfile.python
volumes:
- .:/src
Related
I'm new in docker and I want to setting-up a docker-compose for my django app. in the backend of my app, I have golang packages too and run that in djang with subprocess library.
But, when I want to install a package using go install github.com/x/y#latest and then copy its binary to the project directory, it gives me the error: package github.com/x/y#latest: cannot use path#version syntax in GOPATH mode
I searched a lot in the internet but didn't find a solution to solve my problem. Could you please tell me where I'm wrong?
here is my Dockerfile:
FROM golang:1.18.1-bullseye as go-build
# Install go package
RUN go install github.com/hakluke/hakrawler#latest \
&& cp $GOPATH/bin/hakrawler /usr/local/bin/
# Install main image for backend
FROM python:3.8.11-bullseye
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Install Dist packages
RUN apt-get update \
&& apt-get -y install --no-install-recommends software-properties-common libpq5 python3-dev musl-dev git netcat-traditional golang \
&& rm -rf /var/lib/apt/lists/
# Set work directory
WORKDIR /usr/src/redteam_toolkit/
# Install dependencies
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
# Copy project, and then the go package
COPY . .
COPY --from=go-build /usr/local/bin/hakrawler /usr/src/redteam_toolkit/toolkit/scripts/webapp/
docker-compose.yml:
version: '3.3'
services:
webapp:
build: .
command: python manage.py runserver 0.0.0.0:4334
container_name: toolkit_webapp
volumes:
- .:/usr/src/redteam_toolkit/
ports:
- 4334:4334
env_file:
- ./.env
depends_on:
- db
db:
image: postgres:13.4-bullseye
container_name: database
volumes:
- postgres_data:/var/lib/postgresql/data/
environment:
- POSTGRES_USER=user
- POSTGRES_PASSWORD=password
- POSTGRES_DB=redteam_toolkit_db
volumes:
postgres_data:
the get.py file inside /usr/src/redteam_toolkit/toolkit/scripts/webapp/ directory, to just run the go package, and list files in this dir:
import os
import subprocess
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
print(f"Current path is: {BASE_DIR}")
def go(target_url):
run_go_package = subprocess.getoutput(
f"echo {target_url} | {BASE_DIR}/webapp/hakrawler -t 15 -u"
)
list_files = subprocess.getoutput(f"ls {BASE_DIR}/webapp/")
print(run_go_package)
print(list_files)
go("https://example.org")
and then I just run:
$ docker-compose up -d --build
$ docker-compose exec webapp python toolkit/scripts/webapp/get.py
The output is:
Current path is: /usr/src/redteam_toolkit/toolkit/scripts
/bin/sh: 1: /usr/src/redteam_toolkit/toolkit/scripts/webap/hakrawler: not found
__init__.py
__pycache__
scr.py
gather.py
This looks like a really good candidate for a multi-stage build:
FROM golang:1.18.0 as go-build
# Install packages
RUN go install github.com/x/y#latest \
&& cp $GOPATH/bin/pacakge /usr/local/bin/
FROM python:3.8.11-bullseye as release
...
COPY --from=go-build /usr/local/bin/package /usr/src/toolkit/toolkit/scripts/webapp/
...
Your compose file also needs to be updated, it is masking the entire /usr/src/redteam_toolkit folder with the volume mount. Delete that volume mount to see the content of the image.
GOPATH mode does not work with Golang modules, in your Dockerfile file, add:
RUN unset GOPATH
use RUN go get <package_repository>
I'm facing an issue, am trying to run my go fiber project inside docker with air but am getting this error
uni-blog | /bin/sh: 1: /app/tmpmain.exe: not found
am using
Windows 11
Docker desktop
golang latest
air 1.27.10
fiber latest
Here is my docker compose and dockerfile
# docker-compose up -d --build
version: "3.8"
services:
app:
container_name: uni-blog
image: app-dev
build:
context: .
target: development
volumes:
- ./:/app
ports:
- 3000:3000
FROM golang:1.17 as development
RUN apt update && apt upgrade -y && \
apt install -y git \
make openssh-client
RUN curl -fLo install.sh https://raw.githubusercontent.com/cosmtrek/air/master/install.sh \
&& chmod +x install.sh && sh install.sh && cp ./bin/air /bin/air
RUN air -v
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
EXPOSE 3000
CMD air
I also tried installing air following the readME instructions still it gives me this error
Please help
Thanks in advance
The volumes: mount you have replaces the /app directory in the image with content from the host. If the binary is built in the Dockerfile, that volumes: mount hides it; if you don't have a matching compatible binary on the host in the same place, you'll get an error like what you see.
I'd remove that volumes: block so you're actually running the binary that's built into the image. The docker-compose.yml file can be reduced to as little as:
version: '3.8'
services:
app:
build: .
ports:
- '3000:3000'
If you look the error, you ca notice there is a typo between tmp/main.exe:
/bin/sh: 1: /app/tmpmain.exe: not found
This is coming from .air.toml config file:
bin = "tmp\\main.exe"
Create .air.toml file in project root like so:
root = "."
tmp_dir = "tmp"
[build]
# Build binary.
cmd = "go build -o ./tmp/main.exe ."
# Read binary.
bin = "tmp/main.exe"
# Watch changes in those files
include_ext = [ "go", "yml"]
# Ignore changes in these files
exclude_dir = ["tmp"]
# Stop builds from triggering too fast
delay = 1000 # ms
[misc]
clean_on_exit = true
I have a Dockerfile for a Django and Vue.js app that I use along with Gitlab.
The problem that I'm about to describe only happens when deploying via Gitlab CI and the corresponding .gitlab-ci.yml file. When running the docker-compose up command in my local machine, this doesn happen.
So I run docker-compose up and all the instructions in the Dockerfile run apparently fine. But when I check the production server, the dist folder (where the bundle.js and bundle.css should be stored) doesn't exist.
The logs that are spit out while running the Dockerfile confirm that the npm install and npm run build commands are run, and it even confirms that the dist/bundle.js and dist/bundle.css files have been generated. But for some reason they seem to be deleted.
This is my Dockerfile:
FROM python:3.7-alpine
MAINTAINER My Name
ENV PYTHONUNBUFFERED 1
RUN mkdir /app
# make the 'app' folder the current working directory
WORKDIR /app
# copy project files and folders to the current working directory (i.e. 'app' folder)
COPY ./app .
COPY ./requirements.txt /requirements.txt
RUN apk add --update --no-cache postgresql-client
RUN apk add --update --no-cache --virtual .tmp-build-deps \
gcc libc-dev linux-headers postgresql-dev
RUN pip install -r /requirements.txt
RUN apk del .tmp-build-deps
# copy both 'package.json' and 'package-lock.json' (if available)
COPY app/frontend/package*.json ./frontend/
# Install npm
RUN apk add --update nodejs && apk add --update nodejs-npm
# install project dependencies
WORKDIR /app/frontend
RUN npm install
# build app for production with minification
RUN npm run build
RUN adduser -D user
USER user
CMD ["sh ../scripts/entrypoint.sh"]
This is the .gitlab-ci.yml file:
image: docker:latest
services:
- docker:dind
before_script:
- echo "Runnig before_script"
- sudo apt-get install -y python-pip
- sudo apt-get install -y nodejs
- pip install docker-compose
stages:
- test
- build
- deploy
test:
stage: test
script:
- echo "Testing the app"
- docker-compose run app sh -c "python /app/manage.py test && flake8"
build:
stage: build
only:
- develop
- production
- feature/gitlab_ci
script:
- echo "Building the app"
- docker-compose build
deploy:
stage: deploy
only:
- master
- develop
- feature/gitlab_ci
script:
- echo "Deploying the app"
- docker-compose up --build -d
This is the content of the docker-compose.yml file:
version: "3"
services:
app:
build:
context: .
ports:
- "8000:8000"
volumes:
- ./app:/app
command: >
sh -c "python /app/manage.py runserver 0.0.0.0:8000"
environment:
- DB_HOST=db
- DB_NAME=app
- DB_USER=postgres
- DB_PASS=postgres
depends_on:
- db
db:
image: postgres:10-alpine
environment:
- POSTGRES_DB=app
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
This is the content of the entrypoint.sh file:
#!/bin/bash
(cd .. && ./manage.py collectstatic --noinput)
# Migration files are commited to git. Makemigrations is not needed.
# ./manage.py makemigrations app_name
(cd .. && ./manage.py migrate)
I would like to know why the dist/ folder disappears and how to keep it.
When your docker-compose.yml file says
volumes:
- ./app:/app
that hides everything that your Dockerfile builds in the /app directory and replaces it with whatever's in your local system. If your host doesn't have a ./app/frontend/dist then your container won't have that path either, regardless of whatever the Dockerfile does.
I would generally recommend just deleting this volumes: block entirely. It introduces an awkward live-development path (where all of your tooling needs to know that the actual service runs in Docker) and simultaneously isn't what you'd run in development (you want the image to be self-contained and not to need to copy the application separately from the image).
In your compose file, you set a volume which is going to replace your local environment with the one in your container even after npm run build
volumes:
- ./app:/app
You can either build it in your local or remove volumes
We had a similar issue with a nestjs build. Lately we noticed, that we had excluded the src file in the .dockerignore.
Issue is not with docker file. It issue with your dependency. please check package.json file in root folder.
I added RUN go get to install packages during "docker-compose". However, the following cannot find package error was occurred when I run go build. I found that the packages are saved in /go/pkg/linux_amd64/.
run docker-compose and go build
$ docker-compose up -d
$ docker exec -it explorer-cli /bin/bash
# pwd
/go
# ls
bin pkg src
# echo $GOPATH
/go
# ls /go/pkg/linux_amd64/github.com/
go-sql-driver
# go build -i -o /go/bin/explorer-cli src/main.go
src/main.go:6:2: cannot find package "github.com/go-sql-driver/mysql" in any of:
/usr/local/go/src/github.com/go-sql-driver/mysql (from $GOROOT)
/go/src/github.com/go-sql-driver/mysql (from $GOPATH)
(it worked if I run "go get" manually)
# go get github.com/go-sql-driver/mysql
# ls src/
github.com main.go
# go build -i -o /go/bin/explorer-cli src/main.go
docker-compose.yml
version: '3.4'
services:
mysql:
image: mysql:latest
container_name: database
volumes:
- ./docker/:/etc/mysql/conf.d
- ./docker/:/docker-entrypoint-initdb.d
environment:
- MYSQL_RANDOM_ROOT_PASSWORD=true
- MYSQL_DATABASE=explorer
- MYSQL_USER=admin
- MYSQL_PASSWORD=12dlql*41
app:
build: .
tty: true
image: explorer-cli:latest
container_name: explorer-cli
volumes:
- ./src:/go/src
external_links:
- database
Dockerfile
FROM golang:latest
RUN apt-get update
RUN apt-get upgrade -y
ENV GOBIN /go/bin
RUN go get github.com/go-sql-driver/mysql
main.go
package main
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
func main() {
db, err := sql.Open("mysql", "XUSER:XXXX#(database:3306)/explorer")
if err != nil {
panic(err.Error())
}
defer db.Close()
}
Update 1
I noticed big differences between the following directories.
# ls /go/pkg/linux_amd64/github.com/go-sql-driver/
mysql.a
# ls /go/src/github.com/go-sql-driver/mysql/
AUTHORS connection_go18_test.go packets.go
CHANGELOG.md connection_test.go packets_test.go
CONTRIBUTING.md const.go result.go
LICENSE driver.go rows.go
README.md driver_go18_test.go statement.go
appengine.go driver_test.go statement_test.go
benchmark_go18_test.go dsn.go transaction.go
benchmark_test.go dsn_test.go utils.go
buffer.go errors.go utils_go17.go
collations.go errors_test.go utils_go18.go
connection.go fields.go utils_go18_test.go
connection_go18.go infile.go utils_test.go
Update 2
As #aerokite said, the "volumes" was overwriting the downloaded packages. I changed like the followings and it worked.
Dockerfile
version: '3.4'
FROM golang:latest
RUN apt-get update
RUN apt-get upgrade -y
ENV GOBIN /go/bin
RUN go get github.com/go-sql-driver/mysql
RUN mkdir /go/src/explorer-cli
docker-compose
services:
mysql:
image: mysql:latest
container_name: database
volumes:
- ./docker/:/etc/mysql/conf.d
- ./docker/:/docker-entrypoint-initdb.d
environment:
- MYSQL_RANDOM_ROOT_PASSWORD=true
- MYSQL_DATABASE=explorer
- MYSQL_USER=XUSER
- MYSQL_PASSWORD=XXXX
app:
build: .
tty: true
image: explorer-cli:latest
container_name: explorer-cli
volumes:
- ./src/explorer-cli:/go/src/explorer-cli
external_links:
- database
go build
go build -i -o /go/bin/explorer-cli src/explorer-cli/main.go
I have tried to recreate your problem.
FROM golang:latest
RUN apt-get update
RUN apt-get upgrade -y
ENV GOBIN /go/bin
RUN go get github.com/go-sql-driver/mysql
You have provided this Dockerfile. I have build it
$ docker build -t test .
Now I exec into this image to run your go build command.
$ docker run -it test bash
Then I have created main.go, you provided, in /go/src directory.
And finally, I have built successfully without any error
$ go build -i -o /go/bin/explorer-cli src/main.go
And I think I have found your problem. I have never used docker-compose. But you will understand.
Problem is here:
app:
build: .
tty: true
image: explorer-cli:latest
container_name: explorer-cli
volumes:
- ./src:/go/src <-- problem is here
external_links:
- database
You are mounting ./src into /go/src directory in your docker. This process is overwriting directory /go/src in your docker with your local ./src. And this is removing data you got from go get github.com/go-sql-driver/mysql
Do you understand?
But when you are running go get github.com/go-sql-driver/mysql, its now getting data again.
Solution (01):
Mount your local volume into somewhere else.
volumes:
- ./src:/tmp/src
And modify your Dockerfile to move this main.go to /go/src
Solution (02):
Copy main.go into your docker. Add this line in Dockerfile
COPY ./src/main.go /go/src
Hi there I am new to Docker. I have an docker-compose.yml which looks like this:
version: "3"
services:
lmm-website:
image: lmm/lamp:php${PHP_VERSION:-71}
container_name: ${CONTAINER_NAME:-lmm-website}
environment:
HOME: /home/user
command: supervisord -n
volumes:
- ..:/builds/lmm/website
- db_website:/var/lib/mysql
ports:
- 8765:80
- 12121:443
- 3309:3306
networks:
- ntw
volumes:
db_website:
networks:
ntw:
I want to install the Yarn package manager from within the docker-compose file:
sudo apt-get update && sudo apt-get install yarn
I could not figure out how to declare this, I have tried
command: supervisord -n && sudo apt-get update && sudo apt-get install yarn
which fails silently. How do I declare this correctly? Or is docker-compose.yml the wrong place for this?
Why not use Dockerfile which is specifically designed for this task?
Change your "image" property to "build" property to link a Dockerfile.
Your docker-compose.yml would look like this:
services:
lmm-website:
build:
context: .
dockerfile: Dockerfile
container_name: ${CONTAINER_NAME:-lmm-website}
environment:
HOME: /home/user
command: supervisord -n
volumes:
- ..:/builds/lmm/website
- db_website:/var/lib/mysql
ports:
- 8765:80
- 12121:443
- 3309:3306
networks:
- ntw
volumes:
db_website:
networks:
Then create a text file named Dockerfile in the same path as docker-compose.yml with the following content:
FROM lmm/lamp:php${PHP_VERSION:-71}
RUN apt-get update && apt-get install -y bash
You can add as much SO commands as you want using Dockerfile's RUN (cp, mv, ls, bash...), apart from other Dockerfile capabilities like ADD, COPY, etc.
+info:
https://docs.docker.com/engine/reference/builder/
+live-example:
I made a github project called hello-docker-react. As it's name says is a docker-react box, and can serve you as an example as I am installing yarn plus other tools using the procedure I explained above.
In addition to that, I also start yarn using an entrypoint bash script linked to docker-compose.yml file using docker-compose entrypoint property.
https://github.com/lopezator/hello-docker-react
You can only do it with a Dockerfile, because the command operator in docker-compose.yml only keeps the container alive during the time the command is executed, and then it stops.
Try this
command: supervisord -n && apt-get update && apt-get install yarn
Because sudo doesn't work in docker.
My first time trying to help out:
would like you to give it a try (I found it on the internet)
FROM lmm/lamp:php${PHP_VERSION:-71}
USER root
RUN apt-get update && apt-get install -y bash