I am migrating a Go 1.10 app to Go 1.11. This also includes migrating from dep to mod for managing dependencies.
As the application depends on a database, I am using a docker-compose to set up the local development environment. With Go 1.10 I simply mounted the local repository (including the vendor folder) into the correct location in the container's GOPATH:
web:
image: golang:1.10
working_dir: /go/src/github.com/me/my-project
volumes:
- .:/go/src/github.com/me/my-project
environment:
- GOPATH=/go
- PORT=9999
command: go run cmd/my-project/main.go
Since Go 1.11 ditches GOPATH (when using modules that is) I thought I could just do the following:
web:
image: golang:1.11rc2
working_dir: /app
volumes:
- .:/app
environment:
- PORT=9999
command: go run cmd/my-project/main.go
This works, but every time I docker-compose up (or any other command that calls through to the Go tool) it will resolve and re-download the dependency tree from scratch. This does not happen (rather only once) when I run the command outside of the container (i.e. on my local OS).
How can I improve the setup so that the Docker container persists the modules being downloaded by the go tool?
This is not mentioned in the wiki article on modules, but from reading the updated docs on the go tool, I found out that when using Go modules, the go tool will still use GOPATH to store the available sources, namely $GOPATH/pkg/mod.
This means that for my local dev setup, I can 1. define the GOPATH in the container and 2. mount the local $GOPATH/pkg/mod into the container's GOPATH.
web:
image: golang:1.11rc2
working_dir: /app
volumes:
- .:/app
- $GOPATH/pkg/mod:/go/pkg/mod
environment:
- GOPATH=/go
- PORT=9999
command: go run cmd/my-project/main.go
You can use a volume instead of your local GOPATH. the docker-compose.yml is like:
version: '3'
services:
web:
image: golang:1.11
working_dir: /app
volumes:
- .:/app
- cache:/go
environment:
- PORT=9999
command: go run cmd/my-project/main.go
volumes:
cache:
The volume cache is going to persist all the changes on the GOPATH for the container.
Related
I have docker-compose.yml on my local machine like below:
version: "3.3"
services:
api:
build: ./api
volumes:
- ./api:/api
ports:
- 3000:3000
links:
- mysql
depends_on:
- mysql
app:
build: ./app
volumes:
- ./app:/app
ports:
- 80:80
mysql:
image: mysql:8.0.27
volumes:
- ./mysql:/var/lib/mysql
tty: true
restart: always
environment:
MYSQL_DATABASE: db
MYSQL_ROOT_PASSWORD: qwerty
MYSQL_USER: db
MYSQL_PASSWORD: qwerty
ports:
- '3306:3306'
The api is NestJS app, app, mysql - Angular and Mysql respectively. And I need to work with this one localy.
How could I make so, that any my changes will be applied without rebuilding containers every time?
You don't have to build an image for a development environment with your sources in it. For NestJS, and since you're using Docker (I voluntary specify this because it exists other container runtimes), you can simply run a NodeJS image from the Docker main registry: https://hub.docker.com/_/node.
You could run it with:
docker run -d -v ./app:/app node:12-alpine /app/index.js
N.B.: I choose 12-alpine for the example. I imagine the file to start your app is index.js, replace it with yours.
You must consider to install the node dependencies yourself and they must be in the ./app directory.
For docker-compose, it could look like this:
version: "3.3"
services:
app:
image: node:12-alpine
command: /app/index.js
volumes:
- ./app:/app
ports:
- "80:80"
Same way for your API project.
For a production image, it is still suggested to build the image with the sources in it.
Say you're working on your front-end application (app). This needs to make calls out to the other components, especially api. So you can start the things it depends on, but not the application itself
docker-compose up -d api
Update your application configuration for this different environment; if you would have proxied to http://api:3000 before, for example, you need to change this to http://localhost:3000 to connect to the container's published ports:.
Now you can develop your application totally normally, without doing anything Docker-specific.
# outside Docker, on your normal development workstation
yarn run dev
$EDITOR src/components/Foo.tsx
You might find it convenient to use environment variables for these settings that will, well, differ per environment. If you're developing the back-end code but want to attach a live UI to it, you'll either need to rebuild the container or update the front-end's back-end URL to point at the host system.
This approach also means you do not need to bind-mount your application's code into the container, and I'd recommend removing those volumes: blocks.
I am currently writing a webapp - java backend, react front end and have been deploying via a docker compose file. I've made changes and when I try to run them via yarn build for my front end server and starting my back end server with maven, the changes appear. However, when running with docker, the changes aren't there.
I've been using the docker compose up and docker compose down commands and I even run docker system prune -a after stopping my docker containers via the docker compose down command but my new changes aren't showing. I'd appreciate any guidance on what I'm doing wrong to help show my changes.
I also have docker desktop and have manually gone and deleted all of the volumes, containers and images so that they have to be regenerated. Running the build commands to specify ignoring cache didn't help either.
I also deleted the .m2 folder so that this gets generated (my understanding is that this is the cache store for the backend). My changes are mainly on the front end but since my front end container depends on this, I thought regenerating the back-end container may have a knock on effect that may help.
I would greatly appreciate any help, please do let me know if there's anything else to help with context. The changes involve removing a search bar and some text, both of which are commented out in the code but still appear whilst I also add another button which doesn't show up.
My docker compose file is below as follows:
services:
mysqldb:
# image: mysql:5.7
build: ./Database
restart: unless-stopped
env_file: ./.env
environment:
- MYSQL_ROOT_PASSWORD=$MYSQLDB_ROOT_PASSWORD
- MYSQL_DATABASE=$MYSQLDB_DATABASE
ports:
- $MYSQLDB_LOCAL_PORT:$MYSQLDB_DOCKER_PORT
volumes:
- db:/var/lib/mysql
networks:
- backend
app_backend:
depends_on:
- mysqldb
build: ./
restart: on-failure
env_file: ./.env
ports:
- $SPRING_LOCAL_PORT:$SPRING_DOCKER_PORT
environment:
SPRING_APPLICATION_JSON: '{
"spring.datasource.url" : "jdbc:mysql://mysqldb:$MYSQLDB_DOCKER_PORT/$MYSQLDB_DATABASE?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC",
"spring.datasource.username" : "$MYSQLDB_USER",
"spring.datasource.password" : "$MYSQLDB_ROOT_PASSWORD",
"spring.jpa.properties.hibernate.dialect" : "org.hibernate.dialect.MySQL5InnoDBDialect",
"spring.jpa.hibernate.ddl-auto" : "update"
}'
volumes:
- .m2:/root/.m2
stdin_open: true
tty: true
networks:
- backend
- frontend
app_frontend:
depends_on:
- app_backend
build:
../MyProjectFrontEnd
restart: on-failure
ports:
- 80:80
networks:
- frontend
volumes:
db:
networks:
backend:
frontend:
Since the issue is on the front end, I've also attached the dockerfile for the front end below:
FROM node:16.13.0-alpine AS react-build
WORKDIR /MyProjectFrontEnd
RUN yarn cache clean
RUN yarn install
COPY . ./
RUN yarn
RUN yarn build
# Stage 2 - the production environment
FROM nginx:alpine
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY /build /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
Update - the cache in the browser was storing some (rookie error) however, not all the changes are still being loaded
If your source code is in the same folder (usually) of your Dockerfile, you can be sure that your last source code will be built and deployed. This feature is one of the cornerstones, which is the base of docker. If this would be failing, it would be the end of the world.
These kind of errors are not related to the docker core. Usually is something at application level and/or its development:
Libraries mistake
Developer mistake
Functional test mistake
Load Balancer mistake
Advice
docker-compose and windows are for development stage. For deployment on real environments for real users, you should use linux and some tool like Kubernetes.
I need to configure docker-compose.yml in a way that will invalidate the local image's docker cache, based on a certain file's checksum.
If it's not possible, I'd like to be able to somehow version the docker-compose.yml or Dockerfile, so that it would rebuild the Docker image of a specific service. I'd want to avoid pushing images to DockerHub. Unless it's an absolute the only solution.
At all costs, I want to avoid bash scripts and in general, writing imperative logic. I'm also not interested in CLI solutions, like passing additional flags to docker-compose up command.
Context:
We use docker-compose during the development of our application.
Our app has also a Dockerfile for building the app localy. We don't push docker images into DockerHub, we just have Dockerfile locally and in docker-compose.yml we declare the sourcecode and package.json (a file that for nodeJS applications use to declare dependencies) as volumes. Now sometimes, we modify the package.json, and docker-compose up throws an error, because the image is already built locally and the previous built doesn't contain the new dependencies. I'd want to be able to tell docker-compose.yml to automatically build a new image if there have been any changes to package.json file since we pull dependencies during the build stage.
docker-compose.yml
version: "3.8"
services:
web:
build:
context: .
ports:
- "8000:8000"
command: npx nodemon -L app.js
volumes:
- ./app:/usr/src/app
- /usr/src/app/node_modules
env_file:
- .env
depends_on:
- mongo
mongo:
image: mongo:latest
container_name: mongo_db
volumes:
- ./config/init.sh:/docker-entrypoint-initdb.d/init.sh
- ./config/mongod.conf:/etc/mongod.conf
- ./logs:/var/log/mongodb/
- ./db:/data/db
env_file:
- .env
ports:
- "27017:27017"
restart: on-failure:5
command: ["mongod", "-f", "/etc/mongod.conf"]
volumes:
db-data:
mongo-config:
Dockerfile:
FROM node:14.15.1
RUN mkdir -p /usr/src/app
# Create app directory
WORKDIR /usr/src/app
#Install app dependencies
COPY package.json /usr/src/app
# Install dependencies
RUN npm install
EXPOSE 8000
CMD ["node", "/app/app.js"]
I am running all of these operations on a remove server that is a
VM running Ubuntu 16.04.5 x64.
My Go project's Dockerfile looks like:
FROM golang:latest
ADD . $GOPATH/src/example.com/myapp
WORKDIR $GOPATH/src/example.com/myapp
RUN go build
#EXPOSE 80
#ENTRYPOINT $GOPATH/src/example.com/myapp/myapp
ENTRYPOINT ./myapp
#CMD ["./myapp"]
When I run the docker container using docker-compose up -d, the Go application exits and I see this in the docker logs:
myapp_1 | /bin/sh: 1: ./myapp: Exec format error docker_myapp_1
exited with code 2
If I locate the image using docker images and run the image like:
docker run -it 75d4a95ef5ec
I can see that my golang applications runs just fine:
viper environment is: development HTTP server listening on address:
":3005"
When I googled for this error some people suggested compiling with some special flags but I am running this container on the same Ubuntu host so I am really confused why this isn't working using docker.
My docker-compose.yml looks like:
version: "3"
services:
openresty:
build: ./openresty
ports:
- "80:80"
- "443:443"
depends_on:
- myapp
env_file:
- '.env'
restart: always
myapp:
build: ../myapp
volumes:
- /home/deploy/apps/myapp:/go/src/example.com/myapp
ports:
- "3005:3005"
depends_on:
- db
- redis
- memcached
env_file:
- '.env'
redis:
image: redis:alpine
ports:
- "6379:6379"
volumes:
- "/home/deploy/v/redis:/data"
restart: always
memcached:
image: memcached
ports:
- "11211:11211"
restart: always
db:
image: postgres:9.4
volumes:
- "/home/deploy/v/pgdata:/var/lib/postgresql/data"
restart: always
Your docker-compose.yml file says:
volumes:
- /home/deploy/apps/myapp:/go/src/example.com/myapp
which means your host system's source directory is mounted over, and hides, everything that the Dockerfile builds. ./myapp is the host's copy of the myapp executable and if something is different (maybe you have a MacOS or Windows host) that will cause this error.
This is a popular setup for interpreted languages where developers want to run their application without running a normal test-build-deploy sequence, but it doesn't really make sense for a compiled language like Go where you don't have a choice. I'd delete this block entirely.
The Go container stops running because of this:
WORKDIR $GOPATH/src/example.com/myapp
RUN go build
#EXPOSE 80
#ENTRYPOINT $GOPATH/src/example.com/myapp/myapp
ENTRYPOINT ./myapp
You are switching directories to $GOPATH/src/example.com/myapp where you build your app, however, your entry point is pointing to the wrong location.
To solve this, you either copy the app into the root directory and keep the same ENTRYPOINT command or you copy the application to a different location and pass the full path such as:
ENTRYPOINT /my/go/app/location
I am using a standard structure for my go application.
It is built like this:
cmd
app
main.go
internal
app
server.go
pkg
users
...
pkg
dependency
...
web
app
...
docker-compose.yml
Dockerfile
The problem however is that with this structure it's very hard to mount and build the application dependencies. For example, it I use a file watcher such as fresh, it only watches a single directory and runs a particular file. If I update say pkg/dependency, it will not see those changes.
docker-compose looks like:
version: "3.1"
services:
core:
build: .
depends_on:
- mongo
- memcached
ports:
- 8080:8080
environment:
APP_ENV: dev
volumes:
- .:/go/src/github.com/me/app
mongo:
image: mongo
ports:
- 27017:27017
memcached:
image: memcached
ports:
- 11211:11211
Dockerfile:
FROM golang:1.10.0
WORKDIR /go/src/github.com/me/app
COPY . .
RUN go get -u github.com/golang/dep/cmd/dep
RUN dep ensure
WORKDIR /go/src/github.com/me/app/cmd/app/
RUN go install
RUN go get github.com/pilu/fresh
CMD ["fresh"]
Any help?
I would push back to that fresh repo and ask them
If your file changes are getting saved to git then you could setup a webhook like https://github.com/adnanh/webhook to listen to these git push actions to trigger your rebuild
However if they are just edits then you could roll your own using something like
https://github.com/hpcloud/tail
to do the functional equivalent to a tail -f on an arbitrary set of files/dirs which I have found to work nicely (my logs trigger a parse daemon for error checking)
but you're right there might be an easier way given your use case