I use ArangoDB as a backend server for my web application. So far I have used the Foxx CLI to deploy my code to the ArangoDB server. I wanted to deploy my entire application using Docker, but I can't figure out how to add my Foxx service source codes to an ArangoDB using containers. Is it possible? If it is so what would be the correct way to do this?
So far I have tried a docker-compose approach: running the official ArangoDB image and building another image equipped with Foxx CLI to install the source files, but I got "connection refused" error from the database server when I ran the "foxx install" method from the container. (The ArangoDB server was working fine and I could run the "foxx install" command successfully outside virtualization).
For development purposes, I just keep the Foxx services in development mode and map the Foxx folder in my ArangoDB container (/var/lib/arangodb3-apps/_db/) to a folder in my machine using docker-compose volume definition.
Here is what a sample docker-compose service for ArangoDB could look like:
services:
arangodb_dev:
image: arangodb
container_name: my_arangodb_dev
environment:
- ARANGO_ROOT_PASSWORD=XXXXXX
ports:
- "8529:8529"
volumes:
- ./Arango/db:/var/lib/arangodb3
- ./Arango/apps_db_system:/var/lib/arangodb3-apps/_db/
Above we map both the Foxx service directory and the DB files directory to a local folder for persistency purposes.
Beyond development, you probably want to copy the files to the correct folder in the container instead of mapping the folder
Related
I'm building a NodeJS application on Docker in Swarm mode (single node). I'm using bind mount volume for NodeJS source code. Everything runs perfectly and I can see the output in localhost from NodeJS and Express, but when I change something in NodeJS code (which is in a bind mount volume), nothing changes. I have to restart my service to observe the changes. Earlier when I was working with Docker Compose only, it never happened, but now when I have switched to Swarm, I'm experiencing problems.
I'm using Docker 18 with Visual Studio Code 1.39 on macOS 10.14.6
Dockerfile
FROM node:12-alpine
WORKDIR /node-dir
COPY package*.json ./
RUN npm install
docker-compose.yml file
# Docker-compose.yml
version: '3.7'
services:
node-service:
image: node-img:1.0
ports:
- 80:3000
working_dir: "/node-dir"
volumes:
- ./node-dir/source:/node-dir/source
networks:
- ness-net
command: npm start
networks:
ness-net:
I also read that it could be due to the inodes, most editors when saving the file breaks the link. But it was working correctly under docker-compose with Visual Studio Code, its behaviour is changed only in Docker Swarm.
Update: I served a static html file using Nginx with bind mount, and I can easily change that file using VS Code and it's reflected. Its only NodeJS which is not detecting changes in a file.
If your volume mapping is correct, the source code changes should reach your node.js app container.
You can verify it by inspecting the source code inside the container after you make a change on docker host.
I'm currently in development mode, and I have to test the source code
repeatedly so I want to use bind mounts to make development and
testing easier.
However, your source code change won't be effective until node process inside the container reloads and picks up the changes.
In order to achieve this you have to use nodemon. Nodemon will pick the changes in the source code and reload node process along with the changes.
Another, longer alternative would be building new docker image and then updating your app using: docker service update --image=...
You can also use tilt to automate all of the above actions.
I've two containers : nginx & angular. The angular container contains the code and is automatically pulled from the registry when there is a new version (with watchtower).
I set up a Shared Volume between angular & nginx to share the code from angular to nginx.
### Angular #########################################
angular:
image: registry.gitlab.com/***/***:staging
networks:
- frontend
- backend
volumes:
- client:/var/www/client
### NGINX Server #########################################
nginx:
image: registry.gitlab.com/***/***/***:staging
volumes:
- client:/var/www/client
depends_on:
- angular
networks:
- frontend
- backend
volumes:
client:
networks:
backend:
frontend:
When I build & run for the first time the environment, everything works.
The problem is when there is a new version of the client, the image is pulled, the container is re-built and the new code version is inside the angular container, but in the nginx container it still the old code version of the client.
The shared volumes does not let me do what i want because we can not specify who is the host, is it possible to mount a volumes from a container to an other ?
Thanks in advance.
EDIT
The angular container is only here to serve the files. We could rsync the built application to the server on the host machine then mouting the volume to the container (host -> guest) but it would go against our CI process : build app->build image->push to registry->watchtower pull new image
Docker volumes are not intended to share code, and I'd suggest reconsidering this workflow.
The first time you launch a container with an empty volume, and only the first time and only if the volume is already empty, Docker will populate it with contents from the container. Since volumes are intended to hold data, and the application is likely to change the data that will be persisted, Docker doesn't overwrite the application data if the container is restarted; whatever was in the volume directory remains unchanged.
In your setup that means this happens:
You start the angular container the first time, and since the client named volume is empty, Docker copies content into it.
You start the nginx container.
You delete and restart the angular container; but since the client named volume is empty, Docker leaves the old content there.
The nginx container still sees the old content.
For a typical browser application, you don't actually need a "program" running: once you've run through a Typescript/Webpack/... sequence, the output is a collection of totally static files. In the case of Angular, there is an Ahead-of-Time compiler that produces these static files. The sequence I'd recommend here is:
Check out your application source tree locally.
Develop your browser application in isolation, using developer-oriented tools like ng serve or npm start. Since this is all running locally, you don't need to fight with anything Docker-specific (filesystem mappings, permissions, port mappings, ...); it is a totally normal Javascript development sequence. The system components you need for this are just Node; it is strictly easier than installing and configuring Docker.
Compile your application to static files with the Angular AOT compiler or Webpack or npm build.
Publish those static files to a CDN; or bind-mount them into an nginx container; or maybe build them into a custom image.
In the last case you wouldn't use a named Docker volume. Instead you'd mount the local filesystem into the container. A complete docker-compose.yml file for this case could look like:
version: '3'
services:
nginx:
image: registry.gitlab.com/***/***/***:staging
volumes:
- ./client:/var/www/client
ports:
- '8000:80'
From your comment:
There is no program running for the client, the CI compile the app and build the custom Image which COPY the application files in /var/www/client. Then watchtower pull this new image and restart the container. The container only run in daemon with (tail -f /dev/null & wait).
Looking at this from a high level, I don't see any need to have two containers or volumes at all. Simply build your application with a multi-stage build that generates an nginx image with the needed content:
FROM your_angular_base AS build
COPY src /src
RUN steps to compile your code
FROM nginx_base as release
...
COPY --from=build /var/www/client/ /var/www/client/
...
Then your compose file is stripped down to just:
...
### NGINX Server #########################################
nginx:
image: registry.gitlab.com/***/***/***:staging
networks:
- frontend
- backend
networks:
backend:
frontend:
If you do find yourself in a situation where a volume is needed to be shared between two running containers, and the volume needs to be updated with each deploy of one of the images, then the best place for that is an entrypoint script that copies files from one location into the volume. I have an example of this in my docker-base with the save-volume and load-volume scripts.
I had Docker for Windows, switched to Docker toolbox and now back to Docker for Windows and I ran into the issues with Volumes.
Before volumes were working perfectly fine and my containers which running with nodemon/tsnode/CLI watching files were restarting properly on source code change, but now they don't at all, so it looks like file changes from host are not populated in the container.
This is docker-compose for one service:
api:
build:
context: ./api
dockerfile: Dockerfile-dev
volumes:
- ./api:/srv
working_dir: /srv
links:
- mongo
depends_on:
- mongo
ports:
- 3030:3030
environment:
MONGODB: mongodb://mongo:27017/api_test
labels:
- traefik.enable=true
- traefik.frontend.rule=Host:api.mydomain.localhost
This id Dockerfile-dev
FROM node:10-alpine
ENV NODE_ENV development
WORKDIR /srv
EXPOSE 3030
CMD yarn dev // simply nodemon, working when ran from host
Can anyone help with that?
C drive is shared and verified with docker run --rm -v c:/Users:/data alpine ls /data showing list of files properly.
I will really appreciate any help.
We experienced the exact same problems in our team while developing nodejs/typescript applications with Docker on top of Windows and it has always been a big pain. To be honest, though, Windows does the right thing by not propagating the change event to the containers (Linux hosts also do not propagate the fsnotify events to containers unless the change is made from within the container). So bottom line: I do not think this issue will be avoidable unless you actually change the files within the container instead of changing them on the docker host. You can achieve this with a code sync tool like docker-sync, see this page for a list of available options: https://github.com/EugenMayer/docker-sync/wiki/Alternatives-to-docker-sync
Because we struggled with such issues for a long time, a colleague and I started an open source project called DevSpace CLI: https://github.com/covexo/devspace
The DevSpace CLI can establish a reliable and super fast 2-way code sync between your local folders and folders within your dev containers (works with any Kubernetes cluster, any volume and even with ephemeral / non-persistent folders) and it is designed to work perfectly with hot reloading tools such as nodemon. Setup minikube or a cluster with a one-click installer on some public cloud, run devspace up inside your project and you will be ready to program within your DevSpace without ever having to worry about local Docker issues and hot reloading problems. Let me know if it works for you or if there is anything you are missing.
I've been stuck into this recently (Feb 2020, Docker Desktop 2.2) and nothing from the base solutions really helped.
However when I tried WSL 2 and ran my docker-compose from inside Ubuntu shell, it became to pick up the changes in the files instantly. So if someone is observing this - try to up Docker from WSL 2.
I'm trying to convert a service (IdentityServer 4 v2) that is a .Net Core 2.0 application to run on Docker for Windows. This is on a dev workstation only.
I did the standard Visual Studio 2017 "Add Docker Support" to the existing Net Core 2.0 service. The service starts up fine, but fails due to these issues:
The service is looking for a certificate installed on the local host machine. Running in docker it doesn't see this cert.
The service is using DNS from the host machine etc/Hosts file to find our database server. The service running on Docker doesn't see the host machine DNS.
I attempted to set the network_mode in docker compose to "host" but that failed to build. What is the suggested way to set up such an environment?
The runtime is setup using docker compose, and the default compose and the configuration is very simple:
version: '3'
services:
identity.server:
image: identity.server
ports:
- "8100:8100"
build:
context: ./src/Identity.Server
dockerfile: Dockerfile
The referenced dockerfile:
FROM microsoft/dotnet:2.0-runtime
ARG source
WORKDIR /app
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "Identity.Server.dll"]
I have read the following answer, but it doesn't appear to work inside of .Net itself: https://devops.stackexchange.com/questions/1501/configure-docker-to-use-ssl-for-a-private-registry-on-windows-10?newreg=0d065d6c37214be4bd1f02a45e1248ba
We are accessing the cert like this and it just comes back null. It is found if the service is run outside of Docker.
var cert = X509.LocalMachine.CertificateAuthority.Thumbprint
.Find(settings.CertificateThumbprint).FirstOrDefault();
Whole point of using docker is to separate container resources from other containers and host itself. From application POV, LocalMachine you're calling is not your host, but the "container", so everything works as intended.
I assume you don't want to put crt files inside image, which is very wise.
To make it work, place crt files in some folder and then mount that dir using docker volumes to the container or if you use docker in swarm, use docker secrets.
I have a Node.js web-application that connects to a Neo4j database. I would like to encapsulate these in a single Docker image (using also a Neo4j Docker container), but I'm a docker novice and can't seem to figure this out. What's the recommended way to do it in the latest Docker versions?
My intuition would be to run the Neo4j container nested inside the app container. But from what I've read, I think the supported / recommended approach is to link the containers together. What I need is pretty well illustrated in this image. But the article where the image comes from isn't clear to me. Anyway, it's using the soon-to-be-deprecated legacy container linking, while networking is recommended these days. A tutorial or explanation would be much appreciated.
Also, how does docker-compose fit into all this?
Running a container within another container would imply to run a Docker engine within a Docker container. This is referenced as dind for Docker-in-Docker and I would strongly advise against it. You can search 'dind' online and discover why in most cases it is a bad idea, but as it is not the main object of your question I won't extend this subject any further.
Running both a node.js process and a neo4j process in the same container
While most people will tell you to refrain yourself from running more than one process within a Docker container, nothing prevents you from doing so. If you want to follow this path, take a look at the Using Supervisor with Docker from the Docker documentation website, or at the Phusion baseimage Docker image.
Just be aware that this way of doing things will make your Docker image more and more difficult to maintain over time.
Linking containers
As you found out, keeping Docker images as simple as you can (i.e: running one and only one app within a Docker container) will make your life easier on the long term.
Linking containers together is trivial when both containers run on the same Docker engine. It is just a matter of:
having your neo4j container expose the port its service listens on
running your node.js container with the --link <neo4j container name>:<alias> option
within the node.js application configuration, set the neo4j host to the <alias> hostname, docker will take care of forwarding that connection to the IP it assigned to the neo4j container
When you want to run those two containers on different hosts, things get more difficult.
With Docker Compose, you have to use the link: key to define your links
The new Docker network feature
You also discovered that linking containers won't be supported in the future and that the new way of making multiple Docker containers communicate is to create a virtual network and attach those 2 containers to that network.
Here's how to proceed:
docker network create mynet
docker run --detach --name myneo4j --net mynet neo4j
docker run --detach --name mynodejs --net mynet <your nodejs image>
Your node application configuration should then use myneo4j as the host to connect to.
To tell Docker Compose to use the new network feature, you would have to use the --x-networking option. Also you would not use the links: key.
Using the new networking feature also means that you won't be able to define any alias for the db. As a result you have to use the container name. Beware that unless you use the container_name: key in your docker-compose.yml file, Compose will create container names based on the directory which contains your docker-compose.yml file, the service name as found in the yml file and a number.
For instance, the following docker-compose.yml file, if within a directory named "foo" would create two containers named foo_web_1 and foo_db_1:
web:
build: .
ports:
- "8000:8000"
db:
image: postgres
when started with docker-compose --x-networking up, the web app configuration should then use foo_db_1 as the db hostname.
While if you use container_name:
web:
build: .
ports:
- "8000:8000"
db:
image: postgres
container_name: mydb
when started with docker-compose --x-networking up, the web app configuration should then use mydb as the db hostname.
Example of using Docker Compose to run a web app using nodeJS and neo4j
In this example, I will show how to dockerize the example app from github project aseemk/node-neo4j-template which uses nodejs and neo4j.
I assume you already have Docker 1.9.0+ and Docker Compose 1.5+ installed.
This project will use 2 docker containers, one to run the neo4j database and one to run the nodeJS web app.
Dockerizing the web app
We need to build a Docker image from which Docker compose will run a container. For that, we will write a Dockerfile.
Create a file named Dockerfile (mind the capital D) with the following content:
FROM node
RUN git clone https://github.com/aseemk/node-neo4j-template.git
WORKDIR /node-neo4j-template
RUN npm install
# ugly 20s sleep to wait for neo4j to initialize
CMD sleep 20s && node app.js
This Dockerfile describes the steps the Docker engine will have to follow to build a docker image for our web app. This docker image will:
be based on the official node docker image
clone the nodeJS example project from Github
change the working directory to the directory containing the git clone
run the npm install command to download and install the nodeJS app dependencies
instruct docker which command to use when running a container of that image
A quick review of the nodeJS code reveals that the author allows us to configure the URL to use to connect to the neo4j database using the NEO4J_URL environment variable.
Dockerizing the neo4j database
Well people took care of that for us already. We will use the official Docker image for neo4j which can be found on the Docker Hub.
A quick review of the readme tells us to use the NEO4J_AUTH environment variable to change the neo4j password. And setting this variable to none will disable the authentication all together.
Setting up Docker Compose
In the same directory as the one containing our Dockerfile, create a docker-compose.yml file with the following content:
db:
container_name: my-neo4j-db
image: neo4j
environment:
NEO4J_AUTH: none
web:
build: .
environment:
NEO4J_URL: http://my-neo4j-db:7474
ports:
- 80:3000
This Compose configuration file describes 2 services: db and web.
The db service will produce a container named my-neo4j-db from the official neo4j docker image and will start that container setting up the NEO4J_AUTH environment variable to none.
The web service will produce a container named at docker compose discretion using a docker image built from the Dockerfile found in the current directory (build: .). It will start that container setting up the environment variable NEO4J_URL to http://my-neo4j-db:7474 (note how we use here the name of the neo4j container my-neo4j-db). Furthermore, docker compose will instruct the Docker engine to expose the web container's port 3000 on the docker host port 80.
Firing it up
Make sure you are in the directory that contains the docker-compose.yml file and type: docker-compose --x-networking up.
Docker compose will read the docker-compose.yml file, figure out it has to first build a docker image for the web service, then create and start both containers and finally will provide you with the logs from both containers.
Once the log shows web_1 | Express server listening at: http://localhost:3000/, everything is cooked and you can direct your Internet navigator to http://<ip of the docker host>/.
To stop the application, hit Ctrl+C.
If you want to start the app in the background, use docker-compose --x-networking up -d instead. Then in order to display the logs, run docker-compose logs.
To stop the service: docker-compose stop
To delete the containers: docker-compose rm
Making neo4j storage persistent
The official neo4j docker image readme says the container persists its data on a volume at /data. We then need to instruct Docker Compose to mount that volume to a directory on the docker host.
Change the docker-compose.yml file with the following content:
db:
container_name: my-neo4j-db
image: neo4j
environment:
NEO4J_AUTH: none
volumes:
- ./neo4j-data:/data
web:
build: .
environment:
NEO4J_URL: http://my-neo4j-db:7474
ports:
- 80:3000
With that config file, when you will run docker-compose --x-networking up, docker compose will create a neo4j-data directory and mount it into the container at location /data.
Starting a 2nd instance of the application
Create a new directory and copy over the Dockerfile and docker-compose.yml files.
We then need to edit the docker-compose.yml file to avoid name conflict for the neo4j container and the port conflict on the docker host.
Change its content to:
db:
container_name: my-neo4j-db2
image: neo4j
environment:
NEO4J_AUTH: none
volumes:
- ./neo4j-data:/data
web:
build: .
environment:
NEO4J_URL: http://my-neo4j-db2:7474
ports:
- 81:3000
Now it is ready for the docker-compose --x-networking up command. Note that you must be in the directory with that new docker-compose.yml file to start the 2nd instance up.