docker-compose with multiple options to enable dbt's serve docs functionality - docker

I am trying to generate/access documentation for dbt via this guide: https://docs.getdbt.com/reference/commands/cmd-docs. The issue is I am getting a 'This Site Cant Be Reached'. So I am referencing this post DBT docker: Docs Served but Not Accessible via Browser - which notes to add --publish to my docker-compose. Currently I have a makefile with the line below:
docker-compose -f docker-compose.yml
I would think to change it to the below but does not seem to work.
docker-compose -f docker-compose.yml -p
as I get the error:
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "-p": executable file not found in $PATH: unknown
As well, I tried to edit the docker-compose.yml file to include the below:
version: '3.9'
services:
localdev:
build: .
image: localdev:latest
ports:
- "80:80"
- "8080"
- "8080:8080
stdin_open: true
tty: true
environment:
- ENV=my-env
- ADD_PATH=/bin/docker
volumes:
- $LOCAL_REPO_DIR:/usr/app/code
- /var/run/docker.sock:/var/run/docker.sock:ro
- /usr/local/bin:/bin/docker:ro
command: /usr/app/entrypoint.sh
While this allowed me to spin up the container, I was still not able to access to webpage locally.

dbt docs serve by default serves on port 8080. If you want the localdev container's port 8080 to be accessible at 8080 on the host machine, you need:
ports:
- "8080:8080"
You should only list port 8080 once in this list; if you list it alone, without a host port:
ports:
- 8080
Docker will expose it on a random port on the host.

Related

How to get around network mapping issue "Bind for 0.0.0.0:8080 failed: port is already allocated"

I'm trying to build a Jenkins docker container by following this page so I can test locally. Problem is with this is that once I've ran docker run -it -p 8080:8080 jenkins/jenkins:lts it seems I cannot use the same port for my docker-compose.yml:
version: '3.8'
services:
jenkins:
image: jenkins/jenkins:lts
container_name: jenkins
user: root
privileged: true
ports:
- 8080:8080
- 50000:50000
volumes:
- .jenkins/jenkins_configuration:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
The error shown in PowerShell (I'm on windows 10 if that's relevant) is:
Error response from daemon: driver failed programming external connectivity on endpoint jenkins (xxxx): Bind for 0.0.0.0:8080 failed: port is already allocated
I've made sure it's not affected from another container, image or volume and have deleted everything apart from this.
I wish to use Jenkins locally but how can I get around this? I'm not familiar with networking and what I've googled so far has not seemed to work for me. I would like this to be able to use Jenkins ui on localhost:8080
If port 8080 is already allocated on your host machine, you can just map a different one to 8080 of the container instead. Two things can't be mapped to the same port on the host machine. In order to map 8081 for example, change your compose to the following:
version: '3.8'
services:
jenkins:
image: jenkins/jenkins:lts
container_name: jenkins
user: root
privileged: true
ports:
- 8081:8080 # a different port is mapped here
- 50000:50000
volumes:
- .jenkins/jenkins_configuration:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
Then, you just need to access the container started by docker-compose with port localhost:8081 rather than localhost:8080.

Docker container can't talk to another container

I have a docker compose file set up with 3 separate containers (Flask, Nginx and Solr)
After starting up all 3 run successfully but my Flask application can't connect to my Solr instance and when I run:
wget -S http://localhost:8983/solr/CORE_NAME/select
I get the error "Connecting to localhost (localhost)|127.0.0.1|:8983... failed: Connection refused."
I am fairly new to docker and been around a few different forums looking at this issue but nothing has worked so far. I have tried creating a network also but running into the same issue.
Here is my docker-compose.yml.
version: "2.7"
services:
nginx:
build:
context: .
dockerfile: Dockerfile-nginx
container_name: nginx
ports:
- "80:80"
- "8181:8181"
volumes:
- ./:/opt/ee1
- ee1-logs-volume:/var/log/ee1
- ./:/usr/local/websites/ee1
- sockets-volume:/tmp
depends_on:
- flask
flask:
build:
context: .
dockerfile: Dockerfile-flask
entrypoint: ["/bin/bash", "./system/start-uwsgi-docker.bash"]
container_name: flask
user: root
restart: always
volumes:
- ./:/opt/ee1
- ./ee1config.ini:/opt/ee1config.ini
- ee1jobs-logs-volume:/var/log/ee1
- ./:/usr/local/websites/ee1
- sockets-volume:/tmp
links:
- solr
solr:
build:
context: .
dockerfile: Dockerfile-solr
container_name: solr
volumes:
- data:/var/solr
entrypoint:
- bash
- "-c"
- "precreate-core ee1_1; precreate-core ee1_2; exec solr -f"
ports:
- "8983:8983"
volumes:
sockets-volume: {}
ee1-logs-volume: {}
data:
Every docker container is - network wise - a separate host with it's own IP.
Traffic to localhost or 127.0.0.1 will definitely never leave that container.
So what you need to find out is the IP of the server container (solr) you actually want to talk to, then configure the client container (flask) accordingly. This can be done by e.g. docker inspect. Be aware that upon container restart the IPs can change. You will want to use something like DNS rather than raw IPs.
Since you use docker compose, each container for a service joins the same network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.
For more details check out
https://docs.docker.com/compose/networking/
https://docs.docker.com/network/

Docker-compose setting problem about Domjudge server

I want to build a domjudge server with mriadb, phpmyadmin, judgehost in the docker base on Debian9,
I've install the docker and docker compose
here is the docker-compose.yml code below.
and I use docker-compose up -d and there are some WARNING and ERROR pop out.
here is the entire docker-compose.yml file code
http://codepad.org/souBFdFz
WARNING and ERROR messages:
WARNING: some networks were defined but are not used by any service: phpmyadmin, dj-judgedameons_1, dj-judgedameons_2
ERROR: dor domjudge_dj-judgedameons_2_1 Cannot start service dj-judgedameons_1 : OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:311:getting Starting domjudge_dj-judgedameons_1_1
...and a lots of error messages that I cant even read(binary code or address i think)
Please help me fix it or if there is a easy way to set up domjudge server with mariadb, phpmyadmin, judgehost
THANKS!
Update
I've tried this file several times and it has a drifferent result but it still can't connect to the server (domjudge & phpmyadmin).
here is the message
https://i.stack.imgur.com/qDcDd.jpg
Unfortunately what you want to do is not really possible because of how the application is built: containers need to wait for each other and some of them need manual actions.
However, this is a sequence of actions that works and will bring all containers up and running.
NOTE: I removed the networks declarations because they don't add any value.
version: '3'
services:
dj-mariadb:
image: mariadb
environment:
- MYSQL_ROOT_PASSWORD=rootpw
- MYSQL_DATABASE=domjudge
- MYSQL_USER=domjudge
- MYSQL_PASSWORD=djpw
command:
--max-connections=1000
dj-domserver:
image: domjudge/domserver:latest
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:ro
environment:
- CONTAINER_TIMEZONE=Asia/Taipei
- MYSQL_ROOT_PASSWORD=rootpw
- MYSQL_DATABASE=domjudge
- MYSQL_USER=domjudge
- MYSQL_PASSWORD=djpw
ports:
- 9090:80
links:
- dj-mariadb:mariadb
dj-judgehost:
image: domjudge/judgehost:latest
privileged: true
hostname: judgedaemon-0
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:ro
environment:
- DAEMON_ID=0
- JUDGEDAEMON_PASSWORD=domjudge
links:
- dj-domserver:domserver
dj-judgehost_1:
image: domjudge/judgehost:latest
privileged: true
hostname: judgedaemon-1
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:ro
environment:
- DAEMON_ID=1
- JUDGEDAEMON_PASSWORD=domjudge
links:
- dj-domserver:domserver
dj-judgehost_2:
image: domjudge/judgehost:latest
privileged: true
hostname: judgedaemon-2
volumes:
- /sys/fs/cgroup:/sys/fs/cgroup:ro
environment:
- DAEMON_ID=2
- JUDGEDAEMON_PASSWORD=domjudge
links:
- dj-domserver:domserver
phpmyadmin:
image: phpmyadmin/phpmyadmin
container_name: myadmin
ports:
- 8888:80
environment:
- PMA_ARBITRARY=1
- PMA_HOST=dj-mariadb
links:
- dj-mariadb:db
Start the database and wait for it to initialize (otherwise the server will exit because it cannot find the schema it needs)
docker-compose up -d dj-mariadb
Start the server:
docker-compose up -d dj-domserver
Get the admin password from the logs:
docker-compose logs dj-domserver
Look for the line saying: Initial admin password is .... and save the password.
Set the judgehost password in the web interface: open http://localhost:9090 and login with user admin and the password you saved from the previous step. Go to Users and click on judgehost user. In there change the password to domjudge (according to what you set in the docker-compose.yml for JUDGEDAEMON_PASSWORD. Save the data.
Start the rest of the containers:
docker-compose up -d
Verify that all containers are up and running:
docker-compose ps
Output should look similar to this:
Name Command State Ports
---------------------------------------------------------------------------------------------------
domjudge_dj-domserver_1 /scripts/start.sh Up 0.0.0.0:9090->80/tcp
domjudge_dj-judgehost_1 /scripts/start.sh Up
domjudge_dj-judgehost_1_1 /scripts/start.sh Up
domjudge_dj-judgehost_2_1 /scripts/start.sh Up
domjudge_dj-mariadb_1 docker-entrypoint.sh --max ... Up 3306/tcp
myadmin /run.sh supervisord -n -j ... Up 0.0.0.0:8888->80/tcp, 9000/tcp

Change Swagger port in docker-compose

I am unable to change the port that Swagger uses in docker compose. It works fine with regular docker, I simply set the -p argument on the run command. It seems that I should just need to set the ports field in the docker-compose file. But no matter what I try it just runs on 8080.
I am using the latest versions of docker and docker-compose. The docker image is called swaggerapi/swagger-ui. I have attempted setting the ports field for the container. Also tried setting the url variable in the swagger definition file. Tried changing the expose port. I tried with the docker-compose run command which lets you start an individual service and has the -p argument. Still nothing.
Ideally I should use this to build and run:
sudo docker-compose up --build --force-recreate
My compose file:
version: '3'
services:
swagger:
build: swagger
network_mode: "host"
ports:
- "8081:8080"
env_file: .env
environment:
- SWAGGER_JSON=/swagger.json
volumes:
data:
driver: "local"
And the docker file for the swagger service:
FROM swaggerapi/swagger-ui
EXPOSE 8081
COPY swagger.json /swagger.json
ENV SWAGGER_JSON "/swagger.json"
No matter what I do it wont change ports.
Just change the port in your docker-compose file
swagger:
build: swagger
network_mode: "host"
ports:
- "8081:"**Port which you want to expose**"
env_file: .env
environment:
- SWAGGER_JSON=/swagger.json

Cannot access virtual hosts between containers with docker-compose using nginx-proxy and dnsmasq

Context
I was planning on simplifying some development setup of multiple docker-compose.yml by introducing virtual hosts locally. I looked around and decided to use nginx-proxy for the reverse-proxy (ability to set VIRTUAL_HOST for each service).
Setup
To expose these on the host machine I went the route of dnsmasq and adding a /etc/resolver/test/ with nameserver 127.0.0.1.
I went and put the above into action using a dev/docker-compose.yml file:
version: '3.5'
services:
nginx-proxy:
image: jwilder/nginx-proxy
restart: 'always'
ports:
- "80:80"
- "443:443"
volumes:
- "/var/run/docker.sock:/tmp/docker.sock:ro"
dnsmasq:
image: andyshinn/dnsmasq
restart: 'always'
ports:
- "53:53/tcp"
- "53:53/udp"
cap_add:
- NET_ADMIN
command: --log-facility=-
volumes:
- ./data/dnsmasq.conf:/etc/dnsmasq.conf
- ./data/dnsmasq.d:/etc/dnsmasq.d
networks:
default:
external:
name: proxynet
The data/dnsmasq.conf file only contains address=/test/127.0.0.1.
I've also created an external network proxynet and use that as the default network for the docker-compose file(s) (docker network create proxynet). This then allows other docker-compose files and services to be linked to the proxy.
I have the following proj1/docker-compose.yml:
version: "3.5"
services:
proj1-web:
image: jwilder/whoami
environment:
- VIRTUAL_HOST=proj1-web.test
networks:
default:
external:
name: proxynet
Having both these of these docker-compose files running (i.e., docker-compose up) I am able to access proj1-web.test from my local machine. Everything works as expected.
Now I want to be able to reference proj1-web.test in another container and have it resolve to the running container.
I'll create proj2/docker-compose.yml (similar to previous just different name):
version: "3.5"
services:
proj2-web:
image: jwilder/whoami
environment:
- VIRTUAL_HOST=proj2-web.test
networks:
default:
external:
name: proxynet
With everything running I can access both proj1-web.test and proj2-web.test from my local machine. I can successfully curl different services using between proj1 and proj2: docker-compose run proj1-web sh -c "apk update -qq; apk add curl -qq; curl -v proj2-web:8000".
Problem
The problem is that I cannot curl the virtual host's name proj2-web.test from proj1: docker-compose run proj1-web sh -c "apk update -qq; apk add curl -qq; curl -v proj2-web.test":
* Rebuilt URL to: proj2-web.test/
* Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to proj2-web.test port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to proj2-web.test port 80: Connection refused
Is there something I'm missing here? It appears the individual containers don't have access to the DNS being provided from dnsmasq to my local machine, I cannot figure out how to grant them that access. Maybe I'm going about this the wrong way -- I am open to suggestions.
I ended up creating a solution which addresses my question. You can see the repository here for the tool:
https://github.com/scoremedia/dcdc
I also created a blog post detailing a bit of this: https://kevinjalbert.com/docker-compose-dns-consistency-dcdc/
Hopefully this helps others.

Resources