cannot connect to nsqd from another service defined in docker-compose file - docker

I have a service connecting to nsqd to produce and consume messages. I have integration tests connecting to it (broadcast address 127.0.0.1) and this works fine executing it locally in the cli or the ide.
Then I have created this service to up with docker-compose, but cannot connect to nsqd.
Following is my docker-compose file
version: '3'
services:
redis:
image: redis:4.0.9-alpine
ports:
- "6379:6379"
nsqlookupd:
image: nsqio/nsq:v0.3.8
command: /nsqlookupd
ports:
- "4160:4160"
- "4161:4161"
nsqd:
image: nsqio/nsq:v0.3.8
command: /nsqd --lookupd-tcp-address=nsqlookupd:4160 --broadcast-address=127.0.0.1
links:
- nsqlookupd:nsqlookupd
ports:
- "4150:4150"
- "4151:4151"
nsqadmin:
image: nsqio/nsq:v0.3.8
ports:
- "4171:4171"
links:
- nsqlookupd:nsqlookupd
command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
creator:
build: "creator/"
depends_on:
- nsqlookupd
- nsqd
- redis
environment:
SERVER_ADDR: ":8080"
NSQ_ADDR: "nsqd:4150"
NSQ_TOPIC: "driver_locations"
NSQ_CHANNEL: "ch"
REDIS: "redis:6379"
ports:
- "8080:8080"
Right now I don't care about the tests locally, just to have all the containers working properly.
I have tried changing the broadcast, removing the broadcast...
As they say in the docs https://nsq.io/deployment/docker.html#using-docker-compose this is the last thing I tried (basically the changes are the commands), with no luck:
version: '3'
services:
redis:
image: redis:4.0.9-alpine
ports:
- "6379:6379"
nsqlookupd:
image: nsqio/nsq
command: /nsqlookupd
ports:
- "4160"
- "4161"
nsqd:
image: nsqio/nsq
command: /nsqd --lookupd-tcp-address=nsqlookupd:4160
depends_on:
- nsqlookupd
ports:
- "4150"
- "4151"
nsqadmin:
image: nsqio/nsq
command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
depends_on:
- nsqlookupd
ports:
- "4171"
creator:
build: "creator/"
depends_on:
- nsqlookupd
- nsqd
- redis
environment:
SERVER_ADDR: ":8080"
NSQ_ADDR: "nsqd:4150"
NSQ_TOPIC: "driver_locations"
NSQ_CHANNEL: "ch"
REDIS: "redis:6379"
ports:
- "8080:8080"

Cache problem, sorry for that :facepalm:
What I told it wasn't working (removing the broadcast) IT IS
I have tried changing the broadcast, removing the broadcast...
As they say in the docs https://nsq.io/deployment/docker.html#using-docker-compose this is the last thing I tried (basically the changes are the commands), with no luck:
version: '3'
services:
redis:
image: redis:4.0.9-alpine
ports:
- "6379:6379"
nsqlookupd:
image: nsqio/nsq
command: /nsqlookupd
ports:
- "4160"
- "4161"
nsqd:
image: nsqio/nsq
command: /nsqd --lookupd-tcp-address=nsqlookupd:4160
depends_on:
- nsqlookupd
ports:
- "4150"
- "4151"
nsqadmin:
image: nsqio/nsq
command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
depends_on:
- nsqlookupd
ports:
- "4171"
creator:
build: "creator/"
depends_on:
- nsqlookupd
- nsqd
- redis
environment:
SERVER_ADDR: ":8080"
NSQ_ADDR: "nsqd:4150"
NSQ_TOPIC: "driver_locations"
NSQ_CHANNEL: "ch"
REDIS: "redis:6379"
ports:
- "8080:8080"

Related

Got problem deploying docker-compose service(port issue)

I want to deploy a service that will allow me to use Spark and MongoDB in a Jupiter notebook.
I use docker-compose to build up the service, and it`s as followed:
version: "3.3"
volumes:
shared-workspace:
networks:
spark-net:
driver: bridge
services:
spark-master:
image: uqteaching/cloudcomputing:spark-master-v1
container_name: spark-master
networks:
- "spark-net"
ports:
- "8080:8080"
- "7077:7077"
environment:
- INIT_DAEMON_STEP=setup_spark
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
spark-worker-1:
image: uqteaching/cloudcomputing:spark-worker-v1
container_name: spark-worker-1
depends_on:
- spark-master
networks:
- "spark-net"
ports:
- "8081:8081"
environment:
- "SPARK_MASTER=spark://spark-master:7077"
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
spark-worker-2:
image: uqteaching/cloudcomputing:spark-worker-v1
container_name: spark-worker-2
depends_on:
- spark-master
networks:
- "spark-net"
ports:
- "8082:8082"
environment:
- "SPARK_MASTER=spark://spark-master:7077"
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
mongo:
image: mongo
container_name: 'mongo'
networks:
- "spark-net"
ports:
- "27017:27017"
mongo_admin:
image: mongo-express
container_name: 'mongoadmin'
networks:
- "spark-net"
depends_on:
- mongo
links:
- mongo
ports:
- "8091:8091"
jupyter-notebook:
container_name: jupyternb
image: jupyter/all-spark-notebook:42f4c82a07ff
depends_on:
- mongo
- spark-master
links:
- mongo
expose:
- "8888"
networks:
- "spark-net"
ports:
- "8888:8888"
volumes:
- ./nbs:/home/jovyan/work/nbs
- ./events:/tmp/spark-events
environment:
- "PYSPARK_PYTHON=/usr/bin/python3"
- "PYSPARK_DRIVER_PYTHON=/usr/bin/python3"
command: "start-notebook.sh \
--ip=0.0.0.0 \
--allow-root \
--no-browser \
--notebook-dir=/home/jovyan/work/nbs \
--NotebookApp.token='' \
--NotebookApp.password=''
"
And the result is something like this:
I dont know why. Even I set these 2` services to listen to a different port.
They are using 8081/tcp at the same time, which caused them both to crash.
I want to solve this.
mongo-express seems to need port 8081 internal, so use another external port to be able to login to the webui.
http://localhost:8092 would then be something like this:
mongo_admin:
image: mongo-express
container_name: 'mongoadmin'
networks:
- "spark-net"
depends_on:
- mongo
links:
- mongo
ports:
- "8092:8091"

saleor backend not available under droplet ip from digital ocean

I'd like to deploy my saleor-shop application completely via docker.
So I've built the respective images for saleor backend, storefront & dashboard.
Running the app locally works fine.
Backend is available on localhost:8000/graphql
Storefront runs at localhost:3000
Dashboard runs at localhost:9000
If I'd like to run the app on the droplet IP --> I get issues with running the saleor backend.
As of now trying to access XXX.XX.XXX.XXX:8000 results in "This site can't be reached".
The storefront and dashboard are accessible on XXX.XX.XXX.XXX:3000 and XXX.XX.XXX.XXX:9000 however without any interaction with the backend cause its not available. Thats why the graphql calls are not functioning on the storefront and logging into the dashboard does not work either cause the backend is not available. I think I'm missing something here and would appreciate any help.
[
Within my droplet I'm using the following docker-compose.yml file to get my docker containers up:
services:
api:
ports:
- 8000:8000
image: XXX/murukku-shop
restart: unless-stopped
networks:
- saleor-backend-tier
depends_on:
- db
- redis
- jaeger
env_file: common.env
environment:
- JAEGER_AGENT_HOST=jaeger
- STOREFRONT_URL=http://XXX.XX.XXX.XXX:3000/
- DASHBOARD_URL=http://XXX.XX.XXX.XXX:9000/
storefront:
image: XXX/murukku-storefront
ports:
- 3000:80
restart: unless-stopped
dashboard:
image: XXX/murukku-dashboard
ports:
- 9000:80
restart: unless-stopped
db:
image: library/postgres:11.1-alpine
ports:
- 5432:5432
restart: unless-stopped
networks:
- saleor-backend-tier
volumes:
- saleor-db:/var/lib/postgresql/data
environment:
- POSTGRES_USER=saleor
- POSTGRES_PASSWORD=saleor
redis:
image: library/redis:5.0-alpine
ports:
- 6379:6379
restart: unless-stopped
networks:
- saleor-backend-tier
volumes:
- saleor-redis:/data
worker:
image: XXX/murukku-shop
restart: unless-stopped
networks:
- saleor-backend-tier
env_file: common.env
depends_on:
- redis
- mailhog
environment:
- EMAIL_URL=smtp://mailhog:1025
jaeger:
image: jaegertracing/all-in-one
ports:
- "5775:5775/udp"
- "6831:6831/udp"
- "6832:6832/udp"
- "5778:5778"
- "16686:16686"
- "14268:14268"
- "9411:9411"
restart: unless-stopped
networks:
- saleor-backend-tier
mailhog:
image: mailhog/mailhog
ports:
- 1025:1025 # smtp server
- 8025:8025 # web ui. Visit http://localhost:8025/ to check emails
restart: unless-stopped
networks:
- saleor-backend-tier
volumes:
saleor-db:
driver: local
saleor-redis:
driver: local
saleor-media:
networks:
saleor-backend-tier:
driver: bridge
I was testing Saleor like you in a docker setup and I've found a solution ! You have to set more env variable, they are all explained on the github page of the storefront and the dashboard.
Here is my config if you want :
version: '2'
services:
api:
ports:
- 8000:8000
build:
context: ./saleor
dockerfile: ./Dockerfile
args:
STATIC_URL: '/static/'
restart: unless-stopped
networks:
- saleor-backend-tier
depends_on:
- db
- redis
- jaeger
volumes:
- ./saleor/saleor/:/app/saleor:Z
- ./saleor/templates/:/app/templates:Z
- ./saleor/tests/:/app/tests
# shared volume between worker and api for media
- saleor-media:/app/media
command: python manage.py runserver 0.0.0.0:8000
env_file: common.env
environment:
# - DEFAULT_CURRENCY=EUR
#- DEFAULT_COUNTRY=
- ALLOWED_CLIENT_HOSTS=localhost,127.0.0.1,192.168.0.50
- ALLOWED_HOSTS=localhost,192.168.0.50
- JAEGER_AGENT_HOST=jaeger
- STOREFRONT_URL=http://192.168.0.50:3000/
- DASHBOARD_URL=http://192.168.0.50:9000/
storefront:
build:
context: ./saleor-storefront
dockerfile: ./Dockerfile.dev
ports:
- 3000:3000
restart: unless-stopped
volumes:
- ./saleor-storefront/:/app:cached
- /app/node_modules/
command: npm start -- --host 0.0.0.0
environment:
- NEXT_PUBLIC_API_URI=http://192.168.0.50:8000/graphql/
- API_URI=http://192.168.0.50:8000/graphql/
dashboard:
build:
context: ./saleor-dashboard
dockerfile: ./Dockerfile.dev
ports:
- 9000:9000
restart: unless-stopped
volumes:
- ./saleor-dashboard/:/app:cached
- /app/node_modules/
command: npm start -- --host 0.0.0.0
environment:
- API_URI=http://192.168.0.50:8000/graphql/
- APP_MOUNT_URI=/dashboard/
- STATIC_URL=http://192.168.0.50:9000/
db:
image: library/postgres:11.1-alpine
ports:
- 5432:5432
restart: unless-stopped
networks:
- saleor-backend-tier
volumes:
- saleor-db:/var/lib/postgresql/data
environment:
- POSTGRES_USER=saleor
- POSTGRES_PASSWORD=saleor
redis:
image: library/redis:5.0-alpine
ports:
- 6379:6379
restart: unless-stopped
networks:
- saleor-backend-tier
volumes:
- saleor-redis:/data
worker:
build:
context: ./saleor
dockerfile: ./Dockerfile
args:
STATIC_URL: '/static/'
command: celery -A saleor --app=saleor.celeryconf:app worker --loglevel=info
restart: unless-stopped
networks:
- saleor-backend-tier
env_file: common.env
depends_on:
- redis
- mailhog
volumes:
- ./saleor/saleor/:/app/saleor:Z,cached
- ./saleor/templates/:/app/templates:Z,cached
# shared volume between worker and api for media
- saleor-media:/app/media
environment:
- EMAIL_URL=smtp://mailhog:1025
jaeger:
image: jaegertracing/all-in-one
ports:
- "5775:5775/udp"
- "6831:6831/udp"
- "6832:6832/udp"
- "5778:5778"
- "16686:16686"
- "14268:14268"
- "9411:9411"
restart: unless-stopped
networks:
- saleor-backend-tier
mailhog:
image: mailhog/mailhog
ports:
- 1025:1025 # smtp server
- 8025:8025 # web ui. Visit http://localhost:8025/ to check emails
restart: unless-stopped
networks:
- saleor-backend-tier
volumes:
saleor-db:
driver: local
saleor-redis:
driver: local
saleor-media:
networks:
saleor-backend-tier:
driver: bridge
PS : It's my first answer on stackoverflow :D Don't forget to tick me as the answer if I solved your problem ;)

Docker elastic stack cannot receive connection error

I have docker-compose file looking like below
version: '3'
services:
redis:
build: ./docker/redis
postgresql:
build: ./docker/postgresql
ports:
- "5433:5432"
env_file:
- .env
graphql:
build: .
command: npm run start
volumes:
- ./logs/:/usr/app/logs/
ports:
- "3000:3000"
env_file:
- .env
depends_on:
- "redis"
- "postgresql"
links:
- "redis"
- "postgresql"
elasticsearch:
build: ./docker/elasticsearch
container_name: elasticsearch
ports:
- "9200:9200"
depends_on:
- "graphql"
links:
- "kibana"
kibana:
build: ./docker/kibana
ports:
- "5601:5601"
depends_on:
- "graphql"
networks:
- elastic
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
metricbeat:
build: ./docker/metricbeat
depends_on:
- "graphql"
- "elasticsearch"
- "kibana"
networks:
- elastic
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
packetbeat:
build: ./docker/packetbeat
depends_on:
- "graphql"
- "elasticsearch"
- "kibana"
networks:
- elastic
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
logstash:
build: ./docker/logstash
ports:
- "9600:9600"
volumes:
- ./logs:/usr/logs
depends_on:
- "graphql"
- "elasticsearch"
- "kibana"
networks:
- elastic
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
networks:
elastic:
driver: bridge
When I run docker-compose build and docker-compose up, I get "unable to revive connection: http://elasticsearch:9200" from every container. I don't think any of the containers are able to talk to each other right now. However, it really feels like everything should work because I have exposed all the ports for elastic components, linked them with same networks and URL is also pointing to the correct alias. What am I doing wrong?
The dockerfile settings are all correct as each container runs correctly in isolation - just not able to talk to each other at all

Can't connect to selenium-hub from Rails sidekiq with docker-compose.yml

Trying to run selenium from sidekiq worker with docker-compose.
It works well if I run job from rails task. But It doesn't work when I run from sidekiq.
I got this error when I run Job from sidekiq.
Errno::EADDRNOTAVAIL: Failed to open TCP connection to localhost:4444 (Cannot assign requested address - connect(2) for "localhost" port 4444)
docker-compose.yml
version: '3'
services:
db:
image: mysql
volumes:
- ./tmp/db:/var/lib/postgresql/data
web:
build: .
command: bundle exec rails s -p 3000 -b '0.0.0.0'
volumes:
- .:/myapp
ports:
- "3000:3000"
depends_on:
- db
redis:
image: redis:latest
ports:
- 6379:6379
sidekiq:
build: .
command: bundle exec sidekiq
volumes:
- .:/myapp
depends_on:
- db
- redis
selenium-hub:
image: selenium/hub:3.12.0-boron
container_name: selenium-hub
ports:
- "4444:4444"
chrome:
image: selenium/node-chrome:3.12.0-boron
depends_on:
- selenium-hub
environment:
- HUB_HOST=selenium-hub
- HUB_PORT=4444
firefox:
image: selenium/node-firefox:3.12.0-boron
depends_on:
- selenium-hub
environment:
- HUB_HOST=selenium-hub
- HUB_PORT=4444
Please suggest me how to fix this problem
I have it working with a docker-compose.yml like this:
version: '3.3'
services:
selenium-hub:
container_name: selenium_hub
image: selenium/hub:3.12.0-cobalt
ports:
- 4444:4444
networks:
- selenium_grid
selenium-chrome:
container_name: selenium_chrome
image: selenium/node-chrome:3.12.0-cobalt
environment:
- HUB_HOST=selenium_hub
- HUB_PORT=4444
volumes:
- /dev/shm:/dev/shm
networks:
- selenium_grid
depends_on:
- selenium-hub
selenium-firefox:
container_name: selenium_firefox
image: selenium/node-firefox:3.12.0-cobalt
environment:
- HUB_HOST=selenium_hub
- HUB_PORT=4444
volumes:
- /dev/shm:/dev/shm
networks:
- selenium_grid
depends_on:
- selenium-hub
networks:
selenium_grid:
driver: bridge

Setting up docker auto build to use docker-compose file

I am trying to set up auto builds using docker cloud/docker hub. It is always looking for Dockerfile when I have a docker-compose.yml. I am unable to find any option to change this. I am wondering whether this isn't possible or am I missing something?
This is my docker-compose.yml
version: '3'
services:
reverse-proxy:
image: traefik
ports:
- "80:80"
- "443:443"
- "${TRAEFIK_DASHBOARD_PORT}:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./traefik/traefik.toml:/etc/traefik/traefik.toml
- ./traefik/certs/journal.crt:/certs/journal.crt
- ./traefik/certs/journal.key:/certs/journal.key
networks:
- web
prisma:
image: prismagraphql/prisma:1.8
restart: always
ports:
- "${PRISMA_PORT}"
networks:
- web
environment:
PRISMA_CONFIG: |
port: ${PRISMA_PORT}
managementApiSecret: ${PRISMA_MANAGEMENT_API_SECRET}
databases:
default:
connector: postgres
host: ${PRISMA_DB_HOST}
port: ${PRISMA_DB_PORT}
database: ${PRISMA_DB}
user: ${PRISMA_DB_USER}
password: ${PRISMA_DB_PASSWORD}
migrations: ${PRISMA_ENABLE_MIGRATION}
graphql-server:
build:
context: ./graphql-server/
args:
- PORT=${GRAPHQL_SERVER_PORT}
networks:
- web
ports:
- "${GRAPHQL_SERVER_PORT}"
volumes:
- ./graphql-server:/usr/src/app
depends_on:
- prisma
command: ["./wait-for-it.sh", "prisma:${PRISMA_PORT}", "--", "./bootstrap.sh"]
environment:
- PRISMA_SERVICE_NAME=prisma
- PRISMA_PORT
- GRAPHQL_SERVER_PORT
- APOLLO_ENGINE_KEY
- PRISMA_ENDPOINT
- PRISMA_MANAGEMENT_API_SECRET
labels:
- "traefik.backend=graphql"
- "traefik.frontend.rule=Host:api.journal.com"
- "traefik.enable=true"
- "traefik.port=8080"
- "traefik.docker.network=web"
react-client:
build:
context: ./react-client/
args:
- PORT=${REACT_CLIENT_PORT}
ports:
- "${REACT_CLIENT_PORT}"
volumes:
- ./react-client:/usr/src/app
depends_on:
- graphql-server
environment:
- GRAPHQL_SERVER_PORT
- REACT_CLIENT_PORT
networks:
- web
networks:
web:
external: true
Both docker hub and docker cloud are trying to get only the dockerfile and not docker-compose. I also saw a post mentioning docker-compose should be used only for running and not for building; so I am not sure whether I am doing something wrong.

Resources