I have a docker-compose stack running multiple images together.
Once i have the docker stack up, I have one local-stack and one mongo running. In Local-stack, if a lambda is to be executed, a new docker image is launched. from the lambda image i need to connect to the mongo image in the existing stack.
The docker.sock access is already provided to local stack so it creates a new docker image in my host machine. now the connection to the network is not established from lambda to mongo. neither by applying loopback address nor by network alias mentioned in the docker-compose.yml
can you please help me how can i establish connection...?
UPDATE
My docker-compose.yml
version: '2'
services:
mongo:
image: mongo:3.5
networks:
apitests:
aliases:
- mongo
ports:
- "27017:27017"
localstack:
image: localstack/localstack
ports:
- "4567-4583:4567-4583"
- "4050:4050"
env_file:
- localstack-config.list
volumes:
- "/tmp/localstack:/tmp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
networks:
apitests:
aliases:
- localaws
networks:
apitests: {}
My localstack-config.list
SERVICES=sqs,sns,lambda,s3
DEBUG=1
DEFAULT_REGION=eu-west-1
PORT_WEB_UI=4050
LAMBDA_EXECUTOR=docker
DOCKER_HOST=unix:///var/run/docker.sock
LAMBDA_REMOTE_DOCKER=false
Seems like local stack had a bug in the code where the newly spawning docker images did not have the access to the local stack network.
There was no provision also to provide any such access as only a temporary image was created on the go.
Enhanced and raised a PR#883 for the same.
Related
I am starting to use portrainer.io to manage my docker images, instead of Synology DSM Docker GUI.
Background information:
I've used MacVLAN to create an own IP address for my Pihole Docker, overall everything regarding this piHole is running fine with this settings, made by DSM GUI.
environment network volumesports
Problem:
I now would like to use portrainer.io to manage my Docker installation. Including the Stack option which should be docker compose.
I am now struggeling to get my PiHole Image up with this Docker script:
services:
pihole:
container_name: pihole
image: pihole/pihole:latest
networks: docker
ports:
- "53:53/tcp"
- "53:53/udp"
- "67:67/udp"
- "80:80/tcp"
environment:
TZ: 'Europe/Berlin'
WEBPASSWORD: 'password'
ServerIP: "0.0.0.0"
# Volumes store your data between container upgrades
volumes:
- '/pihole/pihole/:/etc/pihole/'
- '/pihole/dnsmasq/:/etc/dnsmasq.d/'
# Recommended but not required (DHCP needs NET_ADMIN)
# https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
cap_add:
- NET_ADMIN
restart: unless-stopped
Does anyone have an idea why I get "Unable to deploy stack" as error message?
You are telling the service to use a network called "docker", but the network is not defined in the compose file. Is this the complete docker-compose file?
If yes, then you are missing the networks section:
networks:
docker:
external: true
I'm trying to pass redis url to docker container but so far i couldn't get it to work. I did a little research and none of the answers worked for me.
version: '3.2'
services:
redis:
image: 'bitnami/redis:latest'
container_name: redis
hostname: redis
expose:
- 6379
links:
- api
api:
image: tufanmeric/api:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- proxy
environment:
- REDIS_URL=redis
depends_on:
- redis
deploy:
mode: global
labels:
- 'traefik.port=3002'
- 'traefik.frontend.rule=PathPrefix:/'
- 'traefik.frontend.rule=Host:api.example.com'
- 'traefik.docker.network=proxy'
networks:
proxy:
Error: Redis connection to redis failed - connect ENOENT redis
You can only communicate between containers on the same Docker network. Docker Compose creates a default network for you, and absent any specific declaration your redis container is on that network. But you also declare a separate proxy network, and only attach the api container to that other network.
The single simplest solution to this is to delete all of the network: blocks everywhere and just use the default network Docker Compose creates for you. You may need to format the REDIS_URL variable as an actual URL, maybe like redis://redis:6379.
If you have a non-technical requirement to have separate networks, add - default to the networks listing for the api container.
You have a number of other settings in your docker-compose.yml that aren't especially useful. expose: does almost nothing at all, and is usually also provided in a Dockerfile. links: is an outdated way to make cross-container calls, and as you've declared it to make calls from Redis to your API server. hostname: has no effect outside the container itself and is usually totally unnecessary. container_name: does have some visible effects, but usually the container name Docker Compose picks is just fine.
This would leave you with:
version: '3.2'
services:
redis:
image: 'bitnami/redis:latest'
api:
image: tufanmeric/api:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- REDIS_URL=redis://redis:6379
depends_on:
- redis
deploy:
mode: global
labels:
- 'traefik.port=3002'
- 'traefik.frontend.rule=PathPrefix:/'
- 'traefik.frontend.rule=Host:api.example.com'
- 'traefik.docker.network=default'
I have 2 applications that are separate codebases, and they each have their own database on the same db server instance.
I am trying to replicate this in docker, locally on my laptop. I want to be able to have both apps use the same database instance.
I would like
both apps to start in docker at the same time
both apps to be able to access the database on localhost
the database data is persisted
be able to view the data in the database using an IDE on localhost
So each of my apps has its own dockerfile and docker-compose file.
On app1, I start the docker instance of the app which is tied to the database. It all starts fine.
When I try to start app2, I get the following error:
ERROR: for app2_mssql_1 Cannot start service mssql: driver failed programming external connectivity on endpoint app2_mssql_1 (12d550c8f032ccdbe67e02445a0b87bff2b2306d03da1d14ad5369472a200620): Bind for 0.0.0.0:1433 failed: port is already allocated
How can i have them both running at the same time? BOTH apps need to be able to access each others database tables!
Here is the docker-compose.yml files
app1:
version: "3"
services:
web:
build:
context: .
args:
volumes:
- .:/app
ports:
- "3000:3000"
depends_on:
- mssql
mssql:
image: 'microsoft/mssql-server-linux'
ports:
- '1433:1433'
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=P455w0rd!
volumes:
- app1_db:/var/lib/mssql/data
volumes:
app1_db:
and here is app2:
version: "3"
services:
web:
build:
context: .
args:
volumes:
- .:/app
ports:
- "3000:3000"
depends_on:
- mssql
mssql:
image: 'microsoft/mssql-server-linux'
ports:
- '1433:1433'
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=P455w0rd!
volumes:
- app2_db:/var/lib/mssql/data
volumes:
app2_db:
Should I be using the same volume in each docker-compose file?
I guess the problem is in each app i am spinning up 2 different db instances, when in reality I guess i just want one, and it be used by all my apps?
The ports part in docker-compose file will bound the container port to host's port which causes port conflict in your case.
You need to remove the ports part from at least one of the compose file. This way, docker-compose can be up for both. And you can have access to both app at same time. But remember both apps will be placed in separate network bridges.
How docker-compose up works:
Suppose your app is in a directory called myapp, and your docker-compose.yml
When you run docker-compose up, the following happens:
A network called myapp_default is created.
A container is created using web’s configuration. It joins the network myapp_default under the name web.
A container is created using db’s configuration. It joins the network myapp_default under the name db.
If you run the second docker-compose.yml in different folder myapp2, then the nework will be myapp2_default.
Current configuration creates two volumes, two datebase containers and two apps. If you can make them run in the same network and run database as the single container it will work.
I don't think you are expecting two database container two two volumes.
Approach 1:
docker-compose.yml as a single compose.
version: "3"
services:
app1:
build:
context: .
args:
volumes:
- .:/app # give the path depending up on the docker file of app1.
ports:
- "3030:3000"
depends_on:
- mssql
app2:
build:
context: .
args:
volumes:
- .:/app # give the path depending up on the docker file of app2.
ports:
- "3032:3000"
depends_on:
- mssql
mssql:
image: 'microsoft/mssql-server-linux'
ports:
- '1433:1433'
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=SqlServer1234!
volumes:
- app_docker_db:/var/lib/mssql/data
volumes:
app_docker_db:
Approach 2:
To Isolate it further, still want to run them as the sepeare composefiles, create three compose file with network.
docker-compose.yml for database with network
version: "3"
services:
mssql:
image: 'microsoft/mssql-server-linux'
ports:
- '1433:1433'
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=SqlServer1234!
volumes:
- app_docker_db:/var/lib/mssql/data
networks:
- test_network
volumes:
app_docker_db
networks:
test_network:
docker-ompose.yml for app1
remove the database container and add below lines to your compose file
version: "3"
services:
app1:
build:
context: .
args:
volumes:
- .:/app # give the path depending up on the docker file of app1.
ports:
- "3030:3000"
networks:
default:
external:
name: my-pre-existing-network
Do the same for another docker-compose by replacing the docker-compose file.
There are many other option to create docker-compose files. Configure the default network and Use a pre-existing network
You're exposing the same port (1433) two times to the host machine. (This is what "ports:..." does). This is not possible as it would block the same port on your host (That's what the message says).
I think the most common way in these cases is that you link your db's to your apps. (See https://docs.docker.com/compose/compose-file/#links). By doing this your applications can still access the databases on their common ports (1433), but the databases are not accessible from the host anymore (only from the container that is linked to it).
Another error I see in your docker compose file is that both applications are exposed by the same ports. This is also not possible for the same reason. I would suggest that you change one of them to "3000:3001", so you can access this application on port 3001.
I'm using docker compose for a web application that I'm creating with asp.net core, postgres and redis. I have everything set up in compose to connect to postgres using the service name I've specified in the docker-compose.yml file. When trying to do the same with redis, I get an exception. After doing research it turns out this exception is a known issue and the work around is using the ip address of the the machine instead of a host name. However I cannot figure out how to get the ipaddress of the redis service from the compose file. Is there a way to do that?
Edit
Here is the compose file
version: "3"
services:
postgres:
image: 'postgres:9.5'
env_file:
- '.env'
volumes:
- 'postgres:/var/lib/postgresql/data'
ports:
- '5433:5432'
redis:
image: 'redis:3.0-alpine'
command: redis-server --requirepass devpassword
volumes:
- 'redis:/var/lib/redis/data'
ports:
- '6378:6379'
web:
build: .
env_file:
- '.env'
ports:
- "8000:80"
volumes:
- './src/edb/Controllers:/app/Controllers'
- './src/edb/Views:/app/Views'
- './src/edb/wwwroot:/app/wwwroot'
- './src/edb/Lib:/app/Lib'
volumes:
postgres:
redis:
Ok, I found the answer. It was something I was trying but didn't realize the address may change everytime you restart the containers.
Run docker ps to get a list of running contianers then copy the id of your container and run docker inspect {container_id} and that will output the ipaddress that you can access it with from within the other running containers.
The reason I was confused was because that address may change when the containers are started. So I had to guess what the ip address was going to be before I started the containers. Luckly after 5 times I guessed correctly.
I have a couple of app containers that I want to connect to the mongodb container. I tried with external_links but I can not connect to the mongodb.
I get
MongoError: failed to connect to server [mongodb:27017] on first
connect
Do I have to add the containers into the same network to get external_links working?
MongoDB:
version: '2'
services:
mongodb:
image: mongo:3.4
restart: always
ports:
- "27017:27017"
volumes:
- data:/data/db
volumes:
data:
App:
version: '2'
services:
app-dev:
restart: Always
build: repository/
ports:
- "3000:80"
env_file:
- ./environment.env
external_links:
- mongodb_mongodb_1:mongodb
Networks:
# sudo docker network ls
NETWORK ID NAME DRIVER SCOPE
29f8bae3e136 bridge bridge local
67d5519cb2e6 dev_default bridge local
9e7097c844cf host host local
481ee4301f7c mongodb_default bridge local
4275508449f6 none null local
873a46298cd9 prod_default bridge local
Documentation at https://docs.docker.com/compose/compose-file/#/externallinks says
If you’re using the version 2 file format, the externally-created containers must be connected to at least one of the same networks as the service which is linking to them.
Ex:
Create a new docker network
docker network create -d bridge custom
docker-compose-1.yml
version: '2'
services:
postgres:
image: postgres:latest
ports:
- 5432:5432
networks:
- custom
networks:
custom:
external: true
docker-compose-2.yml
version: '2'
services:
app:
image: training/webapp
networks:
- custom
external_links:
- postgres:postgres
networks:
custom:
external: true
Yuva's answer above for the version 2 holds good for version 3 as well.
The documentation for the external_links isn't clear enough.
For more clarity I pasted the version 3 variation with annotation
version: '3'
services:
app:
image: training/webapp
networks:
- <<network created by other compose file>>
external_links:
- postgres:postgres
networks:
<<network created by other compose file>>:
external: true
Recently I faced Name resolution failure trying to link 2 containers handled by docker-compose v3 representing gRPC server and client in my case, but failed and with external_links.
I'll probably duplicate some of the info posted here, but will try to summarize
as all these helped me solving the issue.
From external_links docs (as mentioned in earlier answer):
If you’re using the version 2 or above file format, the externally-created containers must be connected to at least one of the same networks as the service that is linking to them.
The following configuration solved the issue.
project-grpc-server/docker-compose.yml
version: '3'
services:
app:
networks:
- some-network
networks:
some-network:
Server container configured as expected.
project-grpc-client/docker-compose.yml
services:
app:
external_links:
# Assigning easy alias to the target container
- project-grpc-server_app_1:server
networks:
# Mentioning current container as a part of target network
- project-grpc-server_some-network
networks:
# Announcing target network (where server resides)
project-grpc-server_some-network:
# Telling that announced network already exists (shouldn't be created but used)
external: true
When using defaults (no container_name configured) the trick with configuring client container is in prefixes. In my case network name had prefix project-grpc-server_ when working with docker-compose and than goes the name itself some-network (project-grpc-server_some-network). So fully qualified network names should be passed when dealing with separate builds.
While container name is obvious as it appears from time to time on the screen the full network name is not easy-to-guess candidate when first facing this section of Docker, unless docker network ls.
I'm not a Docker expert, so please don't judge too strict if all this is obvious and essential in Docker world.