I have a kubernete docker-compose that contains
frontend - a web app running on port 80
backend - a node server for API running on port 80
database - mongodb
I would like to ideally access frontend via a hostname such as http://frontend:80, and for the browser to be able to access the backend via a hostname such as http://backend:80, which is required by the web app on the client side.
How can I go about having my containers accessible via those hostnames on my localhost environment (windows)?
docker-compose.yml
version: "3.8"
services:
frontend:
build: frontend
hostname: framework
ports:
- "80:80"
- "443:443"
- "33440:33440"
backend:
build: backend
hostname: backend
database:
image: 'mongo'
environment:
- MONGO_INITDB_DATABASE=framework-database
volumes:
- ./mongo/mongo-volume:/data/database
- ./mongo/init-mongo.js:/docker-entrypoint-initdb.d/init-mongo.js:ro
ports:
- '27017-27019:27017-27019'
I was able to figure it out, using the docker-compose aliases & networks I was able to connect every container to the same development network.
There was 3 main components:
container mapping node dns server - Grabs the aliases via docker ps and creates a DNS server that redirects those requests to 127.0.0.1 (localhost)
nginx reverse proxy container - mapping the hosts to the containers via their aliases in the virtual network
projects - each project is a docker-compose.yml that may have an unlimited number of containers running on port 80
docker-compose.yml for clientA
version: "3.8"
services:
frontend:
build: frontend
container_name: clienta-frontend
networks:
default:
aliases:
- clienta.test
backend:
build: backend
container_name: clienta-backend
networks:
default:
aliases:
- api.clienta.test
networks:
default:
external: true # connect to external network (see below for more)
name: 'development' # name of external network
nginx proxy docker-compose.yml
version: '3'
services:
parent:
image: nginx:alpine
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
ports:
- "80:80" #map port 80 to localhost
networks:
- development
networks:
development: #create network called development
name: 'development'
driver: bridge
DNS Server
import dns from 'native-dns'
import { exec } from 'child_process'
const { createServer, Request } = dns
const authority = { address: '8.8.8.8', port: 53, type: 'udp' }
const hosts = {}
let server = createServer()
function command (cmd) {
return new Promise((resolve, reject) => {
exec(cmd, (err, stdout, stderr) => stdout ? resolve(stdout) : reject(stderr ?? err))
})
}
async function getDockerHostnames(){
let containersText = await command('docker ps --format "{{.ID}}"')
let containers = containersText.split('\n')
containers.pop()
await Promise.all(containers.map(async containerID => {
let json = JSON.parse(await command(`docker inspect ${containerID}`))?.[0]
let aliases = json?.NetworkSettings?.Networks?.development?.Aliases || []
aliases.map(alias => hosts[alias] = {
domain: `^${alias}*`,
records: [
{ type: 'A', address: '127.0.0.1', ttl: 100 }
]
})
}))
}
await getDockerHostnames()
setInterval(getDockerHostnames, 8000)
function proxy(question, response, cb) {
var request = Request({
question: question, // forwarding the question
server: authority, // this is the DNS server we are asking
timeout: 1000
})
// when we get answers, append them to the response
request.on('message', (err, msg) => {
msg.answer.map(a => response.answer.push(a))
});
request.on('end', cb)
request.send()
}
server.on('close', () => console.log('server closed', server.address()))
server.on('error', (err, buff, req, res) => console.error(err.stack))
server.on('socketError', (err, socket) => console.error(err))
server.on('request', async function handleRequest(request, response) {
await Promise.all(request.question.map(question => {
console.log(question.name)
let entry = Object.values(hosts).find(r => new RegExp(r.domain, 'i').test(question.name))
if (entry) {
entry.records.map(record => {
record.name = question.name;
record.ttl = record.ttl ?? 600;
return response.answer.push(dns[record.type](record));
})
} else {
return new Promise(resolve => proxy(question, response, resolve))
}
}))
response.send()
});
server.serve(53, '127.0.0.1');
Don't forget to update your computers network settings to use 127.0.0.1 as the DNS server
Git repository for dns server + nginx proxy in case you want to see the implementation: https://github.com/framework-tools/dockerdnsproxy
Related
I am facing a problem with my authentication with Keycloak. Everything works fine when my Nuxt app is running locally (npm run dev), but when it is inside a Docker container, something goes wrong.
Windows 10
Docker 20.10.11
Docker-compose 1.29.2
nuxt: ^2.15.7
#nuxtjs/auth-next: ^5.0.0-1637745161.ea53f98
#nuxtjs/axios: ^5.13.6
I have a docker service containing Keycloak and Ldap : keycloak:8180 and myad:10389. My Nuxt app is running on port 3000.
On front side, here is my configuration, which is working great when I launch my app locally with "npm run dev" :
server: {
port: 3000,
host: '0.0.0.0'
},
...
auth: {
strategies: {
local: false,
keycloak: {
scheme: 'oauth2',
endpoints: {
authorization: 'http://localhost:8180/auth/realms/<realm>/protocol/openid-connect/auth',
token: 'http://localhost:8180/auth/realms/<realm>/protocol/openid-connect/token',
userInfo: 'http://localhost:8180/auth/realms/<realm>/protocol/openid-connect/userinfo',
logout: 'http://localhost:8180/auth/realms/<realm>/protocol/openid-connect/logout?redirect_uri=' + encodeURIComponent('http://localhost:3000')
},
token: {
property: 'access_token',
type: 'Bearer',
name: 'Authorization',
maxAge: 300
},
refreshToken: {
property: 'refresh_token',
maxAge: 60 * 60 * 24 * 30
},
responseType: 'code',
grantType: 'authorization_code',
clientId: '<client_id>',
scope: ['openid'],
codeChallengeMethod: 'S256'
}
},
redirect: {
login: '/',
logout: '/',
home: '/home'
}
},
router: {
middleware: ['auth']
}
}
And here are my Keycloak and Nuxt docker-compose configurations :
keycloak:
image: quay.io/keycloak/keycloak:latest
container_name: keycloak
hostname: keycloak
environment:
- DB_VENDOR=***
- DB_ADDR=***
- DB_DATABASE=***
- DB_USER=***
- DB_SCHEMA=***
- DB_PASSWORD=***
- KEYCLOAK_USER=***
- KEYCLOAK_PASSWORD=***
- PROXY_ADDRESS_FORWARDING=true
ports:
- "8180:8080"
networks:
- ext_sd_bridge
networks:
ext_sd_bridge:
external:
name: sd_bridge
client_ui:
image: ***
container_name: client_ui
hostname: client_ui
ports:
- "3000:3000"
networks:
- sd_bridge
networks:
sd_bridge:
name: sd_bridge
When my Nuxt app is inside its container, the authentication seems to work, but redirections are acting strange. As you can see I am always redirected to my login page ("/") after my redirection to "/home":
Browser network
Am I missing something or is there something I am doing wrong?
I figured out what my problem was.
So basically, my nuxt.config.js was wrong for a use inside a Docker container. I had to change the auth endpoints to :
endpoints: {
authorization: '/auth/realms/<realm>/protocol/openid-connect/auth',
token: '/auth/realms/<realm>/protocol/openid-connect/token',
userInfo: '/auth/realms/<realm>/protocol/openid-connect/userinfo',
logout: '/auth/realms/<realm>/protocol/openid-connect/logout?redirect_uri=' + encodeURIComponent('http://localhost:3000')
}
And proxy the "/auth" requests to the hostname of my Keycloak Docker container (note that my Keycloak and Nuxt containers are in the same network in my docker-compose files) :
proxy: {
'/auth': 'http://keycloak:8180'
}
At this point, every request was working fine except the "/authenticate" one, because "keycloak:8180/authenticate" is put in the browser URL and of course, it doesn't know "keycloak".
For this to work, I added this environment variable to my Keycloak docker-compose :
KEYCLOAK_FRONTEND_URL=http://localhost:8180/auth
With this variable, the full process of authentication/redirection is working like a charm, with Keycloak and Nuxt in their containers :)
👍
I have 2 Docker containers running in the same network and I want 1 of them to call another via spring Webclient.
I'm sure they all are in the same network -> docker network inspect <network_ID> proves this.
AFAIK I can ping one container from another to check if they can talk to each other by docker exec -ti attachment-loader-prim ping attachment-loader-sec
If I run this - I see responses from attachment-loader-sec like 64 bytes from 172.21.0.5: seq=0 ttl=64 time=0.220 ms, which means they can communicate.
When I send Postman request to attachment-loader-prim by its exposed port localhost:8085, I expect that after some business logic it calls for attachment-loader-sec via Webclient, but on that step I get a 500 error with such a message:
"finishConnect(..) failed: Connection refused:
attachment-loader-sec/172.21.0.5:80; nested exception is
io.netty.channel.AbstractChannel$AnnotatedConnectException:
finishConnect(..) failed: Connection refused:
attachment-loader-sec/172.21.0.5:80"
Both attachment-loader-prim and attachment-loader-sec can be accessed separately via postman and both send a response, no problem.
This is my docker-compose:
version: '3'
services:
attachment-loader-prim:
container_name: attachment-loader-prim
build:
context: ""
restart: always
image: attachment-loader:latest
environment:
SERVER_PORT: 8085
networks:
- loader_network
expose:
- 8085
ports:
- 8005:8005
- 8085:8085
attachment-loader-sec:
container_name: attachment-loader-sec
build:
context: ""
restart: always
image: attachment-loader:latest
environment:
SERVER_PORT: 8086
networks:
- loader_network
expose:
- 8086
ports:
- 8006:8005
- 8086:8086
networks:
loader_network:
driver: bridge
And this is a Webclient which makes a call:
class RemoteServiceCaller(private val fetcherWebClientBuilder: WebClient.Builder) {
suspend fun getAttachmentsFromRemote(id: String, params: List<Param>, username: String): Result? {
val client = fetcherWebClientBuilder.build()
val awaitExchange = client.post()
.uri("/{id}/attachment", id)
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(params)
.header(usernameHeader, username)
.accept(MediaType.APPLICATION_OCTET_STREAM)
.awaitExchange {
if (it.statusCode().is2xxSuccessful) {
handleSucessCode(it)
} else it.createExceptionAndAwait().run {
LOG.error(this.responseBodyAsString, this)
throw ProcessingException(this)
}
}
return awaitExchange
}
private suspend fun handleSucessCode(response: ClientResponse) {
// some not important logic
}
}
P.S. BasicUri for Webclient defined as Config Bean like http://attachment-loader-sec/list
All my investigations brought me to such problems as:
Calling container using localhost instead of container name
Containers are not in the same network.
All that seems not relevant for me.
Any ideas will be really appreciated.
The problem was in calling a service without its port. The url became now http://attachment-loader-sec:8086/list and it is correct now. In my case I get 404, which means that my url path is not quite correct, but that is outside of current question
I am trying out Dapr for the first time ....refering to the Dapr go sdk at https://github.com/dapr/go-sdk...
... trying to host a Dapr service using golang with Docker Compose on my Windows 10 machine - using VSCode - and running into an issue connecting to ther service.
I have the docker compose file set to do a simple configuration as follows. And trying to connect to the service via the Dapr API using curl
golang service (taskapi service) => Dapr SideCar (taskapidapr)
I based it off of the example from https://github.com/dapr/go-sdk/blob/main/example/Makefile, but using Docker Compose.
When I try to connect connect to the service using
curl -d "ping" -H "Content-type: text/plain;charset=UTF-8"
"http://localhost:8300/v1.0/invoke/taskapi/method/echo"
I am running into the following error.
{"errorCode":"ERR_DIRECT_INVOKE","message":"invoke API is not ready"}
And the Dapr logs in Docker show a 'no mDNS apps to refresh.' - not sure if this is the cause of it and how to handle it.
Anyone can point me to what I am missing - greatly appreciate it.
Thank you
Athadu
golang package
package main
import (
"context"
"errors"
"fmt"
"log"
"net/http"
"github.com/dapr/go-sdk/service/common"
daprd "github.com/dapr/go-sdk/service/http"
)
func main() {
port := "8085"
address := fmt.Sprintf(":%s", port)
log.Printf("Creating New service at %v port", address)
log.Println()
// create a Dapr service (e.g. ":8080", "0.0.0.0:8080", "10.1.1.1:8080" )
s := daprd.NewService(address)
// add a service to service invocation handler
if err := s.AddServiceInvocationHandler("/echo", echoHandler); err != nil {
log.Fatalf("error adding invocation handler: %v", err)
}
if err := s.Start(); err != nil && err != http.ErrServerClosed {
log.Fatalf("error listenning: %v", err)
}
}
func echoHandler(ctx context.Context, in *common.InvocationEvent) (out *common.Content, err error) {
if in == nil {
err = errors.New("invocation parameter required")
return
}
log.Printf(
"echo - ContentType:%s, Verb:%s, QueryString:%s, %s",
in.ContentType, in.Verb, in.QueryString, in.Data,
)
out = &common.Content{
Data: in.Data,
ContentType: in.ContentType,
DataTypeURL: in.DataTypeURL,
}
return
}
docker-compose.yml
version: "3"
services:
taskapi:
image: golang:1.16
volumes:
- ..:/go/src/lekha
working_dir: /go/src/lekha/uploader
command: go run main.go
ports:
- "8085:8085"
environment:
aaa: 80
my: I am THE variable value
networks:
- lekha
taskapidapr:
image: "daprio/daprd:edge"
command: [
"./daprd",
"-app-id", "taskapi",
"-app-protocol", "http",
"-app-port", "8085",
"-dapr-http-port", "8300",
"-placement-host-address", "placement:50006",
"-log-level", "debug",
"-components-path", "/components"
]
volumes:
- "../dapr-components/:/components" # Mount our components folder for the dapr runtime to use
depends_on:
- taskapi
ports:
- "8300:8300"
networks:
- lekha
#network_mode: "service:taskapi" # Attach the task-api-dapr service to the task-api network namespace
############################
# Dapr placement service
############################
placement:
image: "daprio/dapr"
command: ["./placement", "-port", "50006"]
ports:
- "50006:50006"
networks:
- lekha
networks:
lekha:
Daprd shows these mDNS messages in logs - not sure if this is the cause
time="2021-05-24T01:06:13.6629303Z" level=debug msg="Refreshing all
mDNS addresses." app_id=taskapi instance=442e04c9e8a6
scope=dapr.contrib type=log ver=edge
time="2021-05-24T01:06:13.6630421Z" level=debug msg="no mDNS apps to
refresh." app_id=taskapi instance=442e04c9e8a6 scope=dapr.contrib
Additionally, I see the containers on the expected ports ... running fine in Docker desktop...
enter image description here
{
"errorCode": "ERR_DIRECT_INVOKE",
"message": "invoke API is not ready"
}
same as yours
I'm trying to setup a docker-compose script - to start a dummy: website, API, Gateway and RabbitMQ. (micro service approach)
Request pipeline:
Web >> Gateway >> API >> RabbitMQ
My docker-compose looks like this:
version: "3.4"
services:
web:
image: webclient
build:
context: ./WebClient
dockerfile: dockerfile
ports:
- "4000:4000"
depends_on:
- gateway
gateway:
image: gatewayapi
build:
context: ./GateWayApi
dockerfile: dockerfile
ports:
- "5000:5000"
depends_on:
- ordersapi
ordersapi:
image: ordersapi
build:
context: ./ExampleOrders
dockerfile: dockerfile
ports:
- "6002:6002"
depends_on:
- rabbitmq
rabbitmq:
image: rabbitmq:3.7-management
container_name: rabbitmq
hostname: rabbitmq
volumes:
- rabbitmqdata:/var/lib/rabbitmq
ports:
- "7000:15672"
- "7001:5672"
environment:
- RABBITMQ_DEFAULT_USER=rabbitmquser
- RABBITMQ_DEFAULT_PASS=some_password
This pipe works:
Web >> Gateway >> API
I get a response from the API on the website.
But when I try to push a message to rabbitmq from the API, I get the following error:
System.AggregateException: One or more errors occurred. (Connection failed) ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: Connection refused 127.0.0.1:7001
The RabbitMQ managment GUI still works on the defined port 7000.
Requests to port 7001 does not.
However, if I start the API and RabbitMQ manually, it works like a charm. The API I simply start with a debugger (.Net core + IIS - default settings hitting F5 in VS) and this is the command I use to start the docker image manually:
docker run -p 7001:5672 -p 7000:15672 --hostname localhost -e RABBITMQ_DEFAULT_USER=rabbitmquser -e RABBITMQ_DEFAULT_PASS=some_password rabbitmq:3.7-management
Update
This is how I inject the config in the .Net core pipe.
startup.cs
public void ConfigureServices(IServiceCollection services)
{
// setup RabbitMQ
var configSection = Configuration.GetSection("RabbitMQ");
string host = configSection["Host"];
int.TryParse(configSection["Port"], out int port);
string userName = configSection["UserName"];
string password = configSection["Password"];
services.AddTransient<IConnectionFactory>(_ => new ConnectionFactory()
{
HostName = host,
Port = port,
UserName = userName,
Password = password
});
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
}
controller.cs
private readonly IConnectionFactory _rabbitFactory;
public ValuesController(IConnectionFactory rabbitFactory)
{
_rabbitFactory = rabbitFactory;
}
public void PublishMessage()
{
try
{
using (var connection = _rabbitFactory.CreateConnection())
using (var channel = connection.CreateModel())
{
string exchangeName = "ExampleApiController";
string routingKey = "MyCustomRoutingKey";
channel.ExchangeDeclare(exchange: exchangeName, type: "direct", durable: true);
SendMessage("Payload to queue 1", channel, exchangeName, routingKey);
}
}
catch (Exception e)
{
Console.WriteLine(e.InnerException);
}
}
private static void SendMessage(string message, IModel channel, string exchangeName, string routingKey)
{
byte[] body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish(exchange: exchangeName,
routingKey: routingKey,
basicProperties: null,
body: body);
Console.WriteLine($" Sending --> Exchange: { exchangeName } Queue: { routingKey } Message: {message}");
}
I imagine in your caller you set rabbitmq_url to localhost:7001. However, the caller is in a container, which does have anything running on the port 7001. Rabbitmq is running on port 7001 on your host.
You need to change the url to rabbitmq:5672 to use the internal network or use host.docker.internal:7001 if you are using window or mac docker 18.03+
I have two docker-compose setup, the main service is an SPA containing:
nginx proxy for a wordpress on port 80
wordpress + mysql
expressjs for serving a react app port 6000
This runs behind another docker-compose which is basically an nginx-reverse proxy.
The SPA manages to serve website and connects to backend API via reverse proxy just fine. However, when I try to make a separate https request to the backend api from the server.js I get this message:
{ Error: connect ECONNREFUSED 127.0.0.1:443
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1121:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 443 }
And it's not just axios, plain wget to the backend url gives me connection refused as well.
A sample for said request:
axios.put('/wc/v3/orders/934', {
status: "completed"
},{
withCredentials:true,
auth: {
username: process.env.REACT_APP_WC_ADMIN_CK_KEY,
password: process.env.REACT_APP_WC_ADMIN_CS_KEY
}
}).then(function (response) {
console.log(`ok`);
}).catch(function (error) {
console.log(error);
});
Any one knows what might be the problem here?
If you have multiple docker-compose environments, then each brings up its own network by default. You want to share the network between the two to allow for the services in one environment to communicate to the other.
# spa/docker-compose.yml
version: '2'
services:
spa:
...
networks:
- app-net
networks:
app-net:
driver: bridge
.
# express/docker-compose.yml
version: '2'
services:
api:
...
networks:
- spa_app-net
networks:
spa_app-net:
external: true